AI Crawlers Visibility, Analytics & Control

Description

AI Crawlers Visibility, Analytics & Control is the WordPress plugin to monitor AI crawlers, block unwanted bots, and analyze AI bot traffic on your site. See when GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and dozens of other AI crawlers request your pages — then decide what to allow, discourage, or block.

As AI search engines, chatbots, and training crawlers visit WordPress sites more often, you need clear visibility into who is accessing your content. This plugin gives you a dedicated AI crawler analytics dashboard, robots.txt control for AI bots, optional PHP blocking (403), and an AI Visibility Score — without replacing your existing SEO or security plugins.

Why monitor AI crawlers on WordPress?

AI companies use automated AI crawlers to index, train on, or retrieve website content. Common examples include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Bytespider (ByteDance), CCBot (Common Crawl), PerplexityBot, and many others.

With this AI crawlers plugin you can:

• See which AI crawlers visit your site and which pages they request.
• Compare AI bot traffic across date ranges and spot trends over time.
• Block AI crawlers with robots.txt rules or PHP-level 403 responses.
• Keep logs for compliance, debugging, and content licensing decisions.

AI crawler analytics dashboard

• Live dashboard with total requests, unique AI crawlers, pages crawled, and blocked attempts.
• Charts for AI crawler requests over time and traffic by bot category.
• Top AI crawlers and most-crawled pages tables with period comparison.
• Recent AI crawler activity log with filters for bot, HTTP status, and content type.
• Download analytics reports from the dashboard.
• Blocked-attempt logging when PHP blocking is enabled.

AI Visibility Score

• Analyze any page or your entire site for AI readiness.
• 19 automated checks across discovery, structured data, and technical signals.
• 0–100 score with letter grade and prioritized recommendations.
• Site-wide batch scan, scan history, comparison diff, and PDF export.

68+ built-in AI crawlers

• Pre-configured detection for major AI, search, and scraper bots.
• Category labels (training, search, assistant, scraper, and more).
• Per-crawler info with user-agent pattern details.
• Enable or disable monitoring for individual AI crawlers.

Custom AI crawler detection

• Add your own user-agent patterns for bots not in the built-in list.
• Assign company name and category for consistent reporting.

robots.txt control for AI crawlers

• Add Disallow rules for selected AI crawlers.
• Rules are appended to your WordPress robots.txt output.
• Live preview before you publish changes.

Block AI crawlers with PHP (403)

• Hard-block selected AI crawlers before WordPress serves a response.
• Blocked requests are logged in the AI crawler analytics dashboard.
• Works alongside robots.txt for stronger enforcement when needed.

Data management

• Configurable log retention (7 days to 1 year, or keep all data).
• Optional delete-all-data action from the admin.
• Delete plugin data on uninstall (optional setting).

Clean admin experience

• WordPress admin menu: AI Crawlers
• Redesigned UI with grouped sidebar navigation and dark mode.
• Dashboard, Crawlers, Control, Visibility Score, and Settings in one place.
• Built for site administrators — no code required.

Who is this plugin for?

• Bloggers and publishers who want to know when AI crawlers access their articles.
• Business and marketing sites tracking AI bot traffic to key landing pages.
• Developers and agencies managing AI crawler policy across client WordPress sites.
• Site owners exploring robots.txt and PHP options for AI crawler control.