Description
ChatGPT, Claude, Perplexity and Google AI answer questions by reading websites. If they cannot read yours, you are not in the answer.
Grubbomatic Crawl Readiness runs the free readiness check from crawlreadiness.com against your site and shows the result in wp-admin: a score out of 100, a plain-English verdict, and the list of things AI crawlers need that your site does not have yet. Then it fixes most of them for you, from inside WordPress:
- llms.txt — a Markdown summary of your site, built from your own pages and posts and served at
/llms.txt. Nothing is written to disk and nothing goes stale. - AI crawlers in robots.txt — a Content-Signal line that states your AI preferences, and, if you choose “allow AI search and assistants, block AI training”, a block for each crawler that only collects training data (GPTBot, ClaudeBot, Common Crawl and a few more). It never un-blocks a crawler: any rule you or another plugin already wrote stays in force.
- Meta description and Open Graph tags — a description, title, image and canonical address for every page, when you do not already run an SEO plugin.
- Structured data (JSON-LD) — Organization and WebSite for the site, Article for posts, again only when no SEO plugin owns it.
- agents.json — an optional card at
/.well-known/agents.jsonpointing AI systems at your llms.txt, sitemap and feed.
The check itself takes about ten seconds and can be run again any time. The dashboard shows your latest score.
Check first, then fix
Most AI plugins write a file and hope it helps. This one looks first, from the outside, the way an AI crawler does. It tests 51 AI crawlers and scores them by what each one is for, so blocking the ones that only collect training data doesn’t cost you points. Every finding is matched to the switch that fixes it, and checking again shows whether it worked. llms.txt is one of five fixes, not the point of the plugin.
Works with your SEO plugin
If Yoast SEO, Rank Math, All in One SEO, SEOPress, The SEO Framework or Slim SEO is active, the meta tag and structured data switches stay off so nothing is written twice. llms.txt and the robots rules still apply.
Going further
Being readable is step one. Monitor re-checks your site weekly or daily and tells you the day its verdict or its access policy changes, with the signal that moved. The Machine Readiness Audit measures every page, writes the five files for WordPress and runs an Answer Analysis of the questions your customers ask on live AI search. The check and this plugin are free.
External services
This plugin connects to crawlreadiness.com, a service operated by the plugin author, for one purpose: running the readiness check.
- What is sent: your site address (the home URL), when you press “Check this site” or “Check again”. If you enter an API key in the settings, it is sent with the request. Nothing is sent automatically or in the background.
- What comes back: the score, verdict and list of findings, which the plugin stores in one option so the page and the dashboard widget can show them.
- What the service does with it: fetches your site’s robots.txt, llms.txt, home page and agents.json from the outside, scores them, and keeps the result so the “Full report” link works.
Screenshots
Installation
- Install and activate the plugin.
- Go to Tools AI Readiness and press Check this site.
- Turn on the fixes you want and press Save settings. Check again to see the difference.
An API key is optional. Without one, the check shares a free daily allowance with other sites on your network. A free account at crawlreadiness.com gives you a key with 50 checks a month.
FAQ
-
Does the plugin write files to my server?
-
No. llms.txt, agents.json and the robots.txt rules are served by WordPress when they are requested. If a real llms.txt or robots.txt already exists in your web root, the web server serves that file and the plugin says so.
-
Will blocking AI training crawlers hurt my score?
-
No. The check counts only the crawlers that fetch pages for AI search and answers. Blocking the ones that only gather training data (GPTBot, ClaudeBot, Applebot-Extended, Common Crawl’s CCBot and a few more) is listed in the report but doesn’t lower the score, and that is exactly what the “block AI training” setting does.
-
My site is on a staging or local address. Can I check the live site instead?
-
Yes. Filter the address:
add_filter( 'crawlready_check_url', function () { return 'https://www.example.com/'; } ); -
Does it slow my site down?
-
No. The head tags are built from data WordPress already has, and llms.txt and agents.json are only built when something requests them.
Reviews
There are no reviews for this plugin.
Contributors & Developers
“Grubbomatic Crawl Readiness – AI Crawler Check and Fixes” is open source software. The following people have contributed to this plugin.
Contributors
Translate “Grubbomatic Crawl Readiness – AI Crawler Check and Fixes” into your language.
Interested in development?
Browse the code, check out the SVN repository, or subscribe to the development log by RSS.
Changelog
1.0.2
- The robots.txt switch no longer gives each AI crawler an “Allow: /” rule of its own. A crawler with its own rule ignores the site’s “User-agent: *” rules, so those lines let AI crawlers into /wp-admin/ and into anything else those rules keep out, and could un-block a crawler the owner had blocked. AI crawlers now follow the site’s own rules; “block AI training” still blocks the training-only crawlers by name.
- “Allow AI crawlers” mode leaves ai-train out of the Content-Signal line instead of writing ai-train=yes, so it no longer grants training on the owner’s behalf.
- A blocked-crawler finding no longer points at the robots switch.
1.0.1
- The closing card and this readme describe Monitor and the Machine Readiness Audit; the old LLM Monitor pitch is gone. No functional change.
1.0.0
- First release: the check, llms.txt, AI crawler rules with a Content-Signal line, meta and Open Graph tags, JSON-LD, agents.json, dashboard widget.