What matters
- Test production, because local HTML cannot reveal an edge challenge.
- Compare page bodies as well as status codes.
- A copied bot name does not prove who sent the request.
The clinic
Choose one article, one tool, robots.txt, and the sitemap. Save the time and final URL for each request.
For every tested identity, record:
- Status and redirect chain.
- Content type and cache result.
- Whether a CAPTCHA, JavaScript check, login, or other block appears.
- Whether the first HTML has the same H1, answer, body, links, and main URL.
- Whether robots permits the intended route.
- Whether the main URL appears in the sitemap.
The result is a small proof table. It is more useful than a green badge from a tool that hides its request.
Download the bot access proof table. It includes the request, purpose, result, proof location, owner, and retest fields.
The common failure
Teams often allow a bot in robots.txt while a CDN or firewall blocks it. The reverse can also happen. The site returns a page, but robots rules block the bot. Both layers matter.
The full technical AEO audit explains bot checks, schema, speed, and repeat tests.
Sources
- Publishers and developers FAQ, OpenAI