The Technical AEO Audit: A Practical Protocol

Audit the path, not only the page

A source file can be perfect while the public page is invisible. DNS, TLS, redirects, origin permissions, CDN rules, bot management, browser rendering, and application routing all participate in delivery. Audit the production URL through the same boundary a crawler reaches.

Step 1: inventory the intended public surface

Create a list of canonical page types: homepage, biography, guide, article, case study, tool, service, policy, sitemap, feed, robots file, and error page. Add representative old URLs and query-string variants. For every item, record the intended status, canonical, robots state, content type, and sitemap inclusion.

This becomes a contract the deployment can test.

Step 2: inspect response behavior

For each representative URL, verify:

• HTTPS and one canonical hostname.
• No redirect chain.
• Correct 200, 301, 404, or 410 response.
• Correct Content-Type and useful caching headers.
• No cookie or authentication dependency for public knowledge.
• A real not-found response rather than a soft-404 homepage.
• Consistent response for ordinary browser and legitimate crawler identities.

Repeat from more than one network if a firewall or regional edge is involved.

Step 3: inspect initial HTML

Download the response without running JavaScript. Confirm it contains:

• A unique title and description.
• One descriptive H1.
• The primary answer and complete important body copy.
• Named author, published or updated date, and source links.
• Standard anchor elements for navigation and internal references.
• A self-referencing absolute canonical.
• Matching index directives.
• Valid JSON-LD that describes visible facts.

Client-side JavaScript can enhance a tool. It should not be required to discover the topic or read the central answer.

Step 4: test the crawler policy matrix

Run the same public URL using documented identities for:

• Googlebot and Bingbot.
• OpenAI’s OAI-SearchBot, GPTBot, and ChatGPT-User.
• Anthropic’s Claude-SearchBot, ClaudeBot, and Claude-User.
• PerplexityBot and Perplexity-User.
• A neutral, honest custom user agent.

User-agent strings alone do not verify identity, so do not create insecure allow rules from this test. The purpose is to detect accidental challenges, blocks, or divergent content. Production firewall rules should use provider-published verification or trusted bot categories where available.

Step 5: validate discovery files

Check that robots.txt is plain, reachable, and points to the canonical sitemap. Confirm every sitemap URL is absolute, canonical, indexable, and successful. Validate XML and ensure retired or redirected URLs disappear promptly.

An llms.txt file can be a small human-readable index for consumers that choose to use it. It is not a search directive, not an access-control mechanism, and not a substitute for navigation or a sitemap.

Step 6: compare visible content and schema

Parse every JSON-LD block. Resolve referenced @id nodes. Then compare material properties with the page:

• Person name, roles, affiliations, and profiles.
• Article headline, dates, author, and canonical page.
• Breadcrumb names and destinations.
• Organization identity and contact details.
• Service descriptions, offers, and regions.

Missing optional schema is usually less dangerous than false schema.

Step 7: enforce performance budgets

Measure field Core Web Vitals when traffic exists and lab diagnostics before launch. A practical static-site budget is:

Resource	Internal target
Initial HTML	under 100 KB compressed
Critical CSS	under 25 KB compressed
Default JavaScript	0 KB on reading pages
Hero image	under 160 KB at the served size
Fonts	system stack or tightly subset, self-hosted files

The numbers are budgets, not ranking factors. Their job is to prevent accidental regressions and provide headroom for real-world devices.

Step 8: test accessibility and agent usability

Navigate by keyboard. Inspect heading order, focus visibility, link names, form labels, errors, landmarks, and reflow at 320 CSS pixels. Review the accessibility tree. A stable semantic interface helps assistive technology and agents while improving ordinary usability.

Step 9: produce an evidence-backed backlog

For each failure, keep the URL, timestamp, request identity, response headers, relevant body excerpt or screenshot, expected behavior, impact, owner, and retest date. Classify findings as blockers, important, or improvements. Avoid an opaque “health score” that cannot tell a team what failed.

Use the free robots policy builder for the policy portion. The commercial AEO Visibility Snapshot adds production testing, entity review, prompt baselining, and a prioritized implementation map.