Search Bots, Training Bots, and User Fetchers Are Different

Three different decisions

Search bots collect public pages for search or answer tools. Training bots may collect public text to improve models. User fetch bots open a page when a person asks an assistant to do so.

A site owner may set a different rule for each job. One “block AI” switch misses that key choice.

Purpose	Example bots	Policy question
Traditional search	Googlebot, Bingbot	Do you want the page considered for search?
AI search	OAI-SearchBot, Claude-SearchBot, PerplexityBot	Do you want the source available to supported answer search?
Model training	GPTBot, ClaudeBot, Google-Extended	Do you permit the listed model improvement use?
User fetch	ChatGPT-User, Claude-User, Perplexity-User	May the service open a page for a user request?

These are examples, not a complete registry. Check each provider’s current source before publishing a policy.

The initial policy here

Bobbodily.com lets known search, user fetch, and training bots read the same public pages as people. The goal is broad access. This is not a ranking trick and does not promise inclusion.

Private systems, costly tools, form controls, and harmful traffic stay protected by other rules.

Robots is only one layer

Robots rules state your crawl choices. Site security must still tell real services from people who copy a bot name. When identity matters, use the provider’s IP checks or a trusted bot check.

Use the crawler policy builder to make a starting file. Then check it against your site paths and risks.

For a full release check, use the technical AEO audit and save each result in the bot access proof table.