Search Bots, Training Bots, and User Fetchers Are Different

Three different decisions

Search discovery answers, “May this provider crawl public pages for a search or answer index?” Training crawling asks whether public material may be collected for model improvement. User fetch asks whether an assistant acting at a person’s request may retrieve a page now.

A publisher can reasonably answer those questions differently. A single “block AI” toggle throws away the distinction.

The initial policy here

Bobbodily.com allows documented search, user-fetch, and training crawlers to access the same public pages available to people. That reflects a preference for broad public accessibility. It is not presented as a ranking technique and carries no inclusion guarantee.

Private systems, future compute-heavy endpoints, form administration, and abusive traffic remain protected outside that policy.

Robots is only one layer

Robots rules express crawl preferences. Edge security still needs to distinguish verified services from clients merely copying a provider’s user-agent string. Where access rules depend on identity, use provider-published IP verification or a reputable verified-bot facility.

Use the crawler policy builder to generate a starting file, then review it against your actual routes and risk model.