Free browser tool
AI crawler starter policy builder.
Choose Allow or Disallow for every purpose. Your browser makes the file. No choice is sent to this site.
Generated robots.txt
What the builder covers
It writes rules for each bot listed above and one fallback for unlisted bots. It does not claim to list every crawler. Provider names and controls can change.
Check the direct provider sources
- Google crawler overview and Google-Extended
- Bing crawlers
- OpenAI bots
- Anthropic crawler controls
- Perplexity bots
Custom and verified agents
This starter does not add a custom user-agent field. A custom name may be wrong or may hide how a vendor really fetches pages. Check the provider record, add any needed group to the downloaded file, and test the final policy on the live site.
What it cannot do
It cannot prove a bot is real, change a firewall, protect private data, or promise search or AI results. Keep private data behind access control and test the final file through the live edge.