AI Robots.txt Generator: Block or Allow AI Crawlers
Choose which AI crawlers may read your site, from training bots like GPTBot and ClaudeBot to search bots like OAI-SearchBot. Get a ready robots.txt block. Free, in your browser.
Pick a policy
Then fine-tune each bot below. The first preset keeps your site eligible for AI search results and user-requested fetches while opting out of model training.
| Bot | Operator | Type | Rule |
|---|
Your robots.txt
If you already have a robots.txt, add these groups to it rather than replacing the file. Upload it to the root of your domain as /robots.txt.
How to use the AI Robots.txt Generator
- Click a preset: block training bots and allow search bots, block every AI bot, or allow all.
- Adjust any single bot with its Allow, Block or Not listed menu.
- Set the path to block (the default / blocks the whole site) and add your sitemap URL if you like.
- Copy the output or download robots.txt.
- Merge it into the robots.txt at the root of your domain and test it with the robots.txt tester of your choice.
Frequently asked questions
What is the difference between training, search and user-triggered AI bots?
Training bots collect content that may be used to train AI models, for example GPTBot and ClaudeBot. Search bots index pages so they can appear in AI search answers, for example OAI-SearchBot and Claude-SearchBot. User-triggered bots fetch a page when a person asks the assistant about it, for example ChatGPT-User and Claude-User. Each operator treats these as separate settings.
If I block GPTBot, am I removed from ChatGPT search?
No. OpenAI documents GPTBot (training) and OAI-SearchBot (search) as separate user agents, and says opting out of OAI-SearchBot is what keeps a site out of ChatGPT search answers. Anthropic likewise lists ClaudeBot, Claude-SearchBot and Claude-User as three separate bots.
Do all AI bots obey robots.txt?
Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores robots.txt, and Meta says meta-externalfetcher may bypass it. To enforce a block you need server or firewall rules, not just robots.txt.
What are Google-Extended and Applebot-Extended?
They are control tokens, not crawlers. Google-Extended has no separate user agent string and controls whether content Google crawls may be used for Gemini training and grounding. Applebot-Extended does not crawl pages; it controls how data crawled by Applebot is used. They never appear in server logs.
Will blocking AI bots affect my Google rankings?
Blocking the bots in this tool does not block Googlebot. Google-Extended is documented as a separate token from Google Search crawling. Always keep your general Googlebot access as you want it.
How current is the bot list?
The list follows each operator’s own documentation, last checked on 2026-11-01. New AI crawlers appear often, so recheck the vendor pages linked in the AI Bot User Agent Checker.