AI Crawler robots.txt Generator
Decide who gets to read your site
Choose which AI crawlers may search, fetch or train on your site, get an AI recommendation, and export a clean robots.txt. Free, no signup.
Free · live preview · AI policy advisor1. Start with a policy
The AI advisor is not switched on yet. The presets and the manual controls below work fully without it.
2. Read your current robots.txt (optional)
We fetch your live file, show what it already says about each AI crawler and load those choices below so you can edit them. Your other rules can be kept.
Fetching robots.txt…
3. Set each crawler
All other crawlers (
User-agent: *)
Leave this as "Not mentioned" unless you know you need it. "Block everything" also blocks Google and Bing.
4. Private paths and sitemap
Your robots.txt
Upload this as robots.txt in your site root so it loads at yourdomain.com/robots.txt, then use Read robots.txt above to confirm what crawlers will see. Back up your existing file first.
Good to know: robots.txt is a request, not a lock. Reputable crawlers follow it, but some do not, and user-triggered fetchers may be treated differently by their operators. For a firm block, also use server or firewall rules. Crawler names change, so check each company's documentation before you rely on the list.
How to use the AI Crawler robots.txt Generator
- Pick a preset (allow all, show up in AI answers but block training, block all) or describe your site and let the AI advisor suggest a policy.
- Optionally enter your domain and click Read robots.txt to load your current rules and see how each AI crawler is treated today.
- Fine-tune any crawler with Allow, Block or Not mentioned, and add private paths and your sitemap URL.
- Copy or download the generated file and upload it to your site root as
robots.txt. - Run Read robots.txt again to confirm the live file says what you expect.
About the AI Crawler robots.txt Generator
Your robots.txt file tells crawlers which parts of your site they may request. AI companies now run several different crawlers, and they do different jobs: AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes behind AI answers; assistant fetchers such as ChatGPT-User and Claude-User load a page when a person asks about it; and training crawlers such as GPTBot, ClaudeBot and CCBot collect content that may be used to train models. Blocking all of them with one rule is rarely what a site owner wants, because blocking search crawlers can remove you from AI answers while blocking training crawlers does not.
This AI crawler robots.txt generator lets you set each crawler to Allow, Block or Not mentioned, groups them by purpose, and writes a tidy file with shared rules merged together. You can add private paths such as /admin/ or /cart/ for the crawlers you allow, add your sitemap, and keep any existing rules for other bots. Everything updates live as you click.
The AI policy advisor turns a plain description of your site and goals into a recommended setting for every crawler, with a short reason for each, and loads it into the controls so you can adjust it. The advisor can only choose from the crawlers listed on this page, and it is told that robots.txt is voluntary. If AI is not enabled, the presets and manual controls work the same way.
The robots.txt reader fetches your live file safely, applies the same longest-match, allow-wins-ties rules that major search engines use, and shows whether each AI crawler is currently allowed or blocked and by which rule, so you can see the effect of your file before and after the change. Two names in the list, Google-Extended and Applebot-Extended, are control tokens rather than separate crawlers; blocking them limits how the companies use your content for AI but does not change your normal Google or Apple search presence.
Frequently asked questions
How do I block GPTBot in robots.txt?
Add a group with "User-agent: GPTBot" followed by "Disallow: /". Set GPTBot to Block in the generator and it writes the correct lines for you.
Will blocking AI crawlers hurt my Google rankings?
Blocking training crawlers or the Google-Extended token does not change normal Google Search ranking. Blocking AI search crawlers can remove you from the AI answers that use their index, so choose deliberately.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot collects content for model training, OAI-SearchBot builds the index for ChatGPT search, and ChatGPT-User fetches a page when a person asks ChatGPT about it. They can be set independently.
Do all AI crawlers obey robots.txt?
No. Reputable operators say they follow it, but some crawlers have been reported to ignore it and user-triggered fetchers may be handled differently. For a firm block, add server or firewall rules too.
What are Google-Extended and Applebot-Extended?
They are control tokens, not separate crawlers. They let you opt your content out of certain AI uses by Google and Apple without changing how your pages appear in normal search.
Should I add a Crawl-delay for AI bots?
Most major crawlers do not honor Crawl-delay, so this tool does not add it. Use server rate limiting if a crawler is overloading your site.
Can I keep my existing robots.txt rules?
Yes. Read your current file, tick "Keep the other rules" and the generator will carry over rules for bots that are not in its list and your sitemap lines.
Does the AI advisor store my description?
The description is sent to the configured AI provider to produce a recommendation and is not saved by this tool. Do not include private information.