Skip to content

Free AI tool · runs in your browser

Robots.txt generator for AI crawlers.

Choose a preset or set each crawler yourself: allow everything, block AI training but keep AI search, or block every AI crawler on the list. Add the paths to keep private and your sitemaps, then copy or download the file. It is built and checked in your browser, with the same matcher the AI crawler checker uses.

Preset

Blocks the training and data-collection crawlers; AI search and assistants stay allowed.

One per line, e.g. /admin/ or /*.pdf$. Leave empty to allow all.

Full addresses, one per line.

Crawlers60 blocked
Training and data collection
Training and data collection 60 of 60 blocked
AI search and assistants
AI search and assistants 0 of 73 blocked
Other or unclear
Other or unclear 0 of 47 blocked
robots.txt70 lines
Test in the checker
Training and data collection
60 of 60 blocked
AI search and assistants
0 of 73 blocked
Other or unclear
0 of 47 blocked

Rules applied, and where we read them

How the file is built

A robots.txt that does what the preset says.

  • 01

    One group per blocked crawler

    Each blocked AI crawler is named in its own group that ends in Disallow: /, so it never falls back to the rules for every other crawler. Search engines and everything else keep reading the * group as before.

  • 02

    A list you can trace

    The crawler list is a dated snapshot of the open ai.robots.txt project, with each crawler’s stated purpose. Which group a crawler sits in is our reading of that purpose, and the page says so beside it.

  • 03

    Checked before you copy it

    The generator runs its own output through an RFC 9309 matcher and counts how many crawlers end up blocked. Paste the file into the AI crawler checker to test any URL against it before it goes live.

FAQs · Robots.txt generator

Generating robots.txt · frequently asked questions

Blocking AI training while staying in AI search, where the file goes, and what it cannot do.

Choose "Block AI training, allow AI search". It names each training and data-collection crawler, such as GPTBot, ClaudeBot, CCBot and Google-Extended, in a group ending in Disallow: /, and leaves search and assistant crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot on your normal rules.
The list does not include Googlebot or Bingbot, so the presets leave classic search crawling alone. Google says blocking Google-Extended does not affect inclusion or ranking in Google Search.
At the top level of each host, as /robots.txt in lower case, served as UTF-8 plain text: RFC 9309 requires that location. A file on www.example.com does not cover shop.example.com, so each subdomain needs its own.
Because the ai.robots.txt list includes it. Apple says Applebot powers search in Spotlight, Siri and Safari, and that Applebot-Extended is the token for keeping content out of Apple’s model training. Use the custom setting if you want to block one and not the other.
robots.txt asks crawlers what not to fetch. It does not delete anything already collected, and RFC 9309 says its rules are not a form of access authorization. Ask the operator directly about content it already holds.
It is a snapshot of the ai.robots.txt project, with the source commit and date printed under the generator. New crawlers appear often, so check the date and test your live file with the AI crawler checker.

Next step

Rebuilding the site the file sits on?

QalbIT builds custom web platforms with the crawling, indexing and structured data set up from the first release, not added after launch.

Custom web development

Keep exploring

More free tools

More free tools

Each one opens on a worked example and shows its answer without a signup.

All free tools

Further reading