Robots.txt Generator & Tester
Build robots.txt, block AI crawlers and test any URL. Runs in your browser.
About this robots.txt generator & tester
This robots.txt generator builds the file that tells crawlers which parts of your site to skip, and its built-in tester shows whether any URL is allowed or blocked for any bot. It's for site owners, SEOs and developers who want a correct file without memorising the syntax.
Start from a preset such as Allow all, Block all or Block AI training, then edit the rule groups. A checklist covers about 30 AI crawlers, sorted into training bots, AI search bots and assistant fetchers, so you can opt out of model training without disappearing from AI answers that cite sources.
The tester follows Google's matching rules from RFC 9309: the most specific user-agent group applies, the longest matching path wins, and * and $ work as wildcards. Paste any robots.txt, list your URLs, and you see the exact line that decided each one. Nothing is fetched, so it works for staging sites and files you haven't uploaded yet.
Features
Presets for common setups
Allow all, Block all, Block AI training, Block all AI bots and a WordPress preset that hides /wp-admin/ but keeps admin-ajax.php reachable.
Current AI crawler list
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, PerplexityBot, Bytespider, meta-externalagent and more, each with its operator and what it collects.
Rule groups you can edit
Add groups for any user agents, then Allow and Disallow rows, an optional Crawl-delay and as many Sitemap lines as you need.
A tester that shows its reasoning
Test up to 200 URLs or paths at once. Each result names the group that applied and highlights the matching line with its line number.
Checks on any file
Flags rules placed before a User-agent line, unknown or misspelled directives, unsupported fields like noindex, relative sitemap URLs and files over Google's 500 KiB limit.
Build, test, download
One button sends the generated file to the tester. When it looks right, copy it or download a ready robots.txt.
How to use the robots.txt generator & tester
- 1Pick a preset such as Allow all or Block AI training.
- 2Edit user agents, Allow and Disallow rules and sitemap URLs.
- 3Copy or download robots.txt, or test URLs against it in the Test tab.
When to use it
Opting out of AI training
Block GPTBot, ClaudeBot, CCBot and Google-Extended while Googlebot, Bingbot and AI search bots keep access.
Locking down a staging site
Use Block all on test.example.com so crawlers stay away. Add a password too, since robots.txt is only a request.
Debugging a page that won't get crawled
Paste your live file, enter the URL and Googlebot, and see which Disallow line is catching it.
Cutting crawl waste
Block filter and sort URLs such as /*?sort= so crawlers spend their time on real pages.
How does Google decide if a URL is blocked?
First, Google picks one group. It uses the group whose User-agent line best matches the crawler, like Googlebot. Only if no group names it does the * group apply. The groups are not combined, so rules under * don't apply to Googlebot when a Googlebot group exists. That surprises a lot of people.
Then, inside that group, the longest matching rule wins. If an Allow and a Disallow match with the same length, Allow wins. Paths are case-sensitive, so Disallow: /Admin doesn't block /admin.
- * matches any run of characters: Disallow: /*?sort= blocks /shoes?sort=price.
- $ marks the end of the URL: Disallow: /*.pdf$ blocks /guide.pdf but not /guide.pdf?v=2.
- A trailing slash matters: Disallow: /blog/ leaves /blog itself open.
- An empty Disallow: blocks nothing.
Which AI crawlers should you block?
It depends on what you want to stop. Training crawlers such as GPTBot, ClaudeBot, CCBot and meta-externalagent collect pages to train models. Google-Extended and Applebot-Extended aren't separate crawlers; they are tokens that tell Google and Apple not to use your content for AI training. Blocking them doesn't affect Google Search or Apple's search results.
AI search bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot build indexes for answers that link to sources. Assistant fetchers like ChatGPT-User visit a page when a person asks about it. Block those and your pages can drop out of AI answers. The Block AI training preset leaves them open; Block all AI bots closes everything.
What can't robots.txt do?
It can't hide a page from search results. A blocked URL can still be indexed from links, showing just the address with no description. To keep a page out of results, let it be crawled and add a noindex tag, which you can make with the meta tag generator.
It also can't enforce anything. Well-behaved crawlers follow it; scrapers ignore it. For private content, use a login. And each host needs its own file: example.com/robots.txt doesn't cover shop.example.com.
Why use Mavros
| Mavros | Typical upload sites | |
|---|---|---|
| AI crawlers | About 30, grouped by training, search and assistant use | None, or a short list without explanation |
| Bot list | Current search and AI user agents | Old entries like DMOZ and Teoma |
| Tester | Built in, many URLs at once, shows the matching line | Separate tool, one URL, needs a live site |
| Matching rules | Google's longest-match with * and $ | Often simple prefix checks |
| File checks | Misplaced rules, typos, 500 KiB limit, relative sitemaps | Rarely |
| Privacy | Nothing fetched or uploaded | Fetches your site from their server |
Tips
- Put the file at the root of the host, exactly at /robots.txt. A file in a subfolder is ignored.
- List your sitemap with a full URL. If you don't have one yet, build an XML sitemap from your page list first.
- Don't block CSS or JavaScript files. Google needs them to render your pages properly.
- Google ignores Crawl-delay. Bing reads it, but slow crawling usually means a slow server, which is worth fixing instead.
- Paths with spaces or accents are compared in percent-encoded form. If a rule looks right but doesn't match, percent-encode the path and compare.
Questions
What is a robots.txt file?
It's a plain text file at the root of your site that tells crawlers which paths they may visit. The format is standardised in RFC 9309, published in 2022, and every major search engine follows it.
Where do I put robots.txt?
At the root of each host, so it loads at https://example.com/robots.txt. Subdomains and other protocols need their own file. A robots.txt inside a folder is ignored.
How do I block GPTBot and other AI crawlers?
Add a group with their user agents and Disallow: /. The Block AI training preset does this for 19 training crawlers, including GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended, while normal search engines stay allowed.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether Google may use your content to train and ground Gemini models. Googlebot crawls for Search separately, so rankings and indexing are unaffected.
Will robots.txt remove a page from Google?
No. Disallow stops crawling, not indexing, so a blocked URL can still appear in results if other sites link to it. Use a noindex meta tag on a crawlable page to remove it.
Which rule wins when Allow and Disallow both match?
The rule with the longer path wins. If both are the same length, Allow wins. So Allow: /admin/help/ beats Disallow: /admin/ for /admin/help/faq.
Does Google support Crawl-delay?
No, Googlebot ignores Crawl-delay. Bing reads it. Google adjusts its crawl rate automatically based on how fast your server responds.
Can I test my live robots.txt here?
Yes, by pasting its contents. The tester doesn't fetch URLs, so open yoursite.com/robots.txt in a new tab, copy the text and paste it in. That also lets you test changes before you upload them.