Robots.txt Studio
Build a robots.txt with AI-crawler policy presets and lint it as you type — then test any URL against any file with real RFC 9309 longest-match semantics, with the winning line highlighted. 100% in-browser, no AI, no uploads.
AI crawler user-agents — one-click policy
| Operator | User-agent | Purpose | robots.txt | In your file |
|---|---|---|---|---|
| OpenAI | GPTBot | training | honors | |
| OpenAI | OAI-SearchBot | AI search | honors | |
| OpenAI | ChatGPT-User | user-triggered | honors | |
| Anthropic | ClaudeBot | training | honors | |
| Anthropic | Claude-SearchBot | AI search | honors | |
| Anthropic | Claude-User | user-triggered | honors | |
| Perplexity | PerplexityBot | AI search | declared | |
| Perplexity | Perplexity-Useruser-triggered fetchers may act on behalf of humans and can ignore robots.txt by design | user-triggered | mixed history | |
| Google-ExtendedGemini training opt-out token — does NOT affect Google Search | training | honors | ||
| Google-CloudVertexBot | training | honors | ||
| Apple | Applebot-Extendedtraining opt-out token; Applebot itself powers Siri/Spotlight search | training | honors | |
| Common Crawl | CCBotopen dataset that feeds many LLMs | training | mixed history | |
| Meta | Meta-ExternalAgent | training | declared | |
| Meta | Meta-ExternalFetcher | user-triggered | declared | |
| ByteDance | Bytespiderfrequently ignores robots.txt — server/WAF rules needed for a hard block | training | often ignores | |
| Amazon | Amazonbot | training | honors | |
| DuckDuckGo | DuckAssistBot | AI search | honors | |
| Mistral | MistralAI-User | user-triggered | declared | |
| (legacy) | anthropic-aideprecated token, kept for compatibility blocks | training | honors | |
| (legacy) | Claude-Webdeprecated token, kept for compatibility blocks | training | honors |
Registry v1.0 (reviewed quarterly). Buttons edit standalone User-agent blocks in the Build editor above.
RFC 9309 matching rules
Among all Allow/Disallow rules that match a path, the one with the most matched characters decides — file order is irrelevant.
If an Allow and a Disallow match with equal length, Google resolves the tie in favor of Allow.
* matches any character sequence; $ anchors the end of the URL. Disallow: /*.pdf$ blocks PDFs but not /file.pdfx.
A crawler obeys only the group whose User-agent token best matches its name; * applies only when nothing else does.
Robots.txt in 2026: crawl control meets AI policy
robots.txt is the oldest contract on the web — a plain-text file at the site root that asks crawlers what not to fetch — and in 2022 it finally became a real standard, RFC 9309. The formalization matters because the folk understanding (“rules apply top to bottom”) is wrong: compliant crawlers pick the most specific user-agent group, then apply the longest matching rule, with Allow winning ties. That's why a one-character change to a path pattern can flip a verdict, and why this studio tests with the actual algorithm and shows you the exact winning line rather than asking you to reason about it.
The stakes changed with AI crawlers. Beyond Googlebot and Bingbot, your file now speaks to training bots (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent), AI-search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetchers (ChatGPT-User, Claude-User). These are genuinely different policy decisions: many sites want to be findable in AI search while opting out of model training — which is exactly what the “search-visible, no training” preset emits, built from the bundled registry. Google-Extended deserves special mention: blocking it opts you out of Gemini training without touching your Google Search presence.
Be honest about what the file can and cannot do. It is a request, not a lock: Bytespider has a documented history of ignoring it, and user-triggered fetchers act on behalf of a human who could open your page in a browser anyway. If you need enforcement, that's a server-level or WAF decision. Also remember two classic traps: a page blocked in robots.txt can still appear in results (Google just can't read it — including any noindex tag on it), and noindex inside robots.txt has been ignored by Google since 2019.
Keep the file lean — Google parses only the first 500 KiB — declare your sitemap with a Sitemap: line so discovery doesn't depend on links alone, and re-test after every edit. Then verify the rest of the stack: run the full SEO Audit Pro on key pages, build the sitemap itself in the XML Sitemap Studio, and pair your AI policy with an llms.txt so the crawlers you allow know where the good content is.
Trusted for Crawl Control
“The winning-line highlight is the feature every robots tester should have had years ago. I batch-tested 80 URLs from a migration against the new file and caught two patterns where a shorter Disallow was being beaten by an old Allow. The RFC semantics are actually correct, which I cannot say for most testers.”
“We used the 'search-visible, no training' preset as our AI policy baseline across 14 properties. Having the registry with honest honors/ignores labels — including the Bytespider caveat — saved me a day of research. The lint caught a rule sitting above our first User-agent line that every crawler had been ignoring.”
“Client staging site got indexed because of a robots typo — now I lint every file here before deploy. The byte counter against Google's 500 KiB cap and the BOM detection are details nobody else bothers with. Paste, test, ship. No signup, nothing uploaded.”
“The e-commerce facet preset was 90% of what our store needed; I added two site-specific patterns and verified them against real product URLs in test mode. Would love saved profiles someday, but the export is clean and the tester's reasoning strings make code review of robots changes actually possible.”
Love using our calculator?
More of SEO Forge
Similar Calculators
More tools in the same category
SEO Audit Pro
Paste HTML or fetch a live URL and get a unified 85-check audit — on-page, technical, content, links, images, structured data and security — with weighted scores, a fix-first list and JSON/CSV/Markdown export.
SERP & Social Preview
Pixel-accurate Google SERP preview (desktop + mobile, canvas-measured Arial with query bolding) plus Facebook, X, LinkedIn, WhatsApp and Discord card mockups.
Meta Tags Studio
Build a complete, correct <head>: title and description with live character + pixel meters, canonical, robots directives, Open Graph and Twitter cards — copy one clean block.
Keyword Density & N-Grams
Exact keyword density, a placement matrix (title, H1, first 100 words, URL, alts), keyword proximity, 1–4-gram tables and top-word analysis — computed, not guessed.
Readability Scorer
All seven readability formulas — Flesch, Flesch–Kincaid, Gunning Fog, SMOG, Coleman–Liau, ARI, Dale–Chall — computed from first principles with per-sentence highlighting.
XML Sitemap Studio
Generate, validate, count and split XML sitemaps against the full protocol: namespace, absolute ≤2,048-char URLs, W3C dates, 50k/50MB limits, freshness histogram and index files.
Often Used Together
Complementary tools for complete analysis
Related Articles
Dive deeper with our expert guides and tutorials related to Robots.txt Studio
RFC 9309 longest-match tester · live lint · AI-crawler registry & presets · sitemap lines · in-browser · Last reviewed: 2026-07