Should I block AI crawlers?

Depends on goal. If you want AI traffic / citations: ALLOW (GPT, Claude, Perplexity send users to citing sites). If you don't want your content used for training: BLOCK GPTBot, ClaudeBot. ChatGPT-User and PerplexityBot are USER-AGENT for live queries — blocking them costs you traffic. Trend 2026: allow user-agent bots, sometimes block training bots.

What's Google-Extended?

Separate user-agent for Bard/Gemini training. Disallowing Google-Extended does NOT affect Google Search ranking — only AI features. Many sites allow regular Google but disallow Google-Extended.

Does robots.txt actually work for AI bots?

OpenAI (GPTBot), Anthropic (ClaudeBot), Google (Google-Extended), Perplexity respect robots.txt. Bytespider (TikTok) ignored historically. CCBot (CommonCrawl) respects. Bottom line: most major bots respect it; some don't. Combine with cloudflare bot mgmt for stronger control.

Indietro Indietro to AI

🛂

Generatore robots.txt per Bot AI — Strumento online gratis

Consenti / blocca GPTBot, ClaudeBot, PerplexityBot, Google-Extended

Genera robots.txt con regole per tutti i principali crawler AI: GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, CCBot, Amazonbot, Applebot-Extended.

Quick presets

Per-bot policy

GPTBot OpenAI (training)

Used to train GPT models. Most blocked. Blocking does not affect ChatGPT search/queries.

ChatGPT-User OpenAI (live ChatGPT browse)

When ChatGPT user clicks "browse" — fetches your page in real-time. Blocking = no ChatGPT citations.

OAI-SearchBot OpenAI (SearchGPT)

OpenAI search index — newer SearchGPT product. Allow for visibility.

ClaudeBot Anthropic (training)

Anthropic's training crawler. Block if you don't want content used for training.

claude-web Anthropic (live Claude)

Live Claude.ai user agent for web browsing. Allow for citations.

anthropic-ai Anthropic (legacy)

Older Anthropic identifier. Block.

PerplexityBot Perplexity AI

Perplexity sends users to your site as citations. Almost always allow.

Google-Extended Google (Bard/Gemini training)

Separate from Googlebot — does NOT affect search ranking. Many sites block this.

Bytespider ByteDance / TikTok

Aggressive crawler. Often ignored robots.txt historically. Block + use server-level deny.

CCBot CommonCrawl

Used by many AI training sets (originally trained GPT-3). Block to opt out of public training corpora.

Amazonbot Amazon (Alexa, AI)

Used by Alexa + Amazon AI products. Generally allow.

Applebot-Extended Apple (Apple Intelligence training)

New Apple AI training crawler. Separate from Applebot. Block if opting out of training.

cohere-ai Cohere

Cohere training crawler.

Meta-ExternalAgent Meta (Llama training)

Meta's training crawler for Llama models.

MistralAI-User Mistral AI

Mistral live user agent.

DuckAssistBot DuckDuckGo (DuckAssist)

DuckDuckGo AI feature.

Optional paths + sitemap

Disallow paths (one per line, e.g. /admin/, /private/)

Sitemap URL

📄 Generated robots.txt

📚

Scopri di più

Punti chiave

AI Bot robots.txt Builder is a free, browser-based ai tool — allow / block gptbot, claudebot, perplexitybot, google-extended.
No signup, no downloads, no file uploads. Strumentos run in your browser; the few that need an outside service say so on the page.
Works on desktop, tablet, and mobile. Install as a PWA for offline access.

How to Use AI Bot robots.txt Builder

Open the tool: Launch AI Bot robots.txt Builder on Strumentoolis — no account or download needed.
Enter your data: Paste text, enter values, or select a file directly in your browser.
Get instant results: Everything is processed locally — results appear immediately.
Copy or download: Save your output or share it. Bookmark for quick access next time.

AI Bot robots.txt Builder — Quick Facts

Prezzo: Gratis — senza limiti, senza filigrana, senza paywall
Privacy: Elaborato nel tuo browser. Gli strumenti che richiedono un servizio esterno lo indicano nella pagina. Le visite vengono conteggiate (Google Analytics).
Piattaforma: Qualsiasi browser moderno — desktop, tablet o mobile
Categoria: AI Strumentos on Strumentoolis
Offline: Works offline after first visit (Progressive Web App)

Caratteristica	Dettagli
Strumento	AI Bot robots.txt Builder
Categoria	AI
Registrazione richiesta	No
Caricamento file	Nessuno — elaborato nel browser
Supporto mobile	Completamente adattivo
Costo	Gratis per sempre

Why Use AI Bot robots.txt Builder?

You should try AI Bot robots.txt Builder for a quick, private way to allow / block gptbot, claudebot, perplexitybot, google-extended. Processing happens in your browser and files are never uploaded; the few tools that need an outside service say so on the page.

On the other hand, dedicated APIs or desktop tools suit batch processing better. They also handle server-side automation. For everyday tasks, browser tools offer the best speed, privacy, and convenience.

llms.txt Builder

Generate llms.txt + llms-full.txt for AI crawlers

Open

AI Team Costo Calcolatore

Estimate monthly AI API costs for teams of any size

Open

EU AI Act Risk Classifier

Classify your AI system — Prohibited, High, Limited, Minimal risk

Open

🔒

100% Privacy. Questo strumento funziona interamente nel tuo browser. I tuoi dati non vengono mai caricati su alcun server.

Generatore robots.txt per Bot AI — Strumento online gratis

Quick presets

Per-bot policy

Optional paths + sitemap

📄 Generated robots.txt

AI Bot robots.txt Builder

How to use this tool

Pick policy per AI bot

Add paths

Copy robots.txt

Frequently Asked Questions

Punti chiave

How to Use AI Bot robots.txt Builder

AI Bot robots.txt Builder — Quick Facts

Why Use AI Bot robots.txt Builder?

You might also like

llms.txt Builder

AI Team Costo Calcolatore

EU AI Act Risk Classifier