Should I block AI crawlers?

Depends on goal. If you want AI traffic / citations: ALLOW (GPT, Claude, Perplexity send users to citing sites). If you don't want your content used for training: BLOCK GPTBot, ClaudeBot. ChatGPT-User and PerplexityBot are USER-AGENT for live queries — blocking them costs you traffic. Trend 2026: allow user-agent bots, sometimes block training bots.

What's Google-Extended?

Separate user-agent for Bard/Gemini training. Disallowing Google-Extended does NOT affect Google Search ranking — only AI features. Many sites allow regular Google but disallow Google-Extended.

Does robots.txt actually work for AI bots?

OpenAI (GPTBot), Anthropic (ClaudeBot), Google (Google-Extended), Perplexity respect robots.txt. Bytespider (TikTok) ignored historically. CCBot (CommonCrawl) respects. Bottom line: most major bots respect it; some don't. Combine with cloudflare bot mgmt for stronger control.

Atrás Atrás to AI

🛂

Generador robots.txt para Bots IA — Herramienta online gratis

Permite / bloquea GPTBot, ClaudeBot, PerplexityBot, Google-Extended

Genera robots.txt con reglas para todos los rastreadores IA principales: GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, CCBot, Amazonbot, Applebot-Extended.

Quick presets

Per-bot policy

GPTBot OpenAI (training)

Used to train GPT models. Most blocked. Blocking does not affect ChatGPT search/queries.

ChatGPT-User OpenAI (live ChatGPT browse)

When ChatGPT user clicks "browse" — fetches your page in real-time. Blocking = no ChatGPT citations.

OAI-SearchBot OpenAI (SearchGPT)

OpenAI search index — newer SearchGPT product. Allow for visibility.

ClaudeBot Anthropic (training)

Anthropic's training crawler. Block if you don't want content used for training.

claude-web Anthropic (live Claude)

Live Claude.ai user agent for web browsing. Allow for citations.

anthropic-ai Anthropic (legacy)

Older Anthropic identifier. Block.

PerplexityBot Perplexity AI

Perplexity sends users to your site as citations. Almost always allow.

Google-Extended Google (Bard/Gemini training)

Separate from Googlebot — does NOT affect search ranking. Many sites block this.

Bytespider ByteDance / TikTok

Aggressive crawler. Often ignored robots.txt historically. Block + use server-level deny.

CCBot CommonCrawl

Used by many AI training sets (originally trained GPT-3). Block to opt out of public training corpora.

Amazonbot Amazon (Alexa, AI)

Used by Alexa + Amazon AI products. Generally allow.

Applebot-Extended Apple (Apple Intelligence training)

New Apple AI training crawler. Separate from Applebot. Block if opting out of training.

cohere-ai Cohere

Cohere training crawler.

Meta-ExternalAgent Meta (Llama training)

Meta's training crawler for Llama models.

MistralAI-User Mistral AI

Mistral live user agent.

DuckAssistBot DuckDuckGo (DuckAssist)

DuckDuckGo AI feature.

Optional paths + sitemap

Disallow paths (one per line, e.g. /admin/, /private/)

Sitemap URL

📄 Generated robots.txt

📚

Saber más

Puntos clave

AI Bot robots.txt Builder is a free, browser-based ai tool — allow / block gptbot, claudebot, perplexitybot, google-extended.
No signup, no downloads, no file uploads. Herramientas run in your browser; the few that need an outside service say so on the page.
Works on desktop, tablet, and mobile. Install as a PWA for offline access.

How to Use AI Bot robots.txt Builder

Open the tool: Launch AI Bot robots.txt Builder on Herramientaolis — no account or download needed.
Enter your data: Paste text, enter values, or select a file directly in your browser.
Get instant results: Everything is processed locally — results appear immediately.
Copy or download: Save your output or share it. Bookmark for quick access next time.

AI Bot robots.txt Builder — Quick Facts

Precio: Gratis — sin límites, sin marcas de agua, sin paywall
Privacidad: Se procesa en tu navegador. Las herramientas que necesitan un servicio externo lo indican en la página. Se cuentan las visitas (Google Analytics).
Plataforma: Cualquier navegador moderno — escritorio, tablet o móvil
Categoría: AI Herramientas on Herramientaolis
Sin conexión: Works offline after first visit (Progressive Web App)

Característica	Detalles
Herramienta	AI Bot robots.txt Builder
Categoría	AI
Requiere registro	No
Subida de archivo	Ninguno — procesado en el navegador
Compatible con móvil	Totalmente adaptable
Coste	Gratis para siempre

Why Use AI Bot robots.txt Builder?

You should try AI Bot robots.txt Builder for a quick, private way to allow / block gptbot, claudebot, perplexitybot, google-extended. Processing happens in your browser and files are never uploaded; the few tools that need an outside service say so on the page.

On the other hand, dedicated APIs or desktop tools suit batch processing better. They also handle server-side automation. For everyday tasks, browser tools offer the best speed, privacy, and convenience.

llms.txt Builder

Generate llms.txt + llms-full.txt for AI crawlers

Open

AI Team Coste Calculadora

Estimate monthly AI API costs for teams of any size

Open

EU AI Act Risk Classifier

Classify your AI system — Prohibited, High, Limited, Minimal risk

Open

🔒

100% Privacidad. Esta herramienta funciona enteramente en tu navegador. Tus datos nunca se suben a ningún servidor.

Generador robots.txt para Bots IA — Herramienta online gratis

Quick presets

Per-bot policy

Optional paths + sitemap

📄 Generated robots.txt

AI Bot robots.txt Builder

How to use this tool

Pick policy per AI bot

Add paths

Copy robots.txt

Frequently Asked Questions

Puntos clave

How to Use AI Bot robots.txt Builder

AI Bot robots.txt Builder — Quick Facts

Why Use AI Bot robots.txt Builder?

You might also like

llms.txt Builder

AI Team Coste Calculadora

EU AI Act Risk Classifier