An AI crawler is a bot operated by an AI company, such as GPTBot or PerplexityBot, that fetches web pages to power retrieval or training. Blocking one removes your eligibility for its citations.
What it means in practice
AI companies run several crawlers, and they do different jobs. OpenAI, for example, documents GPTBot for collecting training data, OAI-SearchBot for ChatGPT search results, and ChatGPT-User for pages a user asks ChatGPT to open. Perplexity runs PerplexityBot and Anthropic runs ClaudeBot. Google-Extended is not a separate crawler but a robots.txt token that controls whether Google may use your content for Gemini.
This matters because blocking the wrong one has the wrong effect. Blocking GPTBot keeps your content out of training, but only blocking OAI-SearchBot keeps you out of ChatGPT search answers.
Example
Say a store blocks every bot with "AI" in its name after reading about scraping, then wonders why ChatGPT recommends its competitors. The search crawler was blocked along with the training crawler.
How to check it
- Open yoursite.com/robots.txt and read every rule that names a bot.
- Check your firewall or CDN bot settings too. Many block AI crawlers by default, whatever robots.txt says.
- Look for the crawlers' user agents in your server logs to confirm they are reaching your pages.
Read our full guide on this topic
Related AI Search terms
-
AI Overviews
AI Overviews are Google-generated summaries shown above the traditional results, synthesized from several sources and linked back to the pages they were drawn from.
-
AI Mode
AI Mode is Google's conversational search experience, where a user asks follow-up questions and Google returns generated answers rather than a standard list of ranked links.
-
Grounding
Grounding is the practice of tying an AI-generated statement to a specific retrieved source, which is what produces a citation instead of an unattributed claim.
-
Citation share of voice
Citation share of voice is how often an AI engine names your business relative to competitors across a fixed set of test prompts. It is the primary outcome metric for Generative Engine Optimization.
-
Hallucination
A hallucination is a confident but factually wrong statement produced by an AI engine. Publishing clear, sourced facts about your business reduces the chance of engines inventing details about you.
Every term we define is in our AEO, GEO and SEO glossary. To see how your own site does on these, run the free AI visibility checker.
