Module 3 · Make your site answerable · Lesson 7
Crawler access: robots.txt, AI user agents and llms.txt
Decide, agent by agent, which AI crawlers may read your site and for what purpose, then check that your CDN and firewall respect that decision.
In this lesson
- Separate AI crawlers that fetch for search from those that collect training data
- Make a deliberate robots.txt decision per user agent
- Confirm in your logs and CDN settings that the decision is applied
Your robots.txt file tells crawlers which parts of your site they may fetch. AI companies operate several crawlers with different purposes: some collect pages for model training, others fetch pages so an assistant can answer a question with current sources. Decide for each one whether you want it, write that decision in robots.txt, and then check that your CDN or firewall is not silently overriding it.
Search purposes and training purposes
The distinction matters because the trade-off is different. Blocking a crawler that fetches pages for search answers can remove you from those answers. Blocking a crawler that collects training data affects what future models learn from your pages, not today's searched answers. Several operators now document separate user agents for the two purposes, so you can make the decisions independently.
| User agent | Operator | Purpose | Decision to make |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT's search answers | Allow if you want to appear as a source in those answers |
| GPTBot | OpenAI | Collects pages that may be used to train models | Allow or block based on your stance on training |
| ClaudeBot | Anthropic | Collects pages that may be used to train models | Allow or block based on your stance on training |
| PerplexityBot | Perplexity | Indexes pages for Perplexity's answers | Allow if you want to appear in those answers |
| Google-Extended | A robots.txt token controlling use of content for Gemini models; it does not change Google Search | Decide on model use separately from Search | |
| Googlebot | Crawls for Google Search, which AI Overviews and AI Mode draw on | Keep allowed if you want to be in Google Search |
Writing the rules
Each rule names a user agent and the paths it may not fetch. A blanket “disallow everything” for an agent blocks it entirely; an empty disallow allows it. Keep the rules simple and comment why each one exists, so the next person to edit the file understands the decision. After any change, request the live robots.txt from your domain and confirm the server returns exactly what you wrote.
The blocks you did not choose
Many sites block AI crawlers without anyone deciding to. Bot-protection settings on a CDN or web application firewall can challenge or block unfamiliar agents by default, and the crawler gets an error page instead of your content while robots.txt says it is welcome. Check your CDN's bot settings for rules that mention AI crawlers, and confirm with your logs.
Reading your bot logs
Your server or CDN logs record every request, including the user agent and the response code. Filter them for the AI user agents in the table. Requests that receive a success response are reaching your content; challenge pages, forbidden responses or no requests at all point to a block. Be aware that a user agent string can be faked, so treat logs as a signal and use the operators' published verification methods when it matters.
What about llms.txt?
llms.txt is a proposed plain-text file at the root of a site that summarises the site and links to its most useful pages, written for language models. It is easy to publish and does no harm.
Worked example: Northwind Dental
Northwind's robots.txt is a template from its web agency and blocks nothing. Its logs, though, show no requests from OAI-SearchBot or PerplexityBot at all, and a handful of challenge responses for GPTBot. The CDN's bot-protection mode is set to challenge unfamiliar agents. Northwind decides to allow the search crawlers and Googlebot and to block training crawlers, writes that into robots.txt with comments, and adds allow rules for the search agents in the CDN. A week later, the logs show successful requests from both search crawlers.
Key takeaways
- AI crawlers serve different purposes; decide search access and training access separately.
- robots.txt states your decision, but CDN and firewall settings can override it, so check your logs.
- llms.txt is an unproven experiment; it is no substitute for crawlable pages.
Practice task
Adds to your plan: Site fixes
- Open your live robots.txt and list every rule that mentions an AI user agent.
- For each agent in the lesson's table, write allow or block, the reason and today's date.
- Check your CDN bot settings and a week of logs for those agents, and note any block you did not choose.
Knowledge check
Three questions. Pick an answer to see whether it is right and why.