November 16, 2026 · ChimpanSEO

What does robots.txt actually control?

llms.txt and robots.txt do different jobs, and confusing them leads to bad decisions. robots.txt is a formalized protocol (RFC 9309) that tells crawlers which parts of your site to skip, while llms.txt is a Markdown file that curates content for AI systems at inference time. Neither one enforces anything on its own: robots.txt works only because compliant crawlers choose to respect it, and llms.txt is a proposal, not a standard. Understanding this split is the first step to controlling how AI systems interact with your site in 2026.

What does robots.txt actually control?

robots.txt is a plain text file at your domain root that tells crawlers which URLs they may fetch, but it has no technical enforcement behind it.

According to Kinsta, robots.txt is formalized as RFC 9309 and works only because crawlers decide to follow it. AIOSEO puts it plainly: robots.txt is a protocol, not an enforcement mechanism. A bot that ignores your rules can still reach any public URL.

The file itself follows strict rules. Prerender notes that robots.txt must be UTF-8 encoded, one file per site, with case-sensitive rules and groups starting with User-agent followed by Allow and Disallow lines. Get the syntax wrong and compliant crawlers may misread your intent.

What is llms.txt designed to do?

llms.txt is a Markdown file at yourdomain.com/llms.txt that gives AI models a curated, structured map of your best content at the moment they answer a query.

Similarweb explains that llms.txt targets inference time, when an LLM or agent is actively answering a user, not training or broad web indexing. The Prerender docs describe the companion file llms-full.txt as a complete text file for models with large context windows or RAG systems.

The official spec, proposed by Jeremy Howard of Answer.AI, defines a simple structure:

  • An H1 title at the top
  • A blockquote summary
  • Optional H2 sections
  • A Markdown list of links with descriptions
  • An optional “## Optional” section for lower priority links

llms.txt vs robots.txt: what is the real difference?

robots.txt is access control and llms.txt is curation: robots.txt restricts what crawlers fetch, while llms.txt recommends what AI systems should read and cite.

Similarweb frames the trio clearly: robots.txt handles access control, sitemap.xml handles discovery, and llms.txt handles curation. The spec states explicitly that llms.txt coexists with robots.txt and sitemap.xml without replacing them. AIOSEO confirms that adding or removing a page from llms.txt does not grant or revoke permission to crawl it, since that job stays with robots.txt.

File Primary job Enforcement Status
robots.txt Access control for crawlers None, relies on crawler compliance RFC 9309
llms.txt Curation for AI at inference time None, advisory only Proposal from 2024, not a standard
sitemap.xml Discovery of URLs None, advisory only Widely supported convention

The ARP v2.0 draft adds a useful distinction: robots.txt controls crawler access while ARP does not, and llms.txt provides raw text while ARP provides structured claims. That framing reinforces the split between permission and presentation.

Does blocking a URL in robots.txt remove it from AI answers?

No, blocking a URL in robots.txt does not guarantee exclusion from AI generated answers or search results, because crawling and indexing are separate processes.

Prerender states that stronger control requires noindex directives, password protection, or removing the page entirely. AIOSEO makes the same point about Google: a blocked URL can still be discovered through external links and remain in the index. If your goal is removal, robots.txt alone will not get you there.

Do major AI crawlers actually use llms.txt?

Most major AI crawlers largely ignore llms.txt, and adoption sits at roughly 9 to 10 percent of sites according to current tracking.

Kinsta reports that GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended largely are not fetching llms.txt. Google’s own search advocates have stated flatly that llms.txt is not used as a ranking or crawling signal, comparing it to the old keywords meta tag. Similarweb reports the same public position from Googlers: no Gen AI system currently uses llms.txt.

That does not make the file useless. It means you should treat it as documentation for agents that choose to read it, not as a ranking lever. Publish it for the systems that respect it and keep your expectations calibrated.

What actually enforces your preferences?

Bot protection is the only layer that truly blocks, challenges, or rate limits a crawler that ignores your preferences, because robots.txt and llms.txt are requests rather than controls.

Kinsta points to Kinsta Bot Protection and Cloudflare AI Crawl Control as the tools that apply real enforcement. If a crawler disregards your robots.txt, those systems are what stop it. A practical stack looks like this:

  1. Write a valid robots.txt in UTF-8 with clear User-agent, Allow, and Disallow groups.
  2. Add noindex directives or password protection for pages you truly need excluded.
  3. Publish llms.txt and llms-full.txt to guide agents toward your best content.
  4. Turn on bot protection at the edge to enforce what the files only request.

1 Write a validrobots.txt in UTF-8 with… 2 Add noindexdirectives or password… 3 Publish llms.txtand llms-full.txt to… 4 Turn on botprotection at the edge to…

The ARP v2.0 draft notes that ARP uses the /.well-known/ convention, the same approach as security.txt, which shows how newer machine readable files are being standardized while robots.txt remains the access control baseline.

Frequently Asked Questions

Can I use llms.txt instead of robots.txt?

No. llms.txt curates content for AI systems and does not control crawling. Permission to fetch a URL stays with robots.txt, and the llms.txt spec explicitly says the two files coexist rather than replace each other.

Does Google use llms.txt for ranking?

No. Google’s search advocates have stated publicly that llms.txt is not used as a ranking or crawling signal and compared it to the old keywords meta tag. Treat it as guidance for agents, not an SEO lever.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a structured list of links with descriptions, while llms-full.txt is a complete text file built for models with large context windows or RAG systems that need the full content in one place.

Does blocking a page in robots.txt remove it from Google?

Not necessarily. Crawling and indexing are separate, and a blocked URL can still be discovered through external links. Use noindex, password protection, or removal if you need the page gone from results.

If you run a site that AI systems read, treat these files as layers rather than alternatives. robots.txt sets the rules, sitemap.xml aids discovery, llms.txt curates what matters, and bot protection enforces the boundaries. Audit your stack in 2026 and make sure each layer does its own job.

Related reading

This article was written and published with ChimpanSEO

Write SEO/AEO articles and publish them to WordPress: it usually takes 1-2 minutes. Try it free, no card required.

Related articles