Guides

llms.txt, robots.txt and AI crawlers: what actually matters

A file does not automatically make a website visible in AI answers. Search, training, user-triggered actions, advertising and technical access must be separated before deciding what to allow or block.

In brief

robots.txt manages crawling by bots that respect it; noindex controls indexing when the crawler can read the page; sitemap.xml supports discovery; llms.txt is an optional proposal and Google says it is not used for Search or AI features. OpenAI separates OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot: configure them according to purpose, not all in the same way.

Gian Luca Partengo
Web developer since 1995 · custom websites · Updated 5 August 2026

There is no single switch called “AI visibility”

The correct decision starts with four separate questions. Skipping one produces contradictory configurations.

01Do you want to appear in results and citations?
02Do you want to exclude content from training?
03Do you want to allow user-requested actions?
04Is the page actually reachable beyond robots.txt?

Search, training, actions and infrastructure access are separate layers. The same answer does not automatically apply to all of them.

robots.txt, noindex, sitemaps, schema, llms.txt and WAFs do different jobs

Their names are often mixed together, but each layer answers a different question.

Tool	What it does	What it does not guarantee	When to use it
robots.txt	Declares which paths a compliant crawler may request.	It is not access control and does not always prevent a URL from being known.	Manage crawling and separate purposes by user-agent.
noindex	Asks search engines not to show a resource in their results.	It must be read: if the page is blocked from the crawler, the rule may remain unseen.	Exclude publicly reachable pages from an index.
sitemap.xml	Lists canonical URLs and dates that support discovery and recrawling.	It cannot force crawling, indexing, ranking or citation.	Consistently signal the pages that matter.
Structured data	Describes entities and properties through a machine-readable vocabulary.	It does not grant access and no special schema guarantees AI visibility.	Make information already present in the content explicit and consistent.
llms.txt	Proposes a curated Markdown index for tools and models at inference time.	It is not a crawling directive, ranking factor or universally adopted standard.	Complex documentation contexts where a real consumer and maintenance process exist.
WAF, CDN and authentication	Technically allows or denies requests and protects private resources.	Overly aggressive rules can block legitimate crawlers as well.	Security, rate limits, bot verification and actual protection.

A professional setup coordinates these layers; it does not choose one file as a replacement for the others.

llms.txt is a useful proposal in some contexts, not the new robots.txt

The specification published by Jeremy Howard in 2024 proposes a Markdown file at the site root containing a summary, instructions and selected links.

The idea addresses a real problem: complex pages, navigation, scripts and large bodies of documentation are difficult to condense within a model’s context. A short index can point to the most authoritative resources and their Markdown versions.

The file can make sense for technical documentation, APIs, universities or knowledge bases where users and tools explicitly know to read it. It can also become an internal artefact for testing how an assistant retrieves curated information.

It neither authorises nor prohibits access. It does not replace robots.txt, sitemaps, canonicals, structured data, readable HTML or complete content. Above all, the file’s presence does not prove that a specific engine consumes it.

Adding it is inexpensive; keeping it correct requires attention. An outdated index that conflicts with the website creates one more source of ambiguity.

2024

Open proposal

Its official page presents it as a standardisation proposal, not a universal web protocol.

OPT

Optional implementation

It has value only when it describes genuine, stable, maintained sources and the target system uses it.

0

Declared Google benefit

Google says Search ignores llms.txt: it neither helps nor harms rankings or generative features.

My practical position

I do not sell it as an SEO or GEO shortcut. I assess it as an optional documentation layer after fixing content, HTML, linking, sitemaps, metadata, accessibility and access for crawlers that are actually relevant.

Google Search and AI Mode still rely on Googlebot, not llms.txt

Google’s documentation is explicit: AI Overviews and AI Mode use the requirements and controls of traditional Search.

Official guidance

Technical SEO remains the basis of generative visibility

To be eligible for Google’s AI features, a page must be indexed, eligible to appear with a snippet and meet normal Search requirements. No AI-specific file or markup is required.

Google uses Googlebot to crawl for Search. Google-Extended is a separate token for controlling certain training and grounding uses in Gemini and does not affect inclusion or ranking in Google Search.

Googlebot

Controls access to content for Search, including generative features integrated into search.

Snippet eligibility

nosnippet, data-nosnippet and max-snippet control how much of the page may be shown or directly used in Google Search AI features.

Google-Extended

Expresses a choice about training and grounding in certain Gemini products without opting out of Google Search.

No guarantee

Crawlability, indexing and compliance make a page eligible; they do not guarantee crawling, ranking, citation or traffic.

For citable content, authority and editorial structure, also read how to appear on ChatGPT and AI search.

OpenAI uses different crawlers because their purposes are different

The distinction makes it possible, for example, to participate in ChatGPT Search while indicating that content should not be used to train foundation models.

OAI-SearchBot

Search and citations

Used to surface websites in ChatGPT Search. If excluded, the site does not enter search answers, although it may still appear as a navigational link.

GPTBot

Potential training

Crawls content that may be used to improve the usefulness and safety of foundation models. It can be blocked independently of OAI-SearchBot.

ChatGPT-User

User-triggered action

May visit a page when a person or Custom GPT requests it. It does not crawl automatically and, because the action is user-initiated, robots.txt may not apply.

OAI-AdsBot

Ad validation

Visits landing pages submitted as advertising destinations, checks their safety and may assess relevance. The data it gathers is not used to train foundation models.

robots.txt is not noindex—and neither makes a page private

Define the desired outcome first, then select the control capable of producing it.

Goal	Control	Expected effect	Caution
Reduce or prohibit crawling	Disallow in robots.txt by user-agent and path.	A compliant crawler does not request the prohibited content.	The URL may be discovered elsewhere and the protocol does not protect against non-compliant bots.
Stay out of search results	A robots noindex meta tag or X-Robots-Tag.	The engine removes or does not show the resource after reading the rule.	Do not simultaneously block the crawler that needs to read noindex.
Limit excerpts and direct inputs	nosnippet, data-nosnippet or max-snippet where supported.	Reduces or excludes portions that can be used to present the result.	Support and effects are engine-specific and may reduce visibility.
Make content genuinely private	Authentication, authorisation and server-side access control.	The resource is not served without valid credentials.	robots.txt is public; never list secret paths and assume they will remain hidden.

The most dangerous mistake

Combining Disallow and noindex on the same page in an attempt to strengthen the block: the crawler cannot enter and may therefore never read noindex. Sensitive information must not be publicly served without authentication.

Four starting configurations to adapt to the actual website

These are readable examples, not fragments to copy blindly: precedence, existing groups, CDN rules and project goals must be checked.

robots.txt

Search allowed, training excluded

Allows OAI-SearchBot, excludes GPTBot and Google-Extended, and signals the sitemap. Googlebot remains allowed when no applicable prohibition exists.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

HTML

Reachable page kept out of the index

The crawler must be able to receive the HTML and read the directive. Non-HTML files can use an X-Robots-Tag response header.

<meta name="robots" content="noindex">

HTML

Controlled preview, not a complete opt-out

Limits the text excerpt while allowing a large image preview. It is an editorial trade-off to test, not an automatic improvement.

<meta name="robots" content="max-snippet:160, max-image-preview:large">

robots.txt

Landing page reachable for ChatGPT Ads

Allows relevant crawlers on the advertising path. The page must still pass firewalls, CAPTCHAs, geoblocks, authentication and rate limits.

User-agent: OAI-AdsBot
Allow: /landing-page/

User-agent: OAI-SearchBot
Allow: /landing-page/

Control search, snippets, training and emerging signals without mixing them up

Each platform documents its own controls. A sensible editorial choice weighs visibility, content reuse and commercial consequences.

nosnippet

For Google, this prevents snippets and direct use of the content as input for AI Overviews and AI Mode. It can therefore substantially reduce potential visibility.

max-snippet

Limits the character count of the excerpt. It does not select the passage and is not a universal control for every AI system.

data-nosnippet

Excludes specific portions from Google’s presentation while leaving the rest of the page available—useful for sensitive but non-private elements.

Google-Extended and GPTBot

Express preferences about potential training or grounding without forcing a publisher to leave Google Search or ChatGPT Search respectively.

Evolving standards

Content Signals are not yet a universal lock

Cloudflare proposes search, ai-input and ai-train directives to express permitted purposes. They may add contractual or operational clarity within the relevant ecosystem.

Cloudflare itself states that the signals express preferences rather than technical enforcement: a crawler can ignore them. Treat them as an emerging layer, not a replacement for server-side access control and applicable agreements.

A crawler can be allowed in robots.txt and still be blocked

The real journey crosses DNS, network, CDN, firewall, application and rendering. One inconsistent layer is enough to return an empty page or an error.

01

WAF and bot management

Anti-bot rules may classify a legitimate request as hostile automation and return a 403.

02

CAPTCHAs and JavaScript challenges

A human browser completes the check; a crawler may receive only the challenge page.

03

Rate limiting

Overly strict thresholds or crawl spikes produce 429 responses and interrupt resource retrieval.

04

Geoblocks and IP addresses

The page works in one country but not from the network ranges published by the crawler provider.

05

Redirects and authentication

Chains, logins, deep links or conditional destinations prevent access to the final content.

06

Script-dependent HTML

The initial response contains little content and requires execution, interaction or APIs that the crawler does not complete.

Allow does not mean successful access

Verification must inspect HTTP status, redirects, received HTML, resources, security logs and official IP ranges. Trusting a user-agent alone is unsafe because it can be spoofed.

How to verify what a search engine or AI can actually read

A serious review produces repeatable evidence and separates declared behaviour from observed behaviour.

01
Open robots.txt
Check the 200 response, text format, groups, paths, sitemap directives and automatically generated rules.
02
Test canonical URLs
Check the homepage, service pages, guides, product pages and landing pages—not just one sample URL.
03
Read headers and redirects
Record HTTP status, Location, content-type, X-Robots-Tag, canonical and the final destination.
04
Inspect initial HTML
The title, main content, links and essential data should exist without requiring human interaction.
05
Inspect logs
Look for requests, frequency, status, URLs, timings and WAF blocks associated with the relevant crawlers.
06
Verify bot identity
Compare IP addresses or official provider verification procedures: anyone can copy a user-agent string.
07
Use engine tools
Search Console and URL Inspection help explain what Google receives and which directives it recognises.
08
Monitor outcomes and referrals
Observe citations, utm_source=chatgpt.com traffic, landing pages and conversions without assigning causality to llms.txt.

No special file can replace an understandable website

Crawlers may reach the page; they still need to find useful, consistent, verifiable and connected information.

The correct hierarchy

The web document comes first, accessory indexes second

A good website exposes complete content, descriptive titles, ordered sections, real links, consistent entities, dates and editorial responsibility in its HTML. Sitemaps and structured data reinforce that foundation.

llms.txt may point to the most important resources, but if those resources are generic, contradictory or invisible without JavaScript, it cannot create authority or understanding on their behalf.

Readable HTML

Main information in the response, semantic structure and links with genuine href destinations.

Non-commodity content

Experience, data, cases, limitations and an editorial position that add more than a generic summary.

Entity consistency

Names, services, prices, locations and properties agree across text, schema, feeds, profiles and linked pages.

Performance and accessibility

People, assistive technologies and agents can find and use the same information without avoidable barriers.

Explore visibility in the SEO and GEO guide and interaction in the AI agent-ready website guide.

Eight mistakes that turn configuration into empty marketing

Experimenting is not the problem. The problem is assigning an effect to a file that the platform does not declare or that no one has measured.

“llms.txt improves Google rankings”

Google says it ignores the file for visibility and ranking, including generative Search features.

Generate a file and forget it

Removed URLs, changed offers and outdated descriptions create an unreliable parallel index.

Block every AI bot together

Search, training, user actions and advertising have separate purposes and controls.

Use robots.txt to hide secrets

The file is public and voluntary and may even reveal the paths that were meant to stay hidden.

Combine Disallow and noindex blindly

The crawl restriction may stop the search engine from reading the exclusion directive itself.

Allow the bot but block it in the WAF

The declaration is correct, yet the real request ends with a 403, 429 or challenge page.

Trust the user-agent alone

The string can be spoofed; allowlisting and diagnosis should rely on official procedures and ranges.

Confuse eligibility with a guarantee

Correct configuration removes technical blockers; it cannot buy rankings, citations or conversions.

Checklist before publishing or changing the rules

Ten checks connecting business goals, content, infrastructure and measurement.

01
Define objectives
Write separate choices for search, training, user actions, advertising and private content.
02
Inventory crawlers
Use official names and documentation and record the date, purpose and source page.
03
Map URLs
Separate public, indexable, excluded, transactional and private areas.
04
Review robots.txt
Check precedence, duplicate groups, wildcards, sitemaps and third-party generated instructions.
05
Review meta and headers
Verify robots meta tags, X-Robots-Tag, canonicals and snippet controls on every relevant template.
06
Decide on llms.txt
Implement it only with a defined purpose, owner, canonical sources and update procedure.
07
Align CDN and WAF
Compare declared rules with firewalls, challenges, geoblocks, rate limits and official IP ranges.
08
Test from outside
Check status, redirects and HTML from the network, not only within the developer’s authenticated browser.
09
Collect evidence
Keep test output, logs, verification date and the published configuration.
10
Plan maintenance
Review again when engines, products, CDNs, URL structures, content or business purposes change.

On a custom website these controls belong to the architecture, not a plugin

The advantage is not writing robots.txt by hand; it is understanding the full path through content, routing, headers, security and measurement.

Project method

One reality, expressed across every layer

When I build or review a website, directives are not added at the end as GEO decoration. They derive from the pages that must remain public, the commercial goals and the tools actually in use.

Controlled routing and canonical URLs
Complete, semantic HTML
Consistent sitemaps and metadata
Crawlers separated by purpose
Verified WAF rules and headers
Testing, logs and maintenance

Primary sources and current documentation

AI crawlers and products change quickly. This guide separates protocols, official statements, community proposals and project decisions.

llms.txt — original proposal
Purpose, Markdown format, relationship with sitemaps and robots.txt, and its status as an open proposal.

Open source
Google — Optimising for generative AI features
Technical SEO, content, AI Mode, myths and Google’s explicit statement that llms.txt is irrelevant to Google Search.

Open source
Google — AI features and your website
Googlebot, eligibility, snippet controls, noindex and the relationship with Google-Extended.

Open source
Google — Introduction to robots.txt
The file’s purpose, limitations, disallowed URLs and the difference between crawl management and removal from the index.

Open source
Google — Robots meta and X-Robots-Tag
noindex, nosnippet, data-nosnippet, max-snippet and the requirement that a crawler can read directives.

Open source
Google — Google-Extended
A token for training and grounding in Gemini products, separate from Google Search and not used as a ranking signal.

Open source
IETF — Robots Exclusion Protocol RFC 9309
Technical protocol specification, matching, groups, file access and compliant crawler behaviour.

Open source
OpenAI — Overview of crawlers
OAI-SearchBot, GPTBot, ChatGPT-User, OAI-AdsBot, purposes, user-agent strings and published IP ranges.

Open source
OpenAI — Publishers and Developers FAQ
ChatGPT Search, noindex, OAI-SearchBot, GPTBot and referral measurement through utm_source=chatgpt.com.

Open source
OpenAI — Advertiser crawler access
OAI-AdsBot, OAI-SearchBot, WAFs, CDNs, CAPTCHAs, rate limits, 403, 429 and verification through official IP ranges.

Open source
Cloudflare — Content Signals Policy
Proposed search, ai-input and ai-train signals and clarification of their technical enforcement limits.

Open source

Sources checked on 5 August 2026. Crawler names, products, IP ranges and directive support may change: always recheck official documentation before making changes.

Frequently asked questions

Direct answers that separate documented behaviour from shortcuts being marketed as facts.

Does llms.txt improve Google rankings?

No. Google explicitly says Search does not use llms.txt and that the file neither helps nor harms visibility or rankings, including generative Search features.

Does that make llms.txt useless?

Not necessarily. It may serve as a Markdown index for documentation or tools that state they use it. It still needs a real consumer, canonical sources and maintenance and should not be presented as a universal SEO or GEO requirement.

Does robots.txt stop a page from being indexed?

Not reliably. It manages crawling and a blocked URL may be discovered through other links. To exclude a page from results, use noindex while leaving it accessible to the crawler that needs to read the directive.

Can robots.txt protect confidential data?

No. The file is public and directives are voluntarily followed by compliant crawlers. Confidential content requires authentication, authorisation and server-side controls.

Can I appear in ChatGPT Search without allowing training?

Yes, according to OpenAI documentation: allow OAI-SearchBot and disallow GPTBot. The two settings are independent.

Does ChatGPT-User respect robots.txt?

OpenAI explains that ChatGPT-User acts at a user’s request and does not perform automatic crawling; robots.txt may therefore not apply. It is not used to determine Search inclusion.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot concerns content that may contribute to training foundation models; OAI-SearchBot supports ChatGPT’s search and citation features.

What is Google-Extended?

It is a robots.txt token for controlling certain training and grounding uses in Gemini products. Google says it does not affect inclusion or ranking in Google Search.

Will blocking Google-Extended remove my site from AI Overviews?

No. AI features integrated into Google Search use Googlebot and snippet controls. Google-Extended is separate from Search.

Why does a bot receive a 403 when robots.txt allows it?

Because a WAF, CDN, bot manager, CAPTCHA, authentication, geoblock or rate limit may deny the request after robots.txt has been evaluated.

Does structured data replace llms.txt?

No: their purposes differ. Structured data describes entities and properties found on the page; llms.txt proposes a curated index of resources. Neither grants access or guarantees citations.

How do I know whether the configuration works?

Test URLs and user-agents, inspect status codes, redirects, headers, HTML and logs, verify official IP ranges and observe Search Console and referrals. Repeat the review when the site, CDN or provider documentation changes.