GuideAI Search

What is llms.txt? A Complete Guide for 2025

llms.txt is a new open standard that lets website owners tell AI language models which pages to read and cite. Think of it as robots.txt for ChatGPT, Perplexity, Claude, and Google Gemini — but instead of blocking crawlers, it curates your best content for AI citation.

Updated May 2025·8 min read

Check your llms.txt score instantly: Paste your URL into our free AI SEO Analyzer — we check llms.txt presence, validity, and AI crawler access in under 10 seconds.

What is llms.txt?

llms.txt is a plain-text Markdown file placed at the root of your website at /llms.txt. It provides a structured, human-readable summary of your site's most valuable content — written specifically for AI language models to consume.

The standard was proposed by Jeremy Howard (co-founder of fast.ai) and has been adopted by thousands of websites including Anthropic, Cloudflare, and major developer documentation sites. As of 2025, AI assistants including Perplexity and Claude actively look for and use llms.txt files when generating answers.

The format is intentionally simple: a short description of your site, followed by Markdown sections linking to your key pages with one-line descriptions.

Why llms.txt matters for AI search visibility

When someone asks ChatGPT or Perplexity a question your site can answer, the AI system needs to decide which pages to retrieve and cite. Without guidance, it may:

✗Miss your best content entirely and cite a competitor instead
✗Retrieve low-quality pages (tag pages, login pages, old drafts)
✗Get confused by JavaScript-heavy pages it can't fully parse

An llms.txt file solves this by giving AI systems a clean map of your highest-value, most-citable content — the pages you want to be cited from.

As AI-powered search grows (Perplexity, ChatGPT, Google AI Overviews, and Apple Intelligence combined now handle billions of queries daily), being easily citable by AI systems is becoming as important as ranking in traditional Google search.

The llms.txt format explained

A valid llms.txt file has three parts:

H1 title — your site or project name
Blockquote summary — a one-paragraph description of what your site does, written for an AI to understand
Sections with links — Markdown ## headers grouping your key URLs with brief descriptions

Here is the llms.txt for this site as an example:

# AI SEO Analyzer

> Free dual-lens SEO analysis tool. Score any public URL for both
> Google search rankings and AI search engine readiness.
> No login required.

## Tool

- [Analyzer](https://ai-seo-analyzer.com/): Paste a URL, get 30+ SEO
  checks with dual Google + AI scores in seconds.

## Guides

- [What is llms.txt?](https://ai-seo-analyzer.com/guides/llms-txt):
  Complete guide to llms.txt — what it is, why it matters, how to create one.

- [AI SEO Checklist](https://ai-seo-analyzer.com/guides/ai-seo):
  How to optimize your site for ChatGPT, Perplexity, and Google AI Overviews.

How to create your llms.txt file (step by step)

Step 1: Identify your most-citable pages

List the pages you most want AI assistants to cite when answering questions in your niche. These are typically your best guides, product pages, and documentation — not your homepage, tag pages, or login pages.

Step 2: Write a clear site summary

The blockquote at the top of your llms.txt is what AI systems read first. Write 1–3 sentences that precisely describe what your site does and who it's for. Avoid marketing language — be factual and specific.

Step 3: Group pages into logical sections

Use ## headers to group related pages. Common sections: Docs, Guides, API Reference, Blog, Products. Each page needs a descriptive one-liner — this is what the AI reads to decide if the page is relevant.

Step 4: Place it at /llms.txt

The file must be accessible at https://yourdomain.com/llms.txt with content type text/plain. For most static sites, this means placing a file named llms.txt in your public root.

Step 5: Verify it with our analyzer

Run your homepage through the AI SEO Analyzer to confirm your llms.txt is detected and valid. We check for file presence, correct content-type, and whether your robots.txt allows AI crawlers.

llms.txt vs robots.txt — what's the difference?

Dimension	robots.txt	llms.txt
Purpose	Block or allow crawlers	Curate content for AI citation
Audience	All web crawlers	AI language models specifically
Format	Key-value directives	Markdown with links
Effect	Crawl permissions	Citation prioritization
Required?	Recommended	Optional but increasingly important

You need both. robots.txt controls whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) are allowed to visit your site at all. llms.txt tells them which pages are worth citing once they're in.

Common llms.txt mistakes to avoid

1.
Listing too many pages. Include only your best 10–30 pages. AI systems have context limits — a 500-URL dump defeats the purpose.
2.
Vague page descriptions. "About page" tells the AI nothing. Write what the page actually contains: "Company history, team, and contact information."
3.
Blocking AI crawlers in robots.txt while having llms.txt. If GPTBot is disallowed, it doesn't matter how good your llms.txt is — the AI can't fetch your pages.
4.
Never updating it. llms.txt should evolve with your best content. Set a quarterly reminder to add new guides and remove outdated pages.
5.
Forgetting the /llms-full.txt companion file. The standard also supports an optional /llms-full.txt with expanded descriptions. Useful for documentation-heavy sites.

Frequently asked questions

Do all AI systems support llms.txt?

Not yet — but adoption is growing fast. Perplexity confirmed support in late 2024. Claude's web-browsing mode respects it. OpenAI's GPT-4 with browsing and SearchGPT are expected to follow. Even where explicit support isn't confirmed, AI crawlers that fetch your content will still encounter the file.

Will having llms.txt hurt my Google ranking?

No. Google's crawler (Googlebot) ignores llms.txt — it's not a format Google uses. llms.txt is only consumed by AI systems. It won't change your Google crawl behavior or rankings.

How long does it take to create one?

For most sites, 20–30 minutes. The hardest part is deciding which pages to include and writing tight one-line descriptions. The technical setup is trivial — it's just a text file.

Should I link to llms.txt from my homepage?

You can add a link in your footer or a <link> element in your HTML head, but it's not required. AI crawlers know to check /llms.txt automatically, similar to how they check /robots.txt.

Check if your llms.txt is working

Our free AI SEO Analyzer checks your llms.txt presence, validates the format, and confirms that AI crawlers are allowed by your robots.txt — all in one report.

Analyze your site for free