An AI-ready website: llms.txt, crawlers in robots.txt and structured data

The two previous articles covered technical SEO and how to write to be cited. This one closes the series with the infrastructure: what needs to be on the server so AI systems can access, understand and cite your site.

Who the AI crawlers are

Each company runs more than one crawler, with different jobs. The main ones, according to each company’s documentation:

Company	User-agent	Purpose
OpenAI	GPTBot	Collection for model training
OpenAI	OAI-SearchBot	ChatGPT search index
OpenAI	ChatGPT-User	Visits made on a user’s request
Anthropic	ClaudeBot	Collection for model training
Anthropic	Claude-SearchBot	Claude search
Anthropic	Claude-User	Visits made on a user’s request
Perplexity	PerplexityBot	Perplexity search index
Google	Google-Extended	Control token for use in Gemini (not a separate crawler)
Apple	Applebot-Extended	Control token for training use
Common Crawl	CCBot	Open dataset used by many models

The distinction that matters is training vs. search. Blocking training is a decision about how your content is used; blocking search means giving up on being cited.

robots.txt: decide and declare

For a portfolio or services site, where being found is the goal, I allow search crawlers and make the training decision explicit:

# AI search: allowed (we want to be cited)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Training: the site owner's call
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Allow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://dominio.com.br/sitemap.xml

Switch Allow to Disallow in the second block if you do not want your content used for training. Two notes:

• robots.txt is a request, not a lock. Major companies state they honor it, but it is not access control.
• User-triggered agents (ChatGPT-User, Claude-User) follow their own rules, described in each company’s documentation.

Cloudflare: the block you did not turn on

If your site goes through Cloudflare, open the dashboard. Since July 2025 Cloudflare has offered, and enabled by default on new domains, AI crawler blocking. It is good protection if you do not want to be scraped, but it can silently remove your site from AI search answers. Check Security → Bots and align it with your robots.txt.

HTML a crawler can read

AI crawlers generally do not execute JavaScript. They download the HTML and extract the text. To them, an SPA that builds its content in the browser is an empty page.

What I make sure of:

• Pre-rendered content (SSG or SSR). All text must be in the final HTML.
• Semantic HTML: <article>, <nav>, <main>, ordered headings, real lists, tables with <th>.
• Nothing important inside images. Text in images is not read; use descriptive alt when an image carries information.
• Fast server responses. Crawlers have timeouts; slow pages get dropped.

llms.txt: a site map for models

llms.txt is a September 2024 proposal by Jeremy Howard (llmstxt.org): a Markdown file at the site root with the project name, a summary and links to the pages that matter. The idea is to give a model, in a single read, the context it would otherwise need dozens of pages to build.

To be honest about where things stand: no major AI provider has publicly committed to reading llms.txt for search or ranking. Where it already helps is with agents, coding assistants and tools that build context from documentation. Since the cost is close to zero, I publish one.

The format is simple:

# Adryan Cosmo

> Full-stack web developer. Websites and digital products
> with applied AI.

## Articles

- [Technical SEO in 2026](https://dominio.com.br/en/blog/seo-tecnico-checklist/): crawling, indexing and Core Web Vitals checklist
- [GEO](https://dominio.com.br/en/blog/geo-motores-generativos/): how to get cited by generative engines

## Projects

- [MindUni](https://github.com/CosmoAdryan/minduni): mental health app for university students

On this site, the file is generated at build time from the articles and projects themselves, so it never goes stale. You can see the result at /llms.txt.

Structured data: what to state explicitly

Models try to answer “who said this, when and with what authority?”. JSON-LD answers without ambiguity. The minimum I publish:

• Person (or Organization) on every page, with name, jobTitle, url, knowsAbout and sameAs pointing to GitHub and LinkedIn. That is what ties the site to your other profiles.
• BlogPosting on each article, with headline, author, datePublished, dateModified, image and inLanguage.
• BreadcrumbList to show where the page sits in the site hierarchy.
• FAQPage when the content has real questions and answers.

{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "An AI-ready website: llms.txt, crawlers in robots.txt and structured data",
  "author": { "@type": "Person", "name": "Adryan Cosmo", "url": "https://dominio.com.br" },
  "datePublished": "2026-10-14T00:00:00.000Z",
  "dateModified": "2026-10-14T00:00:00.000Z",
  "inLanguage": "en"
}

Validate with Google’s Rich Results Test and the schema.org validator.

AI-ready site checklist

• AI search crawlers allowed in robots.txt
• Decision on training crawlers made and declared
• Cloudflare (or CDN) bot rules aligned with robots.txt
• Main content in server HTML, not dependent on JavaScript
• Semantic HTML, ordered headings, nothing important only inside images
• /llms.txt generated at build time
• Person JSON-LD with sameAs and BlogPosting with dates
• Takeaways and FAQ on key content (GEO techniques)

None of this is magic, and none of it replaces good content. But it is the difference between a site AI can read and cite and one it simply does not see.