Designing Websites for AI Agents - llms.txt, Structured Data, and Agent-Readable Web Practices

First Published:
Last Updated:

For fifteen years the question behind every markup decision was the same: how does this page look to a person, and how does Google's crawler summarize it for other people? That question has not gone away, but it now has a sibling. A growing share of the systems that read the web are not rendering pixels for a human at all. They are language models building training corpora, answer engines assembling a synthesized response, and agentic browsers fetching a single page because a user asked their assistant a question. These readers do not see your typography. They see your bytes, your headings, your structured data, and whatever machine-readable surface you chose to expose.

This article is a practical guide to designing a website so those readers can find it, parse it, and cite it accurately. I am going to work through the whole stack: the discovery layer (robots.txt, AI crawler user-agents, sitemaps, and feeds), the llms.txt proposal, structured data, HTML semantics, serving strategy, the governance decisions around allowing or blocking AI access, and finally how to verify how your own site is actually being read.

I run this site as a static site, and I use its public surface as the worked example throughout, because the one thing this genre of "how to build for AI" article usually lacks is an author whose own site you can open in view-source and check. Every claim I make about my implementation is verifiable by fetching a public URL. Every claim I make about a vendor's crawler or a specification is footnoted to that vendor's own documentation, checked on 2026-07-24. This is a fast-moving area, so where a fact is likely to change I say so and date it.

One promise up front, because it shapes everything below: there is no magic markup that makes an AI cite you. The durable levers are boring and old — be crawlable, be semantically clean, be factually dense, keep your URLs stable, and be honest in your structured data. The interesting parts are the new specifications layered on top and the governance choices you now have to make deliberately. I will not sell you the hype version. Google, as we will see, publicly states that it ignores llms.txt entirely; I publish one anyway, and I will explain exactly why that is a defensible bet and not a contradiction.

1. Introduction: From Human-Only SEO to the Agent-Readable Web

Search engine optimization, as a discipline, optimized for one consumer: a ranking algorithm that produced a list of ten blue links for a human to click. The entire feedback loop — keywords, backlinks, click-through rate, dwell time — assumed a human at the end of it.

That assumption is fracturing in two directions at once.

First, the reader is increasingly a machine that produces an answer instead of a link. Google's AI Overviews and AI Mode, Perplexity, ChatGPT's search features, and Bing's AI-powered results all synthesize content from multiple pages into a single response, sometimes with a citation link and sometimes without. The page is no longer the destination; it is an input to a generated answer.

Second, the fetcher is increasingly an agent acting on a person's behalf in real time. When a user asks their assistant to "read this page and summarize the pricing table," a user-triggered fetcher retrieves the raw HTML, and — as we will see — the usual crawler rules may not even apply to it.

The term for optimizing toward this new class of reader is Generative Engine Optimization (GEO). The phrase comes from a 2023 research paper, "GEO: Generative Engine Optimization," by researchers from Princeton and collaborating institutions, later presented at KDD 2024.[1] The paper studied how content changes — citations, quotations, statistics, clear structure — affected a source's visibility inside a generated answer. Its headline figure (a visibility improvement of up to roughly 40% for certain techniques) is specific to the study's 2023–2024 experimental pipeline and should not be read as a durable, universal number; the value of the paper is that it named the problem and showed that source-side structure measurably matters.

The scope of this article is the technical and structural surface of GEO — the parts a site owner fully controls. I am deliberately not covering classic SEO tactics (keyword research, link building), because those are well documented elsewhere and largely unchanged. I am covering the machine-readable surface: what to expose, how to structure it, and how to decide who gets to read it.

A note on honesty, since it is the whole game in this genre. Readers of an article like this — especially the technically skeptical ones — will open my site and check whether I do what I recommend. So I have constrained myself: I recommend a practice as a "should" only where I actually do it and can show you the public output. Where I have chosen not to adopt something (or where the practice is unproven), I say so and explain the trade-off. A guide that tells you to do things its own author does not do is worth nothing, and it is trivially falsifiable by anyone with a browser.

2. How AI Agents Read the Web

Before deciding what to expose, it helps to be precise about who is reading, because "AI" is not one reader. There are three broad access modes, and they consume different surfaces of your site.
How AI agents read the web: three access modes - bulk crawlers, answer-engine synthesis, and user-triggered agentic fetch - and the site design elements each one depends on.
How AI agents read the web: three access modes - bulk crawlers, answer-engine synthesis, and user-triggered agentic fetch - and the site design elements each one depends on.

Mode 1 — Crawler-type access (bulk, automated). These are conventional web crawlers operated by AI companies. They fetch pages en masse and store them, either to build a training corpus (for pre-training or fine-tuning a model) or to build a search index that an answer engine later queries. They read your HTML as delivered, they honor (or claim to honor) robots.txt, and — critically — most of them do not execute JavaScript. What they can see is your server-rendered HTML, your headings, your links, your structured data, and your discovery files. Examples: OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot and Claude-SearchBot, Google's Googlebot, Perplexity's PerplexityBot, and Common Crawl's CCBot.

Mode 2 — Answer-engine synthesis. Here the model does not fetch your page at request time; it draws on an index built by Mode 1 crawlers and composes an answer. Google AI Overviews and AI Mode are the largest example. From the site owner's side, you influence this mode almost entirely through the index — meaning through crawlability, snippet controls, and content quality — rather than through anything served at query time. Google is explicit that there are "no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary" beyond ordinary indexability.[2]

Mode 3 — Agentic, user-triggered fetch. A person asks an assistant to act on a specific URL, and a user-triggered fetcher retrieves that page live. These fetchers behave more like a browser opening a tab than like a crawler indexing the web, and several vendors state that robots.txt "may not apply" to them because a human initiated the request. Examples: OpenAI's ChatGPT-User, Anthropic's Claude-User, Perplexity's Perplexity-User, and Meta's meta-externalfetcher. This mode is growing fastest and is the least governed by the discovery-layer conventions.

The practical consequence of this taxonomy is that the surfaces you optimize differ by mode. Mode 1 rewards clean server-rendered HTML and discovery files. Mode 2 rewards indexability and factual density (and is governed by snippet directives). Mode 3 rewards stable URLs and semantic HTML that survives a single live fetch. The rest of this article walks the stack that serves all three.

3. The Content Discovery Layer: robots.txt, AI Crawler User-Agents, Sitemaps, and Feeds

The discovery layer is the set of files, served at well-known paths, that tell an automated reader what exists, where it is, and whether it is welcome. Four things live here: robots.txt, the crawler user-agents it addresses, sitemaps, and feeds.

3.1 robots.txt and RFC 9309: An Advisory Protocol, Not Access Control

robots.txt is a plain-text file at your site root that tells crawlers which paths they may fetch. Since September 2022 it has an actual IETF standard — RFC 9309, the Robots Exclusion Protocol, authored by Koster, Illyes, Zeller, and Sassman.[3] RFC 9309 formally defines exactly three directives: User-agent, Disallow, and Allow.

The single most important property of robots.txt, and the one most often forgotten in "block the AI bots" advice, is stated in the RFC itself: these rules "are not a form of access authorization," and the protocol "is not a substitute for valid content security measures."[3] A crawler obeys robots.txt because it chooses to. Every vendor documents its compliance as something its bot "respects" or "honors" — never as something the file forces. If you need to actually prevent access, you need an application-layer control: authentication, IP allow-listing, a WAF rule, or rate limiting. robots.txt is a sign on an unlocked door, and a cooperative visitor reads the sign.

My own robots.txt is deliberately minimal. Fetched from the live site, it is:
User-agent: *
Allow: /

Sitemap: https://hidekazu-konishi.com/sitemap.xml

Three lines: address every user-agent, permit the entire site, and point to the sitemap. I will justify the "allow everything, enumerate nobody" decision in the governance section (§8); for now, note that the Sitemap: line is doing discovery-layer work. It is worth being precise about it: the Sitemap: directive is not defined by RFC 9309. The RFC permits crawlers to "interpret other records that are not part of the robots.txt protocol," and Sitemap: is exactly such an other record — a widely adopted convention that originates from the Sitemaps protocol, not from the robots standard.[3][4] It works everywhere, but describe it accurately: a de facto extension, not a standardized directive.

The same "other records" door is where the newest robots.txt evolution lives. In September 2025, Cloudflare published its Content Signals Policy, which extends robots.txt with three machine-readable content signals - search, ai-input, and ai-train - so a site can state how its content may be used after it has been accessed, not just whether it may be fetched.[3b] Treat it the way this article treats llms.txt: a single-vendor proposal layered on top of RFC 9309, not an IETF standard - and, like every other rule in this file, an expressed preference rather than an enforced control. A standards-track counterpart is in progress: the IETF AI Preferences (AIPREF) working group is defining a common vocabulary for expressing AI usage preferences and mechanisms for attaching them to content via robots.txt and an HTTP header; as of this writing its drafts are still working documents, not a published standard.[3d]

3.2 The AI Crawler Taxonomy: Who Is Actually Fetching Your Pages

If you are going to make governance decisions about AI crawlers, you need to know their user-agent tokens and what each one is for. The tokens below are taken from each vendor's own documentation, verified on 2026-07-24. Treat this table as a dated snapshot: these lists change several times a year, and legacy tokens (Anthropic's older anthropic-ai and Claude-Web, for instance) fall out of the official docs while lingering in old robots.txt files.

A pattern emerges across vendors — three functional roles, plus a fourth that is not a crawler at all:

* You can sort the table by clicking on the column name.
User-agent (as documented)VendorRoleHonors robots.txt?
GPTBotOpenAITraining crawlerYes
OAI-SearchBotOpenAISearch-index crawlerYes
ChatGPT-UserOpenAIUser-triggered fetchMay not apply
ClaudeBotAnthropicTraining crawlerYes
Claude-SearchBotAnthropicSearch-index crawlerYes
Claude-UserAnthropicUser-triggered fetchYes
GooglebotGoogleSearch crawlerYes
Google-ExtendedGoogleTraining-use permission token (not a crawler)N/A
PerplexityBotPerplexitySearch-index crawlerYes
Perplexity-UserPerplexityUser-triggered fetchGenerally ignores
ApplebotAppleSearch crawlerYes
Applebot-ExtendedAppleTraining-use permission token (not a crawler)N/A
CCBotCommon CrawlOpen-corpus crawlerYes
meta-externalagentMetaTraining / product crawlerYes
meta-externalfetcherMetaUser-triggered fetchMay not apply
AmazonbotAmazonSearch / product crawlerYes

The three functional roles are worth internalizing because they generalize:[5][6][7][8][8b][8c]
  1. Training / data-collection crawlers (GPTBot, ClaudeBot, meta-externalagent, CCBot) fetch in bulk to build corpora. They uniformly claim robots.txt compliance.
  2. Search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot) build the index that an answer engine later queries. They also claim compliance, and vendors like OpenAI explicitly recommend allowing their search bot if you want to appear in that engine's results.
  3. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher) fire when a live user asks the assistant to read a page. Multiple vendors state that robots.txt "may not apply" or is "generally ignored" for this role, on the theory that it is equivalent to a human opening the page in a browser.

The fourth pattern is a frequent source of confusion. Google-Extended and Applebot-Extended are not crawlers. They have no separate HTTP user-agent and they fetch nothing. They are permission tokens you place in robots.txt to control whether content already crawled by the main bot may be reused — for Google, to train Gemini models and ground Vertex AI; for Apple, to train Apple Intelligence.[6][9] Disallowing Google-Extended does not remove you from Google Search and is not a ranking signal; it only opts your content out of that downstream training and grounding use. People routinely misread these as "a bot that visits my site," and then write robots.txt rules that do not do what they think.

One caveat worth stating because it undermines the whole premise of robots.txt-based control: compliance is a claim, not a guarantee. In August 2025, Cloudflare publicly reported crawler behavior it attributed to Perplexity that appeared to contradict Perplexity's stated robots.txt compliance.[8d] Whatever the resolution of that specific dispute, it is a useful reminder that the discovery layer is cooperative, and adversarial or careless actors exist. If access genuinely matters, return to §3.1: use a real control.

3.3 Sitemaps: Prefer loc + lastmod, and Drop the Rest

A sitemap is an XML file listing your URLs so crawlers can discover them without following every internal link. The Sitemaps protocol (sitemaps.org) defines one required child element per URL, <loc>, and three optional ones: <lastmod>, <changefreq>, and <priority>.[4]

Here is the practical, verified guidance most sitemaps get wrong: omit <changefreq> and <priority> entirely. Google's own documentation states that it "ignores <priority> and <changefreq> values," and uses <lastmod> only if that value "is consistently and verifiably … accurate."[10] Bing says the same on its webmaster blog — the optional priority and changefreq tags "are ignored by Bing and do not influence how your content is crawled or ranked" — and explains why: because most sites set those fields uniformly or aspirationally, they carry no signal, so both engines revamped their scheduling around <lastmod> instead.[11]

My sitemap follows this exactly. Each entry is just a location and an accurate last-modified timestamp:
<url>
    <loc>https://hidekazu-konishi.com/entry/designing_websites_for_ai_agents.html</loc>
    <lastmod>2026-07-26T00:00:01+00:00</lastmod>
</url>

No changefreq, no priority. The discipline that makes <lastmod> valuable is that it must be truthful: bump it when the content, structured data, or links meaningfully change, and not for a copyright-year edit in the footer. An engine that catches you setting <lastmod> to "now" on every crawl will simply stop trusting the field. Honesty is, again, the operative constraint.

3.4 Feeds: Atom and RSS as a Second Discovery Channel

Feeds predate the AI era but remain a clean, machine-first way to announce new and updated content. Atom is standardized as RFC 4287 (a Proposed Standard since December 2005); RSS 2.0 is maintained at rssboard.org.[12][13] Both are XML "web content syndication" formats — structured lists of entries with titles, links, timestamps, and content, designed to be consumed by "user agents" rather than rendered for a human.

Two details matter for machine consumption. First, declare the language: my Atom feed carries xml:lang="en" on the root <feed> element and my RSS carries <language>en</language>. That is a small, correct signal about who the content is for. Second, link the feeds from every page's <head> with rel="alternate", so a reader that lands on any article can discover the whole feed:
<link rel="alternate" type="application/atom+xml" title="Atom" href="https://hidekazu-konishi.com/feed.xml">
<link rel="alternate" type="application/rss+xml" title="RSS2.0" href="https://hidekazu-konishi.com/rss.xml">

Feeds are a low-cost, high-longevity channel. They cost nothing to maintain once generated from your source of truth, and unlike proprietary APIs they are not going to be deprecated out from under you.

4. llms.txt and llms-full.txt

llms.txt is the newest and most-hyped item in the discovery layer, and it is the one where being precise matters most, because a lot of what is written about it is wrong. I publish both an llms.txt and an llms-full.txt, so I have skin in the game — which is exactly why I want to describe them accurately rather than sell them.

4.1 What the Proposal Actually Is

llms.txt was proposed by Jeremy Howard of Answer.AI on 2024-09-03.[14] The word "proposed" is not mine — it is llmstxt.org's own framing. The site describes itself as "A proposal to standardise on using an /llms.txt file," and the body repeatedly says "we propose." It calls the document an "informal overview" that is "open for community input."[14] So the first accurate statement to make is this: llms.txt is a proposal, not a ratified standard. Anyone who calls it "the llms.txt standard" is overstating its status.

The format itself is a small, specific Markdown structure. Per the spec, a compliant file contains, in order:[14]
  • an optional byte-order mark;
  • an H1 with the name of the project or site — the only required section;
  • an optional blockquote with a short summary;
  • zero or more free Markdown sections (of any type except headings) giving detail;
  • zero or more H2-delimited "file list" sections, each a Markdown list of links of the form [name](url): optional notes.

There is one special case: an H2 section literally titled "Optional" has defined semantics — its URLs may be skipped when a shorter context is needed. The idea is a curated, human-and-machine-readable map of your site's most useful content, expressed as clean Markdown so a model can ingest it cheaply.

4.2 What llms.txt Is Not (and Where llms-full.txt Comes From)

Now the corrections, because this is where the honesty pays off with a technical audience.

Google states that it ignores llms.txt. Google's Search Central guidance on AI optimization (last updated 2026-07-10 at the time I checked) says you "don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search…," and that maintaining llms.txt-type files "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."[2b] Google's John Mueller has compared it, informally, to the old keywords meta tag. If your goal is Google ranking, llms.txt does nothing.

No major AI vendor documents consuming third-party llms.txt. OpenAI and Anthropic both publish llms.txt-style files for their own documentation — you can fetch them — but I could find no official statement from OpenAI, Anthropic, or others committing their models or crawlers to read and prioritize an arbitrary site's llms.txt. Claims that "ChatGPT reads your llms.txt" are, as of this writing, community-reported and unconfirmed by vendor documentation. Adoption directories exist (llmstxt.site, directory.llmstxt.cloud), but they are community-run listings, not evidence of consumption.

llms-full.txt is not in the original spec. This one trips up almost everyone. The llmstxt.org specification describes derivative files named llms-ctx.txt and llms-ctx-full.txt, generated by the reference tool llms_txt2ctx from the FastHTML project.[14] The widely seen /llms-full.txt convention — a single file containing a site's full expanded content — is a downstream vendor convention, popularized by documentation platforms (notably Mintlify) and adopted by vendors like Anthropic for their own docs. It is a reasonable idea, but attribute it correctly: llms-full.txt is community/vendor practice, not part of the original proposal.

4.3 My Implementation, Honestly

Given all that, why do I publish these files? Because the cost is near zero, the downside is nil (Google ignores them rather than penalizing them), and there is a plausible, unproven upside if any answer engine ever does prefer a clean Markdown map. It is a cheap option on an uncertain future, not a proven tactic — and I want you to adopt it, if you do, for that honest reason.

My llms.txt follows the spec form: an H1 site name, a blockquote summary, a short descriptive paragraph, and H2 sections grouping every article as annotated links. A condensed, illustrative view of the file's shape (abridged and lightly paraphrased here - fetch the live file for the exact text):
# hidekazu-konishi.com

> AWS, generative AI, Amazon Bedrock, security, and software engineering
> deep-dives and free client-side developer tools.

Last generated: 2026-07-26.

## AI, Generative AI, and Amazon Bedrock

- [Amazon Bedrock Converse API Deep Dive](https://hidekazu-konishi.com/entry/amazon_bedrock_converse_api_deep_dive.html): A deep dive into the unified request and normalized response, conversation state, tool use, and streaming.

I generate it from the same source of truth that produces my feeds and sitemap, so it never drifts, and I stamp it with a "Last generated" date so a reader knows how fresh it is.

Now the honest trade-off I promised. My llms-full.txt — the full-content variant — is about 13 MB. That is large enough to exceed the practical fetch and context limits of many of the very agents it is meant to serve; a model with a bounded context window cannot ingest a file that size in one pass, and some fetchers will refuse or truncate it. So it is a genuinely mixed asset: valuable as a complete, single-URL archive of the site's text, but not something I would claim every agent can actually use. If I were optimizing purely for agent ingestion I would consider splitting it or capping it. I keep it because it is complete and cheap to serve, and I am telling you its size so you can judge the trade-off for your own site rather than cargo-culting a 13 MB file into existence. The curated llms.txt (about 137 KB) is the one that is actually ingestible.

5. Structured Data for AI

If llms.txt is the speculative bet, structured data is the proven, boring one. It has been machine-consumed for over a decade, it is a first-class Google input, and — done correctly — it tells any parser exactly what your page is.

5.1 JSON-LD and the @graph

Structured data can be expressed three ways (JSON-LD, Microdata, RDFa), but Google recommends JSON-LD, and so do I: it lives in one <script> block in the head, decoupled from your visible markup, so it is easy to generate and maintain.[15] Google's documentation labels JSON-LD as "(Recommended)" and notes all three formats are otherwise equally acceptable.

Every article on my site carries a JSON-LD @graph with three nodes: a WebSite (describing the site), an Article (describing this page), and a Person (describing the author). A trimmed view of the Article node:
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Designing Websites for AI Agents - llms.txt, Structured Data, and Agent-Readable Web Practices | hidekazu-konishi.com",
  "image": ["https://hidekazu-konishi.com/images/designing_websites_for_ai_agents.jpg"],
  "author": { "@type": "Person", "name": "Hidekazu Konishi", "url": "https://hidekazu-konishi.com/" },
  "datePublished": "2026-07-26T00:00:00+00:00",
  "dateModified": "2026-07-26T00:00:01+00:00"
}

The relevant schema.org types are all long-established: Article (with subtypes BlogPosting and TechArticle), WebSite, WebPage, Person, Organization, and BreadcrumbList.[16] For an article, Google supports the types Article, NewsArticle, and BlogPosting, states there are no required properties, and lists as recommended: author, dateModified, datePublished, headline, and image.[17] My template supplies all of those.

5.2 Choosing Types and Matching Visible Text

Two rules keep structured data useful rather than harmful.

First, choose the type that matches the content. An article is an Article; a homepage/profile is well described by WebSite plus Person; a data reference could add Dataset (whose Google-supported form requires name and description).[18] Do not stack types you cannot justify — a parser that sees Recipe markup on a networking article learns that your structured data lies.

Second — and this is the one Google explicitly calls out for AI features — your structured data must match the visible text on the page. Among the ordinary best practices Google says continue to matter for AI Overviews and AI Mode is "making sure your structured data matches the visible text on the page."[2] Structured data is a machine-readable restatement of what a human sees, not a hidden channel for extra claims. Marking up a price, date, or rating that does not appear on the page is a spam signal, not an optimization.

What structured data does not do is worth stating plainly, because the hype gets this wrong: there is no Google statement that adding structured data boosts your inclusion in AI Overviews. Its documented benefit is helping search engines understand your page and become eligible for rich results — not a lever that increases AI-answer visibility. Use it because it is correct and machine-legible, not because someone promised it buys AI citations.

5.3 What the Retired FAQ Rich Result Means for FAQPage JSON-LD

This section is a concrete example of designing for the current reality rather than a cached memory of it.

For years, the advice was to wrap any Q and A content in FAQPage structured data to earn an expandable FAQ rich result in Google. That advice is now obsolete, and following it is worse than doing nothing. The timeline, from Google's own changelog:[19]
  • September 2023 — Google restricted FAQ rich results to "well-known, authoritative government and health websites." Overnight, most sites lost the feature.
  • May 2026 — Google announced it was deprecating the FAQ rich result entirely: the feature "will no longer appear in Google Search starting May 7, 2026."
  • June 2026 — Google removed the FAQ rich result documentation altogether; the old documentation URL now redirects to the changelog.

As of this writing (2026-07-24), the FAQ rich result is gone for everyone — not restricted, discontinued. So injecting FAQPage JSON-LD to chase a rich result is chasing a feature that no longer exists.

Every article on my site presents frequently-asked questions as ordinary body content under a heading (this one does, in §10), so the answers are fully readable by any human or agent whether or not the markup is present. Beyond that, my own practice splits by page type, and it is worth stating exactly rather than tidily, because you can check it: this guide carries no FAQPage JSON-LD at all, while my timeline and reference pages — roughly a fifth of the site — still emit it inside the article body, where it is part of that page type's fixed structure. A handful of older explainers, written while the rich result still existed, carry it too. I have not gone back to strip any of it out, because markup that faithfully mirrors visible text is inert rather than harmful once the rich result is gone; Google simply ignores it. What I do not do — on any page type — is add FAQPage in order to win a rich result, because that result no longer exists. The distinction matters more than the markup: the failure mode is not "has FAQPage," it is "shipped structured data on the strength of advice nobody re-checked."

6. HTML Semantics and Content Design

Underneath the discovery files and the structured data is the thing every reader in every mode actually consumes: your HTML. For the non-Google crawlers especially — which, as we will see in §7, largely do not run JavaScript — the server-rendered HTML is your content. Getting it right is the highest-leverage, lowest-glamour work in this whole article.

Heading hierarchy. Use one <h1> for the page, then <h2>/<h3>/<h4> in strict nesting order that reflects the document's actual structure. Headings are how both assistive technology and language models segment a page into sections; a page that jumps from <h2> to <h4>, or that uses heading tags for visual size rather than structure, is harder to chunk correctly. A model that ingests your page will often split it on headings to build retrievable passages — so your heading tree is, in effect, your retrieval schema.

Factual density and self-contained passages. Answer engines extract and quote passages. A sentence that only makes sense three paragraphs up is hard to lift accurately; a sentence that states a complete fact — subject, claim, and the qualifier that bounds it — survives extraction. Concretely: prefer "RFC 9309 defines three directives: User-agent, Disallow, and Allow" over "it defines three of them." Tables, definition lists, and short declarative sentences all raise the odds that a passage is quoted correctly. This is the single most actionable finding of the GEO literature: sources that are quotable, cited, and structured are more likely to be used.[1]

Stable URLs. A URL is a promise. When an answer engine cites you, or a user bookmarks you, or another site links you, that citation is only as durable as the URL behind it. Do not restructure paths casually; when you must move a page, return a 301 redirect from the old URL, and reserve 404/410 for content that is genuinely gone. Google warns specifically against "soft 404s" — pages that return 200 OK while telling the reader the content does not exist — because they waste crawl budget and confuse indexing.[20] Every URL on my site is a stable, absolute https://hidekazu-konishi.com/entry/<slug>.html, and it does not change after publication.

Explicit, consistent dates. Machines need to know how fresh a fact is, and they should not have to guess. On every article I express the publication and update dates three times, in agreement: as human-readable text, as a machine-readable datetime attribute, and inside the JSON-LD.
<p>
First Published: <time datetime="2026-07-26T00:00:00+00:00">2026-07-26</time><br>
Last Updated: <time datetime="2026-07-26T00:00:01+00:00">2026-07-26</time><br>
</p>

The <time datetime> value, the visible text, the article:published_time/article:modified_time meta tags, and the JSON-LD datePublished/dateModified all carry the same instant. When those disagree, a parser has to pick a winner, and you have lost control of your own metadata.

Canonicalization. Every page declares one canonical URL, and I keep <link rel="canonical"> byte-for-byte identical to <meta property="og:url">. Google treats rel="canonical" as a strong hint (not an absolute rule) alongside other signals like HTTPS, redirects, and sitemap presence when it picks the representative URL for a piece of content.[21] Giving one unambiguous, self-consistent answer removes a decision a crawler would otherwise make for you.

None of this is new. All of it is exactly what made a page accessible and durable before generative engines existed. That is the point: the agent-readable web is, to a first approximation, the well-built web.

7. Serving Strategy: Static Generation, Low JavaScript, and Performance

How you serve the HTML determines whether Mode 1 crawlers ever see your content at all. This is where a lot of otherwise-good sites quietly fail.

Do AI crawlers execute JavaScript? Here the honest, sourced answer is more useful than the confident one. Only two data points are vendor-confirmed. Googlebot does render JavaScript, using an evergreen headless Chromium in a crawl → render → index pipeline; Google's own documentation says "Google Search runs JavaScript with an evergreen version of Chromium," and, tellingly, adds that "not all bots can run JavaScript."[22] At the other end, Common Crawl's CCBot does not: its FAQ states plainly that "JavaScript is not executed and Cookies are not used."[23]

For the big AI crawlers in between — GPTBot, ClaudeBot, PerplexityBot — none of the vendors document their JavaScript behavior either way. Independent testing has repeatedly found that these crawlers do not execute client-side JavaScript in practice (a widely cited Vercel analysis from December 2024 is the usual reference), but I want to label that correctly: it is an observed community finding as of a point in time, not a vendor guarantee, and it could change without announcement.[24]

The design conclusion holds regardless of the exact list: if your content only exists after client-side JavaScript runs, you are betting your machine-readability on rendering behavior that most AI crawlers do not provide and none of them promise. Server-side rendering or static site generation removes the bet entirely. My site is fully static — every article is complete HTML on disk, served directly, with JavaScript limited to two small deferred scripts — one that registers a service worker for offline caching, one that adds a table sorter, a sample-code loader, and an email-address de-obfuscator — none of which the article text depends on. Disable JavaScript and the entire article is still there. That is the property you want: JavaScript as progressive enhancement, never as the delivery mechanism for text.

Performance still matters, for humans and for crawl budget. Core Web Vitals — Largest Contentful Paint (loading), Interaction to Next Paint (interactivity, which replaced First Input Delay on 2024-03-12), and Cumulative Layout Shift (visual stability) — remain the standard performance metrics.[25] A statically served, low-JavaScript page tends to do well on all three almost by construction: fast HTML delivery helps LCP, minimal main-thread work helps INP, and reserved image dimensions help CLS. Fast, lightweight pages are also cheaper for a crawler to fetch, which matters when a crawler is deciding how much of your site to bother with. Serving strategy is where accessibility, performance, and agent-readability turn out to be the same problem.

8. Governance Decisions: Allowing, Limiting, and Blocking AI Access

Everything so far has been about being readable. This section is about the deliberate choice of by whom — a choice that did not really exist ten years ago and now has to be made on purpose. I am going to present it as genuinely two-sided, because it is, and reasonable site owners land in different places.

The case for allowing AI access. If you want to appear in AI-generated answers and be cited as a source, the entry ticket is being crawlable. Google is explicit that appearing in AI Overviews and AI Mode requires being indexed and snippet-eligible — nothing more, nothing AI-specific — and that means not blocking Googlebot.[2] OpenAI similarly recommends allowing OAI-SearchBot if you want to surface in ChatGPT's search. For a site whose purpose is to be read and referenced — a blog, documentation, a public knowledge base — blocking the crawlers that feed answer engines is blocking your own distribution.

The case for limiting or blocking. If your content is a business asset you sell, or if you object to it training commercial models without compensation, you have levers of increasing bluntness:
  • Opt out of training use while staying in search. Disallow the permission tokens — Google-Extended, Applebot-Extended — to remove your content from those vendors' model-training and grounding, without affecting your search presence.[6][9] This is the surgical option.
  • Limit what is shown from your pages. Google's robots meta tags let you cap or forbid snippets: nosnippet prevents a text snippet and, per Google's current documentation, also prevents the content from being "used as a direct input for AI Overviews and AI Mode"; max-snippet:[number] limits how much may be shown or used as AI input; noindex removes the page from results entirely.[26] That AI Overviews/AI Mode language is a recent, dated addition to Google's snippet documentation and is worth re-checking as it evolves.
  • Block specific crawlers. Disallow individual training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent) in robots.txt — remembering §3.1: this is a request, honored only by cooperative bots, and it is not access control.
  • Move the control to the network layer. CDN-level tools extend what a lone robots.txt can do: Cloudflare's Content Signals Policy (September 2025) adds search / ai-input / ai-train usage preferences to robots.txt,[3b] and its pay per crawl program uses HTTP 402 Payment Required so a site can charge AI crawlers for access rather than merely asking them to leave.[3c] Content signals are still an expressed preference, but a 402 response is enforced at the edge — the first lever in this list that is not honor-system. Both are vendor features, not web standards, so weigh the lock-in accordingly (the IETF AIPREF working group is standardizing this kind of usage-preference signaling, so a standards-based alternative is on the way[3d]).
  • Express a legally grounded opt-out for EU text and data mining. The W3C community-developed TDM Reservation Protocol (TDMRep) publishes a machine-readable /.well-known/tdmrep.json file that expresses the text-and-data-mining opt-out defined by Article 4 of the EU Copyright (CDSM) Directive. Unlike the purely voluntary signals above, it is tied to a legal framework in EU member states, though the protocol itself is a W3C Community Group final report rather than a full W3C standard.[3e]

Why my robots.txt allows everything and enumerates nobody. I made the allow-everything choice for two reasons. First, my content exists to be read and cited; being in the answer layer is the goal, not the threat. Second — and this is the maintenance argument — an enumerated block list is a liability. New crawler tokens appear several times a year; vendors rename them; a hand-maintained Disallow list for two dozen AI user-agents is stale within months and gives a false sense of control while quietly missing the newest bots. A blanket Allow: / with no per-bot rules is honest about what robots.txt can and cannot do, and it has nothing to rot. If my calculus were different — if I sold my writing — I would reach for the permission tokens and snippet controls first, and treat a crawler block list as the leaky, high-maintenance last resort it is.

The meta-point: there is no universally correct governance posture. There is only the one that matches what your content is for, applied with an accurate understanding of what each control actually does.

9. Verification: How to Check the Way Your Site Is Actually Read

Designing for agents is worth nothing if you never check the result. The failure mode of this entire genre is the author who recommends a practice, never verifies their own implementation, and ships a page that contradicts its own advice. Here is how to hold yourself to it.

Fetch your discovery files and check the status and content-type. Every machine-facing file should return 200 with the right Content-Type. A one-line check:
for f in robots.txt sitemap.xml feed.xml rss.xml llms.txt; do
  curl -sS -o /dev/null -w "%{http_code}  %{content_type}  $f\n" \
    "https://hidekazu-konishi.com/$f"
done

If your sitemap returns text/html instead of application/xml, or your llms.txt 404s, no amount of careful authoring matters — the file is not really there.

See the page as a non-rendering crawler sees it. Because most AI crawlers do not run JavaScript (§7), view the unrendered HTML: curl the URL, or use "View Source" (not "Inspect," which shows the rendered DOM). If your article text is missing from the raw HTML, it is missing for those crawlers. You can also fetch as a specific bot to confirm you are not blocking or cloaking it:
curl -sA "GPTBot" -o /dev/null -w "%{http_code}\n" \
  "https://hidekazu-konishi.com/entry/designing_websites_for_ai_agents.html"

Validate your structured data. Paste the URL into Google's Rich Results Test or the Schema.org validator and confirm the @graph parses, the types are what you intended, and — the check that catches real bugs — the values match the visible page.

Read your own robots.txt and sitemap as a parser would. Confirm the Sitemap: line resolves, confirm the sitemap lists the URLs you expect with truthful <lastmod> values, and confirm no path you care about is accidentally disallowed.

Watch your logs. The ground truth of who reads your site is your access logs. The user-agent tokens in §3.2 are what you will see; their frequency tells you which engines actually fetch you, and their absence tells you when a token you blocked has quietly been renamed. Logs are also how you would detect the kind of non-compliant crawling that the Perplexity/Cloudflare episode in §3.2 was about.

I ran exactly these checks against my own site while writing this article — that is how I can state, rather than hope, that its robots.txt, sitemap, feeds, and llms.txt all return 200 with correct content-types, that the sitemap carries only loc and lastmod, and that the feeds declare their language. If you take one habit from this article, take this one: verify, then claim.

10. Frequently Asked Questions

Does publishing an llms.txt improve my Google ranking?
No. Google states it ignores llms.txt-type files and that they neither help nor harm your Search visibility (§4.2).[2b] Publish one as a cheap, unproven bet on other answer engines if you like, but not for Google.

Is llms.txt an official standard?
No. llmstxt.org describes it as a "proposal" and an "informal overview," open for community input (§4.1).[14] Call it a proposal, not a standard.

Should I add FAQPage structured data to get an FAQ rich result?
No. Google discontinued the FAQ rich result feature for all sites in May 2026 and removed its documentation in June 2026 (§5.3).[19] The feature you would be marking up for no longer exists.

Do I need special markup or an AI-specific schema to appear in AI Overviews?
No. Google states there are no additional requirements and no special structured data for AI Overviews or AI Mode beyond being indexed and snippet-eligible (§2, §5.2).[2] Standard indexability and honest structured data are the whole job.

Will blocking GPTBot or ClaudeBot in robots.txt actually stop them?
Only if they choose to comply. robots.txt is advisory; RFC 9309 says it "is not a form of access authorization" (§3.1).[3] For enforcement you need an application-layer control such as authentication or a WAF rule.

Should I block AI crawlers or allow them?
It depends on what your content is for (§8). If you want to be cited in AI answers, allow them; if your content is a monetized asset, prefer training-opt-out permission tokens (Google-Extended, Applebot-Extended) and snippet controls before blanket blocks. There is no single right answer.

My site is a JavaScript single-page app. Is that a problem for AI crawlers?
Likely yes for non-Google crawlers. Googlebot renders JavaScript, but Common Crawl's CCBot does not, and the major AI crawlers are observed not to either (§7).[22][23][24] Serve your content as server-rendered or static HTML so it exists without client-side execution.

What is the difference between Google-Extended and Googlebot?
Googlebot is the crawler that fetches your pages for Search. Google-Extended is not a crawler at all — it fetches nothing; it is a robots.txt permission token that controls whether already-crawled content may be used to train Gemini and ground Vertex AI (§3.2).[6]

11. Summary

The agent-readable web stack: a discovery layer (robots.txt, sitemap, feeds, llms.txt), a semantics and structured-data layer (clean HTML and JSON-LD), a serving layer (static generation and low JavaScript), and a governance layer (crawler permissions and snippet controls).
The agent-readable web stack: a discovery layer (robots.txt, sitemap, feeds, llms.txt), a semantics and structured-data layer (clean HTML and JSON-LD), a serving layer (static generation and low JavaScript), and a governance layer (crawler permissions and snippet controls).

Designing a website for AI agents turns out to be mostly the discipline of designing a well-built website, plus a few new files and a few deliberate governance choices. The stack, from the bottom up:
  • Know your readers. Three access modes — bulk crawlers, answer-engine synthesis, and user-triggered agentic fetch — consume different surfaces (§2).
  • Get the discovery layer right. A minimal robots.txt that is honest about being advisory; an understanding of the AI crawler taxonomy; a sitemap with only loc and truthful lastmod; and language-tagged Atom/RSS feeds (§3).
  • Treat llms.txt accurately. A proposal, not a standard; ignored by Google; llms-full.txt is a vendor convention, not part of the spec. Publish it as a cheap option if you like, and be honest about size trade-offs (§4).
  • Use structured data because it is correct. JSON-LD @graph, types that match the content, values that match the visible text — and never FAQPage added for a rich result that no longer exists (§5).
  • Write clean, dense, stable HTML. Strict heading hierarchy, self-contained factual passages, stable canonical URLs, and dates that agree across text, datetime, and JSON-LD (§6).
  • Serve it statically. Do not make your text depend on client-side JavaScript that most AI crawlers do not run; fast static pages serve humans, crawlers, and Core Web Vitals at once (§7).
  • Decide governance on purpose. Allow, limit with permission tokens and snippet controls, or block — matched to what your content is for, with an accurate model of what each control does (§8).
  • Verify, then claim. Check your own status codes, unrendered HTML, structured data, and logs before you assert anything (§9).

The unifying thread is honesty — with the machine (structured data that matches the page, lastmod that is truthful, dates that agree) and with the reader (calling a proposal a proposal, saying when Google ignores a file, admitting when a 13 MB file is too big to be useful). The engines change every quarter; the honest, well-built page is what survives all of them.

12. References

All external sources verified on 2026-07-24. AI-crawler behavior and search-engine guidance change frequently; re-check the live documentation before relying on any specific detail for production configuration.

Related reading on this site:
Tool Use and Agent Protocol History and Timeline - Function Calling, MCP, and A2A
MCP Server Ecosystem Reference 2026
HTTP Security Headers Complete Reference - CSP, HSTS, COOP, COEP, Permissions Policy
Web Performance Checklist for Core Web Vitals - LCP, INP, and CLS
Self-Hosted Static Site Analytics - A Privacy-First Implementation Guide

References:
Tech Blog with curated related content

Written by Hidekazu Konishi