Structured data for AI: schema, llms.txt, and crawler policy

Making a site legible to AI means publishing an explicit entity graph in Schema.org JSON-LD, summarizing the site in llms.txt, allowing AI crawlers deliberately in robots.txt, and keeping one stable canonical URL per idea.

By Robert Shumake (Ajarn Shaman Shu)

The entity graph

Schema is how you state facts in a form a machine does not have to infer. A Person node with alternateName, knowsAbout, sameAs, and a stable @id, referenced from every page about that person, is worth more than any amount of keyword-tuned prose.

  • One canonical Person @id, referenced everywhere else
  • Book nodes for catalog titles with author and store links
  • BreadcrumbList on deep routes so hierarchy is explicit
  • Self-referencing canonical and og:url on every page

llms.txt and crawler policy

llms.txt gives an assistant a hand-written map of the site: what exists, what each page answers, and how the entity should be described. Robots policy decides who may read it. Blocking every AI crawler removes you from the answer layer entirely — a choice worth making deliberately, not by default.

This site allows GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, and others explicitly, while keeping admin and auth routes disallowed.

Programmatic pages done responsibly

Templated topic pages earn their place only when each one answers a distinct question with material a reader could not get from the others. Spinning one page into fifty near-duplicates is the classic failure — it degrades both search performance and the model's picture of the entity.

Continue reading

Stay connected

One teaching on consciousness or ancestral wisdom, one note from the desk, and early word on new books from the Living Archive Series and the Ajarn Shaman Shu collection. No spam, and one click to leave.