What is an AI-readable web index?
What an AI-readable web index is, how it differs from scraping and from a classic search index, and why one crawl should serve every machine reader.
"AI-readable web index" is a term we use constantly, so this post pins it down. An AI-readable web index is an index of the public web in which every page is stored as a clean, structured, machine-readable document, built to be read by machines rather than rendered for humans. The web already has indexes and it already has ways to fetch pages, so the definition is easiest to see by contrast. It is not a search engine's index, which serves links and snippets to people. It is not scraping, which fetches a page once per consumer and hands back raw HTML. It sits between the two: the crawl-once economics of an index, with the document itself as the thing served.
Lyrenth is one of these. Our index holds over 1.2 billion documents across more than 127 million domains (live counters at /web-index/stats), and every one of them is stored as an AIDocument: the readable content of the page inside a stable, structured envelope. This post walks through why that shape exists, what it contains, and where the index model differs from the alternatives.
The problem with HTML for machines
HTML is a set of rendering instructions. It tells a browser where the navigation goes, which scripts to load, how the cookie banner behaves, and which class names style the footer. A person never sees any of that; the browser consumes it and paints something readable. A machine reading the same page gets no such courtesy. It receives the instructions themselves, and if the machine is a language model, it pays for every one of them by the token.
The numbers are not subtle. The Stripe API reference weighs 307,902 tokens as raw HTML. The same page as an AIDocument is 2,000 tokens, which is 99.4 percent smaller. At $3.00 per million input tokens, that is $0.92 versus $0.006 every time the page goes into a model's context. The rest of our measured set (Vercel, Cloudflare, GitHub, and Kubernetes documentation pages) lands between 94.3 and 99.4 percent smaller; the full table is at /benchmarks.
Cost is only half the damage. The model also has to wade through all of that scaffolding to find the paragraph that matters, so retrieval gets noisier at the same time it gets more expensive.
And the waste repeats. When every consumer fetches and strips a page independently, the same conversion work is redone for every reader of every page, forever. That observation is the core of the index idea.
What an AI-readable web index is
Three properties make an index AI-readable rather than merely large.
It stores documents, not pages. Each URL is converted once into a clean, structured document: content extracted, boilerplate stripped, JavaScript rendered when the page needs it. What remains is the part a machine wants, in a shape it can rely on.
It is an index. The web is crawled by an identified bot, the documents are stored, and every caller is served from the same stored copy. Retrieval is keyed by URL: you ask for a page, you get that page's document.
It is machine-first. The document carries not just readable content but a set of facts about itself: where it came from, when it was fetched, what it cost to read, how it is structured.
At Lyrenth the document is the AIDocument, and reading one is a single call:
curl -X POST https://api.lyrenth.com/v1/aidocument \
-H "Authorization: Bearer $LYRENTH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/docs/getting-started"}'
The response is one aidocument envelope. Alongside the readable content, it carries:
| Block | What it tells you |
|---|---|
source | The URL you asked for, the canonical_url, when it was fetched (fetched_at), how it was rendered (render_mode), and the upstream status_code |
cache | The read's status (cached index copy or freshly indexed) and whether the origin site was contacted (origin_contacted) |
identity | The page's title and identifying metadata |
economics | output_tokens_approx, raw_html_tokens_approx, token_savings_percent, and estimated_cost_usd for this read |
signals | Machine-usable signals about the page |
structure | The document's structure, including its links |
The economics block is the honesty mechanism behind the benchmark claims above: every response reports its own token counts, so the savings are checkable on the exact pages you read. The full field reference is at /docs/aidocument, and we wrote a dedicated walkthrough of the shape in What is an AIDocument?.
Index, not scraper
Scraping is fetch-per-consumer. If a thousand agents need the same documentation page this week, a scraping approach hits the site a thousand times, usually anonymously, and each consumer runs its own extraction with its own bugs. Sites respond the way you would expect: rate limits, bot walls, and markup that shifts under your parser. JavaScript-heavy sites add a second failure mode entirely; we covered it in why scrapers get empty HTML from JavaScript sites.
An index inverts the economics: one crawl serves every caller. The serving semantics are worth stating precisely, because they are the part people ask about. Everything Lyrenth serves is served from the index. On each read, cache.status tells you which of two things happened. A hit means the copy already existed: you are reading the cached index entry, the response is instant, and the site was not contacted. A page that is not yet in the index is crawled, indexed, and served from that fresh index entry, reported as freshly indexed. In both cases the thing you receive is an index entry, and the crawl that created it serves every subsequent caller who asks for that page.
Freshness then becomes a plan property instead of a per-request gamble. Each tier sets a maximum age for a cached copy (90 days on the free tier, 7 days on Starter, 24 hours on Pro; details at /pricing), and "force_refresh": true re-crawls a specific URL on demand for two credits instead of one. Failed fetches cost nothing.
The other difference from scraping is identity. LyrenthBot does not pretend to be a browser. It sends its own user agent, signs its requests with Web Bot Auth, publishes its IP ranges, and resolves via reverse DNS, and it respects robots.txt; everything a site needs to verify it is at /bot. It reads only the public web and never signs in to accounts, and Lyrenth does not train foundation models on crawled content. When a page genuinely cannot be read anonymously, the API says so instead of pretending:
{
"error": "upstream_unauthorized",
"message": "example.com requires authentication (401). Lyrenth fetches as an anonymous client; URLs behind a login, paywall, or IP allowlist can't be retrieved this way.",
"upstream_status": 401
}
That comes back as an HTTP 422, it costs no credits, and it is accurate. An identified crawler plus honest failure semantics is what lets the index model coexist with the sites it reads. It also gives site owners something to work with: verify a domain and you get an AI-readiness score for it, per-domain stats, and analytics on LyrenthBot's actual visits (/docs/bot-tracking).
Not a search engine either
A classic search index also crawls once and stores a copy of the web, so the confusion is understandable. The difference is what gets served, and to whom. A search engine's product is a ranked list of links and snippets, rendered for a person who clicks one and reads the page in a browser. The stored copy exists to build the ranking; the pointer is the product.
An AI-readable web index serves the document itself, to machines. There is no results page and no snippet. The caller arrives with a URL and leaves with the full content of that page in machine shape, plus the provenance and economics to account for it. The consumer is a model, an agent, or a pipeline, not a person scanning links.
The two share the crawl-once foundation, which is why the scale looks familiar: over 1.2 billion documents across more than 127 million domains, growing as the crawl runs. What changes is the unit of retrieval. A search index answers "which pages match this query?" An AI-readable index answers "give me this page, in a form I can use."
What it enables for agents
Once the web is available as documents rather than markup, several standing agent problems stop being problems.
Token budgets become plannable. A research loop that reads ten pages can know, per page, what each read put into context and what it cost, because every response carries its own economics block. At two orders of magnitude fewer tokens per page, the same context window holds the whole loop instead of two pages of boilerplate.
One shape across the whole web. An agent reading pages from 127 million domains cannot maintain per-site parsers. The envelope is the same whether the source was a static page or a JavaScript-rendered app, so the code that consumes it is written once.
Reading replaces browsing. An agent does not need a viewport, screenshots, or simulated clicks to consume content; it needs the document. We made that argument at length in Agents don't browse, they read.
It plugs into the tools you already run. Lyrenth ships an MCP server exposing a read_url tool, and one config block gives Claude Desktop, Claude Code, or Cursor the ability to pull any public URL as an AIDocument. The setup is covered in our MCP guide.
The shortest way to evaluate an index is to read pages you already know through it and check the envelope against your expectations. The free tier is 2,000 AIDocuments a month with no card, so that experiment costs nothing: create a key and POST your first URL, or start from the quickstart.