Two billion documents, and why the number matters more after September 15
Lyrenth's index crossed two billion AI-ready documents this week, on a public live counter. Here is what an index that size changes for agents as the web's defaults turn restrictive.
This is part of Countdown to September 15, a short series about the web's new defaults for machine access. Part one covered what actually changes on the date.
This week the counter on our front page crossed two billion.
Two billion documents, each one a web page already fetched, rendered when it needed rendering, cleaned of its navigation and cookie banners and scripts, and stored as an AIDocument: the page's actual content in a stable, compact shape an agent can read in one request. The counter is live on lyrenth.com, it moves while you watch it, and it only moves one way.
A big number is easy to announce and hard to check, which is why ours is public and always current rather than a press-release snapshot. But the interesting question is not how big the number is. It is what the number does for an agent, and the answer changed this summer.
An index is a different promise than a fetch
Most tools that read the web for AI work like a browser without the human: you ask for a URL, they go get it, right then, from the origin. That works until it does not. The origin is slow, or down, or behind a challenge page, or it simply says no to bots, and says no more often every month. Every read is a fresh negotiation with a stranger.
An index makes the opposite promise: the work of fetching, rendering, and cleaning happened before you asked. When the page you want is one of the two billion, your read is served from the index in a fraction of a second, at a fraction of the tokens, without the origin being asked again. The measured numbers live on our benchmarks page, built from real index data, and the token side is documented in what famous pages cost to read.
Scale is what turns that promise from occasional luck into the normal case. At two billion documents, growing by tens of millions a day, the page an agent asks for is increasingly one we already hold, already clean, already cheap. And when it is not, the miss is fetched by a crawler that identifies itself, respects the site's rules, and adds what it learned to the index, so the next agent's ask is a hit.
Why this matters more after September 15
Everything in part one of this series compresses into one sentence: the cost of an anonymous live fetch is going up. Sites are inheriting defaults that block unexplained bots, and the share of the web willing to answer a stranger's request is shrinking by policy, not by accident.
An index absorbs that shift on your behalf. The fetching that fills it is done by one named, verifiable, policy-published bot with a standing relationship to the sites it visits, described openly on our bot page. Your agent never has to be the stranger at the door, because the knocking already happened, politely, once, for everyone.
That is the quiet argument the counter makes as it climbs. Not that big is impressive, but that ready is different from reachable. After September 15, a growing part of the web will be reachable only by those who prepared. Two billion documents is what prepared looks like.
Reading from the index starts free: 2,000 documents a month, no card. Point your agent at any URL and see what comes back.