One crawl serves everyone: the economics of a shared web index

Why one crawl serving every reader changes the cost of machine-read web data: real token numbers, cache math, and what shared infrastructure means for origins.

Every AI product that reads the web today re-solves the same problem in private. Thousands of teams fetch the same popular pages, strip the same boilerplate, parse the same markup, and pay for the same tokens, independently, over and over. It is as if every household that wanted electricity ran its own wire to the power plant.

This post is the economic argument for the alternative: a shared index, where a page is fetched once, cleaned once, stored once, and read by everyone. The argument is simple enough to fit in three numbers.

Number one: what a page costs to read raw

Web pages are built for browsers, so most of their bytes are not content. From our public benchmarks, measured on the live API against real pages:

• Stripe's API reference is about 307,902 tokens of raw HTML. The same page as a clean AIDocument is about 2,000 tokens: a 99.4 percent reduction.
• Vercel's Functions docs: 237,390 raw tokens against 2,149 clean, 99.1 percent.
• Kubernetes' Pods concepts page: 131,977 against 7,475, 94.3 percent.

At the $3.00 per 1M input token price of a mid-class model, the Stripe page costs roughly 92 cents every time an agent reads it raw, and about half a cent as an AIDocument. An agent that reads fifty pages per task is either a rounding error or a real invoice, depending entirely on which format it reads. The full measured table is in the token cost of famous pages.

Number two: how many times the web gets re-fetched

Cleaning the page solves the token cost. It does not solve the duplication, because every reader still fetches and cleans privately. The popular pages, the exact ones most likely to be asked about, are the ones fetched redundantly the most: every price checker hits the same product pages, every research agent hits the same documentation, every assistant hits the same news.

A shared index collapses that entire multiple to one. The first request for a URL causes one fetch by one identified crawler; the page becomes one canonical AIDocument; every subsequent reader, from any customer, is served from that copy. When a thousand agents ask for the same URL, the origin sees one crawl. We wrote up the serving mechanics, including what happens on a request for a page the index has never seen, in crawl-on-miss, explained.

Freshness stays in the reader's control rather than being the price of sharing: the default serves the indexed copy with its age declared, and force_refresh makes the index refresh its copy before serving when the task genuinely needs the newest version.

Number three: what the index already holds

Sharing only works if the shared thing has coverage, because the first URL an agent asks for and does not get ends the session. As of this week the index holds more than 2 billion pages across 150 million domains, growing by over 50 million pages a day, with live counters on the stats page. Coverage is the product: it is the difference between a cache that sometimes helps and an index you can build on.

The side of the ledger nobody prices: origins

The same math that saves readers money is what makes the model sustainable for the websites being read, and this half matters just as much.

Redundant fetching is a cost the web absorbs silently. Every private scraper that reads a page burns the origin's bandwidth and compute to produce value the origin never sees, multiplied by every company doing it in parallel. The shared index inverts that: aggregate machine demand reaches the origin as a single, rate-limited, robots-respecting crawl from an identified crawler, and everything after that is served from the index without touching the site at all. Attribution and source links travel with every document, we do not train foundation models on crawled content, and site owners can see, verify, and control how they are read.

This is why the shift the web is going through right now, defaults that welcome search-class indexing while walling off anonymous per-agent fetching, which we covered in the metered web, points toward shared infrastructure rather than away from it. An index is the version of machine reading that origins can live with.

Why per-read pricing follows from the architecture

Shared infrastructure is also what makes honest pricing possible. When the expensive events (fetching, rendering, cleaning, storing) happen once per page instead of once per reader, the marginal cost of a read is small and measurable, so the product can charge per read, with the free tier a real allowance rather than a teaser. You pay for what your agents actually consume, not for the duplicated machinery behind it. That is the entire pricing philosophy, and the pricing page is short because the architecture did the work.

The one-sentence version

Reading the web is a shared problem wearing a thousand private costumes: fetch once, clean once, serve everyone, and the cost per read collapses for readers while the load on origins collapses with it. That is the economics of an index, and it is live today: the quickstart is one API call, and the free tier needs no card.