We measured the token cost of famous pages

What is the token count of a web page for an LLM? We measured five famous pages: raw HTML vs AIDocument tokens, with real per-read costs.

Nobody prices a web page before an agent reads it. Pages do not publish their token counts, and a page that loads in a second in a browser can be a third of a million tokens once you fetch the HTML and hand it to a model. So we measured five pages most developers have open in a tab on any given week: Stripe's API reference, Vercel's Functions docs, Cloudflare's Workers docs, GitHub's REST quickstart, and the Kubernetes Pods concepts page.

The short version: raw HTML for these pages runs from roughly 89,000 to 308,000 tokens each. The same pages as AIDocuments run from about 1,400 to 7,500. The rest of this post is the method, the table, and what that gap means for anyone budgeting an agent's context window.

How we measured

Every AIDocument response includes an economics block that does the accounting for you: the approximate token count of the page's raw HTML, the token count of the AIDocument body we actually return, and the savings between them. So the whole method is one API call per page:

curl -s -X POST https://api.lyrenth.com/v1/aidocument \
  -H "Authorization: Bearer $LYRENTH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "PAGE_URL"}'

From each response we took economics.raw_html_tokens_approx and economics.output_tokens_approx, then priced both at a basis of $3.00 per million input tokens, the same basis used across /benchmarks. Here is the actual block from the Stripe read:

"economics": {
  "raw_html_tokens_approx": 307902,
  "output_tokens_approx": 2000,
  "token_savings_percent": 99.4,
  "estimated_cost_usd": 0.006
}

Two notes on what a read is, so the numbers are interpretable.

First, everything is served from the index. When a page is already indexed (ours holds over 1.2 billion documents across more than 127 million domains), cache.status comes back "hit": the copy already existed, the response is instant, and the site is not contacted. A page not yet in the index is crawled, indexed, and served from that fresh index entry. One crawl serves every caller, so a popular page is fetched once, not once per user. When we do fetch, the crawler identifies itself as LyrenthBot, with Web Bot Auth signed requests, published IP ranges, and reverse DNS (details), and it reads only the public web.

Second, the raw-HTML number is not hypothetical. It is what the same page costs if you fetch it yourself with an HTTP client and put the response straight into the model's context, which is exactly what most homegrown "read this URL" tools do.

(Failed fetches, a 4xx or 5xx from the origin, cost nothing, so scripting this across a URL list is safe.)

The results

Sorted by raw size, priced per single read into context at the $3.00 per million basis.

There are two ways to read the table. As a percentage, the reduction ranges from 94.3% to 99.4%. As money, the raw Stripe reference costs $0.92 every time it enters a context window, and the AIDocument costs $0.006. Neither number is dramatic on its own. Multiply by how many pages your agent reads per task, and how many tasks it runs per day, and the two columns describe two very different bills.

Where the waste lives

It is worth being precise about what those hundreds of thousands of raw tokens are, because almost none of them are the documentation.

Navigation. Documentation apps ship the sidebar on every page, and the sidebar is frequently the entire table of contents of the product: every section, every endpoint, every guide, as nested lists of links. The model pays to read the whole manual's index before it reaches the first paragraph of the page it asked for.

Scripts and hydration payloads. Modern docs sites are JavaScript applications. The HTML carries the framework runtime plus a serialized copy of the page state so the client can rebuild the UI. On these pages you often pay for the content twice: once as markup, once again inside a JSON blob that the browser was supposed to consume, not your model.

Chrome. Headers, footers, search overlays, keyboard-shortcut modals, theme switchers, feedback widgets, version pickers. Useful in a browser, dead weight in a context window.

Consent and tracking. Cookie banners, consent-manager markup, analytics snippets, tracking pixels. Necessary for compliance, useless to a model.

The markup itself. Class-name soup, wrapper divs, inline SVG icons. Every angle bracket and every utility class gets tokenized and billed like prose.

An AIDocument is what is left after all of that goes: the title, the body, the structure (headings, links, tables, code blocks), plus source and economics metadata. The full shape is documented at /docs/aidocument. And the cost is only half the damage from the waste: a model reading raw HTML has to get past all of it to find the sentence you wanted, which makes the answer noisier at the same time it makes it more expensive.

What this does to a context budget

The starkest row is the first one. A 200,000-token context model cannot hold the raw Stripe API reference at all: 307,902 tokens is past the window before you add a system prompt, tool definitions, or the question you were trying to answer. The read does not degrade gracefully; it simply does not fit, and whatever truncation your middleware applies is as likely to keep the navigation as the content.

Even pages that do fit crowd out the work. An agent doing real research does not read one page; it reads five or ten across a loop. The five pages in our table sum to 860,846 raw tokens, more than four 200k windows laid end to end. As AIDocuments, the same five pages sum to 17,877 tokens. All five fit in a single context at once, next to a long conversation, with most of the window still free for reasoning.

That difference changes how you design the loop. With raw HTML you ration reads, summarize aggressively (and inherit the hallucinations of the summarizer), and evict pages between turns. With reads this small you can keep every page the agent has seen in context for the whole task and let the model quote from the source instead of from its own compressed memory. We walked through that pattern in how to feed web pages to an LLM without blowing the context window.

Caveats, honestly

These five rows are real measurements, not a claim about the whole web.

• Savings track page type. Heavy documentation apps, the ones with the big sidebars and hydration payloads described above, sit at the top of the range: that is the 99% club. Pages that are mostly prose with modest chrome, like the Kubernetes concepts page or the GitHub quickstart, save less, in the mid-90s, because more of their raw HTML was content to begin with. Measure your own URLs before you budget around a single figure.
• The token counts are approximate by design (the fields are named _approx). Tokenizers differ across models, so treat the numbers as close estimates, not invoices.
• The $3.00 per million basis is just a basis. Your model's input rate may be higher or lower; the dollar columns scale linearly with it, and the percentage column does not move.
• This is the public web only. A page behind a login or paywall returns an honest HTTP 422 (upstream_unauthorized) rather than a guess, and a failed read costs nothing.

The full measured set lives at /benchmarks, so you can check any row against the current numbers.

If you want these figures for the pages your agent actually reads, run the curl above against your own URLs and read the economics block that comes back. The free tier is 2,000 AIDocuments per month with no card, and the quickstart covers the setup in about five minutes.