Lyrenth vs building your own scraper stack: the real 12-month bill

What web scraping infrastructure cost actually looks like over 12 months: fetch workers, browser pools, engineer time, and the shared-index alternative.

Every team that gives its product the ability to read web pages runs into the same build-or-buy question, and the build side hides most of its cost below the waterline. The visible part looks like a week of work: an HTTP client, an HTML parser, a queue. Twelve months later the same team is operating a small distributed system with a browser fleet, a politeness layer, an extraction pipeline that fails silently, and an on-call rotation. Nobody remembers deciding to build that.

We run a web index for a living, so we have both experience and an obvious interest here. This post tries to keep both honest. There are no invented "industry average" dollar figures below. Where we estimate, we label it as an engineering estimate and show the reasoning. Where we cannot estimate honestly, we give you the component list and the cost drivers so you can price it against your own infrastructure and your own salaries. And at the end we describe the cases where building your own really is the right call, because they exist.

What a production stack actually requires

A production pipeline for reading arbitrary public web pages is not one component. It is at least seven, and they fail independently.

1. Fetch workers

The part everyone budgets for. An HTTP client with sane timeout behavior, redirect handling, compression, character-set detection, retries with backoff, and enough concurrency management that one slow host does not stall the whole queue. This is real work, but it is the cheap part in both compute and maintenance. If the entire job were fetch workers, building it yourself would usually be fine.

2. The headless browser pool

The largest hidden cost, in compute and in babysitting. A large fraction of the modern web returns a hollow page to a plain HTTP fetch and only produces its content after JavaScript runs; we wrote up the mechanics in why scrapers get empty HTML from JavaScript sites. So a production stack needs headless browsers, and headless browsers are the most expensive thing you can run per page.

We operate a rendering fleet as part of the index, so this section is experience rather than theory. The cost drivers: each concurrent rendering session wants dramatically more CPU and memory than a plain fetch; sessions leak and must be recycled; the browser updates on its own schedule, and each update can change page behavior; pages hang, spin, or generate endless network activity, so you need settle heuristics and hard timeouts; and crash recovery has to be automatic, because crashes are routine rather than exceptional. A rendering pool is not a library you install. It is a service you operate, and it pages you.

3. IP and politeness management

Per-host rate limiting, robots.txt parsing and caching, backoff on 429s and 503s, and a policy for what happens when a site does not want automated traffic. This is where many DIY stacks drift somewhere uncomfortable: rotating proxies, residential IP pools, disguised user agents. That path is an arms race you fund forever, and it carries reputational and legal questions your future enterprise customers will ask about.

For the record, we made the opposite bet: LyrenthBot identifies itself in its user agent, signs its requests with Web Bot Auth, publishes its IP ranges with matching reverse DNS, respects robots.txt, and reads only the public web; it never signs in to accounts. Lyrenth does not train foundation models on crawled content. The details are at /bot. Whichever way you go, this layer is code plus policy plus ongoing attention, and it is not optional.

4. Extraction and cleaning

Fetching the page is half the job. The other half is turning a modern page (navigation, consent overlays, script tags, footer chrome, div soup) into text you would actually hand to a model or a human. This code has the worst failure mode in the stack: it fails silently. The pipeline keeps returning 200s while a template change on a source site quietly turns your extracted "content" into navigation links. You find out from a downstream user, or you do not find out.

5. Storage and deduplication

Read pages at any volume and you need to keep them: a cache so you do not re-fetch the same URL for every caller, canonical-URL resolution so the same document under three URLs is stored once, content hashing for dedup, and a retention policy. This is a real database with real growth, not a dictionary in memory.

6. Monitoring

Success rate per domain, render-versus-static ratios, latency distributions, queue depth, and alerting on all of it. Without per-domain monitoring you cannot see the silent extraction failures from component four, so this is load-bearing, not nice-to-have.

7. The engineer time that keeps it alive

The line item that dominates, and the one the initial estimate always omits. Every component above sits downstream of the open web, and the open web changes without notifying you. Sites redesign, frameworks change how they render, bot policies shift, the browser ships a new major version. The steady state of a DIY stack is not "done"; it is a recurring tax, measured in engineer-days per month, paid for as long as the stack exists.

We will not pretend to know your salaries, so here is the arithmetic template instead of a fake number: take your honest estimate of maintenance days per month across all seven components, multiply by a fully loaded engineer-day rate, multiply by twelve. For most teams that product is the largest number in this post, and it recurs every year.

The other model: one crawl serves every caller

A shared index changes the economics because the expensive work is amortized. Lyrenth crawls a page once, converts it into an AIDocument (clean, structured, model-ready content), and everything is served from the index. When you request a URL, cache.status: "hit" means the copy already existed: a cached index entry, returned instantly, with the site not contacted at all. A page not yet in the index is crawled, indexed, and served from that fresh index entry. Either way you read from the index. One crawl serves every caller, so no individual reader pays for a private browser fleet.

The index currently holds over 1.2 billion documents across more than 127 million domains (live stats), which means for a large share of URLs your first read is already a hit.

The call is one request:

curl -X POST https://api.lyrenth.com/v1/aidocument \
  -H "Authorization: Bearer $LYRENTH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.python.org/3/library/json.html"}'

The response carries the document plus the accounting you would otherwise build yourself: source (fetch metadata, render mode, status code), cache (hit or freshly indexed, and whether the origin was contacted), and economics (approximate AIDocument tokens versus raw HTML tokens). On real pages that last field is stark. The Stripe API reference measures 307,902 raw HTML tokens against 2,000 AIDocument tokens, 99.4 percent smaller; at $3.00 per million input tokens, that is the difference between $0.92 and $0.006 every time the page is read into a model's context. The full measured set is at /benchmarks.

The 12-month bill, where numbers are honest

Our prices are public at /pricing, so this part needs no estimating:

One credit per successful read, two if you force a refresh, and failed fetches (4xx and 5xx) cost nothing, so error pages never run up the bill. Tiers also set the freshness floor and the rate: Free serves cached entries no older than 90 days at 1 request per second, Starter guarantees 7-day freshness at 5 per second, Pro guarantees 24-hour freshness at 25 per second.

Now set that against the structure of the DIY bill. The DIY bill is mostly fixed: the browser pool, the storage, and the monitoring run whether you read ten pages or ten million, and the maintenance tax accrues either way. The index bill is entirely variable and bounded: at Pro volume, 2.4 million reads a year cost $948. We do not need to invent your cloud bill to make the comparison. We only note, as an engineering judgment you should redo with your own numbers, that $948 a year is less than one engineer-day per month at any rate you can realistically hire for, and one engineer-day per month is an optimistic maintenance estimate for the seven components above.

When building your own is the right call

Honesty cuts both ways. There are jobs Lyrenth deliberately does not do, and if your job is one of them, you need your own stack or a different tool.

Authenticated content. Lyrenth fetches as an anonymous client and never signs in to accounts. A URL behind a login returns an honest HTTP 422 instead of a workaround, and it costs no credits:

{
  "error": "upstream_unauthorized",
  "message": "example.com requires authentication (401). Lyrenth fetches as an anonymous client; URLs behind a login, paywall, or IP allowlist can't be retrieved this way.",
  "upstream_status": 401
}

If the data you need lives behind your own account on a partner site, or inside a VPN or IP allowlist, that is a build-it-yourself problem by design.

Exotic interactions. Filling forms, clicking through multi-step flows, taking screenshots, exercising a page the way a user would. Lyrenth reads pages; it does not drive them.

If those are your requirements, budget for the full component list above with clear eyes, and staff the maintenance line honestly.

When the shared index wins

If what you want is the readable content of public pages (documentation, articles, product pages, reference material, the things agents and pipelines actually consume), then the seven-component stack is overhead standing between you and the text. The index has already paid for the crawl, the render, and the cleaning once, on behalf of everyone. The API surface is small and documented at /docs, and if your agent framework speaks MCP, Lyrenth ships an MCP server with a read_url tool that takes one config block to add; setup is covered in our MCP guide.

Run the arithmetic on your own workload before deciding anything: the free tier is 2,000 AIDocuments a month with no card, enough to replay a real slice of your pipeline against the index. Sign up or start with the five-minute quickstart.