Caching web content for AI agents: what to cache, for how long, and when to force a refresh

A practical guide to caching the web for agents: why every agent team ends up building a cache, what staleness actually costs, and what a shared index changes.

Every team that builds an agent which reads the web goes through the same three stages. First they fetch every page live, every time, and pay full price in latency and tokens for identical content. Then they notice that most of what their agent reads was read before, and they build a cache. Then they discover that caching the web properly is its own product.

This post is about stage three, and about the shortcut past it.

Why agents re-read the same web

Agent traffic is not like human browsing. A human reads a page once. A fleet of agents researching companies, monitoring competitors, or answering support questions hits the same popular pages over and over: documentation, pricing pages, news, reference material. Left uncached, that repetition multiplies three costs: the seconds each live fetch takes, the tokens of re-processing the same content, and the load on the origin site, which is exactly the behavior that gets bot traffic blocked.

What a good cache has to get right

Building your own cache means answering questions that have nothing to do with your product. How long is a cached page trustworthy: a news homepage changes hourly, an API reference changes monthly, and nothing in the URL tells you which is which. What do you store: raw HTML bloats storage and still needs cleaning on every read, while cleaned text must be re-extracted whenever your parser improves. When do you invalidate: too eager and you are back to live-fetching everything, too lazy and your agent quotes a price that changed last week. And every answer has to hold across the whole team, or each service builds its own contradictory copy.

The shared-index answer

An index is what a cache becomes when it is the product instead of the workaround. Every page Lyrenth serves is already fetched, rendered when needed, and stored as a clean AIDocument, and the same stored copy serves every caller, so the economics beat any private cache: the fetch cost was paid once, for everyone.

Freshness is explicit instead of accidental. Each plan documents the maximum age of a cached read, and when your agent must have this minute's version, a force-refresh fetches the page anew and updates the stored copy for the next reader. You choose per request, which is the control a hand-built cache rarely reaches.

When not to cache

Honesty requires the other half: some reads should never be cached. Anything personalized, anything behind authentication, anything where the answer changes per viewer has no meaningful shared copy. For the public web that agents actually read, though, the pattern holds: cache by default, refresh on demand, and let someone whose whole product is the cache carry the invalidation problem.

Start with the free tier: 2,000 reads a month, no card, cached reads included.