Crawl-on-miss: how an index reads pages it has never seen

Crawl on miss, explained: how Lyrenth serves every page from its index, whether cached or freshly indexed, and when a re-crawl happens before serving.

Give an agent the ability to read the web and it will eventually ask for a URL nobody has requested before. An index that could only answer for pages it already holds would make that first read a dead end. Lyrenth's answer is crawl-on-miss: request any public URL and you get an AIDocument back, whether the index has held that page for months or first learned about it because of your call.

The index currently holds over 1.2 billion documents across more than 127 million domains (live counts at /web-index/stats), and crawl-on-miss is the reason those numbers are a floor rather than a ceiling on what you can read through the AIDocument API. This post explains the two serving states, what the cache block in every response tells you, and how freshness windows and force_refresh decide when we go back to the origin.

One rule: everything is served from the index

Start with the rule that makes the rest of the design easy to reason about: every AIDocument is served from the index. There is no separate live-fetch mode and no code path that pipes raw crawler output at you. What varies between requests is only whether the index entry existed before you asked.

Cached index. The copy already existed. cache.status comes back as hit, the response is instant, and the site was not contacted. Most reads land here, because popular pages are exactly the pages the index has already seen.

Freshly indexed. The index has never seen the URL. Lyrenth crawls the page, cleans it into an AIDocument, writes it into the index, and serves you that fresh entry. Your response is the first read of a brand-new index record.

Note what the second state is not: the crawl is not the thing serving you. The crawl is how the entry gets into the index, and the index is what serves it, to you and to everyone after you. That distinction sounds pedantic until you see the consequence: one crawl serves every caller. The second person to request that URL gets a cached-index hit, and the origin site never hears about it.

Reading the cache block

Every response carries a cache block, so you never have to guess which state you got. Here is the call:

curl -s https://api.lyrenth.com/v1/aidocument \
  -H "Authorization: Bearer $LYRENTH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.python.org/3/library/json.html"}'

And here is the relevant slice of the response, with the body and the other fields trimmed:

{
  "aidocument": {
    "source": {
      "url": "https://docs.python.org/3/library/json.html",
      "fetched_at": "2026-07-16T09:14:03Z",
      "render_mode": "static",
      "status_code": 200
    },
    "cache": {
      "status": "hit",
      "origin_contacted": false
    }
  }
}

A cache.status of hit says the copy already existed, and origin_contacted: false confirms the site was not touched to answer you. On a freshly indexed read, the cache block records instead that the origin was contacted, and source.fetched_at carries a timestamp from moments ago. On a hit, fetched_at is the honest capture time of the stored copy, so you always know how old the page you are reading actually is.

(For what sits in the rest of the response, the cleaned body, the structure, and the token economics, see what an AIDocument is.)

What this means for an agent

Three practical consequences follow.

Hits return instantly. A cached-index read involves no origin fetch, no render, and no waiting on someone else's server. For an agent making several reads per task, this is the difference between a loop that is bounded by your own model calls and a loop that is bounded by the slowest website in the chain.

A first-ever read pays one origin fetch. When you request a URL the index has never seen, your call waits for the crawl and the cleaning before the fresh entry comes back. That is the honest cost of being first, and you pay it once per page, not once per caller.

After that, every caller shares the copy. Your first read of an obscure changelog page becomes everyone's instant hit, and the same is true in reverse. This is the core difference between an index and a fetch-on-demand tool: the work compounds. It is also why the read pattern beats the browse pattern for agents, which we cover in agents don't browse, they read.

Both states cost the same: one credit per successful read, whether it was a hit or freshly indexed. And the single origin fetch behind a miss is made by an identified crawler. LyrenthBot signs its requests with Web Bot Auth, publishes its IP ranges, resolves via reverse DNS, and respects robots.txt (verification details at /bot). It reads only the public web and never signs in to accounts. Lyrenth does not train foundation models on crawled content; the crawl exists to serve the page to callers, nothing else.

Freshness windows: when a hit is not fresh enough

A stored copy is only useful if it is recent enough for your purpose, so every plan carries a freshness window, and the serving rule extends naturally: if the copy in the index is older than your plan's window, Lyrenth re-crawls the page before serving you. You still get an index entry, just a newer one. A re-crawl triggered by your freshness window is a normal read and costs one credit.

On Free, the copy you are served is never older than 90 days; on Pro, never older than 24 hours. Rate limits and the rest of the plan details are on the pricing page.

force_refresh, for this-second freshness

Sometimes even 24 hours is too old: a changelog published this morning, a status page mid-incident, a price you are about to act on. Pass "force_refresh": true in the request body and Lyrenth re-crawls and re-indexes the page before serving it, regardless of how fresh the stored copy is. A forced refresh costs 2 credits instead of 1, which keeps it a deliberate choice rather than a default. The response is still served from the index; you are just insisting the entry be seconds old.

The misses that stay misses

Crawl-on-miss covers the public web, and only the public web. Because LyrenthBot fetches as an anonymous client, a URL behind a login, paywall, or IP allowlist cannot become an index entry. Rather than hand you a stripped login shell, the API says so plainly with an HTTP 422:

{
  "error": "upstream_unauthorized",
  "message": "internal.example.com requires authentication (401). Lyrenth fetches as an anonymous client; URLs behind a login, paywall, or IP allowlist can't be retrieved this way.",
  "upstream_status": 401
}

Failed fetches cost nothing. A 4xx or 5xx outcome does not consume a credit, so probing a URL that turns out to be gated is free.

The simplest way to internalize the two states is to trigger them yourself: read a page, read it again, and watch cache.status flip to hit on the second call. The free tier includes 2,000 AIDocuments a month with no card required, and the quickstart gets you to that first call.