Agentic browsers vs crawlers: which one does your agent actually need?
Crawlers read the web so a question can be answered later. Agentic browsers use the web now. Picking the wrong one costs latency, tokens, and access.
Two things that look alike are being built at once right now, and the difference between them is worth being precise about, because most teams end up paying for the wrong one.
A crawler reads the web. It fetches a page, turns it into text a model can use, and stores it so a question can be answered later. An agentic browser uses the web. It drives a real browser session: it clicks, fills forms, waits for a modal, compares two prices, completes a checkout. One produces knowledge. The other produces a change in someone else's system.
The reason this matters commercially is that the browser is the more impressive demo and the more expensive habit. It is very easy to reach for a browser session because it obviously works, and then discover you are paying browser prices to read three paragraphs of text.
The test that decides it
Ask what the outcome of the step is.
If the outcome is knowledge, read. "What does this page say about pricing." "Which of these ten suppliers ships to Ireland." "Summarise this documentation." Nothing changes on the far side. You need the text, correctly extracted, and you need it cheaply enough to do it a thousand times.
If the outcome is a state change, use. "Book the appointment." "Submit the form." "Add it to the basket and pay." Something is different afterwards on a system you do not own. That genuinely needs a session, a real browser, and a human's authority behind it.
Most agent products are ninety percent the first thing and ten percent the second. The ten percent is what gets demoed.
What reading through a browser actually costs
Take a concrete page. Here is the Wikipedia article on retrieval-augmented generation, read through the AIDocument API:
curl -s -X POST https://api.lyrenth.com/v1/public/aidocument \
-H "Content-Type: application/json" \
-d '{"url":"https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}'
The economics block on that response, measured on 24 August 2026:
"economics": {
"output_tokens_approx": 14924,
"raw_html_tokens_approx": 44191,
"token_savings": 29267,
"token_savings_percent": 0.662,
"estimated_cost_usd": {
"our_output": 0.044772,
"raw_html": 0.132573,
"savings": 0.087801
},
"pricing_basis": {
"input_price_per_1k_usd": 0.003,
"model_class": "mid-tier frontier model class"
}
}
The raw page is about 44,000 tokens. The readable document is about 15,000. Two thirds of what a browser would hand your model is navigation, scripts, markup and boilerplate that costs money to think about and contributes nothing to the answer.
That is the cost per page, before you have counted the seconds. A browser session has to start, load, execute scripts, and settle before it can tell you anything. A read from an index answers in milliseconds because the work already happened once, for everybody.
The third cost, which is newer
Access rules are tightening. Through 2026 the default posture of the web towards automated traffic has moved from "allow and see" to "declare what you are and we will decide", and the September 15 changes are the clearest example of that turning into infrastructure.
A browser session driven by an agent is, from the far side, unattributed traffic with a real browser fingerprint. It is not obviously a robot and it is not obviously a person. That ambiguity was an advantage in 2024. In 2026 it is the thing that gets you a challenge page, then a block, then an abuse report to your hosting provider.
Reading through an index inverts that. The fetch happens once, from a crawler that identifies itself, publishes its address ranges, and honours the site's own rules. Your product then reads from the index rather than from the site. You are not one more anonymous browser at the door.
That is not only a politeness argument, though it is that too. It is an availability argument: the access you have not asked for is the access you can lose overnight.
What "read" should return
If you are going to read rather than browse, the thing you read should be worth the trip. Handing a model cleaned-up HTML is only a small improvement on handing it HTML.
An AIDocument is one JSON shape for any URL: the markdown a model should actually see, the structure of the page (headings, tables, links), the source trace saying where it came from and when it was fetched, cache truth saying honestly whether the origin was contacted, the structured data the publisher already declared, and the token economics above so you can see what the read cost you.
The structure matters more than people expect. Roughly a tenth of pages carry usable JSON-LD, and when a publisher has already declared that a page is a product with a price, or a recipe with a cook time, an agent that reads that declaration is more reliable than one that infers it from prose. It is the difference between parsing a sentence and reading a field.
A decision table you can actually use
| The step | Read or use | Why |
|---|---|---|
| Answer a question from a page | Read | Knowledge out, nothing changes |
| Compare fifty products | Read | Fifty browser sessions is a bill, not a design |
| Check whether something changed | Read | Freshness is a property of the index |
| Log in and download an invoice | Use | Authority and a session are the point |
| Fill and submit a form | Use | State change on someone else's system |
| Extract the whole documentation set | Read | Bulk reading through a browser is the classic mistake |
The pattern in real products is a wide read layer and a narrow use layer. Read broadly and cheaply to decide what to do; drive a browser only for the handful of steps that genuinely act.
The hybrid, which is what most products end up as
Say you are building a procurement assistant. It has to find suppliers, compare them, and then place an order.
Finding and comparing is reading. Fifty supplier pages, extracted to text and structure, is fifty cheap reads that can happen in parallel and be cached for the next user who asks a similar question. None of it needs a session, because nothing changes on the supplier's side when you look at their catalogue.
Placing the order is using. One browser session, authenticated as the customer, driving a real checkout. It is slow and expensive and that is fine, because it happens once at the end and it is the step the customer is actually paying for.
Teams that get this backwards build fifty browser sessions to compare prices and then wonder why the feature costs more to run than it earns. The read layer should be wide and cheap. The use layer should be narrow and careful.
Where MCP fits
The reason this is worth writing down now is that the plumbing finally agrees. The Model Context Protocol has become the common way to hand an agent a capability, and that includes the capability to read a URL. It means the read layer does not have to be a bespoke integration in every framework you try: it is a tool the agent already knows how to call.
If you are wiring this up, the quickstart is one call, and the same read is available as an MCP tool for agent frameworks that prefer it. We wrote separately about what an index actually promises about freshness, which is the question that decides whether reading from one is safe for your use case.
The short version
Use a browser when the agent has to act. Read from an index when the agent has to know. Reading is cheaper by roughly two thirds on tokens for a typical page, faster by orders of magnitude, and it does not spend access you cannot get back.
The impressive demo is the agent that books the flight. The product that survives contact with a thousand users is mostly the boring half that reads.