The web is getting metered for machines: what agent builders should know before September 15

Cloudflare's new defaults block agent fetchers on ad pages and per-fetch access is getting priced. How agent builders keep reading the web through an index.

If you build AI agents that read the web, two announcements this month changed your operating environment, and most of the coverage was written for publishers, not for you. Here is the agent builder's view.

What changed

First: agent fetchers lose default access to a slice of the web. On July 1, Cloudflare announced a new classification for automated traffic (Search, Agent, Training) and new defaults arriving September 15, 2026: for new domains on Cloudflare, the Agent and Training categories are blocked by default on pages that display ads. "Agent" here means exactly the thing many of us build: chat fetchers and browser-driving automation acting in real time on a user's behalf. Cloudflare names the pattern explicitly, and the default answer for it on ad-monetized pages becomes no.

Second: per-fetch access is becoming per-use pricing. Cloudflare's pay-per-crawl marketplace is evolving toward what press coverage describes as pay-per-use, where publishers charge when content actually contributes to an answer rather than merely when it is fetched (TechCrunch's coverage has the details). The precise mechanics will evolve, but the direction is not subtle: the anonymous, free, per-request read is on its way out.

Put together: the raw fetch, the thing every DIY agent stack is built on, is becoming both less allowed and more expensive.

Why this breaks the DIY fetching stack

The standard agent architecture fetches pages directly: the agent hits the origin, parses whatever HTML comes back, and burns context window on markup. That stack already had problems we have written about before: JavaScript-heavy sites return empty shells, and the true cost of running your own fetching infrastructure is a full-time job hiding inside a weekend project.

September 15 adds a policy wall on top of the technical ones. Every direct fetch your agent makes is, in Cloudflare's taxonomy, Agent traffic, and on a growing share of the web the default answer to that traffic on monetized pages is a block page. Retrying harder is not a strategy; being evasive about identity is how bot operators end up on the wrong lists permanently.

The architecture that keeps working

Notice which category stays allowed by default: Search. Indexing, the behavior where content is collected once, served with attribution, and funnels discovery back to the origin, is the one the new defaults treat as welcome. That is not an accident, and it points at the durable architecture for agents that read the web:

Do not fetch. Read from an index.

When your agent reads through an index, the fetch that touched the origin happened once, was made by an identified, verified, search-class crawler that honors robots.txt, and now serves every reader from the same copy. Your agent gets the page as a clean, structured AIDocument instead of raw markup, and the origin sees one polite crawl instead of a thousand agent fetches. We wrote up the model in detail in agents don't browse, they read.

The numbers are not close. From our public benchmarks, measured on the live API:

• Stripe's API reference: 307,902 tokens of raw HTML, 2,000 tokens as an AIDocument, a 99.4 percent reduction.
• Vercel's Functions docs: 237,390 raw tokens, 2,149 as an AIDocument, 99.1 percent.
• Kubernetes Pods concepts: 131,977 raw tokens, 7,475 as an AIDocument, 94.3 percent.

At a $3.00 per 1M input token model price, that Stripe page costs about 92 cents to read raw and well under a cent as an AIDocument. Multiply by every page your agent reads per task, and the metered web stops being a threat and becomes an argument for reading efficiently.

What one read looks like

One call, URL in, structured document out:

curl -s https://api.lyrenth.com/v1/aidocument \
  -H "Authorization: Bearer $LYRENTH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.stripe.com/api"}'

If the index holds the page (it holds more than 2 billion of them across 150 million domains, live counters on the stats page), the response is instant and the origin is never contacted. If it does not, the index fetches the page itself, under its own crawler policy, indexes it, and serves it; every future reader shares that copy. When your task needs the newest version, pass force_refresh and the index refreshes its copy before serving you. The full serving model is in crawl-on-miss, explained.

The checklist for agent builders

1. Audit where your fetches actually go. If your agent hits origins directly, count how much of that traffic lands on ad-monetized pages. That share is where September 15 bites first.
2. Split reading from acting. Reading a page to extract information is index work; submitting a form on a user's behalf is genuinely agentic and will always be a direct interaction. Architect the two differently instead of shipping one fetcher that does both badly.
3. Stop spending identity as a cost. Unidentified traffic is the thing the whole web is arming against. Read through infrastructure that identifies itself cryptographically and honors robots.txt, so your product is not betting its uptime on staying unnoticed.
4. Price your reads. Whatever your stack, know your cost per page read, in tokens and in dollars. The metered web punishes architectures that never measured this.

The web is not closing to machines. It is closing to anonymous, redundant, per-agent fetching, and it is staying open to identified reading that gives something back. Build on the second one. The quickstart takes about five minutes, and the free tier needs no card.