The September 15 readiness checklist for agent builders
Five checks to run on your agent's web access before Cloudflare's new crawler defaults land: inventory, identity, failure handling, freshness, and fallback.
Part of Countdown to September 15. Part one covered what actually changes; this is the practical follow-up for people who ship agents.
On September 15, a growing share of ad-supported sites starts refusing unidentified AI crawlers by default. If your product reads the web, spend thirty minutes on these five checks before the date does it for you.
1. Inventory every place your product touches the web
Most products have more fetch paths than anyone remembers: the main reading pipeline, a link-preview feature, an enrichment job, a health check, that one script from the demo. List them. The paths you forget are the ones that will fail silently, and a block on a minor path looks like a flaky feature rather than what it is, a policy decision at the origin.
2. Ask the identity question about each path
For each path: when this code fetches a page, does the site know who is asking and why? A generic client with no name and no contact page is a stranger, and strangers are what the new defaults refuse. Paths that cannot answer the question need either a real bot identity of your own, with a published policy and verifiable requests, or a switch to reading through infrastructure that already has one.
3. Make blocks visible instead of mysterious
A blocked fetch rarely says so. It shows up as an empty page, a challenge screen your parser cannot read, or a timeout. Whatever your agent does on those, make sure it is not "retry forever" and not "pretend the page was empty." Log the failure class distinctly, surface it, and decide the product behavior deliberately. The teams that struggle after policy changes are the ones whose failures all look identical.
4. Separate must-be-fresh from can-be-cached
When live fetching gets harder, the question "does this read need this minute's version?" starts having a price. Most agent reads are of stable content where a cached copy is indistinguishable from a live one, and a shared index serves those without asking the origin at all. Reserve live fetching for the reads that genuinely need it, and your exposure to the new defaults shrinks to a fraction of your traffic.
5. Decide your fallback before you need it
If a source your product depends on goes dark to your fetcher on the 15th, what happens? The wrong time to answer is the morning it happens. The answers that work: an indexed copy that predates the block, an identified fetcher the site still admits, or a graceful product message. The answer that does not: hoping.
The short version
Know your fetch paths, be identifiable on all of them, see your failures, cache what can be cached, and have a fallback. Builders who do this will notice September 15 as a news story. The rest will notice it as a pager alert.
Reading through Lyrenth covers the identity, caching, and fallback checks in one move: 2,000 reads a month free, no card.