September 15 is coming: what actually changes for AI crawlers
Cloudflare's new defaults land on September 15. Here is what gets blocked, who is affected, what stays allowed, and what agent builders and site owners should do before the date.
This is part one of Countdown to September 15, a short series about the biggest change to machine access to the web this year, and how to be ready for it.
On September 15, 2026, a large part of the web changes its default answer to AI crawlers from yes to no.
Cloudflare announced the change in July: crawlers that mix jobs, the ones that blend search indexing, AI training, and agent browsing into one bot, will be blocked by default from pages that carry advertising. The block applies to new Cloudflare customers, to new sites added by existing customers, and to every site on the free plan. Paying customers keep a dashboard switch to let specific crawlers back in.
Two things make this bigger than one company's policy update.
The defaults are the law of the web
Most site owners never touch bot settings. Whatever the default does, the web mostly does. When the default was "allow everything that speaks HTTP," the open web was open to every bot, polite or not. When the default becomes "block mixed-use AI crawlers on ad-supported pages," a meaningful share of the web goes dark to any bot that cannot explain itself, and the owner of that site never has to lift a finger for it to happen.
Cloudflare sits in front of roughly a fifth of the web, so its defaults are not one vendor's preference. They are infrastructure policy.
The question every bot now has to answer
The deadline forces a separation that the web has needed for years: what is this bot, and why is it here? A crawler that indexes for search is treated differently from one that harvests training data, and both are treated differently from an agent acting for one user. Search stays allowed by default. Training and mixed-use face the block. Bots that identify themselves honestly, state their purpose, and respect the site's terms keep a path in. Bots that will not, lose theirs.
That is the real story of September 15. It is not anti-AI. It is the end of anonymous machine access to other people's content, and the beginning of access as a relationship: identity, purpose, permission, and increasingly payment. Cloudflare's own Pay Per Use program, launched with named AI partners, points the same direction: publishers compensated when their content is actually used in AI answers.
If you build agents
Ask one question about every web access path in your product: when your agent fetches a page, does the site know who is asking and why? If the answer is no, some share of your reads starts failing on September 15, and that share grows every time another site inherits the new defaults. The symptoms will look like flaky scraping: more empty responses, more challenge pages, more silent blocks. The cause will be policy, not code, and no retry loop fixes policy.
The durable answer is to read the web through infrastructure that identifies itself: a named bot with a published policy, cryptographically signed requests, and published pages a site owner can check. That is how Lyrenth's reading works today, described openly on our bot page and crawler policy, and it is why reads served from an index do not turn into a new burden on the origin every time an agent asks.
If you own a site
You are about to get real control, and it is worth spending ten minutes deciding what to do with it. Our earlier guides walk through the practicalities: which AI bots read your site and what robots.txt can and cannot do in the agent era. The short version: allow the bots that identify themselves and bring you value, block the ones that will not say who they are, and know that after September 15 the second part increasingly happens for you.
The countdown
We will follow this to the deadline and past it. Next in the series: a readiness checklist for agent builders, a site-owner's guide to the new controls, what a two-billion-document index changes when the walls go up, and, on the day itself, what actually shifted.
The web is not closing. It is starting to ask who is at the door.