Reading the web without getting blocked: why politeness beats tricks

Agents get blocked for behaving like strangers. What polite machine access looks like, why evasion is a treadmill, and how identified reading stays welcome.

Sooner or later, every agent that reads the web directly gets blocked, and the team behind it faces a fork in the road. One path is the treadmill: rotate addresses, disguise the client, retry harder. The other is the one this post argues for: stop being a stranger.

Why sites block bots

From a site owner's side, unidentified bot traffic is pure risk. It consumes bandwidth without ever seeing an ad or buying anything, it might be harvesting content for someone else's product, and it does not answer when asked who it is. Site owners respond rationally: challenges, rate limits, and blocks, applied more aggressively every month as infrastructure providers make blocking the default.

The blocks are not aimed at your use case. They are aimed at your anonymity. A site cannot tell a helpful research agent from a content thief when neither introduces itself, so both get the same door.

Why evasion is a losing trade

Tricks work until they are noticed, and being noticed is guaranteed at any useful scale. Each round of the arms race costs engineering time, makes your traffic look more like an attack, and burns the one thing you cannot rotate: the site's willingness to serve machines at all. Evasion by some bots is exactly why defaults are hardening against all bots. It is a strategy that poisons its own well.

What polite access looks like

The durable alternative is boring and it works: identify yourself, state your purpose, respect the site's rules, and limit your rate. Lyrenth's reading works this way by design. Every fetch identifies itself with a named agent and a published policy page a site owner can check, requests carry a cryptographic signature so the identity cannot be spoofed, robots.txt is honored on every fetch, and per-domain rate limits keep any single site's load negligible.

The deeper advantage is structural. Because reads are served from a shared index, the popular page a thousand agents want was fetched once, politely, instead of a thousand times, rudely. The origin sees less traffic, not more, as usage grows. That is what makes the politeness sustainable rather than performative.

The practical takeaway

If your agent reads the web through your own fetching code, you own the identity problem: name your bot, publish a policy, respect robots, and accept that some doors stay closed. Or read through infrastructure that already has a standing, verifiable identity and takes the whole problem off your plate. Either way, the strangers get blocked first, and after September 15, by default.

A free key takes a minute: 2,000 reads a month, no card.