Blog.
Engineering notes on building the AI-readable web index: AIDocuments, agents that read the web, and the infrastructure behind Lyrenth.
An independent measurement of AI search crawlers, and where Lyrenth sits
Known Agents measures which crawlers do the most AI search indexing across thousands of sites. The list is Huawei, Amazon, Apple, Anthropic, Meta, OpenAI, and, a few rows down, us.
The token economics of RAG over live web data
Most RAG cost is not the model, it is what you feed it. A measured look at raw HTML versus a readable document, and where the money actually goes.
MCP or a REST API: which one should your agent use to read the web?
MCP and a plain HTTP API do the same job in different places. When to expose a capability as a tool, when to call the endpoint, and why most teams run both.
Agentic browsers vs crawlers: which one does your agent actually need?
Crawlers read the web so a question can be answered later. Agentic browsers use the web now. Picking the wrong one costs latency, tokens, and access.
How fresh is an indexed page? Staleness, recrawls, and forcing a refresh
Cached reads are fast and cheap, but pages change. How freshness works in an AI web index: documented cache ages, change-driven recrawls, and per-request refresh.
More than markdown: getting structure from web pages your agent can use
Clean text is half the job. How AIDocument's structure, headings, links, and media, lets agents navigate, cite, and act on pages instead of just reading them.
The September 15 readiness checklist for agent builders
Five checks to run on your agent's web access before Cloudflare's new crawler defaults land: inventory, identity, failure handling, freshness, and fallback.
Reading the web without getting blocked: why politeness beats tricks
Agents get blocked for behaving like strangers. What polite machine access looks like, why evasion is a treadmill, and how identified reading stays welcome.
Caching web content for AI agents: what to cache, for how long, and when to force a refresh
A practical guide to caching the web for agents: why every agent team ends up building a cache, what staleness actually costs, and what a shared index changes.
Two billion documents, and why the number matters more after September 15
Lyrenth's index crossed two billion AI-ready documents this week, on a public live counter. Here is what an index that size changes for agents as the web's defaults turn restrictive.
September 15 is coming: what actually changes for AI crawlers
Cloudflare's new defaults land on September 15. Here is what gets blocked, who is affected, what stays allowed, and what agent builders and site owners should do before the date.
One crawl serves everyone: the economics of a shared web index
Why one crawl serving every reader changes the cost of machine-read web data: real token numbers, cache math, and what shared infrastructure means for origins.
llms.txt, explained for developers
What llms.txt is, the exact file format, how it differs from robots.txt and sitemaps, who adopted it, and how to write one that AI assistants actually use.
How robots.txt applies to AI agents in 2026
What robots.txt actually governs in 2026, how AI crawler tokens work, recipes to allow search and block training, and why verification matters as much as rules.
The web is getting metered for machines: what agent builders should know before September 15
Cloudflare's new defaults block agent fetchers on ad pages and per-fetch access is getting priced. How agent builders keep reading the web through an index.
Cloudflare will block AI crawlers by default on September 15: what site owners should actually do
Cloudflare blocks Training and Agent crawlers by default on ad pages from September 15, 2026. What changes, who it affects, and a site-owner checklist.
See exactly which AI bots read your site (and which just say they do)
User-agent strings are easy to fake. Here is how to verify which AI bots really read your site, and how AI bot traffic analytics show it without grep.
Crawl-on-miss: how an index reads pages it has never seen
Crawl on miss, explained: how Lyrenth serves every page from its index, whether cached or freshly indexed, and when a re-crawl happens before serving.
LangChain document loaders for live web pages: a clean pattern
A custom LangChain web loader that reads pages as clean AIDocuments instead of raw HTML: about 30 lines of Python, with real token numbers.
Lyrenth vs building your own scraper stack: the real 12-month bill
What web scraping infrastructure cost actually looks like over 12 months: fetch workers, browser pools, engineer time, and the shared-index alternative.
Is your site readable by AI? The readiness score, explained
Run an AI readability website check with Lyrenth's readiness score: 0 to 10, the mean of per-page audits. What it measures and how to raise it.
Give your agent a web-reading tool in 30 lines
LLM tool use for web reading, end to end: one tool schema for OpenAI and Anthropic plus a 30-line Python handler that returns clean AIDocument Markdown.
We measured the token cost of famous pages
What is the token count of a web page for an LLM? We measured five famous pages: raw HTML vs AIDocument tokens, with real per-read costs.
What is an AI-readable web index?
What an AI-readable web index is, how it differs from scraping and from a classic search index, and why one crawl should serve every machine reader.
Add web reading to Claude Desktop, Claude Code, or Cursor with one MCP block
A five-minute quickstart: drop one MCP block into your assistant and it reads any URL as a clean AIDocument. Covers read_url, read_urls, and check_usage.
Agents don't browse, they read: what an index for machine readers looks like
Humans browse the web through layout and pixels; AI agents read it as text, structure, and provenance. That difference inverts how an index should be built.
How to feed web pages to an LLM without blowing the context window
Raw HTML burns your context window on nav, scripts, and boilerplate. Here is how to feed web pages to an LLM as clean AIDocuments, with real token numbers.
The publisher's guide to AI bots, robots.txt, and getting represented correctly
Who reads your site now, what a well-behaved AI-index bot looks like, how robots.txt and llms.txt apply, and how to verify your domain with Lyrenth for free.
What is an AIDocument? One clean shape for agents that read the web
An AIDocument is one stable JSON shape for any web page: Markdown body, title, description, structure, and measured economics. Here is how it works.
Why your scraper gets empty HTML from JavaScript sites (and what rendering actually takes)
Fetch a React or SPA site and you get a hollow div, not content. Here is why JavaScript sites return empty HTML, and what real rendering takes at index scale.