The crawler behind the Lyrenth index.
AIWebIndex/2.0 is the crawler that builds Lyrenth, the AI-readable web index: a search index of the public web that serves every page as one clean, attributed AIDocument. It is an indexer, not a model trainer: we do not train foundation models on the content we crawl. It identifies itself honestly and is verifiable three ways on this page, honors RFC 9309 robots.txt including Crawl-delay, keeps per-domain rate limits, and fails closed when robots.txt cannot be read. Agents read the resulting AIDocuments through POST /v1/aidocument.
Verified by Cloudflare
Cloudflare lists both of our user agents, AIWebIndex and AIWebIndex-Agent, as verified bots in its Search category.
All Lyrenth bots on Cloudflare Radar
Also listed in
- Known Agents AI search crawler
- IPScanner AI search crawler
- udger Search engine bot
AIDocument is
an open format.
The index is a universal adapter for the existing web, no website changes required. Our crawler returns the AIDocument shape, which we publish openly so anyone can implement, fork, or extend it.
- ›User-Agent identifier (
AIWebIndex/2.0) - ›AIDocument JSON format for extracted content
- ›Verification (
.well-known/aiwebindex-verify.txtand DNS TXT)
Lyrenth, operated by Aleksma Ai, Inc., is the reference commercial implementation. The format itself is open: Aleksma Ai, Inc. holds no patents on the core AIDocument shape or the verification mechanics, and pledges not to seek such patents. Anyone is free to implement, fork, or extend AIDocument.
User-Agent strings
On-demand index fetches (a customer asked the index for that exact URL) identify separately so your logs can tell the two apart; both honor robots.txt the same way:
Verification probes use a slightly different UA so you can distinguish crawl fetches from ownership checks:
UA strings can be spoofed. Confirm a request is genuinely ours three ways:
- 1Source IP is on our published list at /bot/ip-ranges.json.
- 2Requests carry a Web Bot Auth signature (RFC 9421); our public key directory is at
api.lyrenth.com/.well-known/http-message-signatures-directory. - 3Every IP has forward-confirmed reverse DNS under
lyrenth.com.
robots.txt, backoff, rate limits
Our crawler implements RFC 9309 and honors Disallow rules, Crawl-delay, HTTP 429/503 backoff, and Sitemap directives. Rate: at most one fetch every 2 seconds per domain across our whole fleet, or your Crawl-delay when it is longer. The one exception is pages a verified site owner asks us to fetch again (with Recheck or the change webhook): those skip the wait and the Crawl-delay, and robots.txt still applies. Trap handling: links are read only from a spec-compliant HTML parse (script bodies are never links), known honeypot namespaces such as /.within.website/ are refused before fetch, at admission, and on redirects, and links a page hides from human readers are flagged out of crawl discovery. The simplest opt-out is your robots.txt:
Prefer the opposite? Verify your domain to see how AI reads each of your pages, and send the pages you change to the front of our crawl queue. Verify your domain →
Two methods.
Either works.
Pick whichever propagates first. Our verifier queries 1.1.1.1, 8.8.8.8, and 9.9.9.9 in parallel so misconfigured local resolvers do not block your verification.
DNS TXT record
Recommended. Usually propagates in under 60 seconds.
.well-known file
HTTPS only. Plain HTTP fetches are rejected.
Reading this with
an AI agent?
We publish machine-readable manifests so you do not have to scrape this page. Each one pivots to the same canonical AIDocument shape.
AIDocument
The full JSON envelope, every field and error code, documented.
Read the contract →