You probably landed here from a server log.
Lyrenth is an API: site owners verify a domain, AI agents resolve URLs through POST /v1/aidocument, and we fetch and clean each page once then serve the canonical AIDocument to every agent after that. Crawls are on-demand, never bulk.
AIDocument is an open format.
Lyrenth is a universal adapter for the existing web. No website changes required. Our crawler returns the AIDocument shape, which we publish openly so anyone can implement, fork, or extend it. The open spec covers:
- User-Agent identifier (
AIWebIndex/1.0) - AIDocument JSON format for extracted content
- Verification mechanism (
.well-known/aiwebindex-verify.txtand DNS TXT records)
Lyrenth, operated by Aleksma AI Inc., is the reference commercial implementation. The format itself is open: Aleksma AI Inc. holds no patents on the core AIDocument shape or the verification mechanics, and pledges not to seek such patents. Anyone is free to implement, fork, or extend AIDocument.
Site operators allowlisting AIWebIndexvia robots.txt or firewall rules allow any compliant crawler — not just Lyrenth. This means an open ecosystem of compatible implementations can grow over time.
The full AIDocument spec will be formalized at aiwebindex.org. Today, that domain currently redirects to www.lyrenth.com while the spec site is being built.
User-Agent strings
User-Agent: AIWebIndex/1.0 (+https://www.lyrenth.com/bot; AI-readable web index)
Verification probes use a slightly different UA so you can distinguish on-demand crawls from ownership checks:
User-Agent: AIWebIndex/1.0 verification (+https://www.lyrenth.com/bot)
Backoff and opt-out today, robots.txt soon
Today we honor HTTP 429 / 503 backoff, Sitemap directives in robots.txt, and per-domain rate caps with a 2 second cooldown. Disallow and Crawl-delay enforcement is on the short-term roadmap. The most reliable opt-out right now is firewall blocking or emailing us:
User-agent: AIWebIndex Disallow: /
Add the rule above ahead of time so the block takes effect the moment Disallow handling ships, and contact us in the meantime if you need a hard stop.
Two methods. Either works.
Pick whichever propagates first. Our verifier queries 1.1.1.1, 8.8.8.8, and 9.9.9.9 in parallel so misconfigured local resolvers don't block your verification.
DNS TXT record
_aiwebindex-verify.<your-domain> TXT "aiwi-verify=<token>"
Recommended. Usually propagates in under 60 seconds.
.well-known file
https://<your-domain>/.well-known/aiwebindex-verify.txt contents: aiwi-verify=<token>
HTTPS only. Plain HTTP fetches are rejected.
Reading this with an AI agent?
We publish machine-readable manifests so you don't have to scrape this page. All three pivot to the same canonical AIDocument shape.