◇ AIWebIndex/2.0Bot identification

You probably landed here from a server log.

Lyrenth is a structured index of the public web. Our background crawler discovers and fetches public pages at polite, rate-limited speeds to build the index, and agents also resolve specific URLs on demand through POST /v1/aidocument. Each page is cleaned into one canonical AIDocument and kept fresh, so repeat reads are fast. Our crawler honors robots.txt and per-domain rate limits.

About the open shape

AIDocument is
an open format.

Lyrenth is a universal adapter for the existing web, no website changes required. Our crawler returns the AIDocument shape, which we publish openly so anyone can implement, fork, or extend it.

›User-Agent identifier (AIWebIndex/2.0)
›AIDocument JSON format for extracted content
›Verification (.well-known/aiwebindex-verify.txt and DNS TXT)

Lyrenth, operated by Aleksma AI Inc., is the reference commercial implementation. The format itself is open: Aleksma AI Inc. holds no patents on the core AIDocument shape or the verification mechanics, and pledges not to seek such patents. Anyone is free to implement, fork, or extend AIDocument.

How we identify

User-Agent strings

AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)

User-initiated API fetches (a customer submitted that exact URL) identify separately so your logs can tell the two apart:

AIWebIndex-Agent/2.0 (+https://lyrenth.com/bot; user-initiated fetch)

Verification probes use a slightly different UA so you can distinguish crawl fetches from ownership checks:

AIWebIndex/2.0 verification (+https://lyrenth.com/bot)

UA strings can be spoofed; verify the source IP against /bot/ip-ranges.json.

Rate limits + opt-out

robots.txt, backoff, rate limits

Our crawler implements RFC 9309 and honors Disallow rules, Crawl-delay, HTTP 429/503 backoff, Sitemap directives, and per-domain rate caps with a 2-second cooldown floor. The simplest opt-out is your robots.txt:

User-agent: AIWebIndex Disallow: /

Prefer the opposite? Verify your domain and you author the canonical AIDocument agents receive. Verify your domain →

Verification

Two methods.
Either works.

Pick whichever propagates first. Our verifier queries 1.1.1.1, 8.8.8.8, and 9.9.9.9 in parallel so misconfigured local resolvers do not block your verification.

Method 1

DNS TXT record

_aiwebindex-verify.<your-domain> TXT "aiwi-verify=<token>"

Recommended. Usually propagates in under 60 seconds.

Method 2

.well-known file

https://<your-domain>/.well-known/aiwebindex-verify.txt contents: aiwi-verify=<token>

HTTPS only. Plain HTTP fetches are rejected.

Verify your domain →Already verified? Go to your sites

For agents

Reading this with
an AI agent?

We publish machine-readable manifests so you do not have to scrape this page. Each one pivots to the same canonical AIDocument shape.

Site summary

llms.txt

A compact, machine-readable map of the site for agents.

Open llms.txt →

Full corpus map

llms-full.txt

The expanded manifest with the full page index.

Open llms-full.txt →

The contract

AIDocument

The full JSON envelope, every field and error code, documented.

Read the contract →