Lyrenth.BETA
Crawler policy

Crawler policy.

Last updated: May 9, 2026

Who we are

Lyrenth is operated by Aleksma AI Inc., a Delaware corporation.

Infrastructure operations are primarily hosted from Hetzner infrastructure in Frankfurt and Falkenstein, Germany.

Contact: [email protected]

Aleksma AI Inc.
1111B S Governors Ave # 97667
Dover, DE 19904
USA

What we do

Lyrenth fetches publicly accessible web pages on behalf of AI systems and developers. Our crawler:

  • parses structured data,
  • resolves pages into the canonical AIDocument format,
  • and caches AIDocuments to minimize unnecessary repeated requests.

If 1,000 AI agents request the same URL, our goal is to reduce that to a minimal number of origin fetches whenever technically possible.

What we respect today

Our crawler currently honors:

  • HTTP 429 backoff responses
  • HTTP 503 temporary unavailability responses
  • Sitemap: directives inside robots.txt
  • per-domain rate caps with a 2 second cooldown between requests

On our short-term roadmap and not yet enforced in code:

  • Disallow rules inside robots.txt
  • Crawl-delay directives

Until those land, the most reliable opt-out is to email [email protected] or block our IP range at the firewall. We will not silently downgrade or ignore user-agent rules once enforcement ships.

We do not attempt to bypass:

  • paywalls,
  • login requirements,
  • authentication systems,
  • or technical access controls.
User-Agent

Our crawler identifies itself as:

AIWebIndex/1.0 (+https://www.lyrenth.com/bot)
Data we extract

We may extract:

  • public HTML
  • visible page content
  • OpenGraph metadata
  • JSON-LD structured data
  • schema.org metadata
  • publicly available semantic markup

We do not intentionally extract:

  • passwords
  • authenticated content
  • private user data
  • content behind login walls
Opt-out options

Website operators may opt out through several methods.

robots.txt
User-agent: AIWebIndex
Disallow: /
Firewall blocking

Requests originate from Hetzner infrastructure located in Frankfurt and Falkenstein, Germany.

Email request

Domain owners may request domain-wide opt-out by contacting [email protected].

Verified owner controls

Verified site owners may manage per-path exclusions through future dashboard controls where available.

Abuse reporting

If you believe our crawler:

  • is ignoring robots.txt,
  • causing excessive traffic,
  • hammering your infrastructure,
  • or behaving unexpectedly,

please contact [email protected]. We actively monitor abuse reports and aim to respond within 24 hours.

Operational reference
User-agent details
AIWebIndex/1.0 (+https://www.lyrenth.com/bot)
robots.txt allow example
User-agent: AIWebIndex
Allow: /

User-agent: *
Disallow:
robots.txt full block example
User-agent: AIWebIndex
Disallow: /
Verification methods

Verified domain owners may confirm ownership through:

  • DNS TXT verification
  • meta tag verification
  • future dashboard verification workflows
Agent manifest

Future crawler manifests and operational specifications may be published at:

https://www.lyrenth.com/bot
Contact
Back to homeBot identificationPrivacyTerms