An independent measurement of AI search crawlers, and where Lyrenth sits
Known Agents measures which crawlers do the most AI search indexing across thousands of sites. The list is Huawei, Amazon, Apple, Anthropic, Meta, OpenAI, and, a few rows down, us.
Known Agents runs a bot directory and a measurement panel: their analytics script on thousands of websites, plus daily scans of the top 1,000 sites and their robots.txt files. From that they publish The Agentic Web Index, including a table of which crawlers do the most AI search indexing.
We are on it. That is worth writing down carefully, because the number is small and the company it puts us in is not.
The table
Share of observed AI search crawler activity, by operator, as published in August 2026:
| Operator | Share |
|---|---|
| Huawei | 27.1% |
| Amazon | 18.2% |
| Apple | 14.9% |
| Anthropic | 11.6% |
| Meta | 11.3% |
| OpenAI | 10.6% |
| Linkup | 3.0% |
| Perplexity | 2.1% |
| 0.3% | |
| Microsoft | 0.2% |
| xAI | 0.2% |
| Lyrenth | 0.2% |
| Exa | 0.1% |
| AddSearch | 0.0% |
| Direqt | 0.0% |
Source: knownagents.com/insights,
retrieved 25 August 2026. The crawler token measured for us is
AIWebIndex.
What this measures, and what it does not
Read the table for what it is. It measures observed crawl activity on one panel: how many requests from each AI search crawler their instrumented sites actually saw. It does not measure index size, result quality, or anything about the products behind the crawlers.
Two rows need a caveat, and it would be dishonest to skip it. Google at 0.3% and Microsoft at 0.2% are their AI-search crawlers only (Google-CloudVertexBot and AzureAI-SearchBot). Googlebot and Bingbot are different categories and are not in this table. Those companies crawl the whole web through their main crawlers and their AI-specific ones have little left to do. So this table does not say Lyrenth crawls at Google's scale. It says that among crawlers built specifically for AI search, ours does about as much observed work as theirs.
The like-for-like comparisons are the other rows: Linkup, Perplexity, xAI, Exa. Purpose-built AI search crawlers, measured the same way, on the same panel.
Why 0.2% is the number we wanted to show you
Every other operator on that list above us is a company with hundreds or thousands of people, and most of the ones near us are venture funded. Lyrenth is self-funded. The index behind that 0.2% holds over 2.6 billion documents, and the live counter is public at lyrenth.com/web-index/stats.
We did not measure any of this ourselves, which is the point of citing it. We have no relationship with the panel, no analytics presence on their sites, and no way to influence what they observed. Somebody else's instrument saw our crawler doing real work at the scale of crawlers run by the largest companies in the world, and published the result with our name spelled correctly.
If you found us in your logs
You do not have to take any of this from us. Three bot directories
maintain independent entries on AIWebIndex, and none of them was
written by us:
- Known Agents files it as an AI search crawler and reports that 7% of top websites currently block it in robots.txt.
- IPScanner tracks 150 crawler names. A name in your logs can be faked by anyone, so the only proof is when the operator publishes something checkable, and by their count only 33 of the 150 operators do. We are one of them, and IPScanner ran the check itself: it looked up all 22 of our published crawl addresses and confirmed each one really belongs to lyrenth.com. Their entry records that check, dated August 2026.
- udger classifies both of our user agents, the scheduled crawler and the on-demand fetcher, as search engine bots.
Blocking us is every owner's right, and our crawler policy includes the exact rules for doing it if that is your decision.
Before deciding, you can verify that a request claiming to be us actually is:
- Our crawl addresses are published in machine readable form at lyrenth.com/bot/ip-ranges.json, the same format OpenAI uses for GPTBot.
- Every one of those addresses has forward-confirmed reverse DNS under
lyrenth.com. - Our requests carry Web Bot Auth signatures (RFC 9421), with the public key directory at api.lyrenth.com/.well-known/http-message-signatures-directory.
Known Agents' FAQ answers "Should I block AIWebIndex?" with "Usually not." We will keep earning that answer.