See exactly which AI bots read your site (and which just say they do)

User-agent strings are easy to fake. Here is how to verify which AI bots really read your site, and how AI bot traffic analytics show it without grep.

Ask anyone who runs a site and reads their own access logs: the audience is changing. A growing share of requests come not from browsers but from AI agents, fetching a page to answer a question, ground a coding session, or fill a research loop. The first question every owner asks is which AI bots are reading my site. The sharper second question is which of those bots are who they claim to be. This post covers both: why the obvious approach (grepping for user agents) quietly lies to you, what real verification looks like, and how to see the verified picture on a dashboard instead of in a terminal.

Round one: grep the logs

Every well-behaved bot announces itself in the User-Agent header, so the obvious move is to search your logs for the names you know. Here is what that traffic looks like in a standard combined-format access log. These lines are illustrative, invented traffic against example.com, not real measurements:

198.51.100.7 - - [16/Jul/2026:09:14:07 +0000] "GET /docs/api HTTP/1.1" 200 48213 "-" "SomeAIBot/1.2 (+https://example.net/bot)"
203.0.113.42 - - [16/Jul/2026:09:14:11 +0000] "GET /pricing HTTP/1.1" 200 12904 "-" "LyrenthBot"

From there the classic pipeline is ten minutes of work:

grep -i "someaibot" /var/log/nginx/access.log \
  | awk '{print $1}' | sort | uniq -c | sort -rn

You get a tidy table of bot names and hit counts, and it feels like an answer.

Here is the flaw. A user agent is a plain request header. It is a string the client chooses to send, and nothing anywhere checks it. This one-liner produces a log entry that looks exactly like a visit from our crawler:

curl -A "LyrenthBot" https://example.com/docs/api

Anyone can type any bot's name, and there are real incentives to do so. A known, well-behaved bot name inherits that bot's reputation: if owners have allowed it in robots.txt or in firewall rules, wearing its name is the cheapest way through the door. So a UA-only tally is really the sum of at least three populations you cannot separate: the operator's genuine crawler, scrapers borrowing its name, and one-off scripts someone never bothered to relabel. It is an upper bound with an unknown amount of junk mixed in, and you cannot make allow, throttle, or block decisions from it.

The inverse failure is worth naming too. If you block by user agent, you only block the clients honest enough to keep the name. The dishonest ones switch to a browser string and keep going. UA matching is a courtesy protocol; it works on exactly the actors who never needed policing.

What separates a claimed bot from a proven one

A bot is proven when its identity rests on something a spoofer cannot simply type. Three checks do that work, and they are independent: each one catches cases the others miss.

Published IP ranges

The operator publishes the IP ranges its crawler egresses from, and you check the connecting address of each request against that list. Faking a header costs one command-line flag. Sourcing your traffic from inside someone else's published network is a different class of problem entirely.

Reverse DNS that resolves back

Take the connecting IP and do a PTR lookup. For a genuine bot, the hostname you get back sits under the operator's domain, and resolving that hostname forward returns the same IP you started with. Both halves matter. A PTR record alone can claim anything, since whoever holds the IP sets it. The forward confirmation has to come from the operator's own DNS zone, which the spoofer does not control.

Signed requests (Web Bot Auth)

The strongest check: the bot cryptographically signs its requests, and you verify the signature against the operator's published key. Without the private key there is no valid signature, no matter what the header says or where the packets came from. Signatures also survive infrastructure changes that would make an IP list stale.

LyrenthBot, the crawler that builds Lyrenth's index, does all three. It identifies itself in the user agent, crawls from published IP ranges with reverse DNS that resolves back to us, and signs its requests with Web Bot Auth. The verification details, ranges included, live at /bot. We do this deliberately, precisely so that any owner can separate our real traffic from someone wearing our name. The bot also respects robots.txt, reads only the public web, and never signs in to accounts. And to state the policy plainly: Lyrenth does not train foundation models on crawled content. We build a search index that serves pages as AIDocuments, nothing else.

One structural fact about our traffic is useful when you read your logs. Lyrenth serves every read from the index. If a page is already indexed, the caller gets the cached index copy instantly and your origin is not contacted at all. If a page is not in the index yet, it is crawled once, indexed, and served from that fresh index entry. One crawl serves every caller, so a single LyrenthBot fetch in your log can stand behind many downstream reads of that page.

Seeing it without grep

You can run all three checks by hand. It means keeping per-operator IP lists current, scripting the double-sided rDNS lookups, and implementing signature verification, then redoing pieces of it whenever an operator changes infrastructure. It is tractable, but it is a side project that decays the moment you stop tending it.

We built this into the site-owner side of Lyrenth instead. Verify your domain (a DNS TXT record or an HTML file, your choice) and your dashboard shows bot-traffic analytics through a lightweight worker integration: which AI bots visited, how often they came, and what they read, with the verification already done. The setup is documented at /docs/bot-tracking.

Verification unlocks the rest of the owner tooling at the same time: per-domain stats and an AI-readiness score from 0 to 10, computed as the mean of per-page audits across your site. The score tells you how well AI readers can actually parse what you publish, which is the natural follow-up question once you know they are visiting.

Why this matters more every month

Agent traffic is not a curiosity anymore, and every month it is a larger share of how your pages get read. We see the demand side of this directly: Lyrenth's index now covers over 1.2 billion documents across more than 127 million domains (live index stats), and the callers reading it are agents and applications, not people paging through a browser.

That growth turns bot identity from trivia into an operating decision. You will decide which bots to allow, which to rate-limit, and which to block. You will want to know whether the agents reading your pages are seeing your content correctly. You will write robots.txt rules and want confidence about who actually honors them. Every one of those decisions is only as good as your ability to tell a proven bot from a pretender, because the pretenders are, by construction, the ones your user-agent rules cannot touch.

For the fuller owner playbook, covering robots.txt policy, verification in depth, and what to do about the bots that do not verify, see our publishers' guide to AI bots and verification.

Domain verification is on the free tier: sign up (2,000 AIDocuments a month, no card, one verified domain), prove ownership with a TXT record or an HTML file, and your dashboard starts showing which AI bots really read your site. If you want the API side first, start with the quickstart.