LinkedIn
Our category archive logged 3,318 apparent AI-bot hits in 30 days and earned zero citations in the live ChatGPT and Claude answers we checked. One dated post drew 232 verified live-answer fetches over the same period and appeared repeatedly as a primary citation. The busy page was not the useful one.
I would have made the wrong call from the hit counter alone. Thousands of requests bearing OpenAI and Anthropic User-Agent strings looked like an AI search audience finding its way through our blog. The post that actually fed answers had a fraction of that traffic. Reconstructing the gap meant going back through the requests in order: what we counted, what we could verify, what the bots fetched, and what the pages gave them to use.
This is the postmortem of that mistake. A visit is not a citation, and a bot name in a log line is not proof of a visit.
The setup: a category archive that looked popular with bots
The first view was encouraging. Across a 30-day window, one category archive route collected 3,318 requests stamped with User-Agent strings claiming to be OpenAI or Anthropic bots. If you stopped at that report, you could tell a tidy story: AI crawlers were working through the blog, finding our articles and getting ready to cite them.
That story had an obvious appeal because ordinary analytics do not give you a clean view of bot activity. GA4 strips much of it out, so a team that turns to server and edge access logs can feel as if it has finally found the missing audience. A few thousand GET requests make a better slide than an empty citation report.
The slide would have been wrong. We cross-referenced retrieval logs against live prompt responses for the same window. That category URL produced zero citations in the answers we checked. ChatGPT and Claude did not quote a paragraph from it, pull a metric from it or surface its link. Meanwhile, /post/google-ads-updates-2026 recorded 232 verified fetches associated with live answers and appeared repeatedly as a primary citation.
That was the first sign that the counter was measuring the wrong thing. But before judging the archive’s content, we needed to know who had requested it. The request headers were making claims; the network records had to back them up.
The first sign: 3,318 requests became 249 verified hits
It took about 48 hours in our raw Nginx and Cloudflare logs to puncture the original number. Of the 3,318 requests against the archive, 249 came from verified AI crawler IP ranges. The other 3,069 carried relevant-looking User-Agent strings but did not pass that verification. Calling all 3,318 OpenAI and Anthropic visits had turned an unverified label into a fact.
A User-Agent is just text in a request header. Someone running a short Python script can make a request claim to be ChatGPT-User/1.0. That matters when teams count every matching string or allowlist traffic on the strength of the string alone. As the crawler-identification guide explains, a standard reverse-DNS check is not the way to validate these OpenAI and Anthropic requests. For our count, we matched client IPs against the published crawler IP ranges instead.
I would not put the remaining 3,069 requests in a report as verified AI activity. Nor would I pretend the log alone proves who sent every one of them. The operational distinction is simpler: if the IP does not verify, the User-Agent does not earn the bot’s name in your totals.
Request patterns can help explain what you are seeing after that first check. Analysis of spoofed crawler traffic describes broad, sustained sweeps that look different from sparse, targeted user-initiated fetches. But a pattern is not a substitute for verification. Our first error was counting a claimed identity as a confirmed one, then reading commercial meaning into the inflated count.
The correction still left 249 verified archive hits. That made the next question harder, not easier: if genuine crawlers had reached the page, why had it supplied no citation in the answers we checked?
The wrong call: treating an archive visit as content interest
The archive had a job. It listed posts and helped visitors move through the blog. We let that useful website job stand in for a different one: answering a specific question in a live conversation. A route that helps someone find an article is not necessarily a route an answer engine can cite.
The distinction gets lost because category pages do receive crawler traffic. Log-file analysis across commercial sites reports that a substantial share of AI crawler requests can land on paginated archives and tag indexes. A fetch on one of those routes tells you the crawler requested it. It does not tell you that the page contains a passage worth quoting.
Our archive was made of fragments: short teasers, headlines, thumbnails, bylines and pagination links. A person could scan those cards, choose a promising post and click through. Asked for a concrete answer about a bidding update, though, a retrieval system would have to assemble one from disconnected previews. The archive offered pointers to information, not much information of its own.
The dated post presented the opposite shape. /post/google-ads-updates-2026 opened with a factual summary table and named policy dates instead of making the reader work through an introduction first. It supplied passages that could stand on their own when pulled into an answer. Work on getting cited by AI search engines discusses the value of clear answers, dates and structured page information. Our logs showed the contrast on these two URLs; they did not prove that any one markup field caused the post’s 232 fetches.
That distinction matters. I can say the post’s structure gave a retrieval system something more usable than a grid of teasers. I cannot take two pages and turn them into a universal percentage lift for Article schema. We make the same distinction in our discussion of technical AI-visibility myths: catalog architecture can help people navigate without doing the work of an answer page.
We had mistaken the archive’s place in the site structure for evidence that it could answer a prompt. The 249 verified hits showed that it was reachable. The zero citations showed that reachability was not enough.
The damage: more busy routes, fewer useful signals
Once we looked past the archive, the logs offered other tempting counters. There were verified requests to /robots.txt and the domain root. It would have been easy to file those beside the archive hits as evidence that conversational engines were studying our brand and content. They were requests, certainly. They were not citations, and a request for crawl instructions is not an endorsement of the homepage.
Some of that traffic belongs to the mechanics around a fetch. EdgeComet’s live testing describes root and robots.txt requests around content retrieval and shows why the raw HTML delivered to a bot matters. The practical reading of our logs was narrower than the optimistic one: count those requests as checks on access and delivery, not as proof that a page supplied an answer.
The inverse mistake can hide a real delivery problem. An origin log only records requests that reach the origin. If a CDN blocks or challenges a retrieval bot at the edge, a clean Nginx log does not clear the site. A local curl returning 200 OK does not prove that every bot receives the same response, either. Cloudflare settings such as its AI-scraper controls and bot-fighting features are worth checking when investigating an access gap. A bot served a 403 Forbidden response or an interactive challenge cannot use the article behind it.
That was a diagnostic risk exposed by the investigation, not an explanation we could assign to every missing citation. In this case, we already had verified hits on the archive and verified live-answer fetches on the dated post. The evidence pointed first to a counting error and then to a content-structure gap. Checking the edge still matters because a blocked page and an unhelpful page can look similarly absent from an answer report. The fixes are different.
The larger damage was to our decisions. Raw hit totals encouraged us to give credit to routes that had not appeared in answers, while the useful post’s smaller verified count could have looked unimpressive. Once the counter becomes the goal, you can spend weeks improving the wrong page. I would rather see 232 fetches tied to live answers than 3,318 requests I cannot turn into one citation.
The fix: follow the question back to a fetchable answer
We reviewed customer conversation logs and organic search queries alongside the bot requests. That exposed prompt themes around our category for which the site had high-level marketing copy or brief mentions buried in broader guides, rather than a page built to answer the question directly. Buyers were asking about matters such as fee structures, operational guardrails and technical diagnostics. A generic product page was not a reliable substitute for a clear answer to one of those questions.
The absence of a fetch does not tell us a bot visited a page, rejected its prose and moved on. We cannot see that decision in an empty log. It tells us where our evidence ends. Combined with the gaps we found in our own pages, though, it gave us a useful editorial test: can we point to a fetchable URL that answers this buyer question without making the reader assemble the answer from several pages? If not, adding another archive teaser will not solve the problem.
That test also changed what we wanted from AEO reporting. A dashboard showing a competitor’s name in green and ours in red might reveal an outcome, but it would not explain whether a relevant page existed, whether the bot could retrieve its raw HTML or whether the answer was buried. We needed to keep prompt intent, verified requests and page content in the same view. Otherwise, an access failure could be mistaken for a writing failure, and an unwritten answer could be mistaken for a crawler problem.
The order of the repair followed the order of the evidence. First verify delivery, then judge the page. We check the edge response and the raw HTML a retrieval bot can receive before rewriting copy. Our 45-minute curl and CDN audit walks through the HTTP responses, User-Agent checks and firewall rules involved. A rendered page can look fine in a browser while the response available to a bot is blocked, challenged or missing the content that appears after client-side JavaScript runs.
When delivery is sound, the content work is not to make the archive longer. It is to give an important question a self-contained place to land: a page with the direct answer near the top, the relevant dates or constraints made explicit, and enough context to make the passage useful outside the page’s navigation. The dated post showed what that looked like in our logs. The archive showed what happens when we ask a collection of links to do the same job.
I would still keep the archive. People need paths through a site, and a category page can provide one. I would stop reporting its request count as if it were a measure of answer visibility. Those are different functions, and the 30-day comparison made the difference difficult to ignore.
The rule I now apply is never confuse crawl volume with retrieval eligibility. Verify who made the request, check what the bot could receive, then ask whether the page itself answers the prompt. Our archive passed only part of that test. The dated post gave the answer engine something it could use.