---
title: "3,318 Archive Hits, Zero AI Citations: A Postmortem From Our Edge Logs"
description: "Our category archive logged 3,318 apparent AI-bot hits and no citations. One dated post drew 232 verified live-answer fetches. Here’s what the gap exposed."
url: "https://groas.com/post/postmortem-3-318-bot-fetches-zero-ai-ans"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/1c8afb9a-309d-459b-8636-c734ee684b95.png"
published: "2026-10-10T05:20:22.834Z"
modified: "2026-10-10T05:20:22.897Z"
---

October 10, 2026 · 9 min read

# 3,318 Archive Hits, Zero AI Citations: A Postmortem From Our Edge Logs

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Tiny model crowd in paper robot masks mobs a giant signpost, while a few real figures leave a small lit cottage carrying single pages: busy isn't useful.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/1c8afb9a-309d-459b-8636-c734ee684b95.png)

In this article

1. [The setup: a category archive that looked popular with bots](#the-setup-a-category-archive-that-looked-popular-with-bots)
2. [The first sign: 3,318 requests became 249 verified hits](#the-first-sign-3318-requests-became-249-verified-hits)
3. [The wrong call: treating an archive visit as content interest](#the-wrong-call-treating-an-archive-visit-as-content-interest)
4. [The damage: more busy routes, fewer useful signals](#the-damage-more-busy-routes-fewer-useful-signals)
5. [The fix: follow the question back to a fetchable answer](#the-fix-follow-the-question-back-to-a-fetchable-answer)

Our category archive logged **3,318 apparent AI-bot hits in 30 days** and earned zero citations in the live ChatGPT and Claude answers we checked. One dated post drew **232 verified live-answer fetches** over the same period and appeared repeatedly as a primary citation. The busy page was not the useful one.

I would have made the wrong call from the hit counter alone. Thousands of requests bearing OpenAI and Anthropic User-Agent strings looked like an AI search audience finding its way through our blog. The post that actually fed answers had a fraction of that traffic. Reconstructing the gap meant going back through the requests in order: what we counted, what we could verify, what the bots fetched, and what the pages gave them to use.

This is the postmortem of that mistake. **A visit is not a citation, and a bot name in a log line is not proof of a visit.**

## The setup: a category archive that looked popular with bots

The first view was encouraging. Across a 30-day window, one category archive route collected 3,318 requests stamped with User-Agent strings claiming to be OpenAI or Anthropic bots. If you stopped at that report, you could tell a tidy story: AI crawlers were working through the blog, finding our articles and getting ready to cite them.

That story had an obvious appeal because ordinary analytics do not give you a clean view of bot activity. GA4 strips much of it out, so a team that turns to [server and edge access logs](https://usegeon.com/blog/en/reading-ai-crawlers-in-your-server-logs-a-practical-field-guide) can feel as if it has finally found the missing audience. A few thousand GET requests make a better slide than an empty citation report.

The slide would have been wrong. We cross-referenced retrieval logs against live prompt responses for the same window. That category URL produced **zero citations** in the answers we checked. ChatGPT and Claude did not quote a paragraph from it, pull a metric from it or surface its link. Meanwhile, `/post/google-ads-updates-2026` recorded 232 verified fetches associated with live answers and appeared repeatedly as a primary citation.

That was the first sign that the counter was measuring the wrong thing. But before judging the archive’s content, we needed to know who had requested it. The request headers were making claims; the network records had to back them up.

## The first sign: 3,318 requests became 249 verified hits

It took about 48 hours in our raw Nginx and Cloudflare logs to puncture the original number. Of the 3,318 requests against the archive, **249 came from verified AI crawler IP ranges**. The other 3,069 carried relevant-looking User-Agent strings but did not pass that verification. Calling all 3,318 OpenAI and Anthropic visits had turned an unverified label into a fact.

A User-Agent is just text in a request header. Someone running a short Python script can make a request claim to be `ChatGPT-User/1.0`. That matters when teams count every matching string or allowlist traffic on the strength of the string alone. As the [crawler-identification guide](https://geoiphub.com/blog/gptbot-vs-oai-searchbot-vs-chatgpt-user) explains, a standard reverse-DNS check is not the way to validate these OpenAI and Anthropic requests. For our count, we matched client IPs against the published crawler IP ranges instead.

![Illustration of a bot label peeling back to reveal an unverified request.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/ae6f6fe1-761f-4e00-8d43-0c4fc612999f.png)

I would not put the remaining 3,069 requests in a report as _verified_ AI activity. Nor would I pretend the log alone proves who sent every one of them. The operational distinction is simpler: **if the IP does not verify, the User-Agent does not earn the bot’s name in your totals.**

Request patterns can help explain what you are seeing after that first check. [Analysis of spoofed crawler traffic](https://datadome.co/bots/claude-user/) describes broad, sustained sweeps that look different from sparse, targeted user-initiated fetches. But a pattern is not a substitute for verification. Our first error was counting a claimed identity as a confirmed one, then reading commercial meaning into the inflated count.

The correction still left 249 verified archive hits. That made the next question harder, not easier: if genuine crawlers had reached the page, why had it supplied no citation in the answers we checked?

## The wrong call: treating an archive visit as content interest

The archive had a job. It listed posts and helped visitors move through the blog. We let that useful website job stand in for a different one: answering a specific question in a live conversation. **A route that helps someone find an article is not necessarily a route an answer engine can cite.**

The distinction gets lost because category pages do receive crawler traffic. [Log-file analysis across commercial sites](https://lemniscategrowth.com/blogs/ai-crawler-log-file-analysis.html) reports that a substantial share of AI crawler requests can land on paginated archives and tag indexes. A fetch on one of those routes tells you the crawler requested it. It does not tell you that the page contains a passage worth quoting.

Our archive was made of fragments: short teasers, headlines, thumbnails, bylines and pagination links. A person could scan those cards, choose a promising post and click through. Asked for a concrete answer about a bidding update, though, a retrieval system would have to assemble one from disconnected previews. The archive offered pointers to information, not much information of its own.

![Diagram contrasting an archive card with a self-contained passage from an article.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/cf57bd97-d411-42c7-b8eb-057851d2323d.png)

The dated post presented the opposite shape. `/post/google-ads-updates-2026` opened with a factual summary table and named policy dates instead of making the reader work through an introduction first. It supplied passages that could stand on their own when pulled into an answer. Work on [getting cited by AI search engines](https://www.frase.io/blog/how-to-get-cited-by-ai-search-engines-the-complete-geo-playbook) discusses the value of clear answers, dates and structured page information. Our logs showed the contrast on these two URLs; they did not prove that any one markup field caused the post’s 232 fetches.

That distinction matters. I can say the post’s structure gave a retrieval system something more usable than a grid of teasers. I cannot take two pages and turn them into a universal percentage lift for `Article` schema. We make the same distinction in our discussion of [technical AI-visibility myths](https://groas.com/post/7-technical-ai-visibility-myths-i-see-bu): catalog architecture can help people navigate without doing the work of an answer page.

We had mistaken the archive’s place in the site structure for evidence that it could answer a prompt. The 249 verified hits showed that it was reachable. **The zero citations showed that reachability was not enough.**

## The damage: more busy routes, fewer useful signals

Once we looked past the archive, the logs offered other tempting counters. There were verified requests to `/robots.txt` and the domain root. It would have been easy to file those beside the archive hits as evidence that conversational engines were studying our brand and content. They were requests, certainly. They were not citations, and a request for crawl instructions is not an endorsement of the homepage.

Some of that traffic belongs to the mechanics around a fetch. [EdgeComet’s live testing](https://edgecomet.com/blog/openseotest-how-gptbot-and-chatgpt-user-handle-javascript/) describes root and `robots.txt` requests around content retrieval and shows why the raw HTML delivered to a bot matters. The practical reading of our logs was narrower than the optimistic one: count those requests as checks on access and delivery, not as proof that a page supplied an answer.

![Network engineer examining edge-log errors beside a server rack.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/56a3e82f-2d8b-4678-91f6-71f52aa34002.png)

The inverse mistake can hide a real delivery problem. An origin log only records requests that reach the origin. If a CDN blocks or challenges a retrieval bot at the edge, a clean Nginx log does not clear the site. A local `curl` returning `200 OK` does not prove that every bot receives the same response, either. Cloudflare settings such as its [AI-scraper controls and bot-fighting features](https://www.reddit.com/r/CloudFlare/comments/1tkjv1x/heads_up_the_block_ai_scrapers_toggle_is/) are worth checking when investigating an access gap. A bot served a `403 Forbidden` response or an interactive challenge cannot use the article behind it.

That was a diagnostic risk exposed by the investigation, not an explanation we could assign to every missing citation. In this case, we already had verified hits on the archive and verified live-answer fetches on the dated post. The evidence pointed first to a counting error and then to a content-structure gap. Checking the edge still matters because a blocked page and an unhelpful page can look similarly absent from an answer report. The fixes are different.

The larger damage was to our decisions. Raw hit totals encouraged us to give credit to routes that had not appeared in answers, while the useful post’s smaller verified count could have looked unimpressive. Once the counter becomes the goal, you can spend weeks improving the wrong page. I would rather see 232 fetches tied to live answers than 3,318 requests I cannot turn into one citation.

## The fix: follow the question back to a fetchable answer

We reviewed customer conversation logs and organic search queries alongside the bot requests. That exposed prompt themes around our category for which the site had high-level marketing copy or brief mentions buried in broader guides, rather than a page built to answer the question directly. Buyers were asking about matters such as fee structures, operational guardrails and technical diagnostics. A generic product page was not a reliable substitute for a clear answer to one of those questions.

The absence of a fetch does not tell us a bot visited a page, rejected its prose and moved on. We cannot see that decision in an empty log. It tells us where our evidence ends. Combined with the gaps we found in our own pages, though, it gave us a useful editorial test: **can we point to a fetchable URL that answers this buyer question without making the reader assemble the answer from several pages?** If not, adding another archive teaser will not solve the problem.

That test also changed what we wanted from AEO reporting. A dashboard showing a competitor’s name in green and ours in red might reveal an outcome, but it would not explain whether a relevant page existed, whether the bot could retrieve its raw HTML or whether the answer was buried. We needed to keep prompt intent, verified requests and page content in the same view. Otherwise, an access failure could be mistaken for a writing failure, and an unwritten answer could be mistaken for a crawler problem.

The order of the repair followed the order of the evidence. **First verify delivery, then judge the page.** We check the edge response and the raw HTML a retrieval bot can receive before rewriting copy. Our [45-minute curl and CDN audit](https://groas.com/post/can-chatgpt-actually-read-your-site-a-45) walks through the HTTP responses, User-Agent checks and firewall rules involved. A rendered page can look fine in a browser while the response available to a bot is blocked, challenged or missing the content that appears after client-side JavaScript runs.

When delivery is sound, the content work is not to make the archive longer. It is to give an important question a self-contained place to land: a page with the direct answer near the top, the relevant dates or constraints made explicit, and enough context to make the passage useful outside the page’s navigation. The dated post showed what that looked like in our logs. The archive showed what happens when we ask a collection of links to do the same job.

I would still keep the archive. People need paths through a site, and a category page can provide one. I would stop reporting its request count as if it were a measure of answer visibility. Those are different functions, and the 30-day comparison made the difference difficult to ignore.

The rule I now apply is **never confuse crawl volume with retrieval eligibility**. Verify who made the request, check what the bot could receive, then ask whether the page itself answers the prompt. Our archive passed only part of that test. The dated post gave the answer engine something it could use.

## Frequently Asked Questions

### Delle 3.318 richieste AI-bot registrate sull'archivio, quante provenivano da crawler verificati?

Solo 249 delle 3.318 richieste provenivano da gamme IP verificate di crawler AI. Le altre 3.069 riportavano User-Agent simili a quelli di OpenAI o Anthropic, ma non superavano la verifica IP, quindi non dovrebbero essere conteggiate come attività AI confermata nei report.

### Come posso verificare che una richiesta provenga davvero da un crawler OpenAI o Anthropic?

Abbinando gli IP client alle gamme IP dei crawler pubblicate, anziché fidarsi della stringa User-Agent. Una User-Agent è solo testo nell'intestazione della richiesta: uno script breve può far sì che una richiesta dichiari di essere ChatGPT-User/1.0. Se l'IP non verifica, la User-Agent non guadagna il nome del bot nei vostri totali.

### Perché la mia pagina di categoria riceve visite da crawler AI ma non viene mai citata nelle risposte ChatGPT o Claude?

Perché essere raggiungibile non equivale ad essere citabile. La pagina di archivio riceveva crawler AI verificati ma produceva zero citazioni, mentre un articolo datato con meno traffico veniva citato ripetutamente. Un archivio è composto da frammenti come teaser, titoli e miniature, che offrono puntatori alle informazioni ma poco contenuto autocontenuto che un sistema di recupero possa citare.

### Le richieste dei bot AI al mio robots.txt o alla radice del dominio indicano interesse per i miei contenuti?

No. Quelle richieste appartengono alla meccanica attorno a un recupero: controlli di istruzioni di crawl e verifica di consegna. Un crawler che controlla robots.txt non sta approvando la homepage, quindi non dovrebbero essere presentate come prove di studio del marchio o dei contenuti.

### Il mio log Nginx sembra pulito, quindi i bot AI possono davvero leggere il mio sito?

Non necessariamente. Un log dell'origine registra solo le richieste che raggiungono l'origine: se una CDN blocca o sfida un bot di recupero all'edge, il log non mostrerà nulla di anomalo. Un curl locale che restituisce 200 OK non dimostra che ogni bot riceva la stessa risposta. Vale la pena controllare le impostazioni di Cloudflare come i controlli per gli AI-scraper e le funzionalità anti-bot.

### Se una pagina non viene citata nei risultati AI, dovrei riscrivere prima i contenuti o controllare la consegna?

Prima verificare la consegna, poi giudicare la pagina. Controllate la risposta dell'edge e l'HTML grezzo che un bot di recupero può ricevere prima di riscrivere la copia, perché una pagina renderizzata può sembrare corretta in un browser mentre la risposta per un bot è bloccata, sfidata o priva di contenuto che appare dopo il JavaScript lato client.

### Che tipo di pagina contenuti può effettivamente essere citato come risposta AI?

Una pagina che risponde a una domanda specifica senza richiedere al lettore di assemblare la risposta da più pagine: la risposta diretta vicino alla cima, date o vincoli rilevanti esplicitati, e abbastanza contesto perché il passaggio sia utile al di fuori della navigazione. La nostra pagina di archivio non poteva offrire questo; il nostro articolo datato sì.

### Dovrei eliminare le pagine di categoria se non generano citazioni AI?

No, vale la pena mantenere l'archivio perché le persone hanno bisogno di percorsi attraverso un sito. La correzione è smettere di riportare il conteggio delle richieste dell'archivio come misura della visibilità delle risposte: quelle sono funzioni diverse.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/postmortem-3-318-bot-fetches-zero-ai-ans#webpage","url":"https://groas.com/post/postmortem-3-318-bot-fetches-zero-ai-ans","name":"3,318 Archive Hits, Zero AI Citations: A Postmortem From Our Edge Logs","description":"Our category archive logged 3,318 apparent AI-bot hits and no citations. One dated post drew 232 verified live-answer fetches. Here’s what the gap exposed.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/1c8afb9a-309d-459b-8636-c734ee684b95.png"},"breadcrumb":{"@id":"https://groas.com/post/postmortem-3-318-bot-fetches-zero-ai-ans#breadcrumb"},"dateModified":"2026-10-10T05:20:22.897Z","mainEntity":{"@id":"https://groas.com/post/postmortem-3-318-bot-fetches-zero-ai-ans#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/postmortem-3-318-bot-fetches-zero-ai-ans#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"3,318 Archive Hits, Zero AI Citations: A Postmortem From Our Edge Logs","item":"https://groas.com/post/postmortem-3-318-bot-fetches-zero-ai-ans"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/postmortem-3-318-bot-fetches-zero-ai-ans#article","url":"https://groas.com/post/postmortem-3-318-bot-fetches-zero-ai-ans","mainEntityOfPage":{"@id":"https://groas.com/post/postmortem-3-318-bot-fetches-zero-ai-ans#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"3,318 Archive Hits, Zero AI Citations: A Postmortem From Our Edge Logs","description":"Our category archive logged 3,318 apparent AI-bot hits and no citations. One dated post drew 232 verified live-answer fetches. Here’s what the gap exposed.","datePublished":"2026-10-10T05:20:22.834Z","dateModified":"2026-10-10T05:20:22.897Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/1c8afb9a-309d-459b-8636-c734ee684b95.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"archive, requests, 318, hits, page, verified, citations, logs","wordCount":1940,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/postmortem-3-318-bot-fetches-zero-ai-ans#faq","mainEntity":[{"@type":"Question","name":"Delle 3.318 richieste AI-bot registrate sull'archivio, quante provenivano da crawler verificati?","acceptedAnswer":{"@type":"Answer","text":"Solo 249 delle 3.318 richieste provenivano da gamme IP verificate di crawler AI. Le altre 3.069 riportavano User-Agent simili a quelli di OpenAI o Anthropic, ma non superavano la verifica IP, quindi non dovrebbero essere conteggiate come attività AI confermata nei report."}},{"@type":"Question","name":"Come posso verificare che una richiesta provenga davvero da un crawler OpenAI o Anthropic?","acceptedAnswer":{"@type":"Answer","text":"Abbinando gli IP client alle gamme IP dei crawler pubblicate, anziché fidarsi della stringa User-Agent. Una User-Agent è solo testo nell'intestazione della richiesta: uno script breve può far sì che una richiesta dichiari di essere ChatGPT-User/1.0. Se l'IP non verifica, la User-Agent non guadagna il nome del bot nei vostri totali."}},{"@type":"Question","name":"Perché la mia pagina di categoria riceve visite da crawler AI ma non viene mai citata nelle risposte ChatGPT o Claude?","acceptedAnswer":{"@type":"Answer","text":"Perché essere raggiungibile non equivale ad essere citabile. La pagina di archivio riceveva crawler AI verificati ma produceva zero citazioni, mentre un articolo datato con meno traffico veniva citato ripetutamente. Un archivio è composto da frammenti come teaser, titoli e miniature, che offrono puntatori alle informazioni ma poco contenuto autocontenuto che un sistema di recupero possa citare."}},{"@type":"Question","name":"Le richieste dei bot AI al mio robots.txt o alla radice del dominio indicano interesse per i miei contenuti?","acceptedAnswer":{"@type":"Answer","text":"No. Quelle richieste appartengono alla meccanica attorno a un recupero: controlli di istruzioni di crawl e verifica di consegna. Un crawler che controlla robots.txt non sta approvando la homepage, quindi non dovrebbero essere presentate come prove di studio del marchio o dei contenuti."}},{"@type":"Question","name":"Il mio log Nginx sembra pulito, quindi i bot AI possono davvero leggere il mio sito?","acceptedAnswer":{"@type":"Answer","text":"Non necessariamente. Un log dell'origine registra solo le richieste che raggiungono l'origine: se una CDN blocca o sfida un bot di recupero all'edge, il log non mostrerà nulla di anomalo. Un curl locale che restituisce 200 OK non dimostra che ogni bot riceva la stessa risposta. Vale la pena controllare le impostazioni di Cloudflare come i controlli per gli AI-scraper e le funzionalità anti-bot."}},{"@type":"Question","name":"Se una pagina non viene citata nei risultati AI, dovrei riscrivere prima i contenuti o controllare la consegna?","acceptedAnswer":{"@type":"Answer","text":"Prima verificare la consegna, poi giudicare la pagina. Controllate la risposta dell'edge e l'HTML grezzo che un bot di recupero può ricevere prima di riscrivere la copia, perché una pagina renderizzata può sembrare corretta in un browser mentre la risposta per un bot è bloccata, sfidata o priva di contenuto che appare dopo il JavaScript lato client."}},{"@type":"Question","name":"Che tipo di pagina contenuti può effettivamente essere citato come risposta AI?","acceptedAnswer":{"@type":"Answer","text":"Una pagina che risponde a una domanda specifica senza richiedere al lettore di assemblare la risposta da più pagine: la risposta diretta vicino alla cima, date o vincoli rilevanti esplicitati, e abbastanza contesto perché il passaggio sia utile al di fuori della navigazione. La nostra pagina di archivio non poteva offrire questo; il nostro articolo datato sì."}},{"@type":"Question","name":"Dovrei eliminare le pagine di categoria se non generano citazioni AI?","acceptedAnswer":{"@type":"Answer","text":"No, vale la pena mantenere l'archivio perché le persone hanno bisogno di percorsi attraverso un sito. La correzione è smettere di riportare il conteggio delle richieste dell'archivio come misura della visibilità delle risposte: quelle sono funzioni diverse."}}]}]}
```