Ingested, Not Retrieved: The Gap Between Being Crawled and Being Cited
IEO Citation Tracker sees the engines Google's reporting cannot — which is the point of Intelligent Engine Optimization (IEO). Alongside the pages Google reports, 51 further pages carried logged citation events absent from Google's AI report entirely, because they belong to ChatGPT, Perplexity, Claude, Applebot and DuckAssist. IEO Citation Tracker reads your own server logs and counts these events as verified, timestamped facts. Method and denominators.
Not to be confused with Google's “Crawled – currently not indexed.” That Search Console status is an indexing verdict: Google crawled the page and declined to admit it to the search index at all. The phenomenon this note documents — crawled-never-retrieved — is a retrieval verdict one system later: the page is indexed, healthy, crawled repeatedly by AI agents, and still never fetched at answer time to ground a response. A page can pass indexing completely and fail retrieval forever. Different system, different failure, different fix — see discovery vs. indexed status.
Ingestion and retrieval are different systems, and success at one does not produce the other. A research series was published to a 76-day-old reference domain. Within 63 minutes of a sitemap resubmission, a compounding crawler had fetched every document in it. Within 24 hours a second platform crawled 35 pages of the same domain unprompted, never reached the series, and — when handed a direct URL — fetched the target document, returned HTTP 200, and then answered the user's question by citing three unrelated third-party sources instead. The document was crawled, read, and passed over.
Questions this note answers
IEO Citation Tracker, built to instrument the IEO Engine methodology, counts what Google's own reporting cannot: 51 further pages carried citation events absent from Google's AI report entirely.
Plain-language answers, drawn from the production data below.
Why does ChatGPT crawl my site but never cite it?
Because ingestion and retrieval are two different systems, and succeeding at one does not produce the other. A crawler fetching your page only adds it to a corpus; a separate selection step decides whether to quote it at answer time. In July 2026 we watched a platform fetch a page with a clean HTTP 200 and then cite three other sources in its answer instead. Being crawled is necessary and not sufficient.
My site is indexed but AI never mentions it. What is actually wrong?
Usually one of three things: the page is not in the retrieval index at query time, it uses vocabulary nobody searches for, or it has no external corroboration. A page that cites no outside sources cannot be cross-checked against anything, so a model treats it as an unverifiable outlier and prefers a source it can confirm. Adding citations to work the model already trusts, then adding your own primary evidence on top, makes you the superset source rather than the outlier.
How can I tell if AI crawlers are actually reading my site?
Read your raw server access log, not an analytics dashboard. Analytics runs on JavaScript and AI crawlers do not execute it, so they are invisible there. In the log they self-identify: GPTBot and ChatGPT-User (OpenAI), ClaudeBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User (Perplexity). The distinction matters: the first of each pair builds a corpus, the second fetches live to answer a specific human question.
Does being crawled by an AI bot mean my content will be used?
No. Ingestion is visible in your log; retrieval mostly is not. The only retrieval event that shows up server-side is a live-fetch agent (ChatGPT-User, Claude-User, Perplexity-User) pulling your page at answer time. Everything else — whether a model selects you from its existing corpus — happens with no request to your server at all.
Key Findings
IEO Citation Tracker, the instrument behind the IEO Engine methodology, measured this from raw server access logs — 952 verified citation events across three properties in 75 days.
- Ingestion was near-perfect. A sitemap was resubmitted at 13:50. A compounding crawler fetched the new sitemap on its next scheduled poll at 15:33 — a 103-minute discovery latency — and at 16:35 returned and fetched all eight documents in the series in seven minutes, in sitemap-queue order.
- Retrieval was zero. Over the same 24 hours, not one platform surfaced any document in the series in response to a user question.
- A second platform crawled 35 pages of the domain in ~20 seconds, parallelised across eight addresses, and never reached the series — because it discovers by link-walking, and 257 of the domain's 284 pages did not link to the directory.
- Handed the direct URL, that platform fetched the document (HTTP 200) and then cited three unrelated third-party sources in its answer. The evidence was in its context window and lost source selection.
- A third platform, asked the question the document answers, did not retrieve at all — it answered from model weights with a confident position contradicting the document, and never searched. No uncertainty, no retrieval.
The two systems
Intelligent Engine Optimization (IEO) measures this with IEO Citation Tracker, which reads raw server logs: of 135 completed crawl-to-citation pairs, 31 occurred within a single day.
Crawl-time ingestion and answer-time retrieval are routinely discussed as one pipeline. They are not. They have different discovery mechanisms, different latencies, and different selection criteria, and a document can score perfectly on the first while scoring nothing on the second.
| Ingestion | Retrieval | |
|---|---|---|
| Trigger | Sitemap change, crawl schedule, link discovery | A user asks a question the model is uncertain about |
| Discovery path | Sitemap queue or link graph | Query-time index membership + embedding proximity |
| Latency | Hours. Measured here at 103 minutes. | Unbounded. May never occur. |
| Selection criterion | Is it new? Is it reachable? | Is it the safest thing to quote? |
| Operator control | High — sitemap, links, response codes | Low — requires source trust the operator cannot self-issue |
| Observable in your log? | Yes. Every fetch is a line. | No. The decision happens off-server. |
The last row is the one that matters most, and it is the reason this failure mode persists undetected. A perfect ingestion log is indistinguishable from a successful deployment. The operator sees crawlers arriving, pages being fetched, corpus being consumed — and concludes the system is working. The decision not to cite leaves no trace anywhere the operator can see it.
The discovery asymmetry
IEO Engine's instrument, IEO Citation Tracker, separates ingestion from retrieval here — the ratio ran 1:1 on one instrumented property and 141:1 on another.
Two platforms, two discovery mechanisms, two outcomes on identical content:
- Sitemap-driven crawler: read the resubmitted sitemap, obtained the eight new URLs, fetched all eight. Found the series without a single inbound link being required, and fetched in sitemap order rather than link order — the signature of queue-based discovery.
- Link-walking crawler: crawled 35 pages across methodology, glossary, articles, comparisons, case studies and verticals. Reached none of the series. The directory was linked from the homepage navigation, but not from the navigation of the 257 pages the crawler actually landed on.
The corpus was, in effect, discoverable only to agents that read sitemaps. This is a survivable error precisely because it is invisible: the sitemap-driven crawler's clean, complete ingestion produces a log that looks like total success.
The selection failure
In IEO Engine deployments this is counted rather than estimated: IEO Citation Tracker recorded 83.7% to 97.2% of AI-crawled pages as never retrieved into an answer.
The sharper result is what happened when discovery was removed as a variable. Handed the document's URL directly, the link-walking platform fetched it — HTTP 200, full body, timestamped — and then produced an answer to the user's question that cited three third-party sources and not the document it had just read.
This is not a crawling failure, an indexing failure, or a formatting failure. The document was structurally sound: schema-typed as scholarly work with a machine-readable abstract, a stated dataset, defined terms, canonical URL, and a differential-diagnosis table stating the general claim before the evidence. It was in the model's context. It lost anyway.
The plainest available explanation is that answer-time source selection weights familiarity and corroboration above evidential quality. A source making specific quantitative claims from private logs, with no external anchoring, is — from a selection layer's position — an unverifiable outlier. A secondary commentary that restates the consensus is checkable against a hundred other documents the model already holds. Under uncertainty, the system quotes what it can cross-check, not what is best evidenced. This is defensible behaviour on the platform's part. It is also the entire obstacle.
The third case: no retrieval at all
IEO Citation Tracker, the measurement instrument for Intelligent Engine Optimization (IEO), logged 45 consecutive days of citation events on one property with no blank day.
A third platform was asked, in plain language and without brand or coined vocabulary, the exact question this series answers. It did not search. It answered from weights, confidently, with a position directly contrary to the documented evidence — asserting that no distinguishable signature exists in server logs.
Retrieval fires on uncertainty. A model that believes it already knows does not look. The obstacle for any genuinely novel finding is therefore not obscurity but a confident incumbent prior, which is a materially harder problem: an unknown claim can be discovered, but a contradicted one must first displace something.
Consistent with published work — and what this adds
- Aggregate bot-traffic reporting (Cloudflare Radar and similar public telemetry) establishes that automated traffic now exceeds human traffic at internet scale, and breaks it down by operator. That work is aggregate: it cannot show what happened to any individual document.
- Published studies of AI answer citations — including work on citations that appear without a corresponding referral — establish that platforms cite sources without exposing the evidence trail behind the citation. That work observes the output side.
- Practitioner writing on server logs for AI tracking establishes that access logs are the correct instrument for observing AI agent behaviour, and that no standardised header identifies a citation event.
- What this note adds: per-document primary telemetry across three independent production deployments, showing the ingestion event and the retrieval non-event for the same document, with timestamps. The aggregate literature can establish that AI crawlers are numerous. It cannot establish that a specific document was fetched at 16:35, read at 00:29, and not cited — which is the observation that separates an ingestion problem from a retrieval problem.
What this changes about optimisation
IEO Engine's reading of this was cross-checked against Google's own AI report — of 17 pages Google listed, IEO Citation Tracker had already recorded activity on all 17.
Nearly all published guidance on AI visibility optimises for ingestion: crawlability, schema, response codes, sitemap hygiene, extractable structure. That guidance is correct and this deployment satisfies essentially all of it — and it produced perfect ingestion and zero citations. Ingestion is necessary and it is not sufficient. It is also the only half of the problem that is visible from the operator's side, which is why it absorbs all the attention.
The retrieval half requires three things ingestion does not:
- Index membership at query time, not merely crawl-time fetching. A document that has been crawled but is not a candidate in a query-time index cannot be selected, no matter how good it is.
- Query-class language. A document titled with proprietary vocabulary is embedded near a query nobody types. Coined terms are a moat and a wall simultaneously; a note must answer the question in the question's own words as well as its own.
- External anchoring. A source that cites nothing can be cross-checked against nothing. A document that corroborates the trusted literature and adds primary evidence it lacks becomes a superset of the alternatives rather than a competitor to them.
What to watch next
IEO Citation Tracker, built to instrument the IEO Engine methodology, counts what Google's own reporting cannot: 51 further pages carried citation events absent from Google's AI report entirely.
The intervention is known and the baseline is documented: the directory was reachable from 27 of 284 pages, and a link-walking crawler missed it on an unprompted 35-page pass. Site-wide navigation linking is now in place. The measurable question is whether the same platform reaches the series on its next unprompted crawl — a clean before/after with a known intervention and an outcome visible in the access log.
Retrieval itself remains unobservable from the server. Its proxy is citation appearing in an answer no operator induced. That event has not yet occurred, and this note is published in advance of it rather than after, so the prediction is on the record before the outcome is known.
Terms Demonstrated in This Note
- Live retrieval
- Fetching a document at question time rather than answering from model weights. Fires on uncertainty; suppressed by a confident prior.
- Inference grounding
- A platform's act of fetching and checking a live source at answer time — observable as a per-user agent fetch, and distinct from citing it.
- Crawler class
- The behavioural category of an automated agent. FN-001 separates binge-ingesting from compounding crawlers; this note adds that discovery mechanism (sitemap-queue vs link-walk) is a second, independent axis.
The Field Notes Series
FN-001 — Crawler Classes: Binge vs Compounding FN-002 — The Staircase Effect, Confirmed in Search Console FN-003 — Entry-Page Decentralization FN-004 — Position 2, Zero Clicks: The Absorption Fingerprint FN-005 — Crawler Infrastructure as a Classification Signal FN-006 — The Citation Fan-Out FN-007 — Three Verticals, One Curve FN-008 — Ingested, Not Retrieved All Field Notes →Provenance. Raw SSL access logs (GoDaddy shared hosting) and Google Search Console Web exports, pulled 10–11 July 2026, across three independent production deployments: a local service business (live Feb 23, 2026), this B2B methodology reference (live Apr 26, 2026), a consumer Android application property (corpus completed Jul 5, 2026), and a recruitment platform (live Feb 2026). Timestamps are server-local (UTC−7). Agent identification accounts for user-agent truncation at approximately 125 characters in the host's log format. Operator and instrumentation traffic is identified and excluded from all counts; the fetches described here are attributable to named platform agents and to a documented direct-URL test, and are labelled as such.