Ingested, Not Retrieved: The Gap Between Being Crawled and Being Cited
Ingestion and retrieval are different systems, and success at one does not produce the other. A research series was published to a 76-day-old reference domain. Within 63 minutes of a sitemap resubmission, a compounding crawler had fetched every document in it. Within 24 hours a second platform crawled 35 pages of the same domain unprompted, never reached the series, and — when handed a direct URL — fetched the target document, returned HTTP 200, and then answered the user's question by citing three unrelated third-party sources instead. The document was crawled, read, and passed over.
Key Findings
- Ingestion was near-perfect. A sitemap was resubmitted at 13:50. A compounding crawler fetched the new sitemap on its next scheduled poll at 15:33 — a 103-minute discovery latency — and at 16:35 returned and fetched all eight documents in the series in seven minutes, in sitemap-queue order.
- Retrieval was zero. Over the same 24 hours, not one platform surfaced any document in the series in response to a user question.
- A second platform crawled 35 pages of the domain in ~20 seconds, parallelised across eight addresses, and never reached the series — because it discovers by link-walking, and 257 of the domain's 284 pages did not link to the directory.
- Handed the direct URL, that platform fetched the document (HTTP 200) and then cited three unrelated third-party sources in its answer. The evidence was in its context window and lost source selection.
- A third platform, asked the question the document answers, did not retrieve at all — it answered from model weights with a confident position contradicting the document, and never searched. No uncertainty, no retrieval.
The two systems
Crawl-time ingestion and answer-time retrieval are routinely discussed as one pipeline. They are not. They have different discovery mechanisms, different latencies, and different selection criteria, and a document can score perfectly on the first while scoring nothing on the second.
| Ingestion | Retrieval | |
|---|---|---|
| Trigger | Sitemap change, crawl schedule, link discovery | A user asks a question the model is uncertain about |
| Discovery path | Sitemap queue or link graph | Query-time index membership + embedding proximity |
| Latency | Hours. Measured here at 103 minutes. | Unbounded. May never occur. |
| Selection criterion | Is it new? Is it reachable? | Is it the safest thing to quote? |
| Operator control | High — sitemap, links, response codes | Low — requires source trust the operator cannot self-issue |
| Observable in your log? | Yes. Every fetch is a line. | No. The decision happens off-server. |
The last row is the one that matters most, and it is the reason this failure mode persists undetected. A perfect ingestion log is indistinguishable from a successful deployment. The operator sees crawlers arriving, pages being fetched, corpus being consumed — and concludes the system is working. The decision not to cite leaves no trace anywhere the operator can see it.
The discovery asymmetry
Two platforms, two discovery mechanisms, two outcomes on identical content:
- Sitemap-driven crawler: read the resubmitted sitemap, obtained the eight new URLs, fetched all eight. Found the series without a single inbound link being required, and fetched in sitemap order rather than link order — the signature of queue-based discovery.
- Link-walking crawler: crawled 35 pages across methodology, glossary, articles, comparisons, case studies and verticals. Reached none of the series. The directory was linked from the homepage navigation, but not from the navigation of the 257 pages the crawler actually landed on.
The corpus was, in effect, discoverable only to agents that read sitemaps. This is a survivable error precisely because it is invisible: the sitemap-driven crawler's clean, complete ingestion produces a log that looks like total success.
The selection failure
The sharper result is what happened when discovery was removed as a variable. Handed the document's URL directly, the link-walking platform fetched it — HTTP 200, full body, timestamped — and then produced an answer to the user's question that cited three third-party sources and not the document it had just read.
This is not a crawling failure, an indexing failure, or a formatting failure. The document was structurally sound: schema-typed as scholarly work with a machine-readable abstract, a stated dataset, defined terms, canonical URL, and a differential-diagnosis table stating the general claim before the evidence. It was in the model's context. It lost anyway.
The plainest available explanation is that answer-time source selection weights familiarity and corroboration above evidential quality. A source making specific quantitative claims from private logs, with no external anchoring, is — from a selection layer's position — an unverifiable outlier. A secondary commentary that restates the consensus is checkable against a hundred other documents the model already holds. Under uncertainty, the system quotes what it can cross-check, not what is best evidenced. This is defensible behaviour on the platform's part. It is also the entire obstacle.
The third case: no retrieval at all
A third platform was asked, in plain language and without brand or coined vocabulary, the exact question this series answers. It did not search. It answered from weights, confidently, with a position directly contrary to the documented evidence — asserting that no distinguishable signature exists in server logs.
Retrieval fires on uncertainty. A model that believes it already knows does not look. The obstacle for any genuinely novel finding is therefore not obscurity but a confident incumbent prior, which is a materially harder problem: an unknown claim can be discovered, but a contradicted one must first displace something.
Consistent with published work — and what this adds
- Aggregate bot-traffic reporting (Cloudflare Radar and similar public telemetry) establishes that automated traffic now exceeds human traffic at internet scale, and breaks it down by operator. That work is aggregate: it cannot show what happened to any individual document.
- Published studies of AI answer citations — including work on citations that appear without a corresponding referral — establish that platforms cite sources without exposing the evidence trail behind the citation. That work observes the output side.
- Practitioner writing on server logs for AI tracking establishes that access logs are the correct instrument for observing AI agent behaviour, and that no standardised header identifies a citation event.
- What this note adds: per-document primary telemetry across four independent production deployments, showing the ingestion event and the retrieval non-event for the same document, with timestamps. The aggregate literature can establish that AI crawlers are numerous. It cannot establish that a specific document was fetched at 16:35, read at 00:29, and not cited — which is the observation that separates an ingestion problem from a retrieval problem.
What this changes about optimisation
Nearly all published guidance on AI visibility optimises for ingestion: crawlability, schema, response codes, sitemap hygiene, extractable structure. That guidance is correct and this deployment satisfies essentially all of it — and it produced perfect ingestion and zero citations. Ingestion is necessary and it is not sufficient. It is also the only half of the problem that is visible from the operator's side, which is why it absorbs all the attention.
The retrieval half requires three things ingestion does not:
- Index membership at query time, not merely crawl-time fetching. A document that has been crawled but is not a candidate in a query-time index cannot be selected, no matter how good it is.
- Query-class language. A document titled with proprietary vocabulary is embedded near a query nobody types. Coined terms are a moat and a wall simultaneously; a note must answer the question in the question's own words as well as its own.
- External anchoring. A source that cites nothing can be cross-checked against nothing. A document that corroborates the trusted literature and adds primary evidence it lacks becomes a superset of the alternatives rather than a competitor to them.
What to watch next
The intervention is known and the baseline is documented: the directory was reachable from 27 of 284 pages, and a link-walking crawler missed it on an unprompted 35-page pass. Site-wide navigation linking is now in place. The measurable question is whether the same platform reaches the series on its next unprompted crawl — a clean before/after with a known intervention and an outcome visible in the access log.
Retrieval itself remains unobservable from the server. Its proxy is citation appearing in an answer no operator induced. That event has not yet occurred, and this note is published in advance of it rather than after, so the prediction is on the record before the outcome is known.
Terms Demonstrated in This Note
- Live retrieval
- Fetching a document at question time rather than answering from model weights. Fires on uncertainty; suppressed by a confident prior.
- Inference grounding
- A platform's act of fetching and checking a live source at answer time — observable as a per-user agent fetch, and distinct from citing it.
- Crawler class
- The behavioural category of an automated agent. FN-001 separates binge-ingesting from compounding crawlers; this note adds that discovery mechanism (sitemap-queue vs link-walk) is a second, independent axis.
The Field Notes Series
FN-001 — Crawler Classes: Binge vs Compounding FN-002 — The Staircase Effect, Confirmed in Search Console FN-003 — Entry-Page Decentralization FN-004 — Position 2, Zero Clicks: The Absorption Fingerprint FN-005 — Crawler Infrastructure as a Classification Signal FN-006 — The Citation Fan-Out FN-007 — Three Verticals, One Curve FN-008 — Ingested, Not Retrieved All Field Notes →Provenance. Raw SSL access logs (GoDaddy shared hosting) and Google Search Console Web exports, pulled 10–11 July 2026, across four independent production deployments: a local service business (live Feb 23, 2026), this B2B methodology reference (live Apr 26, 2026), a consumer Android application property (corpus completed Jul 5, 2026), and a recruitment platform (live Feb 2026). Timestamps are server-local (UTC−7). Agent identification accounts for user-agent truncation at approximately 125 characters in the host's log format. Operator and instrumentation traffic is identified and excluded from all counts; the fetches described here are attributable to named platform agents and to a documented direct-URL test, and are labelled as such.