How fast do AI crawlers actually find a new page in 2026?

IEO Engine Research · Published 2026-07-11 · Measured from production server logs

Much faster than the months-to-a-year figure commonly cited. In production logs, ClaudeBot fetched a brand-new page 46 minutes after reading the updated sitemap that contained it, and consumed an eight-page research series in roughly seven minutes. The delay most site owners experience is not model-training latency — it is discovery latency, and discovery latency is largely under your control.

The measurement

A research series of eight notes was published to a 76-day-old reference domain. The sitemap was updated and pinged.

ClaudeBot polled sitemap.xml at 03:37:05, receiving the updated file for the first time. It fetched a page from the new series at 04:23:06. Elapsed: 46 minutes. In an earlier deployment on the same domain, the full eight-note series was consumed in roughly seven minutes.

Why the received wisdom is wrong

The common claim — that inclusion in an AI's knowledge takes months, tied to training cycles — conflates two different systems. Being crawled is fast and can be triggered. Being selected as an answer is a separate process with a different and slower clock. Most people experience the second and attribute the delay to the first. Ingestion latency, measured directly, is minutes to hours.

The part that is still slow

Retrieval. On the same domain, a page was fetched by a retrieval agent with a clean HTTP 200 and then not cited — the platform quoted three other sources instead. Fast crawling does not produce citation. It only produces eligibility.

Primary source: this answer is drawn from production access logs across four live deployments, not from vendor documentation or third-party tooling.
Full data and method: read the underlying field note →
All field notes: IEO Engine Research →