Two Crawler Classes: Binge Ingesters and Compounding Re-Crawlers
Across 140 days and three independent deployments, AI-platform crawlers separated into two clean behavioral classes. Binge ingesters consumed the corpus in a single heavy window and then went silent. One crawler — and only one — returned every month on every deployment, with volume growing each month. Crawler class, not crawl volume, determines whether a platform ever sees your updates.
Key Findings
- GPTBot peaked as the #1 user agent on the local-service deployment in May 2026 (1,400+ requests), then recorded zero requests in June and July.
- PerplexityBot was heavily active in the first 60 days of each deployment it visited, then went to zero — no return visits for 60+ consecutive days.
- PetalBot executed a single 3,100+ request ingestion in one month, then never returned.
- ClaudeBot grew its request volume every single month and by June 2026 ranked #1 or #2 user agent on all three deployments simultaneously — the only agent to do so.
- Traditional search crawlers (Googlebot, Bingbot) maintained a steady maintenance baseline throughout — a third, stable class.
The observation
Every major AI platform assembled on each deployment within days of it becoming ingestible. What happened after first contact is where the platforms diverge — and the divergence is a stable behavioral trait, reproduced identically across a local service business, a B2B reference site, and a consumer application property. This trait is what the glossary defines as crawler class, and it is measured by crawl-revisit-rate.
| Agent | First contact | Peak activity | Status as of Jul 10, 2026 | Crawler class |
|---|---|---|---|---|
| GPTBot (OpenAI) | Week 1 | May 2026 — #1 agent, 1,400+ requests in one month | Zero requests for 40 consecutive days | Binge ingester |
| PerplexityBot | Week 1 | Month 2 of each visited deployment (600–700 requests) | Zero requests for 60+ days | Binge ingester |
| PetalBot (Huawei) | Week 1 | Single month: 3,100+ requests | Zero since that month ended | Binge ingester |
| ClaudeBot (Anthropic) | Week 1 | Grew every month; 2,500+ requests in June on one deployment | Active on all three deployments; #1 or #2 agent on each | Compounding re-crawler |
| Bingbot / Googlebot | Week 1 | Steady | Continuous maintenance baseline | Index maintainer |
Why this matters
A binge ingester's model of your site is a snapshot — frozen at whatever the corpus contained during its ingestion window. Content published after the window does not exist for that platform until its next (unscheduled, possibly never) return. A compounding re-crawler's model is a subscription: updates propagate on roughly a monthly cycle.
The practical consequence: two sites with identical content can have completely different representation across AI platforms depending purely on when each platform's ingestion window intersected the corpus state. Freshness strategy must therefore be planned per crawler class — a single “AI SEO” posture treats a snapshot audience and a subscription audience as if they were the same thing. They are not.
Falsifiability
This is checkable in any raw access log: group requests by verified AI user agents, bucket by month, and plot per-agent volume. If the two-class split is wrong, agents should show mixed or random revisit behavior. Across three deployments and 140 days, we observed no mixed cases: every AI agent fell cleanly into one class and stayed there.
Terms Demonstrated in This Note
- Crawler class
- The behavioral category of a crawling agent — binge ingester, compounding re-crawler, or index maintainer — determined by its revisit pattern rather than its volume.
- Crawl-revisit-rate
- The frequency with which a given agent returns to a corpus after first full ingestion; the measurable variable that separates crawler classes.
Related Field Notes
FN-005: Google Tells You What It Thinks You Are — by Which Crawler Stack It Sends · FN-007: Three Verticals, One Curve: The Ingestion Sequence Replicates