Google Tells You What It Thinks You Are — by Which Crawler Stack It Sends
Google does not announce how it has classified a site — but it doesn’t have to. The classification is legible in raw logs, encoded in which crawler and proxy infrastructure Google assigns. Across three simultaneously observed deployments, three different infrastructure mixes appeared, each matching the property’s actual nature. Source classification can be read directly from reverse DNS.
Key Findings
- Local-service site: crawled exclusively from Google’s legacy 66.249.0.0/16 Googlebot ranges, with the active block rotating monthly — pure index-maintenance treatment.
- Consumer application property: crawled from Google’s newer 192.178.0.0/15 Googlebot range, PLUS near-daily rate-limited user-fetch proxies (fetches made on behalf of real users inside Google products), PLUS mail-proxy fetches, PLUS an app-store verification agent, PLUS Chrome’s privacy-preserving prefetch proxy — the full consumer stack.
- B2B reference site: standard index crawl PLUS logged sessions from the Google Search App on iOS — real humans searching inside Google’s own application and landing on the domain.
- One prolific “Google” source is not Google at all: *.bc.googleusercontent.com hosts are third parties renting Google Cloud VMs. Distinguishing them is the first step of any log audit.
- A UA-parsing trap: common log tools truncate user agents at ~125 characters, hiding the “(compatible; Googlebot/2.1)” suffix — the mobile Googlebot then masquerades as an ordinary Android Chrome visitor at the top of the agent table.
Three properties, three verdicts
| Infrastructure | Local service | Consumer app | B2B reference | What it signals |
|---|---|---|---|---|
| Googlebot, legacy 66.249.0.0/16 (blocks rotate monthly) | Yes — exclusively | No | Yes | Index maintenance |
| Googlebot, newer 192.178.0.0/15 range | No | Yes — heavy | No | Modern-stack crawl assignment |
| Rate-limited user-fetch proxies (108.177.x) | No | Yes — near-daily | No | Real-user demand inside Google products |
| Mail-proxy fetches | No | Yes | No | The property circulating in email |
| App-store verification agent | No | Yes — continuous | No | Store listing ↔ web presence re-verification |
| Chrome privacy-preserving prefetch proxy | No | Yes | Yes | A human on a results page, about to click |
| Google Search App user sessions | No | No | Yes | Humans researching the entity in Google’s own app |
The claim
The infrastructure mix is not random. The consumer property gets the consumer stack; the local business gets maintenance; the reference site gets crawl plus human research sessions. Google’s source classification of a property is therefore observable from the outside, for free, in any raw access log with reverse DNS — months before it is visible in any dashboard. A change in the assigned crawler class mix is a change in classification, and it is the earliest such signal we know of.
How to check it yourself
Reverse-resolve crawler IPs; verify Googlebot ranges against Google’s published IP lists; separate googlebot.com and google.com hosts (Google) from googleusercontent.com hosts (tenants on Google Cloud); and read full user-agent strings from raw logs, not truncated summaries. The verdict is sitting in the log file.
Terms Demonstrated in This Note
- Source classification
- The category an AI or search system assigns to a domain — what kind of thing it believes the property is — which governs the treatment the property receives.
- Crawler class
- The behavioral and infrastructural category of an agent visiting a site; the mix of classes assigned by a platform encodes its classification verdict.
Related Field Notes
FN-001: Two Crawler Classes: Binge Ingesters and Compounding Re-Crawlers · FN-007: Three Verticals, One Curve: The Ingestion Sequence Replicates