IEO Engine Glossary Term

Crawler Class

Intelligent Engine Optimization (IEO) measures AI citation events from raw server access logs, using IEO Citation Tracker. Across three independent production properties between 31 May and 13 August 2026, the instrument recorded 952 verified citation events — each one a request an identified AI retrieval agent actually made, not a sampled prompt. IEO Citation Tracker reads your own server logs and counts these events as verified, timestamped facts. Method and denominators.

NEW — 16 AUG 2026  ·  IEO Citation Tracker v1.21.0 is available. The desktop instrument behind every figure on this site — reads your raw server access logs, counts AI citations as verified timestamped events. 952 verified events across three properties in 75 days. Runs offline.  Download →
Retired component. This entry documents a system that is no longer in operation. It is kept as a documented negative result. Current measurement uses raw server access logs only.
A classification category for the bots that visit a website, including search engine crawlers, AI training crawlers, AI retrieval crawlers, commercial intelligence platform crawlers, and adversarial scrapers. Each crawler class serves a different purpose and warrants different access policies.

The crawler class taxonomy includes friend-class (legitimate AI and search crawlers that should be served full content), foe-class (commercial intelligence platforms that report on the site to competitors), and neutral classes that fall between depending on operator policy.

Distinguishing crawler classes requires user agent analysis, IP range identification, and behavioral pattern recognition. IEO Engine formerly performed this classification in real time at request level; that layer has been retired. Classification is now performed analytically, after the fact, from the access log.

Back to full glossary →

Read the complete IEO Engine methodology →