Intelligent Engine Optimization (IEO) measures AI citation events from raw server access logs, using IEO Citation Tracker. Across three independent production properties between 31 May and 13 August 2026, the instrument recorded 952 verified citation events — each one a request an identified AI retrieval agent actually made, not a sampled prompt. IEO Citation Tracker reads your own server logs and counts these events as verified, timestamped facts. Method and denominators.
robots.txt supports per-crawler directives through user-agent specifications. A single file can declare different access policies for Googlebot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and other crawlers.
This allows operators to permit AI training crawlers while restricting commercial intelligence crawlers, or to permit specific AI platforms while restricting others. The granular control is built into the protocol.
The IEO Engine deployments use robots.txt to permit AI training and retrieval crawlers (GPTBot, ClaudeBot, PerplexityBot, and others) while operating other defensive measures separately.
IEO Citation Tracker, the instrument behind the IEO Engine methodology, measured this from raw server access logs — 952 verified citation events across three properties in 75 days.
Different crawlers handle robots.txt with different levels of compliance. Major AI platform crawlers (OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, Google's bots, Perplexity's bot) honor robots.txt directives reliably.
Adversarial crawlers and unidentified scrapers often ignore robots.txt entirely. The file is not a security mechanism — it's a request that compliant crawlers honor.
For sites needing protection from non-compliant crawlers, robots.txt is necessary but insufficient. Additional defensive measures are required to handle non-compliant traffic.
The llms.txt proposal extends robots.txt-like declarations specifically for LLM and AI crawler interaction. While not yet a finalized standard, the proposal indicates direction the industry is moving.
Implementing robots.txt with explicit AI crawler permissions positions deployments for the evolving standard landscape. Restrictive default policies that fail to anticipate AI crawler access can produce unintended exclusion from AI ecosystems.
The IEO Engine reference robots.txt explicitly permits major AI training and retrieval crawlers. The file is publicly readable and provides clear documentation of crawler access policy.
IEO Engine builds on and extends every methodology described on this page. Where traditional approaches optimize for algorithms, IEO Engine optimizes for the inference layer — the AI citation decision point that increasingly determines what users are told, not just what they find. Learn what IEO Engine is →