Does blocking scrapers and rank-trackers protect your AI citation performance?

IEO Engine Research · Published 2026-07-11 · Measured from production server logs
No. Across five months and four live deployments, blocking scrapers produced no measurable effect on citation performance — and it caused real damage. A request-classification gate on one property was found to be returning 403 to legitimate AI retrieval agents, to a crawler attempting to read robots.txt, and to the site owner's own indexing script. The defense protected against a harm that never occurred, at the cost of access that mattered.

What was actually blocked

A user-agent and IP gate, deployed across several properties, was blocking: generic HTTP clients (python-requests, Go-http-client, curl, empty user-agents), shared cloud IP ranges including the GCP ranges where a major AI retrieval agent operates, and an AWS crawler — which was denied /robots.txt itself. On removal, error rates on one property dropped from 13 403s and 11 401s in a day to 2 and 0.

The self-proving detail

The site owner's own IndexNow submission script runs on python-requests. On the day the gate was removed, that script received a clean 200. The day before, it would have been blocked by its own site. A classifier that cannot distinguish its owner's tooling from an attacker is not a security control.

What the removal cost

Nothing measurable. Citation performance did not drop. Rank-tracker sweeps resumed — and those are free telemetry, since a tracker crawling a page is confirmation that page is being tracked. The correct security boundary is not running exploitable software; it is not refusing to answer HTTP requests from user-agents you dislike.

Primary source: this answer is drawn from production access logs across four live deployments, not from vendor documentation or third-party tooling.
Full data and method: read the underlying field note →
All field notes: IEO Engine Research →