Every time a customer’s AI agent visits your site, you are billing them.
You don’t see the bill. They do. It shows up in their token spend, in the agents that route around you, and in the products their team built on someone else instead. That’s the agent tax — and at frontier model rates of $5 per million input tokens, the meter runs every time an agent grinds through your HTML to figure out what you offer.
The numbers are not subtle. An unoptimized site can cost an agent 120,000 to 500,000 extra tokens per session versus an optimized one. At frontier rates that’s between $0.60 and $2.50 per single interaction. Multiply by ten thousand agent visits a month and you have a five-figure annual gift to the wrong people.
The forty-eight automated BotVisibility web checks measure exactly where that bill comes from.
Where the bill actually comes from
Three categories account for most agent waste:
- Discovery. Agents that can’t find your llms.txt, OpenAPI spec, or MCP server scrape your HTML instead. A homepage that delivers 800 useful tokens routinely pulls in 30,000 raw tokens of scripts and chrome.
- Retries. Agents that hit a 500 with a plain-text error body burn 1,500 tokens trying to interpret what went wrong. Structured JSON errors with a code and a recoverable hint resolve in 80.
- Over-fetching. Agents that can’t request
fields=name,pricefrom your API receive every field on every call. List endpoints without cursor pagination force them to fetch and re-fetch. Each is invisible per call and ruinous at scale.
These map directly onto L1: Discoverable, L2: Usable, L3: Optimized, and L4: Indexable — the four web-scannable levels covering 48 of the 55 BotVisibility checks. Each fail is a line item on the agent bill.
Current pricing, May 2026
| Model | Input / 1M | Output / 1M | Typical use |
|---|---|---|---|
| Claude Opus 4.7 | $5.00 | $25.00 | Frontier reasoning, complex multi-step agents |
| GPT-5.5 | $5.00 | $25.00 | Frontier general-purpose |
| Claude Sonnet 4.6 | $3.00 | $15.00 | Workhorse agent |
| Claude Haiku 4.5 | $1.00 | $5.00 | Classification, routing |
Two things changed this year. Tokenizers got hungrier — the same English page produces about 15% more tokens than it did in early 2025. And frontier prices doubled. The bill your customer absorbs is materially larger than it was twelve months ago, even if nothing on your site changed.
One concrete example
A moderately complex agent session looks like this:
| Step | Unoptimized | Optimized | Savings |
|---|---|---|---|
| Site discovery (no llms.txt) | 80,000 | 1,000 | 99% |
| API discovery (no OpenAPI) | 100,000 | 15,000 | 85% |
| Authentication (OAuth dance vs API key) | 5,000 | 50 | 99% |
| 5 API calls (no sparse fields) | 10,000 | 1,000 | 90% |
| Error handling (1 failed call) | 12,000 | 80 | 99% |
| Pagination (100 items, full payload) | 15,000 | 5,000 | 67% |
| MCP tool overhead (no quality schemas) | 55,000 | 2,000 | 96% |
| Total | 277,000 | 24,100 | 91% |
At Claude Opus 4.7 input pricing, that single interaction costs $1.39 unoptimized vs. $0.12 optimized — a $1.27 swing per task. Same job. Same agent. Same customer. Eleven times the bill.
A thousand such interactions a day works out to about $38,000 a month saved for your customer, not for you. They notice. The procurement team that signed the AI budget notices even faster.
The discovery layer does most of the work
Eighteen of the forty-eight web checks are at L1: Discoverable. Those eighteen account for roughly 70% of the total token savings in the table above. Ship an llms.txt, an OpenAPI spec, a SKILL.md, markdown content negotiation, and a working MCP server — and you’ve already captured the bulk of the win.
The other thirty checks tighten the bill: structured errors, pagination, sparse fields, indexable headings, declared Google-Extended policy, a real Organization entity in your JSON-LD. Each one is small. Together they’re the difference between an agent finishing your job in 24,000 tokens or 277,000.
The bottom line
The agent tax is not a tax you pay. It’s a tax your customers pay every time their agent has to figure you out.
The cheaper you are to automate, the more agents — and the people who deploy them — will choose to build on you. Gartner expects 40% of agentic AI projects to be canceled before production by 2027, largely on cost overruns. The forty-eight checks are the difference between being the cheap path and being the canceled line item.
Run the scan. See what your customers are paying. Fix the discovery layer first.