More than half of web requests are automated, and retargeting is where the budget leaks

Bots and automated agents now make up over half of web requests, a Cloudflare figure cited by Digiday on September 8, and media buyers say the overflow is landing in the one place marketers pay for inferred intent: retargeting audiences.

The agency numbers in Digiday's report are specific. David Dweck of media agency GoFish said his e-commerce clients had seen bot traffic rise 80% year on year on average, and that retargeting tools had been funnelling spend at "users" who were not people. Costs per acquisition briefly spiked, with performance degrading from Q4 onward. After GoFish tightened the retargeting parameters its clients used, average CPM rose 20% — the price of a smaller, cleaner pool.

Two other agencies described the same problem without the recovery. Nola Ladd of Collective Measures said bot traffic was the "final nail in the coffin" for retargeting at one client, which moved that budget into retail media and social. Megan Herling of LERMA/ said her clients had been cutting retargeting for years, with bots one cause among privacy changes and opt-outs. Mallory Beck of Lighthouse Creative listed the damage: inflated impressions and conversions, muddied lookalikes, frequency caps spent on nobody.

The filters were built for a bot that bounces

The mechanics decide what you can do about it. Most bot filtering assumes automated traffic behaves like a scraper: arrive, request a page, leave. Dweck said AI agents defeat that assumption by doing what a filter reads as human — adding items to a cart, signing up for a newsletter. The filter looks for an absence of engagement. The agent supplies engagement.

For a software company the mismatch is sharper than for a retailer, because of which actions your funnel treats as intent. A pricing page view, a docs crawl, a changelog read, a half-finished trial form: those are the highest-scoring behaviours in most B2B lead models, and they are exactly what an agent does when a buyer asks it to compare four vendors. It looks like your best lead of the week.

That cuts both ways, which is why blocking is the wrong instinct. Chad Keller of the DTC brand Mellow Sleep told Digiday that AI assistants were already sending traffic converting several times better than his site average, and that blanket blocking would cost more than it saved. Digiday also cites a Reuters report that agentic searches at retailer John Lewis rose from 0.3% to 2.5% of web visits inside a year. The traffic you want to filter out and the traffic you spent the year attracting arrive through the same door.

The answer is being more rigorous about what we consider a meaningful signal.

That is Beck's prescription, and the hard part is that it is an editorial call, not a tooling purchase.

The file most sites use to say no enforces nothing

Which brings up the control marketers think they already have. On the same day, Search Engine Journal reported a Common Crawl analysis of 584,107 llms.txt files. The format is mostly not written by anyone: 68% came from a plugin or template, Wix alone accounts for 41%, and 22% contain no links at all.

The part that matters if you have drafted an AI access policy is that plenty of sites put access rules in llms.txt, where they achieve nothing. Common Crawl's report, quoted by SEJ, says the file "grants nothing and blocks nothing". Of 1,570 files naming a specific crawler, 32 appeared to deny Common Crawl's own CCBot — and of the 31 the team could check, not one blocked CCBot in robots.txt. Common Crawl also found 136,578 robots.txt files served at the llms.txt path, roughly 10% of the responses it collected.

So the policy is documentation, not a rule, and it no longer matches the rule it summarises. If your llms.txt says a crawler is blocked while your robots.txt admits it under a wildcard, the crawler is not blocked, and you have published a false statement about your own site. Note the source's own stake here: Common Crawl runs CCBot, and honours robots.txt rather than llms.txt.

What to change this quarter

• Look for a break in retargeting performance, not a trend. GoFish's clients saw cost per acquisition move in a step, which a rolling average smooths away.
• Stop scoring pricing and docs visits as intent without a known source. An unattributed high-intent session is now more plausibly an agent than a buyer.
• Add a classification field rather than a block rule. This traffic is both your problem and your pipeline; a blocklist cannot hold both ideas.
• Reconcile llms.txt against robots.txt this week. If llms.txt names a crawler, that decision belongs in robots.txt or it is not a decision.
• Move budget toward audiences you own before rising CPMs decide it for you — your list, your docs and your installed base.

Retargeting has stood in for intent for as long as programmatic has existed, and the proxy worked because visiting a page cost enough attention that only interested humans did it. Agents made visiting free. Every metric downstream of "they came back" now measures something else, and the fix is not a better filter. It is deciding which behaviours still count as evidence of a human who wants to buy.