Only SEO tools are requesting your markdown, and Claude Code reads your docs instead

Two sets of primary data written up on August 31 arrive at the same conclusion from opposite directions: the file format you serve to AI crawlers matters far less than which pages those crawlers come for. Google's John Mueller, answering a Reddit thread about whether anyone had seen a major AI bot ask for Markdown, said that on his own test sites "the only crawlers who claim to accept markdown are SEO tools". Separately, agent-traffic research from Profound found that Claude Code's agent sends nearly three-quarters of its page visits to documentation, informational and pricing pages, while Claude's agent sends 60% of its visits to robots.txt files, sitemaps and home pages.

Put those together and the instruction for a software marketer inverts. A year of AI-search advice has been format-led: serve clean Markdown, add a parallel machine-readable copy of the site. The evidence actually gathered points somewhere duller and more expensive — agents fetch the pages your engineering-adjacent content already lives on, and most B2B software companies treat those as documentation overhead rather than a marketing surface.

One model, two answer engines

The Profound study, published on the company's research blog and written up by Search Engine Journal on August 31, ran 1,724 prompts across Claude and Claude Code between July 13 and July 23, generating 24,135 responses with web search enabled on both. The products share an underlying model. They did not behave like it.

Claude Code invoked web search in 13% of responses; Claude did so in more than 93%. For the same prompt, the two products named the same brands only about a fifth of the time. Both were markedly more consistent with themselves than with each other — roughly half the brands overlapped between two Claude runs, about two in five between two Claude Code runs. Claude Code mentioned slightly more brands per response despite searching far less often, 6.6 against 5.2, and produced shorter, more tabular output: 322 words on average against 459, with tables in 54% of responses against 11%.

The useful conclusion is not about Anthropic. It is that the harness — the product wrapper, its instructions and its tools — determines what gets recommended more than the model underneath. Track "Claude" as one row in an AI-visibility dashboard and you are averaging two populations that agree one time in five. Profound estimates Claude Code alone has two to four million weekly active users, and it is not the only harness of that size. This compounds a problem we reported on August 28: single-run checks were already close to worthless given run-to-run variance, and the surface has now fragmented underneath it.

The page type is the variable

Profound's second dataset compared the 1,000 most-visited pages for each agent across its tracked domains between July 18 and August 18. Claude Code's agent concentrated on documentation, informational and pricing pages — about 75% of observed visits, against 5% for Claude's agent. Claude's agent did the opposite, spending 60% of its visits on robots.txt, sitemaps and home pages, against 4% for Claude Code. One agent is working out what a site contains; the other already knows the page types it wants and goes to fetch specific facts from them.

An agent that skips your home page and reads your pricing table has no interest in your positioning. It wants extractable claims.

Two caveats before anyone rebuilds a docs site. Profound sells AI visibility tracking and publishes this research as marketing for it. Search Engine Journal also noted that the page types were labelled by an AI model with no stated human review, and that the report never tests whether its recommendations actually raise mentions. This is directional evidence about where agents go, not proof that changing those pages changes outcomes.

Why the Markdown answer keeps disappointing

The format question is worth settling because it keeps absorbing engineering time. Cloudflare, which sells on-the-fly HTML-to-Markdown conversion, has written that Markdown "has quickly become the lingua franca for agents and AI systems as a whole" — a claim from a vendor with a product attached to it.

Profound's own controlled test, published in February and therefore older evidence, cuts against that. It randomly split 381 pages across six sites, serving Markdown to bots on 192 and unchanged HTML on 189, then measured bot visits for three weeks. The Markdown group gained roughly one extra median visit per page. The headline mean difference of about 16% came almost entirely from pages already receiving heavy bot traffic, and daily lift was already running about 12% during the implementation window, before any Markdown was served — noise of the same order as the effect.

Mueller's advice was procedural rather than doctrinal: log the accept header and find out whether anything is asking for Markdown before generating it at scale. That is the right sequence, and almost nobody follows it.

What to do differently

The shift is one of ownership, not tooling. Documentation, changelogs, integration pages and the pricing page are usually written by engineers, reviewed by nobody in marketing, and left off the content calendar. If they are the pages coding agents fetch on behalf of the developer evaluating you, they are demand-generation assets maintained by people who have never been briefed as such.

• Split your AI-visibility reporting by harness, not by model — Claude and Claude Code are separate rows.
• Read your own docs and pricing pages as an extraction target: replace "works with your existing stack" with named versions and limits.
• Use question-shaped headings and put the answer in the first sentence beneath them, before context.
• Check your server logs for accept-header requests and agent user agents before commissioning any Markdown project.
• Give documentation and pricing a named owner in marketing, with the same review cadence as the blog.