Cloudflare and OpenAI's pilot surfaced hundreds of millions of pages ChatGPT search had missed

A four-month pilot between Cloudflare and OpenAI has pulled hundreds of millions of previously invisible web pages into ChatGPT's search index, the two companies reported October 15, crediting network-level signals rather than inbound links for the discovery.

The companies published early results on October 15 in a post co-authored by engineers from both sides. The test covers a cohort of roughly 1% of Cloudflare's total network and was built around one question: could signals Cloudflare already collects on page freshness, traffic quality and recent edits help OpenAI's search crawler discover pages it would otherwise miss. Cloudflare and OpenAI's joint post called the early answer "a resounding yes."

What changed

Conventional web discovery leans on links: a page earns visibility partly because other pages point to it. That approach misses a specific kind of content that software companies publish constantly but rarely link between each other, an installation guide for one product version, a changelog entry, a passport-fee schedule, a restaurant's current hours.

"These pages can be valuable even if few other sites link to them."

That line, from the companies' October 15 post, describes exactly the content the pilot's signals are tuned to surface. Search Engine Journal's reporting notes that Cloudflare's earlier July 8 announcement of the pilot named the signals being tested: how recently a page was updated, the quality of traffic it receives, and whether the page's content has actually changed, factors a backlink count never captures. The same signals work in reverse too, filtering out pages not worth a crawler's time. Bots repeatedly request URLs that redirect to a homepage, duplicate other pages, or do not exist at all; change signals now tell OpenAI's crawler when a page is worth revisiting and when it is not. The companies are separately testing signals that isolate one bad page, for instance a login box an intruder slipped onto an otherwise clean site, rather than treating the entire domain as suspect.

Who this affects

Software marketers who maintain long-tail content, API references, integration guides, pricing FAQs, support documentation, are the clearest test case for this kind of discovery. These pages rarely attract inbound links, yet they answer a specific buyer or user question, which is the exact category Cloudflare and OpenAI say the pilot's signals are built to find. If the approach scales past its current cohort, getting a niche support article into ChatGPT's index may depend less on earning links to it and more on whether a site's traffic and freshness signals look legitimate to Cloudflare's network.

Indexing is not the same as being cited, though. OpenAI "independently evaluates and ranks" the pages the pilot surfaces, based on relevance and usefulness, a distinction Search Engine Journal's Matt G. Southern underlined in his reporting on the post: nothing guarantees that a newly indexed page gets surfaced, let alone linked, in an actual ChatGPT answer. The pilot addresses discovery, not ranking, and the gap between the two is where most of a marketing team's actual visibility work still has to happen.

What to do about it

Access is a separate question from discovery, and it remains entirely within a site's control. OpenAI's crawler infrastructure already splits training and search into two robots.txt directives: GPTBot, which can collect content for model training, and OAI-SearchBot, which powers ChatGPT's search results. A site can block one while allowing the other. Per Southern's reporting, blocking OAI-SearchBot removes a site from ChatGPT's search results entirely, though pages may still surface as navigational links.

There is a measurement wrinkle worth flagging alongside this. Southern separately reported that some ChatGPT links were missing both UTM parameters and referrer data earlier in October, which would make referral traffic from the platform invisible in standard analytics; he noted that a fix began rolling out around the same time as the pilot results. A marketing team that watched its "ChatGPT referral traffic" number sit flat or fall this month should treat that number with more suspicion than confidence, independent of anything the Cloudflare pilot changes.

What's still unclear

Cloudflare and OpenAI have not disclosed which sites participated in the pilot, how many of the newly indexed pages were actually surfaced in a ChatGPT answer, or how much referral traffic resulted. Neither company has committed to a date for extending the signals beyond the current 1% cohort, and the October 15 post frames the next phase as refining page selection and measuring business outcomes, not a broader rollout. The practical move for a software marketing team in the meantime is narrower: audit which of your deep, rarely-linked pages, the ones your link-building strategy was never going to reach, are worth having found at all, and make sure your crawler permissions reflect that answer rather than a blanket AI-blocking default set months ago.