Google and OpenAI's API only match a quarter of what ChatGPT actually cites, a new study finds

Google search results and OpenAI's own API, the two shortcuts most AI-visibility and GEO tools use to approximate what ChatGPT cites, together catch only about a quarter of the exact pages ChatGPT actually references, a finding published October 11 in a preprint from a team at the National University of Singapore and Xi'an Jiaotong University.

The paper exists because nobody outside OpenAI can query ChatGPT's production web app at scale. So GEO and AI-visibility vendors routinely substitute Google's organic rankings or OpenAI's Responses API, then treat the overlap as a stand-in for what ChatGPT itself would show a user. The researchers tested that assumption directly, across four ChatGPT Web settings spanning GPT-5.2 Instant through GPT-5.5 Instant, using 80 real queries pulled from Search Arena, a public log of search-enabled chatbot sessions.

The proxies barely agree with ChatGPT on exact pages

The 80 test questions split evenly between English and Chinese, covered recommendation, fact-lookup, synthesis and analysis prompts, and each one went through GPT-5.3 Instant and GPT-5.5 Instant twenty separate times to capture variation. Alongside those runs, the team gathered an average of roughly 43 Google organic listings per question via SerpAPI and queried OpenAI's Responses API with web search turned on, timing both within a three-day window of each ChatGPT test.

Put together, Google and the API accounted for little more than a quarter of what GPT-5.3 Instant and GPT-5.5 Instant actually cited as pages — 28.3% and 26.2%, respectively. Widen the measure to whole domains instead of exact URLs and coverage roughly doubles, to 50.3% and 48.7%, but that still leaves close to half of a brand's real citation footprint outside what rankings or API calls alone would show. The API alone outpaced Google alone, covering 22.9% and 22.5% of cited pages against Google's 9.4% and 6.4%.

ChatGPT doesn't search the way most trackers assume

Part of the gap traces to how ChatGPT actually searches. Wherever the researchers could see the underlying query, neither GPT-5.3 Instant nor GPT-5.5 Instant repeated a user's question word for word; both rewrote it into a fresh search phrase every single time. And when the original prompt was in Chinese, those same two settings still searched in English on 67.0% and 69.6% of runs.

Which pages turned up also varied widely between repeat runs of an identical prompt. For GPT-5.5 Instant, 44.5% of the unique pages surfaced across twenty attempts showed up in just one of them; GPT-5.3 Instant's figure was 40.8%. A model trained to combine both proxy signals could still forecast whether a page would get cited with real accuracy, posting an AUC above 0.74 at the URL, hostname and entity levels alike. Even so, the authors stop short of claiming any single proxy method, commercial tools included, reconstructs ChatGPT's actual source list. Search Engine Journal, which first reported the paper, noted that it did not test any specific commercial tracking product directly.

What this means for anyone paying for GEO tracking

Most AI-visibility and GEO platforms sold to software marketers today build their citation reports from exactly the kind of proxy data this paper tested: ranking pulls, API calls, or both. If a vendor can't say plainly whether its citation numbers come from a real, signed-in ChatGPT session or from an API simulation, this research is a reason to ask before renewing.

That distinction matters most for two decisions marketers actually make with this data: where to spend content budget, and what to tell leadership about progress. A proxy-based report that overstates or understates an exact-page win can send a content team chasing the wrong URL, while a renewal-season chart that looks flat may just reflect a tool's blind spot rather than a real plateau in visibility.

• Ask whether citation data is pulled from live ChatGPT sessions, the Responses API, Google rankings, or a blend of the three
• Ask how often the vendor re-validates its methodology against a new default model
• Read exact-URL citation counts as directional, and weight hostname-level presence more heavily
• Expect month-to-month swings around model changes, not just content changes, before reacting to a drop

The timing compounds the problem: the underlying data runs through July 2026, months before GPT-5.6 Luna became the default for ChatGPT's Free and Go tiers and GPT-6 began reaching paid plans. The volatility is not theoretical — Search Engine Journal reported Sistrix figures showing German-language ChatGPT citations swinging as much as 47% day to day around the GPT-5.5 rollout, against a typical 1% to 2% baseline. A number that looked stable last quarter can mean something different after a silent model swap, whether or not a vendor's dashboard flags it.

None of this makes AI-visibility tracking worthless. Domain-level presence still correlates with being cited, and the researchers found their combined proxy signal predictive enough to serve as a leading indicator even without exact-page matches. What it does mean is that treating any dashboard's precise citation count as ground truth is the wrong way to read it. For a category of tools increasingly priced as a replacement for rank tracking, disclosing methodology is becoming the real differentiator between vendors, not the size of the number on the report.