AI Pick Check asks the assistants through their official APIs. A customer's buyer types into an app or a search page. Those are two different doors into what looks like the same assistant. This post explains the gap, shows what the evidence says, and says plainly what it means for our numbers.
The short answer
An API gives you a model plus a search tool, set up by whoever calls it. An app adds its own instructions, settings and product logic on top. So the answers can differ, sometimes a lot. Our results are close in kind to what buyers see, not identical.
What an API call is
When we call an AI company's API, we pick the model and switch web search on. OpenAI's guide, for instance, says the model can choose whether or not to search based on the prompt, and that search is optional unless the caller requires it. It also says one of its settings does not guarantee a specific number of sources or citations.
The app has settings we do not see. We can name the model we call. We cannot always say which model, instructions and search set-up sit behind the app.
What the Surfer study measured
Surfer, a company that sells AI visibility tracking, collected answers on 4 August 2026. It compared two ways of getting them: scraping the consumer products and calling the APIs. It used 1,000 prompts across five products: ChatGPT, Perplexity, Gemini, Google's AI Mode and AI Overviews. That produced 13,779 answers.
Surfer reports, among other things:
- The brands named overlapped only partly. By its overlap measure, the figure was between 15.5% and 23.8% before cleaning up brand names, and between 21.3% and 31.6% after.
- For ChatGPT, the API answers named 13.8 brands on average, against 7.9 in the app.
- ChatGPT's app answers cited 12.1 sources on average and its API answers 3.1. For Perplexity it was the other way round: 19.5 through the API against 10.27 in the app.
- Web search was triggered for 88% of the prompts scraped from the ChatGPT app and 83% of the ChatGPT API prompts.
- For ChatGPT, only 4.8% of the cited sources matched between the two routes at the domain level.
Surfer's own summary is that roughly 75% of the brand list does not overlap. Its conclusion is: "You cannot reliably track your AI visibility based on APIs." SparkToro's authors also note that whether API results match the variety real users see is only partly explored, with early data.
How much weight to give it
Surfer states the last three limits itself. The first is our reading, and we agree with all four.
- A vendor with a stake. Surfer sells AI visibility tracking and promotes it in the article. Its conclusion favours scraping the apps.
- One sample per prompt. AI answers vary on their own (see AI answers change every time). Surfer says some of the disagreement is not due to the route.
- Models are only roughly matched. Surfer says no vendor exposes the exact model its product serves, so it used the closest purchasable equivalent.
- Some comparisons are borrowed or incomplete. For example, the AI Overviews result reuses the AI Mode API, and one AI Mode method covered 813 of the 1,000 prompts.
Even so, the direction of the finding is hard to wave away. An API result and an app result do not agree on every rule.
How AI Pick Check handles it
- We say how we ask. A report lists, for each assistant, the model and how we reached it, and says the answers come from official APIs with web search on.
- We do not scrape the apps. We use APIs the way any customer would, and we do not pretend they are the apps.
- We label the odd route. Our Gemini answers come from Google's Gemini model, reached through Perplexity's Agent API and searching with Perplexity's web search, not Google Search. The report states that route next to Gemini. The Gemini app, which searches with Google, can answer differently.
- We leave out what we do not measure. We do not measure Google's AI Overviews or AI Mode, or Microsoft Copilot. Four assistants are asked, and no more.
- We compare like with like. Because we ask the same way each time, a re-run is a fair test of whether something changed, even when the absolute level differs from the app.
What to do with this
Treat our numbers as a steady yardstick and not as a mirror. Use them to see whether you are named more or less often than before, and which sources and competitors the assistants lean on. Then check the apps yourself. Our guide on asking the assistants yourself shows how.
What this won't do
It will not tell you what any one buyer saw in an app. We cannot see their history, location or settings. The gap between API and app is real, and it is not the same size for every assistant. We have not measured it ourselves, and we do not claim a number for it. If your decision depends on one app, test that app directly.