AI answers change every time you ask

If you have asked an AI assistant "who is the best for X?" more than once, you may have seen the list change. A study published in January 2026 put numbers on that. This post reports what it found, what we take from it, and what it cannot tell us.

The short answer

The same question, asked again, rarely gives the same list of businesses. It gives an even less stable order. So one answer is a poor measure of whether you are recommended. How often you appear across many answers is a better one.

What the study did

SparkToro published it on 28 January 2026. The author is Rand Fishkin of SparkToro. The article says his friend Patrick O'Donnell joined Gumshoe.ai, an AI tracking start-up, and that Gumshoe took part. Fishkin flags the possible conflict of interest in the article.

As SparkToro reports it:

• 600 volunteers ran prompts in November and December 2025.
• There were 12 prompts, each run through three tools: ChatGPT, Claude, and Google's AI Overview (or AI Mode when no Overview appeared). That came to 2,961 runs in total.
• Each prompt was run "60-100X" per tool.
• Volunteers copied and pasted the answers into survey forms. The team then normalised each answer into an ordered list of brands.

What it found

SparkToro states the main result like this: there is a "<1 in 100 chance" that ChatGPT or Google's AI, "if asked 100X", gives the same list of brands in any two responses. For the same order, it says it is "more like 1 in 1,000 runs". Claude, it says, is "slightly more likely" to give the same list twice, but even less likely to give it in the same order.

A second part looked at people. The study collected 142 prompts written by people for the same headphone request. SparkToro reports their semantic similarity as 0.081 and compares that to a pair of prompts as different as Kung Pao Chicken and Peanut Butter. In other words, real buyers word the same need very differently.

Despite that, some brands kept showing up. Across 994 AI responses to those headphone prompts, brands including Bose, Sony, Sennheiser and Apple appeared 55% to 77% of the time.

The author's conclusion is that visibility, the share of answers that name a brand across many prompts and runs, is "a reasonable metric". He says any tool that gives a "ranking position in AI" is "full of baloney".

What this means for measuring

We read it in three ways, and they shaped our method.

1. Never judge on one answer. That is why we ask every question more than once. See why we ask every question twice.
2. Count how often, not where. We report how often a business is named and show a range. We do track whether a business is in the first three names, but we treat it as a rate, not as a fixed rank.
3. Use several wordings. Our audit asks each buyer need in three wordings, because real people do not word things the same way.

If you want to try this by hand, Gumshoe's own post gives 97 runs of one prompt on one model to land within 10 percentage points, and 385 runs to land within 5. That is its calculation, at 95% confidence and assuming the worst case. It also shows why a handful of manual tries cannot settle anything.

Limits of the study

The authors are open about these, and we would add our own reading.

• Not peer-reviewed, and the authors say more is needed. They say they are not professional researchers or credentialed data scientists, and that they need "far larger sample sizes, thousands more prompts, bigger user panels".
• A vendor was involved. Gumshoe.ai sells tracking. That does not make the result wrong, but it is worth knowing.
• Uncontrolled settings. Volunteers used their default or personalised settings. The study did not make everyone use the same country or device, and it did not adjust temperature. That reflects real life, and it also means some of the variation may come from personalisation, not from the AI itself.
• Old models. The runs were in late 2025. SparkToro notes that newer models may behave differently.
• Only a few topics. Twelve prompts cannot speak for every market. SparkToro itself says other headphone categories gave very different answers.
• API versus app only partly explored. Whether API results match the variety real users see was only touched on, with what SparkToro calls early data.
• No view on what works. The study says nothing about how to get named more often.

What this won't do

It will not tell you whether your business is recommended today, or what to change. For that you need your own measurement, with enough answers to give a range, run in the same way each time. Even then, a number from any tool, ours included, is an estimate and not a promise of what any one buyer will see.