Ask an AI assistant the same question twice and you may get two different lists of businesses. That is not a fault. It is how these systems work, and it changes how a fair measurement has to be done.
One answer is an anecdote
Imagine you ask once and your business is named. You might conclude that you are doing well. Ask again and you may not be named at all. The first answer was real, but it was one draw from a range of possible answers.
An outside study points the same way. SparkToro and Gumshoe.ai had volunteers run the same prompts again and again. SparkToro puts the chance that ChatGPT or Google's AI gives the same list of brands in any two responses at under 1 in 100. We go through what they did, and where the study is thin, in a separate post.
So we never treat a single answer as a result. We look at how often a business is named across many answers.
What the paid audit does
The paid audit asks 20 questions on 4 AI assistants. It asks each of them twice. That makes 160 answers. From those we count how often your business is named, how high on the list, and how often your own website is cited as a source.
Those 20 questions are 5 buyer needs, each worded 3 ways, plus 5 questions about your own facts. The different wordings matter. Real people word the same need in very different ways, so one "perfect" question tells you little.
The free check is deliberately smaller: 5 questions on 2 assistants, once each. It is a first look, and we label it that way. Its ranges are wide, and the report says so.
Why twice, and not a hundred times
Honestly, twice is a compromise between cost and certainty. More runs give a tighter picture. They also cost more for every audit. We chose a number that gives a usable range for most businesses at a price a small firm can pay, and we show the range instead of hiding it.
If a rate rests on fewer than 5 answers, the report does not show it and says "too few answers to say". A re-run adds more answers.
Why we show a range
A rate worked out from a small number of answers is uncertain. As an example we worked out ourselves: with 30 answers, a measured rate of 10% (3 answers) is consistent with a true rate anywhere from about 3% to 26%. We use the Wilson score interval, a standard way to put a margin of error on a proportion. The same method is used for every rate in a report.
That is why a score comes with a range. If the range is wide, the honest reading is that we need more answers before we can say much.
What "real change" means
When you fix something and re-run the audit, the numbers will move a little even if nothing real changed. For each area we run a statistical test (a two-sided, pooled two-proportion z-test) on the answers from before and after. We call a change real when the test says it is bigger than normal variation. Otherwise we say it is within normal variation. The report also shows whether the two ranges overlap, as a second look.
If you re-run before a fix has had time to show, we label the area "too early to tell", because changes can take days or weeks to reach AI answers.
This is stricter than comparing two numbers. It means we will sometimes tell you that a change is within normal variation. We think that is more useful than a number that goes up and down for no reason.
What we cannot see
We ask the assistants the same way every time, through their official APIs, with web search on. The apps that people use add their own history, location and settings. So what a person sees can differ from what we measure. Our results are a fair, repeatable sample, not a photograph of every conversation. Another post explains that gap in detail.
We also cannot say what share of buyers act on any answer. We measure what the assistants say, not what people do next.