Track AI recommendations without fooling yourself

The short answer

Do not trust one answer. Ask the same question many times, count how often you are named, and write that as a rate: "named in 6 of 20 answers". Then remember that a small count can be off by a lot. Treat a change as real only when it is bigger than the usual wobble.

What the research says about variation

SparkToro ran a study with Gumshoe, which it describes as an AI-tracking startup. SparkToro's author discloses that a co-author joined Gumshoe and acknowledges a possible conflict of interest. Roughly 600 volunteers ran 12 prompts on ChatGPT, Claude and Google's AI results, giving 2,961 runs in November and December 2025.

SparkToro reports:

• A chance below 1 in 100 of two identical lists of brands in 100 runs, for ChatGPT and Google's AI. Claude was slightly more likely to repeat a list.
• The same order about once in 1,000 runs.
• Lists that ran from two or three items to ten or more.
• A view that a "visibility %" across many runs is a reasonable metric, while a tool that reports a "ranking position" cannot be trusted.

The authors also list limits. They say they are not professional researchers or credentialed data scientists, they invite others to repeat the work, volunteers kept their default settings, and newer models may behave differently. It is a blog post, not a journal paper. Read it as a strong hint, not a law.

What that means for you

• A single answer that names you is nice. It is not proof.
• A single answer that leaves you out is not proof either.
• Your position in a list moves around too much to track closely. Count whether you appear.

Step 1: count named answers, not impressions

For every question, ask it several times in fresh chats and record each answer. Then work out:

share named = answers that name you / answers you collected

Do this per question and for the whole set.

Step 2: understand the margin of error

A margin of error says how far the true rate could be from the one you measured, because you only looked at a sample.

Gumshoe publishes its own sums, at 95% confidence and assuming the worst case (a true rate of 50%). For one prompt on one model, it says you need 97 answers to be within 10 percentage points, 385 to be within 5, and 9,604 to be within 1. These are one company's figures, so treat them as a guide.

A small example worked out the way our reports do it, with a Wilson interval, which behaves well on small counts. Say you were named in 6 of 20 answers, a rate of 30%. The 95% range is about 15% to 52%. So "30%" could really be anywhere from a little under a sixth to about half. If next month you are named in 9 of 20 (45%), the ranges overlap heavily. That is not a real change yet.

The rule of thumb: if your two ranges overlap, call it "within normal variation". This is cautious. Our reports use a two-proportion test, which can call a change real in a few cases where the ranges overlap a little.

Step 3: spread the questions out

Gumshoe's article also shows that mixing many prompts, wordings and models gives a tighter overall result than repeating one prompt. In its sums, a margin of 5 points needs 385 answers for one prompt, but about 39 per prompt across 10 prompts and about 8 per prompt across 50. That tighter figure is for the whole set. If you look at one question alone, you are back to a small sample.

A tracking sheet you can copy

One row per question per assistant per month.

month | assistant | question id | answers collected | times named | share named | low end | high end | top competitor | notes

Then a summary row for each month:

month | total answers | total named | share named | low end | high end | change vs last month: real / within normal variation

Fill "low end" and "high end" using a free Wilson interval calculator, or leave them blank and just be careful with small counts.

How to read a change

1. Compare the ranges, not the single numbers.
2. If the ranges overlap, write "within normal variation" and do nothing dramatic.
3. If they do not overlap, you probably have a real change. Write down what you changed and when.
4. Wait for a second month before you act on a surprise.

How to check it worked

• Each question has at least two answers per assistant, and a total you can state.
• Your summary shows a range, not only a headline number.
• You can say, for any change, whether it is real or within normal variation, and why.

What this won't do

Tracking cannot tell you what caused a change. A model update, a new article about a competitor or a change in how search is used might each move the rate. And a free hand-run sheet will rarely have enough answers to detect small changes.

It also cannot promise that more answers will name you. It only tells you honestly where you are.

Our reports ask each question twice on each assistant, show a range with every rate that has enough answers, and label a re-run result as real or within normal variation. You can read why in our post on asking every question twice.