How we measure AI recommendations

In short

• We ask 20 questions: 5 buyer needs in 3 wordings each, plus 5 questions about the business's own facts.
• We ask 4 AI models through their official APIs, with live web search on, and ask each question twice: 160 answers in all.
• The AI Pick Score (0 to 100) weighs how often you are named, how often you are in the top 3, how often your own site is cited, and how often the facts about you are right.
• Every rate comes with a 95% Wilson interval, and the score comes with a range built from those intervals.
• A re-run calls a change real only when a statistical test says it is bigger than normal variation.

The questions

We start by reading the business's home page, and its about page if it links to one, from the raw HTML (no JavaScript). One call to Anthropic's claude-haiku-5-5, with no web search, turns that text into a short profile: name and other names, category, what is on offer, the audience, likely competitors and the facts the site states, such as prices or locations. The customer can correct all of it before a paid audit starts.

From the profile, a second call writes the questions:

• 5 buyer needs, each in 3 wordings. These are questions a buyer who does not know the business yet would ask, such as which provider to choose for a need. Different wordings of the same need catch answers that depend on phrasing.
• 5 fact questions. These do name the business, and ask about things its own site states, so we can see whether the assistants get them right.

Every question is checked before use, and the set is written again once if any check fails. Buyer questions must not contain the business name, any of its other names or its domain. No question may contain a placeholder. Each is 8 to 30 words long, in the right language, and the wordings of one need must differ. Fact questions must name the business.

The free check uses the first wording of each of the 5 needs.

The AI assistants

All answers come from official APIs with live web search turned on. We do not scrape consumer apps.

About the Gemini route. We reach Google's Gemini model through Perplexity's Agent API. The model searches with Perplexity's web search, not Google Search grounding. So these answers show how the Gemini model recommends businesses from what Perplexity's search returns. The Gemini app, which searches with Google, can answer differently. Every Gemini answer in a report is labelled with this route.

The free check uses OpenAI and Google Gemini.

Runs and answers

AI answers vary. The same question asked twice can name different businesses in a different order. So the full audit asks each question twice on each assistant: 20 questions × 4 assistants × 2 runs = 160 answers. Of those, 120 are buyer-need answers (30 per assistant), and the rest are fact answers.

The free check asks 5 questions on 2 assistants once: 10 answers. That is a first look, and its ranges are wide.

Each request carries the customer's chosen country as an approximate location wherever the API accepts one. We store every answer with its text, citations, model, date and time, so you can read exactly what was said.

Reading the answers

• Named. The answer text names the business (alias matching, case and accent insensitive, whole words, the domain and its subdomains count).
• Top 3. The business is among the first three brands the answer names. Counted over the same need answers as mention.
• Own site cited. One of the answer's citations is on the business's own domain.
• Facts right. A fact answer counts as accurate when the fact check found no statement that contradicts the facts on the business's own site. Only computed when the business has facts on file.

Name matching is done by code. Listing the other brands in order, judging whether each is recommended, merely mentioned or spoken of negatively, and comparing fact answers with the site's facts is done in batches by claude-haiku-5-5 with a fixed output format. It can make mistakes, so every number in a report links to the answers behind it.

The AI Pick Score

The score is a weighted sum of four rates, scaled to 0 to 100:

score = 100 × (0.5 × named + 0.2 × top 3 + 0.15 × own site cited + 0.15 × facts right)

That is 50% for being named, 20% for being in the top 3, 15% for your own site being cited and 15% for facts being right. Named, top 3 and own site cited are measured on the buyer-need answers. Facts right is measured on the fact answers.

A part with no answers (n = 0) is left out and the other weights are scaled up to sum to 1. The free check has no fact questions, so its score uses mention, top 3 and own-site citation only.

The score's range uses the same formula twice: once with the lower end of every rate's interval, once with the upper end.

Margins of error

Each rate is a count out of a number of answers, so it carries sampling error. We show a 95% Wilson score interval, which behaves well for small samples and for rates near 0% or 100%, where the simpler textbook interval breaks down. For k successes out of n answers, with p = k / n and z = 1.96:

centre = (p + z²/2n) / (1 + z²/n)
half-width = z × √(p(1 − p)/n + z²/4n²) / (1 + z²/n)

Example. Named in 12 of 30 answers on one assistant is 40%, with a likely range of 25% to 58%. The same 40% over all 120 buyer-need answers narrows to 32% to 49%. That is why per-assistant numbers have wider whiskers than the overall ones.

Below 5 answers we do not show a rate at all. We say "too few answers to say" instead.

Site checks

These cost nothing in AI calls and run in every check:

• robots.txt, read with the correct group rules: a crawler that has its own group ignores the * group.
• Crawler access. We load the home page with the user agent of each key AI crawler, to catch firewalls or CDNs that block them even when robots.txt allows them.
• Raw HTML. Are the name, the offer and the price in the page without JavaScript? Many crawlers do not run scripts.
• Structured data. Is there JSON-LD, does it parse, and does it describe the organisation and its products or offers?
• sitemap.xml and llms.txt.

The key crawlers we test are: OAI-SearchBot (OpenAI), ChatGPT-User (OpenAI), Claude-SearchBot (Anthropic), Claude-User (Anthropic), PerplexityBot (Perplexity), Perplexity-User (Perplexity), Googlebot (Google), bingbot (Microsoft). Each was checked against its company's own documentation.

Why llms.txt is not scored

llms.txt is a proposed file that lists a site's important pages for language models. We check for it and report what we find, but it never changes your score or creates a fix. No major AI company says it uses llms.txt, and scoring something with no known effect would send you after the wrong work.

The fix plan

Fixes come from rules applied to the evidence. A model only writes the wording; it does not decide what goes in the plan.

• Access. A robots.txt block, a crawler blocked by a firewall, or key facts that only appear with JavaScript. The fix gives the exact robots.txt lines or settings.
• Facts. An assistant states something that contradicts your site. The fix shows the quote next to the truth, with a correction email and an outline for a facts page.
• Sources. A site cited at least 2 times that lists competitors but not you. The fix names the pages and includes an outreach email.
• Need pages. A buyer need where you are named in 20% of answers or fewer while a competitor is named in 50% or more. The fix is a content brief.
• Comparisons. Answers that name you but rank a competitor above you at least 2 times. The fix is a comparison page.

Each fix has an impact level. For most fixes it comes from the share of missed answers the fix explains: high from 33%, medium from 15%. For a facts fix it comes from the number of wrong statements: high from 3, medium from 2. Each fix also lists the effort, who should do it, the answers behind it, and when a change could show. A plan has at most 10 fixes, and we never pad it: three real fixes beat ten invented ones.

Re-runs and the noise test

A re-run asks the same questions on the same assistants. For each rate we run a pooled two-proportion z-test, two-sided. A change is called real when p < 0.05. Otherwise it is "within normal variation". We also show whether the two intervals overlap. An area is compared only when both runs have at least 10 answers.

Example. Going from 20% to 40% on one assistant (30 answers each time) gives p = 0.091: not enough to call it real. The same jump over all 120 buyer-need answers gives p = 0.001: a real change.

Changes need time to reach AI answers. Areas whose fixes were ticked less than about 7 days ago (access fixes) or 28 days ago (content, facts and sources) are marked "too early to tell". Your report suggests a re-run date from the fixes you tick. The re-run is free once within 60 days.

What we cannot measure

• Consumer apps differ from APIs. ChatGPT, Claude, Perplexity and the Gemini app add their own instructions, search settings and sometimes different models. Our answers come from the APIs and are close to, but not the same as, what someone sees in an app.
• Personalisation. Apps use a person's chat history, memory and settings. We ask as a new user with no history, the same way every time, so results are fair to compare.
• Location. We pass a country as an approximate location where the API allows it. We cannot reproduce a buyer's exact city or what a local search would return.
• Time. Answers reflect the day they were collected. Models and search indexes change, so we record the model and date on every answer.
• Other AI products. We only use official APIs. Products without one that fits, such as Google's AI Overviews and AI Mode or Microsoft Copilot, are not measured.
• Sales. We measure what AI assistants say, not how many buyers act on it.

Version and change log

This page describes methodology version 1.0. Every report records the version it was made with. When the method changes, we add an entry here and raise the version.