LinkedIn
Your AI visibility score is not a score
One prompt, one run, one screenshot is not AI visibility measurement. It is a sample dressed up for a budget meeting.
Picture the slide: your brand has an “AI Visibility Score” of 43% across ChatGPT, Gemini and Google AI Overviews. A vendor assembled 30 prompts, ran each one through an API once, found your name in 13 answers and divided by 30. A rival sits at 65%. The executive team approves another $4,000 a month to close the gap.
That budget is chasing a number the vendor has not earned. If a media buyer evaluated an ad account from one impression on a Tuesday morning, I would revoke their access before lunch. Yet AI visibility trackers routinely treat one batch of answers like a 2015 rank report. They can tell you what appeared in that batch. They cannot tell you that buyers see your brand 43% of the time.
Run the prompts again and the answers may change. In empirical testing across category questions, researchers found that within-prompt resampling alone accounted for 34.8% of output variance, even at a low decoding temperature. A change in your score does not, by itself, mean a competitor published better content or your agency’s schema edits worked. It may be sampling noise with a logo on top.
Why the usual visibility score lies
The answer changes even when the prompt does not
Traditional rank tracking starts with a relatively stable object: a results page assembled from an index. Search results can change, but a tracker can return to the same query and compare positions. An LLM answer is not a position on a shelf. The model generates text token by token, drawing from probabilities; interfaces that retrieve live material add another moving part.
That distinction matters most when the question asks for judgment rather than a simple fact. In tracking of brand outputs under identical prompts, only 37% of featured brands remained consistent across repeated runs. Simple factual queries diverged by 5% to 10%; complex recommendation and evaluation queries showed 40% to 60% output divergence. A weekly slide that drops from 55% to 30% cannot tell you whether your standing degraded unless the tracker first establishes how much the answer normally moves.
Repeat the identical prompt before calling a change a trend.
Buyers do not all ask the same sentence
Suppose a buyer wants warehouse software. One asks for the “best warehouse management software for mid-market retail.” Another wants “top-rated WMS tools for 50-person logistics teams.” A third asks what to buy after outgrowing Fishbowl. A tracker that tests only the first sentence does not measure the whole buying conversation.
Prompt-sensitivity audits put the error margin from evaluating a single query variant at ±40%. Swapping “best” for “top” changed featured vendors by 15% to 30%; intent modifiers changed the list by 40% to 60%. Those are not cosmetic edits. They alter what the buyer is asking the model to solve.
Test variants of the buyer’s problem, not your favorite wording of it.
A blended engine score hides the useful result
Commercial trackers often roll ChatGPT, Gemini, Perplexity and Google AI Overviews into one composite. That makes a clean chart, but it buries which interface actually produced the mention. The systems have different models and retrieval behavior. In an analysis of 300 business software queries across AI interfaces, broad queries showed 70% recommendation variance across platforms. Google AI Overviews also showed greater run-to-run volatility than the standalone chat models in that analysis.
If one engine recommends you and another does not, an average will conceal the difference you need to investigate. Keep results separate by engine. A composite can wait until the underlying answers make sense.
The test I would run before moving a dollar
When I managed search accounts by hand, I did not kill an ad group after three impressions. I waited for volume, checked what else had changed and decided what result would justify an action. AI visibility needs the same operator discipline. Here is the question: does your brand appear consistently in unprompted commercial answers, or did one batch happen to include it?
This is a prospective test, not a claim that I ran your prompts. Set it up before you open the dashboard.
Set up a panel that does not give away the answer
Remove your brand name from every measured query. A prompt such as “How does [Brand Name] compare to Competitor X?” invites the model to repeat your name, then rewards the tracker for spotting it. In one documented audit, a client’s reported 13% visibility came entirely from a prompt that named the company; its unprompted visibility across commercial category searches was zero.
Build a panel of 20 to 30 unbranded buyer prompts from sales conversations, or use a structured starting point such as our 60 AI search prompts sorted by buyer intent. Include problem-aware questions, category comparisons and high-intent questions with specific constraints. Write down which group each prompt belongs to. Do not quietly replace the panel next month because the first version produced an ugly score.
Hold the setup steady, then repeat the runs
For each engine you choose to track, run every prompt 3 to 5 times within a tight testing window. Log the engine, exact phrasing, run number and full answer. Keep the setup consistent between runs and testing dates so a change in your own procedure does not masquerade as a change in visibility.
Twenty iterations per prompt can produce cleaner confidence intervals, as our AI visibility test protocol explains. But a variance decomposition found sharply diminishing returns beyond five repeats per prompt-engine cell. For a lean audit, I would spend the next unit of effort on another buyer phrasing or engine, not another identical run.
Add controls that should stay dark
A PPC operator uses negative keywords to catch irrelevant traffic and checks that test transactions do not count as real conversions. Give the prompt panel the same sanity check: add five to ten control queries where your brand should not appear.
If you sell B2B accounting software for medical clinics, controls might ask about oil-and-gas logistics, residential landscaping invoicing or freelance design templates. A brand mention there is not evidence of astonishing reach. It is a reason to inspect the answer and your testing pipeline. If controls start lighting up, do not celebrate the visibility score. Fix the test.
Measure what the model actually did
“Visible” is too blunt a label. For every answer, record whether your brand was:
- Recommended as a vendor solution.
- Mentioned without a recommendation.
- Cited through a linked source.
- Absent.
Keep the citation field separate; an answer can cite your content while recommending someone else. In an audit of 1,796 commercial category queries, Asva AI reported that its domain was cited 142 times but recommended as a vendor zero times. Crunchjunkie likewise found cases where ChatGPT cited a company’s technical post to explain a problem, then recommended a competitor. A source citation is not a buying recommendation. Count both, but do not sell one to leadership as the other.
Decide what would change your mind
Before comparing months, write down the threshold that would make you investigate or act. My practical rule for this panel: treat a shift of less than 15 percentage points, or one that does not persist across two monthly testing windows, as insufficient grounds for a budget move. That is an operating guardrail, not a universal confidence interval. A different panel may need a different threshold; the important part is choosing it before you see the result.
I expect broad recommendation prompts to move around more than tightly constrained ones because a vague request leaves more plausible vendors in play. I also expect repeats to expose some apparent wins as ordinary variation. If the recommendation frequency rises beyond your threshold, persists on the next check and does not appear in the controls, investigate what changed and whether it reaches pipeline. If it does not, leave the budget alone. Our guide to measuring AI search visibility without trusting one screenshot goes deeper on that decision.
Find the buyer constraints that bring you into the answer
Broad prompts such as “best project management software” are the AI-answer equivalent of broad-match head terms. They attract well-funded incumbents and give the model little reason to pick a specialist. In the 300-prompt analysis, broad queries showed 70% recommendation variance; heavily constrained queries reduced it to 25%.
Mine closed-lost notes and sales transcripts for the friction buyers mention near a decision: headcount limits, compliance requirements, legacy software migrations and integrations. Turn those into unbranded prompt variants. You are not trying to coax the model into saying your name. You are testing whether it recommends you when a buyer describes the problem you actually solve.
That distinction also tells you what to do with a weak result. If your brand never appears for a broad category term but appears repeatedly for a relevant constraint, do not flatten those observations into one “share of voice” score. The constrained question shows where your case is credible. Report visibility by buyer problem, not just by category.
The in-house version: one spreadsheet, one afternoon
You do not need a $500-a-month subscription just to learn whether the same prompts produce the same answers. A single spreadsheet and a two-hour monthly audit can give a small team a usable starting point. Track seven fields:
- Date
- Exact prompt phrasing
- Buyer intent stage
- Engine
- Run number
- Recommendation status: Recommended, Mentioned or Absent
- Footnote URL, if present
Keep the full answers as well. Without them, you cannot check whether your labels captured what the model said. On the first Tuesday of each month, run the same 25 core prompts through each chosen engine five times. For each prompt and engine, calculate Recommendation Frequency: runs that recommended you divided by total runs. Calculate Citation Rate separately: runs with a linked source divided by total runs.
Basic OpenAI or Google Gemini API scripts can automate execution if you want them to. Twenty-five prompts at five runs require roughly 125 calls per engine. Automation saves copying; it does not decide whether a mention is commercially useful or whether a change clears your threshold. Preserve the raw text and the historical log. A colored area chart is not an audit trail.
Pay for action, not another alarm
Marketers buy software for problems that require operational discipline. Agencies charge retainers to watch dashboards change color. Paying hundreds a month for a tool that pings an LLM, plots the answers and sends alerts to an unread Slack channel is an expensive alarm clock.
An alert about a missing citation does not make a pricing table easier to find, clear a blocked crawler or restructure paid ad groups around unbranded demand. Those are execution problems. Monitoring earns its fee only when someone can act on a credible signal.
That is the operating philosophy behind groas: paid search, SEO and AI discoverability belong in one continuous execution system. Specialized models adjust bids, refine landing page assets and expand query coverage around the clock, while a named human strategist owns direction and commercial guardrails. The point is not to give the team another probabilistic chart to decipher. It is to connect what the team learns to work that can affect revenue.
Hand back the single-run report
The next time a vendor puts an AI visibility score in front of you, ask how many runs produced it. If the answer is one, hand it back. If your brand appears in 7 of 20 answers, you observed 35% in that sample. You have not established that buyers see you 35% of the time, or that a later 30% means you lost ground.
Report results by prompt group and engine. Show repeated-run frequencies, keep citations distinct from recommendations, and watch the trend alongside downstream pipeline. Then move money only when a change clears the threshold you set before looking.
Ignore that discipline and you will spend your marketing budget optimizing for whichever answer the model happened to give on Tuesday.