Most “we got you cited in ChatGPT” claims would not survive the control group I used to demand for ad copy tests. One screenshot of your brand in an answer proves nothing: ask again in an hour and you may get a different answer. To find out whether AEO work caused a citation gain, you need treated prompts, untouched prompts, and enough repeats to see through the wobble.

Here is the problem in numbers. Ask the same assistant the same question twice and it repeats only about 51% of cited sources on average. The noise is not equal everywhere: a second ask repeats about 72% of sources on Perplexity but only about 32% on ChatGPT. ChatGPT needs more runs per prompt before you believe a lift. These engines produce variable answers, so a single before-and-after read is sampling error wearing a success-story costume.

This is my lab-notebook version of the fix. Pick 10 to 20 real buyer prompts. Change content for half, leave half alone, and run every prompt several times in ChatGPT and Perplexity before and after, on a fixed schedule. Then compare the change in the treated group with the change in the holdout. Here is what to set up, what to log, and what either result should change.

Why one screenshot cannot carry the claim

I used to tell clients that a winning ad test needed 100 conversions per variant before I would call it. I was wrong about the exact number, but right about the instinct: no control, no claim. A causal claim requires a counterfactual. You apply a change to one group, withhold it from a comparable group, then look at how far the groups move apart. Without that, you have association. The vendor changed a page, ChatGPT cited you once, and everyone takes credit for what might be Tuesday’s sampling noise.

AI answers make that problem worse. Only about 30% of brands stay visible from one AI answer to the next and about 20% across five runs, so your baseline moves even when you touch nothing. Repeated runs show you how much it moves. The holdout shows you whether movement also happened where you made no change. If untreated prompts bounce as much as treated ones, you have not demonstrated a lift. That nothing could save you a retainer.

Lab notebook showing two prompt groups, with one marked as an untouched holdout

Setup: freeze the prompts before you edit a page

Start with 10 to 20 buyer prompts in fixed wording: sentences a customer would type, not phrases invented to make your new page look good. Build the panel across three kinds of intent:

  • Brand: “best [your category] for [job]”
  • Commercial: “[your type] vs [competitor type] for [job]”
  • Problem: “how do I fix [pain you solve]”

That mix matters because ChatGPT names brands far more often than it links. You need prompts where a bare mention still means something to the business, alongside prompts likely to surface a source. Lock the wording in a sheet. Do not improve a prompt halfway through the test because its first answers were disappointing. If you are still guessing which prompts trigger relevant answers in your category, run The Prompt-Trigger Test: Did Your Page Earn the AI Citation? first. A holdout built on dead prompts measures nothing.

Next, pair prompts by intent and split each pair between treatment and holdout. Say you have four plumber comparison prompts: two go to treatment and two stay untouched. Do the same for pricing prompts and best-of prompts. Before assigning them, note which existing page or planned content change each prompt maps to. If both sides depend on the same page, editing that page will contaminate your holdout; pick a different pair or a narrower change. Treatment gets the content change. Holdout gets nothing. That split is the test, not paperwork you tidy up afterward.

Keep the testing conditions consistent within each platform: logged out, the same ChatGPT model and Perplexity setting, a fresh session for each run, and the same day-of-week pattern for both readings. Write those settings down before you start. If you change the question, the setting, and the page together, a better answer will not tell you which change mattered.

Baseline: repeat each prompt until you can see the wobble

For the baseline, run every prompt five times in ChatGPT and three times in Perplexity, logged out and in fresh sessions on the same day. The stability gap above is why I give ChatGPT the extra runs. A 2-in-5 result should not become a victory speech just because the next pass gives you 3-in-5. Trackers run each prompt multiple times to produce a citation-rate percentage rather than treating one answer as a pass or fail. I laid out the repeat-count logic in Run the Prompt 20 Times: A Protocol for Testing AI Visibility. Record rates, not screenshots.

With 10 treatment prompts, that is 50 ChatGPT runs and 30 Perplexity runs for treatment, and the same again for 10 holdouts. Score each run the same way. Your baseline also tells you whether the matched groups start in roughly the same place. If treatment sits at 10% citation rate and holdout at 12% before you edit anything, that closeness makes the later comparison more useful. If one side already dominates, revisit the pairing before you ship. Do not call an uneven starting line an experiment and hope the arithmetic fixes it.

Intervention: make one change, on one side

Choose one change and apply it only to content mapped to the treatment prompts. The options in this test are:

  • An answer-first block at the top of a page, stating the fact in 40 to 60 words.
  • A comparison page that names your category against the alternative.
  • A fix to entity facts, such as name, location, hours, or price range, so they agree everywhere you make the change.
  • A third-party mention on a page the engines already cite.

Pick one, not the whole list. If you ship all four in the same week and citations rise, you cannot tell which intervention mattered. Neither can the vendor sending the invoice. For this protocol, also leave pages mapped to holdout prompts alone. A broad rewrite that answers both groups’ questions is not a treatment-only change just because you label it one in the sheet.

My expectation is that a comparison page or an entity fix may move faster than a generic rewrite. The mechanism is the reason for that expectation: the engine has a short passage tied to a specific question or a consistent fact it can retrieve and attribute. That is a hypothesis, not a result I am reporting. Write down the publication date, request indexing in Search Console, and leave the holdout content untouched until the window closes. One treatment. One date. Everything else frozen.

For each prompt, log three outcomes because they answer different questions. Citation rate is the share of runs where your page URL is linked. Mention rate, in this sheet, is the share where your brand is named without a link. Share of voice records how often you appear relative to named competitors. Do not merge those into a single “AI visibility” score: ChatGPT names brands far more often than it links. Links alone can miss a useful mention; mentions alone cannot prove your page earned a citation. Read the text. Count the links separately.

Give each run one row. Log the date, engine, prompt ID, treatment or holdout, cited yes or no, mentioned without a link yes or no, position in the answer, whether you were cited, recommended, or compared, the cited URL, and the competitors that appeared. I also note sentiment in three words or fewer. “First paragraph,” “middle,” or “list item” is enough for position; you need a consistent record, not a literary review of every answer.

After the window closes, reduce those rows to rates by prompt, group, and engine. If a group has 14 citations in 40 runs, its citation rate is 35%; four in 40 is 10%. That gap is worth examining only alongside each group’s baseline and the matched prompts behind the totals. The repeat-run approach also tracks mention rate, answer position, and cited URL. One good run is gossip. A persistent gap across matched pairs is evidence.

Kitchen timer on a stack of logged AI answers, marking the waiting period

Waiting window: let the bots set the schedule

Publish the change, then wait at least three weeks before the after-read. Rerun the exact baseline: same prompts, run counts, logged-out conditions, and day-of-week pattern. Recrawl lag is why the window matters. The median background re-read is 5.5 days for Perplexity and 14 days for ChatGPT, measured across 17,850 page-bot pairs. A crawl is a floor, not a guarantee that an updated page has made it into the answers you see. Week one tells you little; week two is early for ChatGPT. Week three is the first reading I would use to judge the change.

Fix the calendar before publishing: baseline in week zero, ship in week one, ignore week two, score in week four. If you cannot resist a mid-window spot check, label it as one and keep it out of the scored comparison. Follow the monthly-window approach trackers use for trends rather than promoting a good Tuesday into a case study. Do not move the finish line to the first answer you like.

Reading the result: subtract the holdout’s movement

Calculate the change from baseline for treatment, then calculate it for holdout. The comparison that matters is treatment’s change minus holdout’s change. A treatment rise on its own is not the result. The same engine behavior, news cycle, or newly visible competitor may also have moved the untreated prompts.

A clear win looks like separation across repeated runs and matched pairs: treatment prompts gain citations or mentions while holdout prompts stay near baseline. That supports crediting the content change, provided the holdout remained untouched. Check the cited URLs before you say so. If a supposedly untreated prompt starts citing the treated page, the groups are no longer separate enough to make the clean claim you planned.

A null is useful too. If both groups rise together, or neither moves, the edit has not demonstrated a gain over the holdout. Do not renew work on the theory that a screenshot outranks the comparison you agreed to run. If both groups shift, note any model or interface change you observed, check whether a competitor published or earned coverage, and confirm whether the cited URL is the new page rather than an older one the engine rediscovered. You may not be able to identify the other cause. You can still refuse to assign the lift to your edit.

The middle needs another read. Perhaps treatment gains on Perplexity but not ChatGPT. Perhaps mentions rise without links. Perhaps one matched pair carries the whole gap while the others sit flat. Those are different outcomes, not one win with inconvenient footnotes. When only half the evidence moves, repeat the after-read the following week instead of arguing the first one into a success. Keep the original result in the log. A second reading should test whether the pattern holds, not replace the answer you disliked.

Who should skip the holdout

If you cannot list 10 real buyer prompts without inventing them, skip the holdout and fix coverage first. A business with three service pages and no comparison content has little panel to split. Repeating a prompt nobody types can return clean zeros that prove nothing. The same applies if your category barely triggers cited answers yet, or if your site is weeks old and unindexed. Publish the answer-first, pricing, comparison, or entity information the relevant prompts need; test once the engines have something to retrieve. A holdout measures lift. It does not create baseline visibility from thin air.

Run this protocol the next time someone shows you a screenshot with your logo in it, whether that someone is a vendor or your own optimism. If treatment pulls away from an intact holdout across repeated runs, keep the change and consider rolling it to the holdout prompts. If both groups move together or neither moves, do not pay for a citation gain the test failed to show. I would rather run one honest holdout that kills a weak tactic than renew three months of work on the strength of a screenshot. Cause, then effect. Everything else is storytelling.