---
title: "The AI Citation Holdout Test: Did Your Content Change Cause the Lift?"
description: "Test an AI citation claim with matched prompts, untreated holdouts, repeated ChatGPT and Perplexity runs, and a fixed reading window."
url: "https://groas.com/post/the-holdout-test-how-to-prove-a-content"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/1bb4d7d0-97cb-4b98-88f7-43e6a10f623a.png"
published: "2026-10-09T05:37:20.135Z"
modified: "2026-10-09T05:37:20.204Z"
---

October 9, 2026 · 10 min read

# The AI Citation Holdout Test: Did Your Content Change Cause the Lift?

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Two matched seedlings, one on an expensive fertilizer drip labeled treated, one untouched holdout, grown to the same height: the paid lift changed nothing.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/1bb4d7d0-97cb-4b98-88f7-43e6a10f623a.png)

In this article

1. [Why one screenshot cannot carry the claim](#why-one-screenshot-cannot-carry-the-claim)
2. [Setup: freeze the prompts before you edit a page](#setup-freeze-the-prompts-before-you-edit-a-page)
3. [Intervention: make one change, on one side](#intervention-make-one-change-on-one-side)
4. [Measurement: count links, mentions, and competitors separately](#measurement-count-links-mentions-and-competitors-separately)
5. [Waiting window: let the bots set the schedule](#waiting-window-let-the-bots-set-the-schedule)
6. [Reading the result: subtract the holdout’s movement](#reading-the-result-subtract-the-holdouts-movement)
7. [Who should skip the holdout](#who-should-skip-the-holdout)

Most “we got you cited in ChatGPT” claims would not survive the control group I used to demand for ad copy tests. One screenshot of your brand in an answer proves nothing: ask again in an hour and you may get a different answer. To find out whether AEO work caused a citation gain, you need treated prompts, untouched prompts, and enough repeats to see through the wobble.

Here is the problem in numbers. Ask the same assistant the same question twice and it repeats only about [51% of cited sources on average](https://dev.to/agustaon/how-do-you-measure-something-that-gives-a-different-answer-every-time-55m5). The noise is not equal everywhere: a second ask repeats about [72% of sources on Perplexity but only about 32% on ChatGPT](https://dev.to/agustaon/how-do-you-measure-something-that-gives-a-different-answer-every-time-55m5). ChatGPT needs more runs per prompt before you believe a lift. These engines produce variable answers, so **a single before-and-after read is sampling error wearing a success-story costume**.

This is my lab-notebook version of the fix. Pick 10 to 20 real buyer prompts. Change content for half, leave half alone, and run every prompt several times in ChatGPT and Perplexity before and after, on a fixed schedule. Then compare the change in the treated group with the change in the holdout. Here is what to set up, what to log, and what either result should change.

## Why one screenshot cannot carry the claim

I used to tell clients that a winning ad test needed 100 conversions per variant before I would call it. I was wrong about the exact number, but right about the instinct: no control, no claim. A [causal claim requires a counterfactual](https://www.seosavages.com/glossary/causal-research/). You apply a change to one group, withhold it from a comparable group, then look at how far the groups move apart. Without that, you have association. The vendor changed a page, ChatGPT cited you once, and everyone takes credit for what might be Tuesday’s sampling noise.

AI answers make that problem worse. Only about [30% of brands stay visible from one AI answer to the next and about 20% across five runs](https://www.airops.com/blog/increase-brand-visibility-ai-search), so your baseline moves even when you touch nothing. Repeated runs show you how much it moves. The holdout shows you whether movement also happened where you made no change. **If untreated prompts bounce as much as treated ones, you have not demonstrated a lift.** That nothing could save you a retainer.

![Lab notebook showing two prompt groups, with one marked as an untouched holdout](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/6dd11173-6cb2-467f-b8d7-9b6810ab6a35.png)

## Setup: freeze the prompts before you edit a page

Start with 10 to 20 buyer prompts in fixed wording: sentences a customer would type, not phrases invented to make your new page look good. Build the panel across three kinds of intent:

- **Brand:** “best \[your category] for \[job]”
- **Commercial:** “\[your type] vs \[competitor type] for \[job]”
- **Problem:** “how do I fix \[pain you solve]”

That mix matters because ChatGPT names brands far more often than it links. You need prompts where a bare mention still means something to the business, alongside prompts likely to surface a source. Lock the wording in a sheet. Do not improve a prompt halfway through the test because its first answers were disappointing. If you are still guessing which prompts trigger relevant answers in your category, run [_The Prompt-Trigger Test: Did Your Page Earn the AI Citation?_](https://groas.com/post/the-prompt-trigger-test-find-out-which-b) first. A holdout built on dead prompts measures nothing.

Next, pair prompts by intent and split each pair between treatment and holdout. Say you have four plumber comparison prompts: two go to treatment and two stay untouched. Do the same for pricing prompts and best-of prompts. Before assigning them, note which existing page or planned content change each prompt maps to. If both sides depend on the same page, editing that page will contaminate your holdout; pick a different pair or a narrower change. **Treatment gets the content change. Holdout gets nothing.** That split is the test, not paperwork you tidy up afterward.

Keep the testing conditions consistent within each platform: logged out, the same ChatGPT model and Perplexity setting, a fresh session for each run, and the same day-of-week pattern for both readings. Write those settings down before you start. If you change the question, the setting, and the page together, a better answer will not tell you which change mattered.

### Baseline: repeat each prompt until you can see the wobble

For the baseline, run every prompt five times in ChatGPT and three times in Perplexity, logged out and in fresh sessions on the same day. The stability gap above is why I give ChatGPT the extra runs. A 2-in-5 result should not become a victory speech just because the next pass gives you 3-in-5. Trackers [run each prompt multiple times to produce a citation-rate percentage](https://nationalpositions.com/track-ai-search-visibility/) rather than treating one answer as a pass or fail. I laid out the repeat-count logic in [_Run the Prompt 20 Times: A Protocol for Testing AI Visibility_](https://groas.com/post/run-every-prompt-20-times-a-test-protoco). **Record rates, not screenshots.**

With 10 treatment prompts, that is 50 ChatGPT runs and 30 Perplexity runs for treatment, and the same again for 10 holdouts. Score each run the same way. Your baseline also tells you whether the matched groups start in roughly the same place. If treatment sits at 10% citation rate and holdout at 12% before you edit anything, that closeness makes the later comparison more useful. If one side already dominates, revisit the pairing before you ship. Do not call an uneven starting line an experiment and hope the arithmetic fixes it.

## Intervention: make one change, on one side

Choose one change and apply it only to content mapped to the treatment prompts. The options in this test are:

- An answer-first block at the top of a page, stating the fact in 40 to 60 words.
- A comparison page that names your category against the alternative.
- A fix to entity facts, such as name, location, hours, or price range, so they agree everywhere you make the change.
- A third-party mention on a page the engines already cite.

Pick one, not the whole list. If you ship all four in the same week and citations rise, you cannot tell which intervention mattered. Neither can the vendor sending the invoice. For this protocol, also leave pages mapped to holdout prompts alone. A broad rewrite that answers both groups’ questions is not a treatment-only change just because you label it one in the sheet.

My expectation is that a comparison page or an entity fix may move faster than a generic rewrite. The mechanism is the reason for that expectation: the engine has a short passage tied to a specific question or a consistent fact it can retrieve and attribute. That is a hypothesis, not a result I am reporting. Write down the publication date, request indexing in Search Console, and leave the holdout content untouched until the window closes. **One treatment. One date. Everything else frozen.**

## Measurement: count links, mentions, and competitors separately

For each prompt, log three outcomes because they answer different questions. [Citation rate is the share of runs where your page URL is linked](https://www.airops.com/blog/increase-brand-visibility-ai-search). Mention rate, in this sheet, is the share where your brand is named without a link. Share of voice records how often you appear relative to named competitors. Do not merge those into a single “AI visibility” score: [ChatGPT names brands far more often than it links](https://www.airops.com/blog/increase-brand-visibility-ai-search). Links alone can miss a useful mention; mentions alone cannot prove your page earned a citation. **Read the text. Count the links separately.**

Give each run one row. Log the date, engine, prompt ID, treatment or holdout, cited yes or no, mentioned without a link yes or no, position in the answer, [whether you were cited, recommended, or compared](https://bluejar.ai/blog/track-brand-mentions-ai-search/), the cited URL, and the competitors that appeared. I also note sentiment in three words or fewer. “First paragraph,” “middle,” or “list item” is enough for position; you need a consistent record, not a literary review of every answer.

After the window closes, reduce those rows to rates by prompt, group, and engine. If a group has 14 citations in 40 runs, its citation rate is 35%; four in 40 is 10%. That gap is worth examining only alongside each group’s baseline and the matched prompts behind the totals. The [repeat-run approach also tracks mention rate, answer position, and cited URL](https://www.airops.com/blog/increase-brand-visibility-ai-search/). One good run is gossip. A persistent gap across matched pairs is evidence.

![Kitchen timer on a stack of logged AI answers, marking the waiting period](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/02538c2e-f54c-4fe4-b6f2-72251bd97d1b.png)

## Waiting window: let the bots set the schedule

Publish the change, then wait at least three weeks before the after-read. Rerun the exact baseline: same prompts, run counts, logged-out conditions, and day-of-week pattern. Recrawl lag is why the window matters. The [median background re-read is 5.5 days for Perplexity and 14 days for ChatGPT](https://snorklee.com/blog/en/how-often-ai-crawlers-revisit-your-site), measured across 17,850 page-bot pairs. A crawl is a floor, not a guarantee that an updated page has made it into the answers you see. Week one tells you little; week two is early for ChatGPT. Week three is the first reading I would use to judge the change.

Fix the calendar before publishing: baseline in week zero, ship in week one, ignore week two, score in week four. If you cannot resist a mid-window spot check, label it as one and keep it out of the scored comparison. Follow the [monthly-window approach trackers use for trends](https://nationalpositions.com/track-ai-search-visibility/) rather than promoting a good Tuesday into a case study. **Do not move the finish line to the first answer you like.**

## Reading the result: subtract the holdout’s movement

Calculate the change from baseline for treatment, then calculate it for holdout. The comparison that matters is **treatment’s change minus holdout’s change**. A treatment rise on its own is not the result. The same engine behavior, news cycle, or newly visible competitor may also have moved the untreated prompts.

A clear win looks like separation across repeated runs and matched pairs: treatment prompts gain citations or mentions while holdout prompts stay near baseline. That supports crediting the content change, provided the holdout remained untouched. Check the cited URLs before you say so. If a supposedly untreated prompt starts citing the treated page, the groups are no longer separate enough to make the clean claim you planned.

A null is useful too. If both groups rise together, or neither moves, the edit has not demonstrated a gain over the holdout. Do not renew work on the theory that a screenshot outranks the comparison you agreed to run. If both groups shift, note any model or interface change you observed, check whether a competitor published or earned coverage, and confirm whether the cited URL is the new page rather than an older one the engine rediscovered. You may not be able to identify the other cause. You can still refuse to assign the lift to your edit.

The middle needs another read. Perhaps treatment gains on Perplexity but not ChatGPT. Perhaps mentions rise without links. Perhaps one matched pair carries the whole gap while the others sit flat. Those are different outcomes, not one win with inconvenient footnotes. **When only half the evidence moves, repeat the after-read the following week** instead of arguing the first one into a success. Keep the original result in the log. A second reading should test whether the pattern holds, not replace the answer you disliked.

## Who should skip the holdout

If you cannot list 10 real buyer prompts without inventing them, skip the holdout and fix coverage first. A business with three service pages and no comparison content has little panel to split. Repeating a prompt nobody types can return clean zeros that prove nothing. The same applies if your category barely triggers cited answers yet, or if your site is weeks old and unindexed. Publish the answer-first, pricing, comparison, or entity information the relevant prompts need; test once the engines have something to retrieve. **A holdout measures lift. It does not create baseline visibility from thin air.**

Run this protocol the next time someone shows you a screenshot with your logo in it, whether that someone is a vendor or your own optimism. If treatment pulls away from an intact holdout across repeated runs, keep the change and consider rolling it to the holdout prompts. If both groups move together or neither moves, do not pay for a citation gain the test failed to show. I would rather run one honest holdout that kills a weak tactic than renew three months of work on the strength of a screenshot. Cause, then effect. Everything else is storytelling.

## Frequently Asked Questions

### Why isn't one screenshot of my brand in ChatGPT enough to prove a citation gain?

A single answer proves nothing because AI engines give variable results. Asking the same question twice repeats only about 51% of cited sources on average, and only about 30% of brands stay visible from one AI answer to the next. A screenshot may capture sampling noise rather than a real, persistent gain.

### Does ChatGPT need more repeat runs than Perplexity when testing citations?

Yes. A second ask repeats about 72% of sources on Perplexity but only about 32% on ChatGPT, so ChatGPT's answers wobble more. That is why the protocol runs each prompt five times in ChatGPT but only three times in Perplexity before believing any lift.

### How do I test whether my content change actually caused an AI citation lift?

Pick 10 to 20 real buyer prompts, apply your content change to half of them, and leave half untouched as a holdout. Run every prompt several times in ChatGPT and Perplexity before and after the change, then compare the treated group's movement against the holdout's movement. The result that matters is treatment's change minus holdout's change.

### How should I choose and split the prompts for a holdout test?

Use 10 to 20 real buyer prompts in fixed wording across brand, commercial, and problem intent, and lock the wording before editing anything. Pair prompts by intent and split each pair so treatment gets the content change and holdout gets nothing. If both sides depend on the same page, pick a different pair, because editing that page would contaminate the holdout.

### Can I ship several content changes at once and still attribute the citation lift?

No. If you ship multiple changes in the same week and citations rise, you cannot tell which intervention mattered. The protocol calls for one treatment on one publication date, with pages mapped to holdout prompts left untouched until the window closes.

### What should I log for each AI answer run?

Give each run one row logging the date, engine, prompt ID, treatment or holdout, whether you were cited, whether you were mentioned without a link, position in the answer, whether you were cited, recommended, or compared, the cited URL, and competitors that appeared. Citation rate, mention rate, and share of voice are kept separate because ChatGPT names brands far more often than it links.

### How long should I wait after publishing a change before measuring AI citations again?

Wait at least three weeks. The median background re-read is 5.5 days for Perplexity and 14 days for ChatGPT, and a crawl is only a floor, not a guarantee the updated page is in the answers. Week three is the first reading usable to judge the change; week two is early for ChatGPT.

### What should I do if treatment gains citations on Perplexity but not ChatGPT, or mentions rise without links?

Those are mixed outcomes, not a win with footnotes. Repeat the after-read the following week to see whether the pattern holds, keep the original result in the log, and let the second reading test the pattern rather than replace it.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/the-holdout-test-how-to-prove-a-content#webpage","url":"https://groas.com/post/the-holdout-test-how-to-prove-a-content","name":"The AI Citation Holdout Test: Did Your Content Change Cause the Lift?","description":"Test an AI citation claim with matched prompts, untreated holdouts, repeated ChatGPT and Perplexity runs, and a fixed reading window.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/1bb4d7d0-97cb-4b98-88f7-43e6a10f623a.png"},"breadcrumb":{"@id":"https://groas.com/post/the-holdout-test-how-to-prove-a-content#breadcrumb"},"dateModified":"2026-10-09T05:37:20.204Z","mainEntity":{"@id":"https://groas.com/post/the-holdout-test-how-to-prove-a-content#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/the-holdout-test-how-to-prove-a-content#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"The AI Citation Holdout Test: Did Your Content Change Cause the Lift?","item":"https://groas.com/post/the-holdout-test-how-to-prove-a-content"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/the-holdout-test-how-to-prove-a-content#article","url":"https://groas.com/post/the-holdout-test-how-to-prove-a-content","mainEntityOfPage":{"@id":"https://groas.com/post/the-holdout-test-how-to-prove-a-content#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"The AI Citation Holdout Test: Did Your Content Change Cause the Lift?","description":"Test an AI citation claim with matched prompts, untreated holdouts, repeated ChatGPT and Perplexity runs, and a fixed reading window.","datePublished":"2026-10-09T05:37:20.135Z","dateModified":"2026-10-09T05:37:20.204Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/1bb4d7d0-97cb-4b98-88f7-43e6a10f623a.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"change, prompts, holdout, chatgpt, page, citation, test, content change","wordCount":2064,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/the-holdout-test-how-to-prove-a-content#faq","mainEntity":[{"@type":"Question","name":"Why isn't one screenshot of my brand in ChatGPT enough to prove a citation gain?","acceptedAnswer":{"@type":"Answer","text":"A single answer proves nothing because AI engines give variable results. Asking the same question twice repeats only about 51% of cited sources on average, and only about 30% of brands stay visible from one AI answer to the next. A screenshot may capture sampling noise rather than a real, persistent gain."}},{"@type":"Question","name":"Does ChatGPT need more repeat runs than Perplexity when testing citations?","acceptedAnswer":{"@type":"Answer","text":"Yes. A second ask repeats about 72% of sources on Perplexity but only about 32% on ChatGPT, so ChatGPT's answers wobble more. That is why the protocol runs each prompt five times in ChatGPT but only three times in Perplexity before believing any lift."}},{"@type":"Question","name":"How do I test whether my content change actually caused an AI citation lift?","acceptedAnswer":{"@type":"Answer","text":"Pick 10 to 20 real buyer prompts, apply your content change to half of them, and leave half untouched as a holdout. Run every prompt several times in ChatGPT and Perplexity before and after the change, then compare the treated group's movement against the holdout's movement. The result that matters is treatment's change minus holdout's change."}},{"@type":"Question","name":"How should I choose and split the prompts for a holdout test?","acceptedAnswer":{"@type":"Answer","text":"Use 10 to 20 real buyer prompts in fixed wording across brand, commercial, and problem intent, and lock the wording before editing anything. Pair prompts by intent and split each pair so treatment gets the content change and holdout gets nothing. If both sides depend on the same page, pick a different pair, because editing that page would contaminate the holdout."}},{"@type":"Question","name":"Can I ship several content changes at once and still attribute the citation lift?","acceptedAnswer":{"@type":"Answer","text":"No. If you ship multiple changes in the same week and citations rise, you cannot tell which intervention mattered. The protocol calls for one treatment on one publication date, with pages mapped to holdout prompts left untouched until the window closes."}},{"@type":"Question","name":"What should I log for each AI answer run?","acceptedAnswer":{"@type":"Answer","text":"Give each run one row logging the date, engine, prompt ID, treatment or holdout, whether you were cited, whether you were mentioned without a link, position in the answer, whether you were cited, recommended, or compared, the cited URL, and competitors that appeared. Citation rate, mention rate, and share of voice are kept separate because ChatGPT names brands far more often than it links."}},{"@type":"Question","name":"How long should I wait after publishing a change before measuring AI citations again?","acceptedAnswer":{"@type":"Answer","text":"Wait at least three weeks. The median background re-read is 5.5 days for Perplexity and 14 days for ChatGPT, and a crawl is only a floor, not a guarantee the updated page is in the answers. Week three is the first reading usable to judge the change; week two is early for ChatGPT."}},{"@type":"Question","name":"What should I do if treatment gains citations on Perplexity but not ChatGPT, or mentions rise without links?","acceptedAnswer":{"@type":"Answer","text":"Those are mixed outcomes, not a win with footnotes. Repeat the after-read the following week to see whether the pattern holds, keep the original result in the log, and let the second reading test the pattern rather than replace it."}}]}]}
```