Does AI Overview Optimization Work? Run a Six-Week Test With 20 Pages

An agency adds a schema snippet to your pricing page. Three days later, it sends a screenshot of your domain in a Google AI Overview and calls the job done. I would ask one question before paying for the victory lap: what happened to the pages nobody touched?

Change 10 pages, leave 10 matched pages alone, and check both groups against the same fixed prompts for six weeks. That is how I used to test ad copy: give the proposed improvement a control, then see whether it beats the background noise. Most advice on getting cited in AI Overviews, and most tools promising to do it for you, never clear that bar.

The noise is substantial. SISTRIX tracked 82,619 prompts across AI search engines and found that Google replaced 56% of cited sources in AI Overviews week over week. Profound’s analysis of 80,000 queries reported 59.3% month-over-month citation drift. A URL appearing after an edit may be progress. It may also be Tuesday.

This is a prospective test protocol, not a report of results. Choose one set of edits, lock the prompts and measurement conditions, and decide what would count as useful evidence before looking at the outcome.

Question: did the edits beat ordinary citation churn?

A before-and-after screenshot cannot answer that question. BrightEdge’s tracking across five AI search engines found that 87% of URL movements were abrupt drops rather than gradual declines. In an Ahrefs study of 863,000 SERPs, 45.5% of cited URLs changed between subsequent AI Overview refreshes.

If your edited page gains a citation while comparable, untouched pages gain citations too, the edit has not separated itself from engine movement. If the edited pages improve while controls fall, the difference is more interesting. That is the point of a holdout, not a claim that Google has suddenly become predictable.

Define what the holdout can tell you

Use a matched-page setup, in the spirit of the SEO experimentation framework described by Airbnb engineering. Ten pages receive the edit; 10 comparable pages remain unchanged. Measure citation frequency before and after in both groups, then compare the changes. The Prompt-Trigger Test protocol applies the same discipline to a fixed query set.

The result tests the bundle you deploy, not every ingredient in it. If you change headings, tables, schema, and internal links together, a positive result does not tell you which change mattered. If you need that answer, run a narrower follow-up test. For this run, the question is whether the bundle earns more citation presence than matched pages left alone.

Setup: match 20 pages before changing one

Select 20 informational or commercial-intent URLs that already earn organic impressions. Look for guides, category explainers, or comparison pages that address questions Google might synthesize into an Overview. Leave the homepage and brand-name queries out: their existing prominence makes them poor matches for ordinary topic pages.

Step 1: Make 10 pairs, then assign treatment

Match each page with another on three attributes:

• Topic: Pair pages serving similar kinds of questions.
• Baseline organic traffic: Keep the pages reasonably close in existing search activity.
• Traditional search rank: Avoid pairing a page near the top of results with one far below it.

Ranking deserves particular attention. Conductor’s study of 167.8 million citations found that 57% came from outside the organic top 10, yet position 1 had a 25% citation probability compared with 10.2% for position 10. seoClarity’s study of 362,000 keywords likewise reported a 43% citation rate at ranking #1 and 7% at #20. The studies use different datasets, but both give you the same experimental warning: do not stack your treatment group with stronger-ranking pages.

A pair might contain two CRM comparison pages ranking #4 and #5. Another might contain two onboarding guides ranking #8 and #9. Flip a coin within each pair. One page receives the edits; its partner becomes the control. Assign after matching, not after deciding which pages look most promising.

Step 2: Lock a bank of 60 prompts

Write three exact queries for each page: one definitional, one comparative, and one decision-stage. For example: “what is X,” “X vs Y,” and “how to choose X.” Twenty pages times three queries gives you 60 prompts: 30 associated with treatment pages and 30 with controls.

Choose queries prospective buyers might actually use. Then freeze the wording on Day 0. Changing a prompt halfway through the run changes what you are testing, just as swapping an ad headline halfway through an A/B split would.

Keep the collection method steady too. Use an automated API or clean private browser sessions, with the geographic setting locked to your primary operating region. Avoid changing devices, locations, or logged-in search conditions between runs. You cannot eliminate every source of variation; you can stop adding your own.

Step 3: Deploy one defined edit bundle

Apply changes to the 10 treatment pages within a single 48-hour window. Put the 10 control pages under an editing freeze for the test. Do not change their copy, title tags, or internal links.

For this protocol, use an extractability-focused bundle:

1. Put a 45-to-55-word direct answer beneath the main H2 that defines the subject.
2. Turn narrative comparisons into clear HTML <table> or ordered <ol> structures, with explicit headers where a table needs them.
3. Replace ambiguous opening-section pronouns such as “it” or “this platform” with the entities they refer to.
4. Add TechArticle or Article schema with detailed about entities, and point two internal links from related high-authority articles to each treatment page.

Google’s generative search documentation describes AI features in terms of search and retrieval, not a special tag that guarantees inclusion. Direct answers are worth testing because they make a passage easier to identify and extract; accounts in an r/Blogging discussion also favor putting them early beneath a heading. Neither is a substitute for the holdout. Treat the edits as a hypothesis, not a recipe with a guaranteed citation.

Measurements: log the citation, not the celebration

A yes-or-no screenshot throws away useful context. As the Crawlspace AEO testing framework argues, repeated, timestamped observations matter when retrieval varies between sessions. For every prompt on every run, record:

1. Citation status (0 or 1): Does the target URL appear among the AI Overview’s citation cards or source links?
2. Card position (1 to N): If cited, where does it appear? Leave this blank when it is not cited.
3. Attribution match (Yes, No, or None): Does the answer use information from the new answer passage, use different material from the page, or not cite the page at all?
4. Organic rank (1 to 100): Where does that URL rank in traditional results for the same query during the check?

Keep the observations in one sheet. These columns are enough to reconstruct what changed instead of relying on whichever screenshot made it into a sales deck:

Organic rank helps you interpret a citation gain. If an edited page moves from #12 to #4 and also starts appearing in Overviews, you have not isolated a special generative-search effect from an ordinary ranking improvement. Attribution is another clue: a citation that draws on an unchanged footer is weaker evidence for your new answer passage than one that uses the passage itself. Neither observation proves why Google chose the URL. Both keep you from telling yourself an overly tidy story.

Schedule: three mornings a week through Week 6

Run the full 60-prompt bank on three fixed mornings each measurement week, such as Tuesday, Wednesday, and Thursday, under the same collection conditions. Daily spot checks invite you to react to every fluctuation. A fixed cadence gives both cohorts the same opportunities to rise and fall.

• Week 0 — Baseline: Run all 60 prompts on three mornings before editing. Record the citation frequency for each page across its prompts and runs. A prompt cited once in three checks has a baseline frequency of about 0.33, not 1.
• Week 1 — Deployment: Publish the treatment edits within 48 hours. Submit all 20 pages, treatment and control, for manual re-indexing in Google Search Console to keep that step consistent. Leave this week out of the post-treatment measurement while index updates are being processed.
• Weeks 2–6 — Evaluation: Run the same 60 prompts three mornings per week. That yields 15 post-treatment observations per prompt, or 900 prompt observations across the two cohorts.

Keep the raw log. You will need it when a promising weekly average turns out to be one good morning followed by two disappearances.

Analysis: subtract what happened to the controls

Calculate the citation frequency for each cohort in Week 0 and again across Weeks 2–6. Then compare each cohort’s change:

Net lift = (treatment active rate − treatment baseline rate) − (control active rate − control baseline rate)

Suppose treatment pages move from a 20% baseline citation frequency to 40% during evaluation: a gain of 20 percentage points. Controls move from 20% to 33% without content edits: a gain of 13 points. The treatment’s net lift is 7 percentage points, not 20. The control does not explain every moving part of Google’s systems, but it prevents you from crediting the entire raw gain to your rewrite.

Decide how much evidence you need before rollout

Do not treat a small positive net lift as a verdict. With 20 pages and shifting citations, an 8-point advantage could vanish on another run. Before you start, set a practical rollout bar. For this protocol, I would look for at least 20 percentage points of net lift, sustained across the evaluation weeks, with attribution matches showing the new passages in use. That is a decision rule for whether to invest in a wider rollout, not a statistical guarantee or a claim that 19 points cannot matter.

If the result lands below your bar, inspect the weekly pattern, organic ranks, and attribution log. Do not rewrite hundreds of pages because one aggregate number has a plus sign.

Treat a flat result as a boundary, not a diagnosis

If treatment and control move together, this bundle has not shown a citation advantage in this test. It does not prove that headings, schema, or internal links never help, nor does it identify your real bottleneck. Brand authority, broader topical depth, and technical rendering are possible next places to investigate, not conclusions you can extract from a flat chart.

The useful outcome is narrower and cheaper: you have a reason not to fund a sitewide rewrite on the strength of screenshots alone. Choose the next hypothesis, keep the controls honest, and test again.

Vendor trial: make the tool face the same holdout

Apply the protocol to any agency or AEO tool pitching citation gains. Ask what it will actually change on 10 treatment URLs, leave 10 matched URLs frozen, and measure both against the locked prompt bank. A platform that polls results and plots citation charts may be useful for monitoring. It has not demonstrated optimization merely by drawing the chart. MainTouch’s review of AEO software describes that distinction between tracking and action; our rundown of tools for getting featured in Google AI Overviews makes the same question worth asking.

If a vendor offers a 30-day proof of concept, use that period to inspect what it ships, then keep the six-week evaluation cadence if you need this test’s full readout. Before you buy an AI Overview tool, test what it ships in 30 days. An automated task list that your team must implement is not autonomous optimization, whatever the dashboard calls it.

groas is the stronger fit when you want execution rather than another reporting layer: its specialized models identify citation gaps, revise page structure, address technical barriers, and push changes within defined guardrails, with action logs you can inspect. Put those actions through the holdout too. A better operating model still has to show its work.

Eligibility check: skip the test if there is nothing to observe

This protocol needs pages with an established search footprint. If your domain receives fewer than 10,000 monthly organic visits, your target pages sit outside the traditional top 50, or the pages do not register organic impressions, skip this version of the experiment for now. The Conductor and seoClarity figures show why rank belongs in the setup; a prompt bank that produces zeros in both groups gives you little room to measure a change.

Work on crawlability, indexation, and topical coverage first. Then return when you have comparable pages and queries that produce observable results.

I would not pay an ad agency for a headline test that ignored its control group. I would not pay for AI Overview optimization on the strength of a celebratory screenshot, either. Match the pages, lock the prompts, freeze the controls, and run the six weeks. If treatment beats the holdout by your pre-set bar, you have a reason to expand the edits. If it does not, keep the budget and change the hypothesis.