---
title: "Before You Buy AI Overview Software, Run a Six-Week Split Test"
description: "Pair your landing pages, rewrite half, and measure citations against an untouched control before paying for AI Overview software."
url: "https://groas.com/post/before-you-buy-ai-overview-software-run"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/15e6b7cf-f22b-4941-a504-2f4bee263b20.png"
published: "2026-10-10T05:39:03.968Z"
modified: "2026-10-10T05:39:04.032Z"
---

October 10, 2026 · 9 min read

# Before You Buy AI Overview Software, Run a Six-Week Split Test

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![A tabletop model of twin shops: tiny workers renovate one while its identical partner sits frozen in ice, a split test with an untouched control.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/15e6b7cf-f22b-4941-a504-2f4bee263b20.png)

In this article

1. [Set up pairs before you touch the copy](#set-up-pairs-before-you-touch-the-copy)
2. [Write buyer prompts, then collect a two-week baseline](#write-buyer-prompts-then-collect-a-two-week-baseline)
3. [Apply one treatment; leave its partner alone](#apply-one-treatment-leave-its-partner-alone)
4. [Score citations and check whether the pages were fetched](#score-citations-and-check-whether-the-pages-were-fetched)
5. [Read the result: lift, flat, or noise](#read-the-result-lift-flat-or-noise)
6. [Take the result into the sales demo](#take-the-result-into-the-sales-demo)

A citation screenshot is not a control group. I have sat through the AI Overview software demos: a logo wall, a line going up, a page that got cited. None tells you what happened to the pages nobody touched.

So run the comparison yourself. Pair 10 to 20 of your landing pages by topic and intent. Rewrite one page in each pair to make its answer easy to quote; leave the other alone. Sample the same buyer prompts for two weeks before the edit and four weeks after it. Then count **which half gets cited more often**. If the rewritten half cannot beat the untouched half, software will not rescue that on-page strategy. A quotable passage has to earn the citation. The dashboard only records it.

The question is deliberately narrow: **does changing the page change its citation rate against a comparable, unchanged page?** Not “how do I get cited everywhere?” I used to trust my eyes on this. A cleaner page looked like it ought to win. That is a poor reason to buy a subscription.

## Set up pairs before you touch the copy

Start with pages that already rank in the top 10 for their terms. Pages outside that range rarely get cited at all; TenSpeed’s teardown of B2B content in AI Overviews makes the same point about [ranking eligibility and passages Google can lift cleanly](https://www.tenspeed.io/blog/b2b-content-ai-overviews). If you do not have enough eligible pages, do not pretend a rewrite test will answer a different problem.

Pair pages by topic and buyer intent. Put pricing pages with pricing pages, comparison pages with comparison pages, and location pages with location pages. They need not be twins. They do need to face roughly the same kind of question, or you will be measuring topic demand as much as your rewrite.

Then flip a coin within each pair:

1. Mark one page **treatment** and the other **control**.
2. Record both URLs, their current ranks, and the prompts relevant to that pair.
3. Freeze the controls for all six weeks: no title tweaks, FAQ additions, schema changes, or quiet copy fixes.
4. Record every change to a treatment page and its date. Keep a note of other site changes that could affect either side.

The coin flip matters. If you give the rewrite to every page you already expect to win, the test will mostly confirm your preferences. The untouched partner gives you something less flattering and more useful: a check on what would have happened anyway.

![Lab notebook with landing-page pairs labeled treatment and control](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/0c385c97-3cc4-4e47-8845-7b6c55077b9a.png)

## Write buyer prompts, then collect a two-week baseline

Do not start with 100 generic questions because a keyword tool says they have volume. Write **15 to 25 prompts per pair** in the language customers use. Sales-call transcripts, support tickets, and the Search Terms report are better starting points than an exercise in sounding like a chatbot.

Make roughly half comparison and price questions: “acme vs beta for machine shops” or “how much does industrial floor cleaning cost per square foot.” Make the other half unbranded problem questions, such as “how to keep a warehouse floor dust free between shifts.” A buyer asking the second kind has not chosen a vendor. That is precisely why the answer matters.

Before editing a page, run every relevant prompt three to five times across a two-week baseline. Sample each surface separately: ChatGPT, Claude, Perplexity, and Google AI Overviews. If you also check Google AI Mode, give it its own column. Use the same prompts and sampling routine after the edit. Do not let a change in the questions masquerade as a change in the pages.

A screenshot captures one answer, not a rate. In SparkToro’s January 2026 test, [600 volunteers ran 12 prompts 2,961 times and got the same brand list twice in fewer than 1 in 100 runs](https://ecommercefastlane.com/how-to-choose-an-ai-visibility-tracker/). Repetition is not fussy lab theater here. It is how you avoid building a purchase decision around one lucky response.

Keep the Google surfaces apart, too. Ahrefs compared 730,000 AI Mode and AI Overview pairs and found [86% semantic similarity but only 13.7% citation overlap](https://ecommercefastlane.com/how-to-choose-an-ai-visibility-tracker/). A blended Google score can hide the movement you came to measure. **Count citations by page, prompt, run, and surface.**

## Apply one treatment; leave its partner alone

Once the baseline is logged, rewrite only the treatment page in each pair. The treatment should be consistent enough that you can later say what you tested:

1. Open with a 40-to-60-word answer. State who the page is for and give the relevant price, comparison, or deciding fact already supported by the page.
2. Name the product, company, city, and competitor where those details matter. Make the answer quotable without forcing a reader to decode “our solution.”
3. Follow with question-based H2s drawn from your prompt list. Use a short table or bullets where they make the answer easier to extract.
4. Add FAQ and product schema where they fit the page, and make sure the structured data matches visible copy.

No clever introductory paragraph before the answer. No new facts invented to fill the table. You are testing whether clearer presentation of what the page can already say changes its chances, not whether making a bigger claim wins attention.

SearchPilot’s account of SEO split testing recommends [similar page buckets, server-side changes visible to users and crawlers, and crawl-coverage checks](https://www.searchpilot.com/). Those checks matter here for a plain reason: a page nobody fetched cannot tell you much about the rewrite. Keep the control pages still while you check whether the edited ones were seen.

Run the same prompts for four weeks after the edit. Resist the helpful developer who notices a control page needs an FAQ and adds it midway through. They may be right about the FAQ. They would still break the comparison. **Half gets clearer; half stays still.** Put everything else in the log.

## Score citations and check whether the pages were fetched

Your primary measure is **citation share per page: runs citing that URL divided by relevant prompt runs on that surface**. If a page is eligible for 20 prompts and you run each five times on ChatGPT, that gives you 100 looks at whether ChatGPT cites it. A page cited in 34 sits at 34% for those runs. Calculate the treatment and control averages separately for each surface and week; compare their post-edit movement with the two-week baseline.

Expect small numbers. A meta-analysis of 19 studies puts the average Overview at [5 to 8 citations, with 99% of sources appearing once](https://www.searchenginejournal.com/impact-of-ai-overviews-on-seo/539982/). Overviews also skew toward informational long-tail queries and co-occur with featured snippets about 60% of the time. You are competing for a handful of slots. A move from 12% to 22% can matter even if the screenshot looks unremarkable. The question is whether that movement belongs to the treatment half rather than to the whole set.

Next, check fetches in your server or edge logs. Record when Googlebot and OAI-SearchBot fetch edited URLs, and look for verified ChatGPT-User and Claude-User requests where they occur. Do not treat those requests as interchangeable. Most ChatGPT citations come from [a prebuilt search index built by OAI-SearchBot, not a live ChatGPT-User fetch for each question](https://geolikeapro.com/blog/openai-crawlers-comparison). A missing user-triggered fetch is not proof that a citation was impossible; a confirmed search-crawler fetch helps establish that the rewrite was available to be seen.

**Verify the requester, not just its name in a log.** A user-agent string can be spoofed. Where the provider publishes IP ranges, [check the requesting IP against its published files](https://geolikeapro.com/blog/openai-crawlers-comparison) before counting a fetch as verified. Record the date and the bot separately. “A bot visited” is too vague to explain a citation change.

Finally, track ordinary Google rank for each page. Position can affect Overview appearances: a link in position 1 has [a 53% chance of appearing in Overviews versus 36.9% in position 10](https://searchengineland.com/guide/how-to-optimize-for-ai-overviews). If a treatment page moves from position 6 to position 2 while its control stays put, you cannot pin the citation gain on the answer block alone. The rewrite may have helped rank, or something else may have changed. Note the movement rather than hiding it in the average.

Keep one sheet with pair names, URLs, prompt text, run dates, surface, citations, verified fetch dates, rank, and page edits. Say you are spending $20k a month on search and weighing a tool contract against content work. That sheet is more useful than a polished chart with no record of how the numbers were made. No sheet, no test.

![Bar chart comparing citation share for treatment and control pages across repeated runs](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/527fd801-ae4f-4447-951c-652b190c9f32.png)

## Read the result: lift, flat, or noise

**A useful lift is the rewritten half pulling ahead and staying ahead.** Look for the gap in three of the four post-edit weeks on the same surface, across several pairs rather than one lucky URL. Check that the edited pages were fetched. The mechanism you expected was straightforward: you supplied a passage with the names and facts intact, made it easy to lift, and the relevant crawler had an opportunity to see it.

If that pattern appears, roll the _same treatment_ to the next batch: the answer-first block, entity naming, and table shape. Do not turn a win for this set of pages into proof that every page needs every possible optimization. The test earns a specific scaling decision.

**A flat result is useful, even if it stings.** Perhaps both halves stay where they began. Perhaps both rise together while the answers change around them. I used to respond to that by rewriting the copy again. That is often just more work applied to the one thing already tested. If the edited pages were fetched and still did not pull ahead, look beyond sentence polish: the domain’s standing on the topic, facts competitors publish that you do not, or prompts too far from the informational questions where Overviews cite. Stop fixing sentences and start fixing the constraint.

Noise has its own shape. Treatment jumps in week three; control jumps in week four. One pair carries the average while the others sit flat. I have called a thin gap a win too early before. Do not repeat that mistake because the chart has a pleasing color.

Use these checks before you bet budget on a gap:

- **Enough repeated looks.** Aim for at least 80 to 100 relevant looks per page per period. With only 20 looks per page, a 10-point difference is thin evidence, not a verdict.
- **The same direction twice.** Wait for two consecutive post-edit weeks pointing the same way after the relevant edited pages have been fetched.
- **Movement across pairs.** Look for at least three pairs moving together. Remove the best-performing pair from your calculation and see whether the pattern survives.
- **Controls that behave like controls.** If both halves rise together, log it and keep sampling. Do not award the rewrite credit for a shared move.

These are practical decision rules, not a guarantee that every wobble has disappeared. When the evidence is noisy, collect more runs. **A gap worth buying around survives a refetch, a second week, and the loss of its best page.**

## Take the result into the sales demo

If the rewritten half wins, you have shown that on-page edits can move citations for your pages under these conditions. Ask the vendor to [test what it ships in 30 days](https://groas.com/post/before-you-buy-an-ai-overview-tool-run-t): which pages it changed, when bots fetched them, and how citation share moved against untouched pages. You need the work shipped and measured at scale. A dashboard that hands your team suggestions to implement does not close that gap; you already ran the treatment yourself.

If the result is flat, a subscription promising quotable pages is the wrong next purchase. Read the distinction between [software that gets work done and software that tracks who got cited](https://groas.com/post/will-any-of-this-software-actually-get-m), then put the next dollar toward the constraint your log exposed: pages that rank, mentions on sites models already quote, or relevant facts your competitors publish and you do not.

Run the six weeks. Keep the control half frozen. Buy only the thing that beats it.

## Frequently Asked Questions

### Why isn't a screenshot of a page getting cited enough proof that AI Overview software works?

A screenshot captures one answer, not a rate, and shows nothing about the pages nobody touched. To know whether edits actually move citations, compare rewritten pages against untouched control pages over repeated runs on the same prompts and surfaces.

### How do I set up a split test for AI Overview citations on my landing pages?

Pair 10 to 20 landing pages by topic and buyer intent, starting with pages that already rank in the top 10. Flip a coin to mark one page in each pair as treatment and the other as control, then freeze the controls for all six weeks with no title tweaks, schema changes, or copy fixes.

### What prompts should I use for the baseline, and how often should I run them?

Write 15 to 25 prompts per pair in the language customers use, with roughly half comparison or price questions and half unbranded problem questions. Run every relevant prompt three to five times across a two-week baseline before editing, and sample ChatGPT, Claude, Perplexity, and Google AI Overviews separately, using the same routine after the edit.

### What changes should I make to the treatment page during the rewrite?

Open with a 40-to-60-word answer stating who the page is for and the relevant price, comparison, or deciding fact. Name the product, company, city, and competitor where relevant, add question-based H2s from your prompt list, use tables or bullets where they help extraction, and add FAQ or product schema that matches the visible copy.

### How do I measure whether the rewrite actually changed citation rates?

Calculate citation share per page: runs citing that URL divided by relevant prompt runs on that surface. Compute treatment and control averages separately for each surface and week, and compare post-edit movement against the two-week baseline. Aim for at least 80 to 100 relevant looks per page per period before trusting a gap.

### Why do I need to check crawler logs, and how do I verify a bot is real?

A page nobody fetched cannot tell you much about the rewrite, and a user-agent string in a log can be spoofed. Record when Googlebot and OAI-SearchBot fetch edited URLs, and verify the requesting IP against the provider's published IP ranges. Note that most ChatGPT citations come from OAI-SearchBot's prebuilt index, not live ChatGPT-User fetches.

### Should I also track Google rankings during the test?

Yes. Position affects Overview appearances: a link in position 1 has a 53% chance of appearing in Overviews versus 36.9% in position 10. If a treatment page moves up in rank while its control stays put, you cannot pin the citation gain on the answer block alone, so note the movement rather than hiding it.

### How do I know if my split test result is a real win and not noise?

Look for the rewritten half pulling ahead in three of the four post-edit weeks on the same surface, with at least three pairs moving together, and check that a gap survives removing the best-performing pair. If both halves rise together, log it and keep sampling rather than crediting the rewrite.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/before-you-buy-ai-overview-software-run#webpage","url":"https://groas.com/post/before-you-buy-ai-overview-software-run","name":"Before You Buy AI Overview Software, Run a Six-Week Split Test","description":"Pair your landing pages, rewrite half, and measure citations against an untouched control before paying for AI Overview software.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/15e6b7cf-f22b-4941-a504-2f4bee263b20.png"},"breadcrumb":{"@id":"https://groas.com/post/before-you-buy-ai-overview-software-run#breadcrumb"},"dateModified":"2026-10-10T05:39:04.032Z","mainEntity":{"@id":"https://groas.com/post/before-you-buy-ai-overview-software-run#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/before-you-buy-ai-overview-software-run#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Before You Buy AI Overview Software, Run a Six-Week Split Test","item":"https://groas.com/post/before-you-buy-ai-overview-software-run"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/before-you-buy-ai-overview-software-run#article","url":"https://groas.com/post/before-you-buy-ai-overview-software-run","mainEntityOfPage":{"@id":"https://groas.com/post/before-you-buy-ai-overview-software-run#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Before You Buy AI Overview Software, Run a Six-Week Split Test","description":"Pair your landing pages, rewrite half, and measure citations against an untouched control before paying for AI Overview software.","datePublished":"2026-10-10T05:39:03.968Z","dateModified":"2026-10-10T05:39:04.032Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/15e6b7cf-f22b-4941-a504-2f4bee263b20.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"page, pages, before, overview software, overview, run, test, pair","wordCount":1992,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/before-you-buy-ai-overview-software-run#faq","mainEntity":[{"@type":"Question","name":"Why isn't a screenshot of a page getting cited enough proof that AI Overview software works?","acceptedAnswer":{"@type":"Answer","text":"A screenshot captures one answer, not a rate, and shows nothing about the pages nobody touched. To know whether edits actually move citations, compare rewritten pages against untouched control pages over repeated runs on the same prompts and surfaces."}},{"@type":"Question","name":"How do I set up a split test for AI Overview citations on my landing pages?","acceptedAnswer":{"@type":"Answer","text":"Pair 10 to 20 landing pages by topic and buyer intent, starting with pages that already rank in the top 10. Flip a coin to mark one page in each pair as treatment and the other as control, then freeze the controls for all six weeks with no title tweaks, schema changes, or copy fixes."}},{"@type":"Question","name":"What prompts should I use for the baseline, and how often should I run them?","acceptedAnswer":{"@type":"Answer","text":"Write 15 to 25 prompts per pair in the language customers use, with roughly half comparison or price questions and half unbranded problem questions. Run every relevant prompt three to five times across a two-week baseline before editing, and sample ChatGPT, Claude, Perplexity, and Google AI Overviews separately, using the same routine after the edit."}},{"@type":"Question","name":"What changes should I make to the treatment page during the rewrite?","acceptedAnswer":{"@type":"Answer","text":"Open with a 40-to-60-word answer stating who the page is for and the relevant price, comparison, or deciding fact. Name the product, company, city, and competitor where relevant, add question-based H2s from your prompt list, use tables or bullets where they help extraction, and add FAQ or product schema that matches the visible copy."}},{"@type":"Question","name":"How do I measure whether the rewrite actually changed citation rates?","acceptedAnswer":{"@type":"Answer","text":"Calculate citation share per page: runs citing that URL divided by relevant prompt runs on that surface. Compute treatment and control averages separately for each surface and week, and compare post-edit movement against the two-week baseline. Aim for at least 80 to 100 relevant looks per page per period before trusting a gap."}},{"@type":"Question","name":"Why do I need to check crawler logs, and how do I verify a bot is real?","acceptedAnswer":{"@type":"Answer","text":"A page nobody fetched cannot tell you much about the rewrite, and a user-agent string in a log can be spoofed. Record when Googlebot and OAI-SearchBot fetch edited URLs, and verify the requesting IP against the provider's published IP ranges. Note that most ChatGPT citations come from OAI-SearchBot's prebuilt index, not live ChatGPT-User fetches."}},{"@type":"Question","name":"Should I also track Google rankings during the test?","acceptedAnswer":{"@type":"Answer","text":"Yes. Position affects Overview appearances: a link in position 1 has a 53% chance of appearing in Overviews versus 36.9% in position 10. If a treatment page moves up in rank while its control stays put, you cannot pin the citation gain on the answer block alone, so note the movement rather than hiding it."}},{"@type":"Question","name":"How do I know if my split test result is a real win and not noise?","acceptedAnswer":{"@type":"Answer","text":"Look for the rewritten half pulling ahead in three of the four post-edit weeks on the same surface, with at least three pairs moving together, and check that a gap survives removing the best-performing pair. If both halves rise together, log it and keep sampling rather than crediting the rewrite."}}]}]}
```