Skip to content
Chalk Labs / AI SEO
How-to guide

How to Test Whether Your AI SEO Changes Worked

By Rahil Jain, founder of Chalk LabsUpdated How we research

The short answer

Fix a list of prompts, run each one several times before you change anything, and record how often your brand is mentioned or cited. Make the change, wait four to eight weeks, then repeat the same runs. Count a result only if it moves more than the normal run-to-run swing and beats a control group you did not touch 23.

  • In one study of local business prompts, a business that appeared once reappeared in only about half of repeat attempts 3.
  • Google suggests waiting a few weeks before judging whether SEO work helped 1.
  • Platform changes can shift everything at once: Ahrefs saw top-10 overlap in AI Overview citations fall from about 76% to about 38% 4.
  • Use a control group of prompts or pages you did not change, so market noise does not pass as your win.

How do you know an AI SEO change really worked?

You know it worked when the result moves by more than normal noise and more than a control group that you left alone. A single before and after check cannot tell you that, because AI answers vary.

BrightLocal ran repeat local searches across ChatGPT, Google AI Mode and AI Overviews over 60 days. It found that a business that appears once is seen again in about half of later attempts: 50% for ChatGPT and AI Mode, and 58% for AI Overviews. Average overlap between result sets on repeat searches was 23%, 21% and 33%, and BrightLocal notes that overlap also reflects how many businesses each answer lists 3. These were local business prompts, so non-local prompts may differ. If your own mention rate swings that much without any change, a good week proves little.

Do it step by step

0 of 8 done0%

  1. State what you expect to change and by how much, in one sentence. Choose one main metric, such as mention rate across your prompts.

    Tip: Save this sentence with the date before you look at any data.

  2. Write 30 to 50 prompts real buyers ask, in their words. Include a mix of category, comparison and problem questions, and a few branded ones.

  3. Divide the prompts, and the pages that serve them, into two sets of similar value. You will only change pages for the first set.

  4. Run every prompt at least three times a week, in fresh chats, on the same platforms. Record mention, citation, position and the sources cited.

  5. Find the highest and lowest weekly mention rate in the baseline. Set the pass line just above the highest, and write it down.

  6. Publish the rewrite, new pages or fixes for the change set only. Record the date and what changed. Do not touch the control set.

  7. Continue the same runs, same wording, same platforms. Add Search Console, Bing and GA4 numbers as supporting data 567.

  8. Compare the change set with its baseline and with the control set. Declare a result only if it passes your line for two weeks in a row. Then decide to scale, adjust or stop.

Your ticks are saved in this browser only.

Copy-ready templates

Run log columns

text
Week | Date | Prompt ID | Set (change/control) | Platform | Run # | Mentioned? (1/0) | Linked? (1/0) | Position (1,2,3...) | Sources cited | Notes

One row per run. Use a fresh chat and the same account type each time.

Weekly mention rate formula (Google Sheets)

text
=COUNTIFS(A:A, 5, D:D, "change", G:G, 1) / COUNTIFS(A:A, 5, D:D, "change")

Counts mentioned runs for week 5 in the change set, divided by all runs. Assumes the column order from the run log template, with Week stored as a number. Change 5 to the week number, and "change" to "control" for the control set.

Pre-registered test plan

markdown
## Test plan (written before data)
- Date written:
- Hypothesis (one sentence):
- Main metric: weekly mention rate
- Prompts: 40 (20 change, 20 control)
- Runs: 3 per prompt per week, fresh chats, same account type
- Baseline: 3 weeks. Baseline high point:
- Pass line (just above high point):
- Change date:
- Judge on: week 4 (look) and week 8 (decide)
- Result counts only if: pass line beaten 2 weeks in a row AND control not moved similarly

Fill it in once, then do not edit it during the test.

GA4 regex for AI assistant referrers (check domains are current)

regex
(chatgpt\.com|chat\.openai\.com|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.ai)

For a custom channel or exploration filter on Source. GA4 also has a default AI Assistant channel that excludes AI Overviews and AI Mode . Referrer domains change, so review this list each quarter.

What does a fair test cost in time and tools?

You need a spreadsheet, 30 to 50 prompts that real buyers ask, access to the AI tools you care about, and about three hours to set up the baseline. Manual running is slow: 120 runs (40 prompts x 3) takes roughly 3 to 5 hours a week per platform. To cut it, pilot with 20 prompts x 2 runs, or use a tracker that automates runs.

The test itself lasts at least seven weeks, and usually up to eleven: three weeks of baseline, the change, then a four to eight week wait. Google says changes can show within hours or take months, and suggests waiting a few weeks before judging 1.

What should you measure?

Pick one main metric and two supporting ones before you start. The main metric is usually mention rate: the share of runs in which your brand appears in the answer. The supporting ones are citation rate, meaning how often a page of yours is linked, and enquiries from AI-related sources.

Keep them separate on purpose. A mention without a link can still influence a buyer, and a link without a mention may bring a visit but little trust. Our guide to measuring AI share of voice explains the metric in more depth.

Supporting data comes from the tools you already have. Google counts AI feature clicks and impressions inside the normal Web search totals in Search Console 7. Bing Webmaster Tools has an AI Performance report that shows total citations and grounding queries, but Microsoft says the data is a sample and does not show ranking or placement within an answer 5. In GA4, the AI Assistant channel covers visits from tools like ChatGPT, Gemini and Copilot, and excludes Google AI Overviews and AI Mode, which count as Organic Search 6.

How do you design a fair before-and-after test?

Use four parts: a fixed prompt list, repeat runs, a baseline period and a control group. The prompt list stays the same for the whole test. Run each prompt at least three times per week in fresh chats, with the same account type and location, and record the results.

The control group matters most. Split your prompts or pages into two sets of similar value. Change only the first set. If both sets rise, the cause is probably the market or a platform update. Ahrefs found that the share of AI Overview citations also in Google's top 10 fell from about 76% in July 2025 to about 38% in a study published in March 2026, a shift Ahrefs suggests may be linked to query fan-out 4. Changes like that hit everyone, so only a control group can separate them from your work.

Write the rules down before you see any results. This is called pre-registering. It stops you from picking the metric or the date range that happens to look best.

How much movement counts as real?

Count a change as real only if it is larger than the swing you saw during the baseline. Measure the baseline for three weeks, note the highest and lowest weekly mention rate for each set, and set each set's threshold slightly above its own highest. A simple rule is: the new weekly rate must beat the baseline high point for at least two weeks in a row, and beat the control group by a clear margin.

Profound ran a vendor study of 753 US prompts over two weeks in June 2026. It found that one run a day gave a visibility reading within about 2 percentage points of ten runs a day (78.7% vs 80.4%). It said prompt choice matters more than run frequency, especially for citation share. It sells tracking, and the study covered one country and two weeks, so use it as a hint 2. For most small teams, a few runs a week across a well-chosen prompt list beats many runs of a poor list.

Worked example (illustrative numbers, not a real client)
  • Prompts: 40 buyer questions, 3 runs each per week = 120 runs per week.
  • Change set baseline: 25%, 24%, 24% (high point 25%, swing 1 point). Control set baseline: 21%, 23%, 22% (high point 23%). Each set has its own threshold.
  • After 8 weeks: change set 36%, control 23%.
  • Reading: the change set rose 11 points over its baseline average of 24.3%, the control rose 1. The gain is far above both sets' baseline swing. Likely real, though not proven.

How long should you wait before judging?

Wait at least four weeks for a first look and eight weeks for a decision. Search engines need time to crawl, reprocess and use your changes, and AI products update their sources on their own schedules. Google's own guidance says changes can take from a few hours to several months and that you should wait a few weeks to assess the effect 1.

OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust 8. That is the speed for access changes. Content quality changes take longer, because other sites must also be crawled and weighed.

What goes wrong in these tests?

The common faults are changing five things at once, changing the prompt wording mid-test, using a logged-in account with memory that shapes answers, testing only branded prompts, and declaring victory after one good run. Another is counting only wins: record the prompts where you vanished too.

A quieter fault is confusing correlation with cause. A sales campaign, a press mention or a seasonal peak can raise mentions at the same time. Keep a change log with dates for everything you publish, send or announce.

When is a formal test not worth it?

Skip the full design if you publish fewer than ten pages a quarter, or if your buyers rarely use AI tools. Manual runs at full scale are also too slow for many small teams. In that case, a simple monthly check of 15 prompts, plus an enquiry question asking how people found you, gives enough signal. Use the AI prompt list builder to create the list and the GA4 AI traffic setup for the traffic side.

Frequently asked questions

Why does ChatGPT give different answers to the same question?

Answer engines generate each reply fresh and may search again each time, so the sources and brands can change. BrightLocal's study of local searches found a business reappeared in about half of repeat attempts; results for non-local prompts may differ, and overlap also reflects list length 3. That is why you need repeat runs and a threshold, not one check.

How many prompts do I need?

Thirty to fifty is a practical start for a small business. Profound found one run a day gave a visibility reading within about 2 points of ten a day, and said prompt choice matters more than frequency, especially for citation share. It is a vendor study with limits 2. Choose prompts that match real buyer questions.

Can Search Console prove my AI SEO worked?

Only partly. Google counts AI feature clicks and impressions inside normal Web search totals, with no separate filter 7. Use Search Console as supporting data and combine it with prompt tracking and enquiries.

What if my results rise but my control group rises too?

Then you cannot claim the change worked. A shared rise points to a platform or market shift, such as the large drop in top-10 overlap Ahrefs saw in AI Overview citations 4. Keep the work, keep testing, and report the result honestly.

Is a paid tracking tool necessary?

No. A spreadsheet and manual runs work for a small list, such as a 20-prompt pilot, but 120 runs a week is several hours per platform. Tools save that time. Ask any vendor how it handles repeat runs and what sample sizes it uses, since results vary between runs 3.

About the author

Rahil Jain

Rahil Jain is an AI SEO (answer engine and generative engine optimisation) consultant and co-founder of Chalk Labs.

LinkedIn · Method and credentials

Sources

  1. SEO Starter Guide: The Basics · Google Search CentralOpened October 20, 2026. Supports: Changes may take a few hours to several months; wait a few weeks to assess whether work had beneficial effects. Last updated 2025-12-10.
  2. Is once a day enough? · ProfoundOpened October 20, 2026. Supports: 753 US prompts, 7 platforms, 1-14 June 2026: once-a-day visibility within about 2 percentage points of ten-a-day; prompt choice matters more than run frequency, especially for citation share (78.7% vs 80.4% visibility for once vs ten runs a day). Vendor study that promotes its own service; one country, two weeks.
  3. What 200k local AI searches tell us about what businesses get recommended · BrightLocalOpened October 20, 2026. Supports: A business that appears once is seen again in about half of later attempts (50% ChatGPT and AI Mode, 58% AI Overviews); average overlap on repeat searches 23%, 21%, 33%; roughly 200,000 local prompts over 60 days; published 16 September 2026.
  4. Update: 38% of AI Overview Citations Pull From The Top 10 · AhrefsOpened October 20, 2026. Supports: About 38% of AI Overview citations also in the top 10 in March 2026 vs about 76% in July 2025 (863K SERPs, 4M URLs); Ahrefs attributes the shift to query fan-out.
  5. Introducing AI Performance in Bing Webmaster Tools Public Preview · Microsoft Bing Webmaster BlogOpened October 20, 2026. Supports: Report shows total citations, average cited pages and grounding queries; Microsoft says the data is a sample and counts do not show ranking, authority or placement within an answer. Published 10 February 2026.
  6. [GA4] Default channel group · Google Analytics HelpOpened October 20, 2026. Supports: AI Assistant channel covers visits from sources like ChatGPT, Gemini, Deepseek, Copilot and Grok; it excludes Google's AI Overviews and AI Mode, which count under Organic Search.
  7. AI features and your website · Google Search CentralOpened October 20, 2026. Supports: AI feature traffic is reported inside the Performance report under Web search type. Last updated 2025-12-10.
  8. Overview of OpenAI crawlers · OpenAIOpened October 20, 2026. Supports: It can take about 24 hours from a robots.txt update for OpenAI's systems to adjust.
How this page was made. Built on Rahil Jain's AI SEO method and drafted with AI research assistance. Every statistic links to the page we read it on (opened October 20, 2026 or nearby). Published October 20, 2026; last updated October 20, 2026. Chalk Labs sells AI SEO services, so we have a commercial interest in this topic. Spotted an error? Tell us and we will fix it.