peham

CEO notes · GEO experiments14 min read

GEO experiments.

Most GEO "wins" are week-to-week noise. Here is how to run a controlled test that proves a page change improved AI citations, what our own data at OtterlyAI says about noise, and which of our GEO experiments held up.

TL;DR

To prove a page change improved AI citations, compare changed pages with similar unchanged pages on a fixed prompt set, over equal windows before and after, per AI engine. Only trust a lift that's bigger than normal noise. In OtterlyAI's own data, the citations of a typical top page moved about 19% from one week to the next. Across our GEO experiments, cheap "AI file" tricks like llms.txt and Markdown copies didn't work, schema helped only on Google's surfaces, and freshness looks promising but not yet proven.

Key takeaways

  • Use a control group. In one of our tests, an untouched page rose 63%, more than the pages we edited.
  • Know your noise. Half of the weekly moves on our top pages were 20% or more.
  • Change one thing, choose one metric, and decide both before you start.
  • Read results per engine and per page. Engines often move in opposite directions.
  • Schema isn't required for AI answers. Google says so, and our test agrees outside Google.

The short answer: how to prove a page change improved AI citations

To prove that a page change improved AI citations, compare the pages you changed with similar pages you didn't change, on a fixed set of prompts, over the same time windows, and only call it a win if the changed pages beat the unchanged ones by more than normal week-to-week noise. In practice:

  1. Write the hypothesis and the success metric down first. For example: "Adding an answer-first summary to 10 guides raises their citations in ChatGPT and Perplexity." One metric, chosen before you look at any data.
  2. Pick treatment and control pages. Choose pages that are similar in topic, page type and current citation volume. Change one group and leave the other alone.
  3. Freeze the prompt set. Track the same prompts, in the same country and language, across the same AI engines, before and after.
  4. Measure a baseline. Collect at least 2–4 weeks before the change, ideally daily, so you know how much the numbers move on their own.
  5. Change one thing, and log the date. Make sure the change is actually live and crawlable.
  6. Compare like for like. Use equal windows before and after, then compare the change in the treatment group with the change in the control group. Check each engine and each page separately.
  7. Check the market. If competitors' visibility moved the same way at the same time, the cause is probably the engine, not your edit.

That's the whole method. The rest of this article is about why each step matters, what our own data says about noise, and what the GEO experiments we've run at OtterlyAI actually showed.

Why a simple before-and-after check lies

AI answers are probabilistic. The same prompt can produce a different answer, with different sources, an hour later. On top of that, engines change their models and retrieval all the time, and your competitors publish things. So citation counts move even when you do nothing.

To show how much, I pulled the daily citation history of the 25 most-cited pages on otterly.ai and our help center for four full weeks (21 October – 17 November 2026), across the seven AI engines we track:

~19%
median change in a page's citations from one week to the next
1 in 2
weekly moves were 20% or larger, up or down
1 in 5
weekly moves were 50% or larger

Volume matters a lot. Our homepage, with roughly 2,000 citations a week, stayed within ±7% from week to week. Pages with 100 or more citations a week moved by a median of about 12%. Pages below that moved by about 20%, and some swung by more than half in a single week.

Now picture the typical GEO case study: someone edits one page with 40 citations a week, sees +30% the following week and posts a screenshot. On our numbers, that's within the range of normal noise for a page that size. It could be the edit. It could just be a normal week.

We learned this the hard way. In one of our own experiments, we added the year "2026" to the title tag and H1 of 11 blog posts. Citations rose 56% and 61% in the two test groups, which looked like a clear win until we checked a page we had never touched. That untouched page rose 63% over the same period. Almost all of the lift in the second group came from a single URL. Our honest conclusion: if adding the year does anything, the effect is smaller than the noise.

How to design a controlled GEO experiment, step by step

Our research team bases its approach on the controlled-experiment principles from the A/B testing literature (Kohavi et al.) and adapts them to AI search, where you can't randomize what an engine does. Our methodology is public. This is the practical version I'd give any marketing team.

1. Start with a falsifiable hypothesis

"If we add an answer-first summary to these 10 guides, their AI citations will rise more than those of 10 comparable guides we leave unchanged." If no result could prove it wrong, it isn't a hypothesis.

2. Choose one success metric before you start

Pick one primary metric, such as citations to the treated URLs, the share of tracked prompts that cite them, or brand coverage on the target prompts. Note any secondary metrics, like crawler visits or Google clicks, but don't switch to whichever one looks best afterwards.

3. Build treatment and control groups

Match pages on topic, page type (guide, product page, help article), current citation volume and age. More pages per group beats one hero page, because a single URL can dominate the result. That happened in our year-in-title test. If you can, assign pages to groups by coin flip rather than picking your favourites for treatment.

4. Freeze the prompts, engines and market

Use the same prompt wording, country, language and engines throughout. AI engines behave very differently from each other, so plan to read results per engine. My prompt research playbook covers how to build the prompt set.

5. Measure a baseline long enough to see the noise

Two to four weeks of daily data shows how much each page moves on its own. Small pages need longer baselines and bigger effects before you believe anything.

6. Change one variable, verify it, and log it

Ship the change, confirm it's live in the server-rendered HTML that crawlers fetch, and write down the date. If you fix three things at once, you'll never know which one worked. When you need to know which of several signals an engine picked up, use a "canary": a unique, harmless fact placed only in one version, so you can tell from the answer which version the engine read.

7. Wait for re-crawling, then compare equal windows

Engines need time to re-crawl and re-retrieve a page. Compare equal windows before and after, and leave out the change date itself. You can check when AI crawlers re-visited the page in your logs. I explain how in AI crawlers: how to track them.

8. Read the difference, not the raw change

The result is the treatment group's change minus the control group's change. Look at it per engine and per page, and report how many pages went up, flat and down. "7 of 10 pages gained" tells you more than one percentage.

9. Check whether the whole market moved

If competitors gained visibility on the same prompts at the same time, an engine update is the likelier cause. In our schema experiment, competitors that hadn't changed anything moved just as much as we did. That changed our conclusion.

10. Publish the null results too

Most tactics don't work, or work only on one engine. Writing that down saves your team, and everyone else, from repeating them.

Design choiceWeak testControlled test
ComparisonOne page, before vs afterTreated pages vs matched untouched pages
MeasurementA few manual prompts in your own chat windowA fixed prompt set, tracked daily across engines
Baseline"Last week"2–4 weeks, so you know the normal range
MetricWhatever went upOne metric, chosen in advance
ReadingTotal changeDifference vs control, per engine and per page
ContextIgnoredCompetitor movement and crawl dates checked

What our GEO experiments showed so far

At OtterlyAI we've run more than 30 GEO experiments and studies since 2025, on our own site, on test sites and across large citation datasets. I've presented many of them on stage, including at brightonSEO. Here are the ones that matter most for anyone planning their own tests, with an honest confidence level:

ExperimentWhat happenedHow confident I am
Updating only the publication date on 10 articlesCitations rose 31.5% in the 30 days after, while two unchanged control articles fell 10.6%. Results differed a lot by engine.Directional. Only two controls, and the trend was already moving before the change.
Adding "2026" to titles of 11 posts+56% and +61%, but an untouched page rose 63%.No effect we can separate from noise.
Sitewide schema markup (FAQ, Organization, SoftwareApplication, Review, Article)AI Overviews appearances rose 1,500% over three months. Six of seven AI platforms couldn't fetch or correctly read the schema when asked, and a fact we placed only in FAQ schema was never used in any answer.Helps Google's surfaces through classic SEO. No measurable direct effect on the other AI engines.
llms.txt fileUnder 0.1% of AI bot visits went to /llms.txt over 90 days.Not a meaningful AI crawl signal.
Markdown copies of pagesHTML pages got 4.6% of AI crawler visits; the Markdown copies got 0%.Don't duplicate pages as .md for GEO.
Deleting pages (dead citations)Of 20,092,769 cited URLs we checked, 19.3% no longer returned a live page. Pages we removed still held about 5% citation share eight months later.High. AI forgets much more slowly than you'd think.
Press release distribution (10 releases)First citations from large news portals within days, and brand mentions trended up afterwards.Directional, with no control group.

Two patterns come up again and again. First, engines disagree. In the publication-date test, ChatGPT and Perplexity citations went up sharply while Google AI Mode and Gemini went down. A single blended number would have hidden that. Second, the cheap "AI file" tricks don't work, and boring content work does. Clear answers, current content, FAQs and being present on the third-party pages engines cite keep showing up in the results. Formats invented for AI crawlers don't.

The best AI search experiments to run in 2026

If you want to build your own evidence rather than trust anyone's LinkedIn post, including mine, these are the experiments I'd run first. Each has a clear hypothesis and can be run with treatment and control pages.

  1. Real freshness updates vs date-only updates. Review and update 10 important pages properly, change only the date on 10 others, and leave 10 alone. This tells you whether the work or the signal drives citations. Please don't change dates on content you haven't reviewed.
  2. Answer-first summaries. Add a two- or three-sentence direct answer at the top of guides that target question-style prompts.
  3. FAQ sections in visible HTML. Add real questions your buyers ask as on-page FAQs, not only in schema.
  4. Original data. Add one first-party number or table to a page, and see whether engines start quoting it.
  5. Third-party presence. Get accurately included in a few of the pages engines already cite for your prompts, and track brand coverage on those prompts against a set you didn't work on. I've written about how this fits into GEO overall.
  6. Video companions. Publish a long-form video for a set of articles, and compare citations with articles that don't get a video.
  7. Local-language prompts. If you sell in several markets, track the same prompts in English and the local language. Engines localize very differently.
  8. Crawl access fixes. If your server or firewall blocks AI crawlers on some sections, unblock one section and compare it with the others.

Do you need schema markup for AI answer eligibility?

No, not for AI answers in general. Google says it directly in its guidance on generative AI search: "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." The same guidance says you don't need llms.txt files or other "AI text files", because Google Search doesn't use them.

Our schema experiment points the same way for the other engines. Most AI platforms couldn't even read the markup when asked to, and a fact that existed only in our FAQ schema never appeared in an answer. Where schema did help was on Google's own surfaces, through rich-result eligibility and classic SEO.

So keep valid structured data for Google rich results, and check it with Google's own tools. But if you're auditing a site for AI answer eligibility, check these first:

  • Can AI crawlers fetch the page, according to both robots.txt and your firewall?
  • Is the main content in the server-rendered HTML, not only after JavaScript runs?
  • Does the page answer the question plainly in the first lines?
  • Are the key facts (prices, features, numbers) in visible text?
  • Is the page current, and does it say so?

My take as a CEO: most GEO case studies wouldn't survive a control group

I'm a marketer who builds a measurement company, so I see both sides. Marketers need wins to show, and AI search is new enough that almost any screenshot looks like evidence. That's how llms.txt became a "must-have" before anyone had measured whether engines read it.

My view:

  • A result without a control is an anecdote. That includes some of our own early tests. We now say so in the article itself, as we did with the year-in-title test.
  • Effects in AI search are smaller than the noise, until proven otherwise. If your page moves 20% a week on its own, a 15% "lift" is nothing to celebrate.
  • Measure per engine. "AI search" isn't one channel. A tactic that helps Google's AI Overviews can do nothing for ChatGPT, or the opposite.
  • Spend most of your budget on the boring things that keep working, such as clear answers, current content, original data and earned third-party coverage. Then use experiments to retire the folklore.

And publish your null results. The GEO field is young, and the fastest way for all of us to learn is to share what didn't work.

How we run experiments at OtterlyAI

Disclosure, and what OtterlyAI offers here

I'm the CEO of OtterlyAI, so weigh this accordingly. OtterlyAI tracks a fixed set of prompts daily across ChatGPT, Google AI Overviews, Google AI Mode, Gemini, Perplexity, Microsoft Copilot and Claude, and records every URL cited in the answers. That gives you the baseline, the control pages and the per-engine view this method needs. The Citation Changes view compares each cited URL with the previous period of equal length, so you can see which pages are new, increasing, stable, decreasing or lost. Agent Analytics shows when AI crawlers re-visited a page after you changed it. That's the data behind this article. Plans start at $29/month, with a free trial.

Sources

Related reading: the monthly AI search report I want as a CEO, AI referral traffic: how to measure it and my talks on GEO experiments.

FAQs

How do I run a controlled test to prove a page change improved AI citations?

Pick matched treatment and control pages, freeze a prompt set across the AI engines you care about, and measure a 2–4 week baseline. Then change one thing on the treatment pages only, log the date and compare equal windows before and after. The result is the treatment group's change minus the control group's change, read per engine and per page. Also check whether competitors moved at the same time.

What are the best AI search experiments to run in 2026?

Start with tests that answer real budget questions: genuine content refreshes vs date-only updates, answer-first summaries, visible FAQ sections, adding original data, earning accurate inclusion on third-party pages that AI engines already cite, companion videos, local-language prompts and fixing blocked AI crawler access. Run each with a control group.

How long should a GEO experiment run?

Plan at least 2–4 weeks of baseline and 4 weeks after the change, and leave out the change date itself. Engines need time to re-crawl and re-retrieve a page, and small pages are noisy. In OtterlyAI's data, a typical page's citations moved about 19% from one week to the next, so short tests on low-volume pages rarely show anything you can trust.

Does schema markup help with AI answer eligibility?

Structured data isn't required for AI answers. Google's guidance says there's no special schema.org markup you need to add for generative AI search. In OtterlyAI's experiment, sitewide schema increased AI Overviews appearances, but most other AI platforms couldn't read it, and a fact placed only in FAQ schema never appeared in an answer. Keep schema for Google rich results, and focus AI work on crawlable HTML and clear answers.

Does llms.txt improve AI visibility?

There's no evidence it does. Google says it doesn't use llms.txt or other AI text files. In OtterlyAI's 90-day test, under 0.1% of AI bot visits went to the llms.txt file. It does no harm, but it's not a lever.

How many pages and prompts do I need for a GEO test?

More than one page per group. Ten or more treated pages compared with a similar number of untouched pages is a reasonable start, because a single URL can drive most of a result. Use enough prompts to cover the topics of those pages, and track them daily across engines rather than checking a few prompts by hand.

Keep
reading.

All writing →
OtterlyAI

See how your brand shows up in AI search.

Track mentions, citations and share of voice across ChatGPT, Perplexity, Gemini, Google AI Overviews, AI Mode and Copilot.

Try OtterlyAI ↗

Talks · advisory · press

Want Thomas on your stage or podcast?

Get in touch →