SEO has a measurement problem. You change a title tag, traffic goes up next week, and you have no idea whether your change caused it, or whether a competitor dropped off page one, Google ran an update, or it is just the seasonal swing you get every October.
Without controlled experiments, every change is a guess. SEO testing applies the same experimental rigour that CRO teams use for conversions to ranking and click-through.
Why SEO testing is different#
Standard A/B testing is simple: randomly split your users, show half version A and half version B simultaneously, compare. This works for on-site behaviour because both groups experience identical external conditions at the same time.
SEO testing cannot randomise users, because Google indexes one version of each URL. Instead, the two methods are:
Sequential (before/after): Make a change to a set of pages. Compare performance before and after. The risk: external conditions changed between the two periods (algorithm update, seasonal shift, competitor action, a reporting change).
Split-page (the gold standard): Divide a large set of similar pages into a control group and a variant group. Make the change only to the variant group. Compare performance between groups over the same time period. Because both groups experience the same external conditions simultaneously, you can attribute differences to your change.
Always use split-page testing when you have enough pages. The added rigour is worth the setup time.
What's worth testing#
Title tags#
The highest-leverage, most reliable SEO test. Titles are a ranking input and the main driver of CTR, and they are easy to change at scale. Remember that Google rewrites some titles in the SERP, so check what actually displays (the SERP preview tool helps you draft, Search Console tells you the outcome).
Hypotheses worth running:
- Number in title ("7 ways") vs no number
- Question format vs declarative statement
- Year inclusion ("2026 guide") vs no year
- Specific benefit vs generic power words ("complete", "ultimate")
- Brand name at the start vs at the end
Meta descriptions#
They do not directly influence rankings but can influence CTR when Google uses them. Test length, specificity and calls to action, and expect noise because Google often generates its own snippet.
Content changes#
Adding a comparison table, original data, a summary at the top, or refreshing outdated sections. These take longer to read (8 to 12 weeks) but often move more than titles. See Content Refresh & Decay.
Structured data#
Test only markup that can still change the SERP. FAQ rich results stopped showing on 2026-05-07 and HowTo was retired in 2023, so an "add FAQPage and measure CTR" test now has nothing to measure. Product snippets, merchant listings, review snippets and video markup are still live and testable on eligible templates (structured data guide). Google says no special markup is needed for AI Overviews or AI Mode, so do not frame a schema test as an AI visibility test.
Internal linking#
Testing whether adding prominent contextual links to target pages improves their rankings. Pick a group of target pages, add links from relevant high-authority pages for the variant half only, and measure over 4 to 6 weeks (internal linking).
Setting up a split-page test#
1. Build a large enough page pool#
Aim for at least 50 pages per group, ideally 100 or more, all on the same template with similar traffic. Good candidates:
- Product category pages (e-commerce)
- Blog posts in the same topic cluster
- Location service pages
- Programmatic SEO pages from the same template
2. Assign randomly, then check balance#
Use explicit random assignment (a random number per URL, not "the first 50"). Then confirm both groups had similar clicks and impressions over the previous 8 weeks. If they did not, re-draw.
3. Change only the variant group#
Leave the control group completely untouched throughout the test, and freeze other work on both groups.
4. Set your measurement window in advance#
Title and meta tests: 4 to 6 weeks minimum. Content changes: 8 to 12 weeks. Commit to the window before you look at results: this is how you avoid the "peeking problem." Add a Search Console annotation (right-click the Performance chart, available since November 2025) on the start date so everyone reading the chart later knows a test ran.
5. Measure in Search Console#
Filter the Performance report by page (or use the API or BigQuery export) for each group's URLs. Compare:
- Clicks (the metric that matters most)
- CTR and impressions
- Average position
Compare the difference between groups, not each group against its own past. If control fell 5% and variant rose 5%, the effect is about 10 points.
Reading results without fooling yourself#
SEO data is noisy. Small groups and short windows produce misleading results. A practical rule of thumb (opinion, not a statistical law):
- Under 10% difference in clicks between groups: inconclusive, within normal variation
- 10 to 20% difference held for 4+ weeks: indicative, worth a follow-up test
- Over 20% consistent difference at 6+ weeks: likely a real effect, reasonable to ship
For more rigour, use a time-series method that models the control group as the counterfactual, such as a difference-in-differences comparison or Google's open-source CausalImpact package. These handle the day-to-day correlation in search data better than a simple significance calculator.
Data breaks that invalidate comparisons#
- Confirmed Google updates. Check the Google Search Status Dashboard. In 2026 alone there were core updates in March and May and spam updates in March, June, August and September. A split test survives an update better than a sequential test, but pause and re-check balance if one lands mid-test. See Algorithm Updates.
- The
&num=100break (September 2025). Google stopped honouring the 100-results parameter around 2025-09-12. Rank-tracker impressions largely vanished from Search Console: a Locomotive study of 319 properties found 87.7% saw impressions decline. Impressions and average position before and after that date are not comparable. - Your own releases. A site-wide template change, a migration or a CWV fix mid-test touches both groups unevenly.
Common testing mistakes#
Testing one page at a time. Individual pages have too much variance. Even a great page can have a bad month for reasons unrelated to your change. You need groups.
Reading results before the measurement window closes. This is the peeking problem. Early results are noisy. Commit to the timeline and do not touch the test mid-run.
Running multiple changes simultaneously. If you change the title, add schema and update the H1 in the same test, you cannot attribute the result to any single variable. One change per test.
Ignoring data breaks. A core update or a reporting change mid-test can swamp a small effect. Annotate and adjust.
Calling a winner on day 7. Don't. Check at the end of the pre-committed window.
Building a testing culture#
The compound value of SEO testing is real. Teams that run even one test per month develop an empirical, site-specific understanding of what moves rankings in their niche. That knowledge is worth more than any best-practice guide, including this one.
Start with the simplest possible experiment: title tag changes on 60 to 100 similar blog posts, half changed and half not. Run it for 6 weeks. Measure carefully. The first test teaches the process; the second and third start generating genuine competitive insight. Report results with the framing in SEO Strategy & Stakeholder Reporting.
Related: Analytics & Forecasting, the measurement foundation that makes testing credible.