Chris Raad · 20 November 2026

A/B testing on Shopify: native Rollouts experiments, the testing apps and their prices, and how many weeks a valid test takes at your Australian store's size.

You can A/B test on Shopify with Shopify's own Rollouts experiments on the Grow plan and above, or with an app from about US$15 to US$999 a month. The constraint is traffic: detecting a 10% lift from a 2% conversion rate takes about 161,000 sessions, which is more than three months of traffic for a store turning over AU$1.5M a year.

This guide compares the native option, the apps and the standalone tools with prices read on 20 November 2026, then gives a table of how long a valid test takes at each Australian store size, and what to do instead when the answer is "too long".

Can you A/B test on Shopify without an app?

Yes. Shopify's Rollouts, under Markets in the admin, includes an experiment type that compares a treatment against a control before you launch a change (Shopify Help Center). Shopify announced Rollouts in its Winter '26 Edition, released it to merchants on 31 March 2026, added tests between whole themes and checkout setups on 5 June 2026, and redesigned the setup on 22 September 2026.

How Rollouts experiments work, from Shopify's documentation:

QuestionAnswer
Which plans?Rollouts on Basic and above; experiments on Grow and above. In Australia, Grow is A$149 a month billed monthly
What can you change?One main theme edit or replacement, one checkout and accounts configuration, catalogue changes and one discount change per rollout (rollout types)
Default split and length50% control, 50% treatment, ending 90 days after the start; both adjustable
What does it report?For theme tests: conversion rate, bounce rate, reached checkout rate, add to cart rate. For checkout tests: checkout conversion rate (analytics)
FlickerNone: Shopify assigns each visitor before the page loads
ConsentVisitors who decline tracking still get a version but aren't counted
GA4Rollout and treatment IDs are not sent to GA4 or app pixels
LimitsOnline store only, no vintage themes, no Liquid template changes in a rollout

The gaps matter. A theme experiment reports no revenue, average order value or revenue per visitor, so a change that lifts conversion by pushing cheaper products looks like a win. Shopify lets you pull more metrics through ShopifyQL, and checking revenue per visitor there before you launch is worth the extra step.

For a store on Grow or above testing a new theme, a redesigned product template or a checkout configuration, Rollouts is the cheapest valid option, and assigning visitors before the page loads removes the flicker that script-based tools have to work around.

Which Shopify A/B testing apps are there, and what do they cost?

The Shopify App Store lists testing apps from US$15 to US$999 a month, priced by orders or by monthly visitors. These are the apps with the most reviews and published prices, read on 20 November 2026. Prices are in US dollars, as billed; the App Store shows Australian visitors an AUD estimate (it used 1 AUD = 0.6713 USD that day, so US$99 showed as A$147.48).

AppListed plans (US$ a month)Priced byWhat it testsRating
Intelligems69 (content), 112.10 (merchandise or shipping), 349 (unlimited)Sitewide ordersContent, themes, prices, discounts, shipping rates and thresholds; checkout on the top plan4.8 (204 reviews)
Shoplift99 Core, 399 Advanced, 999 ProMonthly unique visitorsThemes, templates, split URLs; price tests from Advanced4.8 (135)
Elevate A/B Testing49, 99, 499Sitewide ordersTemplates, themes, content; prices, shipping and checkout from 994.9 (147)
ABConvert99 (1,000 test orders), 249, 399, 599Test orders, with overageTemplates, themes, checkout; price, shipping and offers from 2494.4 (84), Built for Shopify
Visually15 (100 orders), 25 (200), 80 (600), higher tiersMonthly ordersPages, carts, checkouts, upsells, themes, discounts, prices, shipping4.8 (148)
Shogun A/B Testing39 (3 live tests), 119 (5), 499 (unlimited)Live testsTemplates, themes, pages; prices from 4994.3 (48), Built for Shopify
Personizely39, 59Monthly unique visitorsSplit URLs and themes; content, price and shipping on 595.0 (94)

Read the plan limits before the headline price. ABConvert notes that its checkout content, cart and upsell tests need Shopify Plus, and its shipping tests need a plan with carrier-calculated rates. Checkout tests on any app depend on what Shopify lets apps change at checkout, which is far more on Plus. Our guide to Shopify checkout customisation covers those limits.

Standalone experimentation tools sit above the apps, and most of them now price on application:

ToolPublished entry priceNote
Convert ExperiencesUS$299 a month paid annually, US$399 monthly100,000 tested users a month on the entry plan; its Shopify listing is a connector
KameleoonUS$495 a monthUp to 10 experiments and 50,000 tested visitors a month
OptimizelyQuote only
VWO and AB TastyQuote onlyNow one company, Wingify; both pricing pages redirect to Wingify's

Google Optimize, the free option many guides still recommend, shut down on 30 September 2023.

How much traffic do you need to A/B test on Shopify?

You need enough sessions to detect the size of change you expect, and for most changes that is more than a store under AU$5M a year gets in a month. The standard two-proportion formula, at 95% confidence (two-sided) and 80% power, is the same one we use in our CRO cost guide:

n per variant = ( z(0.975) × sqrt(2 × p̄ × (1 - p̄)) + z(0.80) × sqrt(p1 × (1 - p1) + p2 × (1 - p2)) )² ÷ (p2 - p1)²

z(0.975) = 1.96, z(0.80) = 0.84, p1 = baseline rate, p2 = p1 × (1 + lift), p̄ = (p1 + p2) ÷ 2

From a 2% baseline, the sessions needed in total (both versions together) are:

Relative lift to detectConversion moves fromSessions per versionTotal sessions
30%2.0% to 2.6%9,79719,595
20%2.0% to 2.4%21,10942,218
10%2.0% to 2.2%80,682161,364
5%2.0% to 2.1%315,206630,412

Now the same test at each Australian store size. Assumptions, which you should replace with your own numbers: a AU$120 average order, a 2% conversion rate, and every session on the site entering the test. Weeks are rounded up, with a two-week floor because shopping differs by day of the week and pay cycle.

Annual online revenueSessions a month30% lift20% lift10% lift5% lift
AU$1.5M52,0002 weeks4 weeks14 weeks53 weeks
AU$3M104,0002 weeks2 weeks7 weeks27 weeks
AU$5M174,0002 weeks2 weeks4 weeks16 weeks
AU$10M347,0002 weeks2 weeks2 weeks8 weeks
AU$20M694,0002 weeks2 weeks2 weeks4 weeks
AU$50M1,736,0002 weeks2 weeks2 weeks2 weeks

The arithmetic for one cell: AU$1.5M ÷ AU$120 ÷ 2% = 625,000 sessions a year, or about 12,000 a week. A 10% test needs 161,364 sessions, so 161,364 ÷ 12,019 = 13.4 weeks, rounded up to 14.

Real tests usually run slower than the table. A test on product pages only sees product page traffic. Most changes move conversion by less than 10%. And the lift has to be relative to the baseline: a 10% lift means 2.0% becoming 2.2%, not 12%.

Why vendors quote smaller numbers

App vendors publish lower thresholds than this formula gives. Intelligems recommends 300 or more orders per group and at least 7 days, and says about 300 orders a group detects a 10% change at 90% confidence. Shoplift's FAQ says most tests need at least 10,000 visitors per variant and 14 days, while another Shoplift help page says several hundred visitors.

The difference is the standard. At a 2% conversion rate, 300 orders is about 15,000 sessions per group. By the formula above, that sample has about a 23% chance of detecting a true 10% lift at 95% confidence. Lower thresholds finish sooner and call more winners, and more of those winners are false.

Is A/B testing worth it?

A/B testing is worth it when your traffic can detect the size of change you are testing, and most changes lose. Optimizely's analysis of more than 127,000 experiments run on its platform between 2018 and 2023 found "around 12% of experiments win on the primary metric", where a win meant statistical significance of 90% or more.

Some of those wins aren't real. Ron Kohavi, who led experimentation at Microsoft and Airbnb, and Nanyu Chen worked through Optimizely's figure in False Positives in A/B Tests (KDD 2024). At Optimizely's default threshold, a 12% win rate implies a true success rate of 9.3% and a false positive risk of 37.8%: more than a third of the celebrated winners would be noise. Kohavi, Deng and Vermeer's earlier paper, A/B Testing Intuition Busters (KDD 2022), shows how that risk grows with success rate and power:

Success rate of ideas testedPublished byFalse positive risk at 80% powerAt 20% power
33%Microsoft5.9%20.0%
10%Booking.com, Google Ads, Netflix22.0%52.9%
8%Airbnb search26.4%59.0%

The 20% power column is the one a small store lives in. A test stopped early, on too few sessions, declares winners that are more often false than true.

The method still pays where the traffic exists. Koning, Hasan and Chatterji followed 35,262 start-ups and found those that adopted A/B testing grew page visits about 10% in the first months and 30% to 100% after a year (Management Science, 2022); they also launched new products 9% to 18% more often. Their sample is technology start-ups, not online stores, and the effect came from testing many ideas over time.

High-traffic operators test sparingly. One r/shopify operator reported "~200k visits/day" and about six tests a month "to try to avoid chasing false signals".

What should a store below the traffic line do instead?

A store below the traffic line should test only large changes and measure everything else against a baseline written down before the change ships. That gets most of the value of testing without months of waiting on results that can't settle.

  1. Fix what is broken without testing it. A button that does nothing, a script error at checkout or a shipping cost hidden until the last step doesn't need a test to justify the fix. Our one-hour diagnosis finds these.
  2. Test big changes only. A new product page layout, a different theme or a reworked cart can move conversion 20% or more, which a AU$1.5M store can detect in about four weeks. A button colour can't.
  3. Measure shipped fixes on revenue per visitor, against the same weeks before the change and the same weeks a year earlier, so the season doesn't get credited as the fix.
  4. Use Rollouts without an experiment: Shopify's event type publishes a change for a set period and then rolls it back, which suits a sale or a seasonal layout.
  5. For price tests, judge on profit per visitor, not conversion. A higher price usually converts lower and can still earn more.

If you do test, set the rules before the test starts: one hypothesis drawn from research, one primary metric, the sample size from the table above, whole weeks only, and no stopping when the dashboard first shows a winner. Check the test's order count against Shopify's orders for the same dates; one r/shopify merchant found their testing dashboard reporting more sales than Shopify recorded, and a result that doesn't reconcile with real orders isn't a result.

Stores under about AU$1.5M a year online, or under 20,000 visits a month, usually get more from fixing the basics themselves than from paying for testing. Our guide to choosing a CRO retainer or audit covers where paid help starts to make sense.

Sources

Questions

Can you A/B test on Shopify without an app?

Yes, on the Grow plan (A$149 a month) and above. Shopify's Rollouts feature, available to merchants since 31 March 2026, runs experiments that split visitors 50/50 between your current theme or checkout setup and a changed version. Theme experiments report conversion rate, bounce rate, reached checkout rate and add to cart rate, but not revenue or average order value.

What do Shopify A/B testing apps cost?

On the Shopify App Store on 20 November 2026, testing apps listed from US$15 a month (Visually, up to 100 orders) to US$999 a month (Shoplift Pro). Common entry plans sit at US$39 to US$99 a month, with price and shipping tests on higher plans. Standalone tools cost more: Convert from US$299 a month paid annually, Kameleoon from US$495 a month.

How much traffic do you need to A/B test on Shopify?

Detecting a 10% relative lift from a 2% conversion rate needs about 161,000 sessions at 95% confidence and 80% power. A store doing AU$1.5M a year at a AU$120 average order has about 52,000 sessions a month, so a whole-site test takes about 14 weeks. Detecting a 20% lift needs about a quarter of that traffic.

Is A/B testing worth it for a small Shopify store?

Usually not for small changes. Optimizely found about 12% of 127,000+ experiments won on their primary metric, and Kohavi and Chen estimate that at Optimizely's default threshold about 38% of those wins are false positives. Below roughly AU$5M a year, test only large changes and measure smaller fixes against a written baseline instead.

Which is better for Shopify A/B testing, an app or Shopify Rollouts?

Start with Rollouts if you are on Grow or above and want to test themes or checkout setups: it costs nothing extra and assigns visitors before the page loads, so pages don't flicker. Use an app when you need price, shipping or offer tests, or revenue per visitor as the result, which Rollouts experiments don't report.

We'll show you what your customers do on it.

Apply for a free teardown