Standard deviation in A/B testing

Noise shrinks with √n, not n

Each group in a test gives you a sample mean: a conversion rate, an average order value, an average session length. How much that mean would wobble if you reran the test is its standard error, the metric's SD divided by the square root of the group size:

SE = σ / √n

A test can only reliably detect a difference that is a few standard errors wide. To halve the smallest difference you can detect, you have to halve the SE, and because of the square root that takes four times as many visitors. Double the metric's SD and you need four times the visitors too. Required sample size grows with the variance, σ², which is why the spread of the metric matters so much. The standard error calculator shows the √n effect on any data you paste in.

The SD of a yes/no metric

Most tests measure conversion: each visitor either does the thing (1) or does not (0). The mean of those 0s and 1s is the conversion rate p, and the standard deviation of a single visitor's outcome follows directly from the binomial set-up:

σ = √(p (1 − p))

You never have to measure this SD separately; it comes from p. What matters for planning is how big it is relative to p, because tests are usually framed in relative terms ("we want to detect a 10% lift").

The absolute SD is largest at 50%, but the relative SD, the SD divided by the rate, is largest for rare events. A 2% conversion rate is seven times noisier, relative to its own size, than a 50% one. That is the reason tests on low-traffic, low-conversion pages take so long.

A worked sample-size calculation

Suppose a sign-up page converts at 5% and you want to detect a lift to 5.5%, a 10% relative improvement. Use the usual settings: a two-sided significance level of α = 0.05 and 80% power. The numbers here are hypothetical; put your own baseline in their place.

The standard formula for the visitors needed in each group when comparing two proportions is:

n = (z₁₋α/₂ + z₁₋β)² × [p₁(1 − p₁) + p₂(1 − p₂)] / (p₂ − p₁)²

Step by step:

1. The z-values: z for α = 0.05 two-sided is 1.960; z for 80% power is 0.842. Their sum squared is (1.960 + 0.842)² = 7.849.
2. The variances: p₁(1 − p₁) = 0.05 × 0.95 = 0.0475, and p₂(1 − p₂) = 0.055 × 0.945 = 0.051975. Together, 0.099475.
3. The difference to detect: 0.055 − 0.05 = 0.005, squared 0.000025.
4. n = 7.849 × 0.099475 ÷ 0.000025 ≈ 31,231 per group (computed with unrounded z-values and rounded up to a whole visitor).

That is 62,462 visitors in total. If the page gets 4,000 visitors a day split evenly, the test needs 62,462 ÷ 4,000 = 15.6 days, so plan for 16, and in practice round up to whole weeks so each weekday is represented equally. The A/B test calculator and the power analysis calculator run the same calculation for other inputs.

Now change only the baseline and the size of the lift:

The first three rows all look for the same 10% relative lift. The low-baseline page needs over twelve times as many visitors as the high-baseline one, and the relative SD column in the earlier table is the reason. The last row shows the other lever: doubling the lift you are trying to detect cuts the sample to about a quarter.

Try it: A/B test calculator

The calculator opens with example counts of 400 and 460 conversions from 5,000 visitors each; enter your own visitors and conversions, and your minimum detectable lift, to get the significance test and a sample-size plan.

Open the full A/B test calculator for the pooled z-test and the sample-size formula explained.

Continuous metrics: where the SD really bites

For a metric like revenue per visitor there is no formula that gives the SD from the mean; you have to estimate it from past data. It is often huge. Most visitors spend nothing and a few place large orders, so the SD can be several times the mean. For two groups, the sample size per group for a difference δ in means is:

n = 2 (z₁₋α/₂ + z₁₋β)² σ² / δ²

Take an illustrative shop with revenue per visitor averaging $3 and an SD of $15. Detecting a 10% lift ($0.30) needs 2 × 7.849 × 225 ÷ 0.09 ≈ 39,245 visitors per group. If a handful of very large orders pushes the SD to $30, the requirement quadruples to about 157,000. The mean did not change at all; only the spread did.

That is why experienced teams spend effort on reducing the metric's variance before they spend it on waiting longer. Capping extreme values at a high percentile, using a pre-experiment measure of each user as a covariate, or switching to a less noisy metric that tracks the same goal can each shorten a test more than any amount of patience. The sample size calculator is useful for the survey-style version of the same question, where you need a given margin of error on a single rate.

A checklist before you launch

Know the baseline rate or, for a continuous metric, an SD from recent data. Decide the smallest lift worth detecting before you see any results. Compute the sample size and convert it into days using real traffic. If the answer is months, either choose a larger lift, a less variable metric, or accept that this page cannot be tested reliably at its current traffic. Each of those decisions comes down to the same ratio: the effect you want to see, against the standard deviation you are measuring it through.