sample size calculator for A/B tests

Here’s a practical sample size calculator for A/B tests you can use.

1) What you need

To calculate sample size, you typically need:

• Baseline conversion rate (p1): current conversion rate for variant A
• Minimum detectable effect (MDE): the smallest lift you want to detect
• Significance level (alpha): usually 0.05
• Power: usually 0.8 or 0.9
• Test type: usually two-sided

---

2) Formula for two-proportion A/B test

For conversion-rate tests, the approximate sample size per group is:

[ n = \frac{\left(z_{1-\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_2-p_1)^2} ]

Where:

• (p_1) = baseline conversion rate
• (p_2 = p_1 + \text{MDE})
• (\bar{p} = (p_1 + p_2)/2)
• (z_{1-\alpha/2}) = z-value for alpha
• (z_{1-\beta}) = z-value for power

---

3) Quick example

Suppose:

• Baseline conversion rate = 10%
• MDE = 2 percentage points (10% to 12%)
• Alpha = 0.05
• Power = 0.80

Then you’ll need roughly:

• ~3,800 users per group
• ~7,600 total users

---

4) Simple Python calculator

import math
from scipy.stats import norm

def ab_sample_size(p1, mde, alpha=0.05, power=0.8, two_sided=True):
    p2 = p1 + mde
    if p2 <= 0 or p2 >= 1:
        raise ValueError("p2 must be between 0 and 1")

    z_alpha = norm.ppf(1 - alpha/2) if two_sided else norm.ppf(1 - alpha)
    z_beta = norm.ppf(power)

    p_bar = (p1 + p2) / 2

    numerator = (
        z_alpha * math.sqrt(2 * p_bar * (1 - p_bar)) +
        z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
    ) ** 2

    denominator = (p2 - p1) ** 2

    return math.ceil(numerator / denominator)

# Example:
n = ab_sample_size(p1=0.10, mde=0.02, alpha=0.05, power=0.8)
print("Sample size per group:", n)
print("Total sample size:", 2 * n)

---

5) If you want a ready-to-use rule of thumb

For conversion tests:

• Low baseline rate (1–5%) → often need large samples
• Moderate baseline rate (5–20%) → thousands per group is common
• Smaller MDE → sample size rises quickly
• Higher power → more sample needed

---

6) Excel / Google Sheets formula

If you want, I can give you an Excel-friendly formula too.

---

If you want, I can also provide:

1. a Google Sheets / Excel template,
2. a calculator for continuous metrics (e.g. revenue, time on site), or
3. a web form / JavaScript version.