Production-grade guide to ab testing statistical rigor covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.
This page is about the statistical integrity layer of A/B testing—the mechanisms that determine when an experiment result can be trusted, when it cannot, and what to do when the math conflicts with business pressure. It matters whenever experiment outcomes influence product decisions, feature gates, or risk calculations. If you have ever looked at a p-value and wondered whether the experiment actually proved anything, or if you have been asked to "just ship it" because the numbers look right despite a small sample, this is the reference for you.
The reader’s first question is usually whether the data provide enough evidence to support a decision. This section covers the mechanics of reaching a conclusion, not the calculation itself.
For two variants with a binary outcome (e.g., conversion yes/no), the standard approach is a two-proportion z-test. The test statistic compares the observed difference in conversion rates to its expected standard error under the null hypothesis of equal rates. In Python with scipy:
from scipy.stats import proportions_ztest
counts = [conversions_variant_a, conversions_variant_b]
total = [total_exposed_variant_a, total_exposed_variant_b]
stat, pval = proportions_ztest(counts, total, alternative='two-sided')The function returns the z-statistic and two-sided p-value. If the p-value falls below the pre-specified significance threshold (commonly α = 0.05), the result is statistically significant. Note that the threshold is a separate decision from the calculation; the test produces a p-value regardless of what you compare it against.
In R, the equivalent function is prop.test():
prop.test(x = c(conversions_a, conversions_b),
n = c(total_a, total_b),
alternative = "two.sided")The output gives the test statistic, p-value, and a confidence interval for the difference in proportions. Many experimentation platforms report a p-value automatically, but the platform’s calculation may or may not account for peeking, multiple variants, or assumption violations. Pulling the raw counts and applying the test in your own code is safer when you need to verify the result.
For continuous outcomes with approximately normal distributions, the equivalent is a two-sample t-test. In scipy:
from scipy.stats import ttest_ind
result = ttest_ind(variant_a_values, variant_b_values, equal_var=False)The equal_var=False flag requests Welch's t-test, which adjusts the degrees of freedom for unequal variances. If you omit this flag, the pooled-variance t-test is used instead, which can inflate the Type I error rate when variances differ.
Two error types govern the risk of acting on experiment results. Understanding them is necessary for interpreting any statistical output.
Type I error (false positive): Concluding a variant outperforms the control when it does not. The probability of this error is bounded by the significance level α, but only if the test assumptions hold and the threshold is not crossed prematurely. α = 0.05 does not mean "there is a 5% chance this result is a false positive" for the specific experiment. It is a long-run frequency property: if the exact same experiment design were repeated infinitely many times, approximately 5% of those repetitions would reject the null when it is true. Any single experiment either has a false positive or it does not; the p-value does not calibrate the risk for that specific case.
Type II error (false negative): Failing to detect a genuine effect. The probability of this error is 1 − power. Power depends on sample size, effect magnitude, α, and the true underlying variance. A underpowered experiment may produce a non-significant p-value even when a meaningful effect exists, simply because the sample size was insufficient to detect it.
A common misconception is that reaching p < 0.05 guarantees the result is "real." It only guarantees that, under the assumptions and given the data, the observed difference would be unlikely under the null. The complement—that p ≥ 0.05 guarantees no effect—is false. A non-significant result may simply indicate low power.
One of the sharp edges in A/B testing is the temptation to check results before the sample size is reached. Each additional observation without a pre-planned interim analysis inflates the Type I error rate. The mathematical assumptions assume the decision to stop is made once, at the end of the planning period. If you look at the data at t=1, t=2, t=3, and stop as soon as p < 0.05, the actual false positive rate can be 2×, 3×, or higher than the nominal level.
Some platforms offer group sequential designs that pre-specify how α is spent across looks at the cost of requiring a larger total sample size. If your tooling does not support this, the safe practice is to fix the sample size in advance and never peek at statistical significance before it is met.
If you must peek, the O'Brien-Fleming and Pocock spending functions describe how to allocate α across looks. O'Brien-Fleming allocates very little α to early looks and most to the final look, making early stopping hard but the final test conservative. Pocock allocates roughly equal α across all looks, allowing earlier stopping but requiring a larger overall sample size to maintain the same false positive rate. Implementing either requires planning before the experiment starts; applying them post-hoc does not retroactively control the error rate.
When more than two variants are tested against a control, or when many metrics are evaluated on the same experiment, the family-wise error rate balloons. Testing k variants at α = 0.05 each gives a family-wise error rate of approximately 1 − (1 − 0.05)^k, which grows quickly with k.
Bonferroni adjusts the per-comparison threshold to α/k. Simple but conservative; power drops, especially when k is large or the variants are correlated.
Holm-Bonferroni is a step-down procedure that is uniformly more powerful than Bonferroni while still controlling the family-wise error rate. It orders the p-values from smallest to largest and compares each to α/(k − i + 1), where i is the rank.
Benjamini-Hochberg controls the false discovery rate (FDR) rather than the family-wise error rate. It is less stringent and often preferred when many hypotheses are tested and some false positives are tolerable. The procedure orders p-values and finds the largest rank i such that p_(i) ≤ (i/k) · q, where q is the desired FDR threshold; all hypotheses with rank ≤ i are rejected.
The choice depends on the cost of a false positive versus the cost of missing a real effect. If the experiment drives a high-stakes decision (e.g., launching a payment flow change), family-wise error control is appropriate. If you are screening many ideas and expect only a few to succeed, FDR control loses less power.
Standard A/B test calculations assume independent observations, approximately normal sampling distributions, and adequate sample sizes for the normal approximation to binomial (for proportion z-tests) or equal variances (for t-tests). When these assumptions fail, the p-value may not have the claimed meaning.
Unequal variances: If the two variants have different underlying variances, Welch's t-test provides a more accurate approximation. In scipy, scipy.stats.ttest_ind(..., equal_var=False) computes this automatically and adjusts the degrees of freedom via the Welch-Satterthwaite equation.
Overdispersion: In conversion rate experiments, observed variance may exceed the binomial variance. This inflates Type I error if unaccounted for. A quasi-binomial or beta-binomial model absorbs the extra-Poisson variation and produces more reliable confidence intervals. The key diagnostic is comparing the deviance residuals to the expected binomial variance; if the dispersion parameter φ is significantly greater than 1, a over-dispersed model is warranted.
Small sample sizes: When total exposures are low, the normal approximation to the binomial breaks down. Exact tests (Fisher's exact test for 2×2 tables) or bootstrapped confidence intervals avoid the approximation error. Fisher's exact test gives a conditional p-value that has no large-sample approximation and is valid for any sample size.
Non-independence: If users are assigned to variants multiple times (e.g., returning visitors), the independence assumption is violated and variance is underestimated. Cluster-robust standard errors or permutation tests that respect the assignment structure are needed. Permutation tests shuffle variant labels within clusters and recalculate the test statistic to build an empirical null distribution.
Not every experiment produces a statistically significant result, and that is information in itself. A non-significant p-value can mean several things:
The worst practice is to re-run the experiment with a lower threshold or to "tweak" the analysis until significance emerges. If the experiment is underpowered, the correct response is to acknowledge the inconclusive result, report the effect size and confidence interval, and either increase the sample size in a new experiment or accept that the effect, if it exists, is below the practical detection threshold.
If the normal approximation is questionable—small samples, binary outcomes with low base rates, or over-dispersed data—exact methods or resampling avoid the approximation error entirely.
Fisher's exact test for a 2×2 contingency table gives a conditional p-value that has no large-sample approximation. In R:
fisher.test(matrix(c(conversions_a, non_conversions_a,
conversions_b, non_conversions_b),
nrow = 2))Python's statsmodels.stats.contingency_tables.Table2x2 provides the same test with odds ratio and confidence limits.
Bootstrapped confidence intervals for the difference in means or proportions avoid the normality assumption by resampling the observed data. In Python:
import numpy as np
def boot_ci(data_a, data_b, n_boot=10000, alpha=0.05):
diffs = []
for _ in range(n_boot):
sample_a = np.random.choice(data_a, size=len(data_a), replace=True)
sample_b = np.random.choice(data_b, size=len(data_b), replace=True)
diffs.append(np.mean(sample_b) - np.mean(sample_a))
lower = np.percentile(diffs, 100 * (alpha / 2))
upper = np.percentile(diffs, 100 * (1 - alpha / 2))
return lower, upperThese methods are slower but more robust when the sampling distribution deviates from the assumed form.
When you need to compute statistics outside the experimentation platform, the data usually lives in a database. A typical query to pull variant-level conversion counts might look like this in psql:
SELECT
variant,
COUNT(*) AS exposed,
SUM(CASE WHEN converted = true THEN 1 ELSE 0 END) AS conversions
FROM experiment_runs
WHERE experiment_id = 'exp-2024-08-01'
AND event_timestamp >= '2024-08-01'
AND event_timestamp < '2024-08-08'
GROUP BY variant;From that result set, you feed the counts into whichever test function is appropriate for your stack. If you are using a platform with an API, the endpoint typically returns the same fields, and you apply the test in your own code rather than relying on the platform's reported p-value—which may or may not account for peeking, multiple variants, or assumption violations.
| Situation | Recommended Approach |
|---|---|
| Two variants, binary outcome, adequate N | Two-proportion z-test or chi-squared test |
| Two variants, continuous outcome, roughly normal | Welch's t-test (unequal variances assumed) |
| More than two variants without correction | Bonferroni, Holm-Bonferroni, or Benjamini-Hochberg depending on error-rate goal |
| Peeking before sample size reached | Pre-specify interim looks with group sequential design, or fix N and never peek |
| Small N, binary outcome | Fisher's exact test or bootstrapped CI |
| Overdispersed conversion data | Beta-binomial model or quasi-binomial GLM |
| Non-independent observations | Cluster-robust SEs or permutation test respecting assignment units |
Statistical rigor in A/B testing is not a property of the p-value alone. It is a property of the entire design—sample size planning, look schedule, correction for multiple hypotheses, assumption checking, and honest reporting of what the data (do and do not) support. When any one of those links breaks, the experiment result becomes unreliable, and the failure is often invisible until a bad product decision is already in production.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy