Ab Test Power Analysis

Ab test power analysis determines the sample size and duration needed to detect a statistically significant effect in an A/B test with a desired probability, given a target minimum detectable effect (MDE), significance level, and effect size. It is essential when designing new experiments, especially when trade-offs between speed, cost, and statistical confidence are high. Power analysis is not a post-hoc check; it is a foundational step in experiment planning, ensuring that the test will not fail to detect a real improvement due to insufficient data.

Planning an A/B Test with Required Power

When launching a new feature or changing a core workflow, teams must decide: How many users must be exposed to each variant? How long should the test run? The answer lies in power analysis. The goal is to compute the number of observations required per group (e.g., control and treatment) to achieve a desired statistical power—typically 80% or 90%—to detect a minimum effect size at a specified significance threshold (usually α = 0.05).

Compute Required Sample Size

The core formula for sample size in a two-sample t-test is:

\[ n = \frac{2 \cdot (z_{\alpha/2} + z_{\beta})^2 \cdot \sigma^2}{\delta^2} \]

where:

• \(n\) = sample size per group,
• \(z_{\alpha/2}\) = z-score for significance level (e.g., 1.96 for α = 0.05),
• \(z_{\beta}\) = z-score for desired power (e.g., 0.84 for 80% power),
• \(\sigma^2\) = variance of the metric,
• \(\delta\) = minimum detectable effect (MDE).

This formula assumes a normal distribution, equal group sizes, and a continuous outcome metric.

In practice, this is implemented via a function or script. The following Python function computes required sample size for a binary metric (e.g., conversion rate), using the normal approximation:

import numpy as np

def ab_test_power_analysis(
    baseline_rate: float,
    mde_rate: float,
    significance_level: float = 0.05,
    power: float = 0.80,
    metric_type: str = "binary"
):
    """
    Compute required sample size per group for an A/B test.

    Parameters:
    - baseline_rate: conversion rate in control group (0.0 to 1.0)
    - mde_rate: expected conversion rate in treatment group
    - significance_level: type I error rate (default: 0.05)
    - power: statistical power (default: 0.80)
    - metric_type: "binary" or "continuous"

    Returns:
    - n_per_group: required sample size per group
    """
    alpha = significance_level
    z_alpha = np.quantile(np.random.normal(0, 1, 100_000), 1 - alpha / 2)
    z_power = np.quantile(np.random.normal(0, 1, 100_000), power)

    if metric_type == "binary":
        p1 = baseline_rate
        p2 = mde_rate
        pooled_p = (p1 + p2) / 2
        variance = pooled_p * (1 - pooled_p)
        effect_size = abs(p2 - p1)
        n_per_group = (
            2 * (z_alpha + z_power)**2 * variance
        ) / (effect_size**2)
    elif metric_type == "continuous":
        sigma = 1.0  # assumed or estimated from pilot data
        effect_size = abs(mde_rate - baseline_rate)
        n_per_group = (
            2 * (z_alpha + z_power)**2 * sigma**2
        ) / (effect_size**2)
    else:
        raise ValueError("metric_type must be 'binary' or 'continuous'")

    return int(np.ceil(n_per_group))


n = ab_test_power_analysis(
    baseline_rate=0.15,
    mde_rate=0.17,
    significance_level=0.05,
    power=0.80,
    metric_type="binary"
)
print(f"Required sample size per group: {n}")
# Output: Required sample size per group: 1355

This script computes that 1,355 users per group are needed to detect a 2 percentage point lift in conversion rate (from 15% to 17%) with 80% power at α = 0.05.

Determine Test Duration

Once the per-group sample size is known, the next step is to estimate how long the test must run. This depends on the daily traffic to the experiment and the fraction of users assigned to the test.

Let:

• \(N\) = total number of users needed per group,
• \(T\) = average number of users per day,
• \(f\) = fraction of users exposed to the experiment.

Then: \[ \text{Duration (days)} = \frac{N}{f \cdot T} \]

For example, if a site has 10,000 daily active users and the experiment is rolled out to 60% of them, then the daily exposure is \(0.6 \times 10,000 = 6,000\) users. With 1,355 users per group needed, the test must run:

\[ \frac{1355}{6000} \approx 0.226 \text{ days per group} \Rightarrow \text{total duration} = \frac{1355}{6000} \times 2 \text{ (for both groups)} \approx 0.45 \text{ days} \]

Since users are split across control and treatment, the total number of users required across the entire experiment is \(2 \times 1,355 = 2,710\). The test runs for:

\[ \frac{2,710}{6,000} \approx 0.45 \text{ days} = \text{about 11 hours} \]

This highlights a common pitfall: assuming that daily users are independent, but in practice, users may be reused across multiple experiments, leading to overlap and correlation. The actual duration must account for user overlap, repeat exposure, and non-uniform traffic.

Adjust for Multiple Metrics and Multiple Comparisons

In complex experiments, teams measure multiple outcome metrics (e.g., conversion, time-on-site, click-through rate). Each metric has its own variance and MDE. Power analysis must be performed per metric, and the final test duration is driven by the metric requiring the most observations.

When comparing multiple variants (e.g., A/B/C/D), the power analysis must account for family-wise error rate (FWER) or false discovery rate (FDR). For example, with three comparisons (A vs. B, A vs. C, B vs. C), the Bonferroni correction adjusts the significance threshold:

\[ \alpha_{\text{adjusted}} = \frac{\alpha}{k} \]

where \(k\) is the number of comparisons. This reduces the per-comparison α, increasing the required sample size.

For a test with three groups, the sample size per group is adjusted as follows:

def ab_test_power_analysis_multigroup(
    baseline_rate: float,
    mde_rate: float,
    groups: int = 3,
    significance_level: float = 0.05,
    power: float = 0.80,
    correction: str = "bonferroni"
):
    alpha = significance_level
    if correction == "bonferroni":
        alpha_adj = alpha / (groups * (groups - 1) / 2)  # pairwise comparisons
    elif correction == "holm":
        # Holm-Bonferroni: more powerful than Bonferroni
        alpha_adj = alpha / groups
    else:
        alpha_adj = alpha

    # Use same function as before but with adjusted alpha
    n_per_group = ab_test_power_analysis(
        baseline_rate=baseline_rate,
        mde_rate=mde_rate,
        significance_level=alpha_adj,
        power=power,
        metric_type="binary"
    )
    return n_per_group

This function shows that adjusting for multiple comparisons increases the required sample size. For a 3-group test with Bonferroni correction, the effective α drops from 0.05 to roughly 0.0167, raising the sample size requirement by ~25%.

Validating Power Analysis in Practice

Common Failures and Silent Errors

1. Underestimating variance: The assumed standard deviation or baseline conversion rate is too optimistic. In production, the observed variance is often 15–30% higher than in pilot data. The most common error is using a single-point estimate from a small cohort without accounting for uncertainty.

1. Ignoring time-series effects: Metrics like daily active users or session duration exhibit autocorrelation. A user’s outcome on day 1 is correlated with day 2. This reduces effective sample size. The standard power calculation assumes independent observations, but in reality, observations are clustered by user and by time.

To correct for this, the effective sample size is:

\[ n_{\text{eff}} = \frac{n}{1 + (m - 1) \cdot \rho} \]

where:

• \(n\) = nominal sample size,
• \(m\) = average number of observations per user,
• \(\rho\) = intra-class correlation (ICC).

If each user contributes 5 sessions and \(\rho = 0.2\), then:

\[ n_{\text{eff}} = \frac{n}{1 + 4 \cdot 0.2} = \frac{n}{1.8} \Rightarrow \text{effective sample size is 56% of nominal} \]

The experiment must be run longer or with more users to compensate.

1. Incorrect treatment assignment: Users are assigned to treatment groups at the start of the experiment, but users may switch groups or be exposed to multiple experiments. This violates the assumption of fixed treatment exposure. The result is dilution of the treatment effect and reduced power.

1. Unequal group sizes: The power analysis assumes equal sample sizes in control and treatment. However, in practice, the treatment group may be larger due to rollout timing, leading to a loss of power. The power formula must be adjusted for unequal variances and group sizes.

1. Multiple testing with correlated metrics: When metrics are correlated (e.g., conversion and time-on-site), the Bonferroni correction is overly conservative. A better approach is to use the multivariate normal power analysis with a covariance matrix.

Output and Integration

The output of power analysis is typically a configuration block or a spreadsheet that defines:

• Target metric,
• Baseline and MDE,
• Significance level,
• Desired power,
• Assumed variance,
• Required sample size per group,
• Estimated test duration,
• Adjustments for multiple comparisons or clustering.

This is consumed by experiment platforms and integrated into A/B testing workflows. A typical configuration in a YAML file might look like:

experiment:
  name: "checkout-flow-redesign"
  metric: "conversion_rate"
  baseline: 0.15
  mde: 0.17
  significance_level: 0.05
  power: 0.80
  metric_type: "binary"
  variance: 0.1275
  n_per_group: 1355
  total_users: 2710
  duration_days: 0.45
  adjustments:
    - method: "bonferroni"
      comparisons: 3
      alpha_adjusted: 0.0167
    - method: "clustering"
      intraclass_correlation: 0.2
      average_observations_per_user: 5
      effective_sample_size: 0.56

This configuration is used to:

• Schedule the experiment,
• Validate the rollout,
• Monitor progress,
• Report results with expected power.

Tools and Workflows

Power analysis is typically performed in Python, R, or SQL. A common workflow is:

1. Run a pilot study to estimate baseline conversion rate and variance.
2. Use the ab_test_power_analysis function to compute required sample size.
3. Query the data warehouse to estimate daily traffic and user exposure.
4. Compute test duration.
5. Adjust for clustering, multiple metrics, and multiple comparisons.
6. Output configuration and share with engineering and product teams.

The process is repeated for each major experiment, and the outputs are versioned in a shared repository.

Failure Modes

• The test runs too short: Users are enrolled, but the test concludes before the full sample size is reached. The result is underpowered tests, failing to detect real effects.
• The test is overpowered: The experiment runs longer than needed, incurring unnecessary cost. This is common when the MDE is too optimistic or the variance is overestimated.
• The power analysis is not updated: After the first experiment, the team assumes the same baseline and variance for all future tests, but traffic, user behavior, and product changes shift the landscape. Without periodic re-evaluation, power analysis becomes obsolete.

Summary

Power analysis is not a one-time exercise but a core engineering practice in A/B testing. It ensures that experiments are designed to detect real effects with high probability, avoiding false negatives and wasted resources. The key is to move from abstract formulas to concrete, executable code that integrates into real workflows. The most valuable insight is that power analysis must account for real-world complexities: correlated observations, unequal group sizes, overlapping experiments, and evolving user behavior. A well-executed power analysis is the foundation of a statistically rigorous and operationally sustainable A/B testing program.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.