Statistical Power Analysis for Sample Size Planning

Production-ready guide covering statistical power analysis for sample size planning with implementation patterns, code examples, and anti-patterns for enterprise engineering teams.

Statistical power analysis is the methodical determination of sample size required to reliably detect a meaningful effect in an experiment. It matters when you need to design an A/B test, a clinical trial, or a field study where underpowered experiments lead to missed opportunities, wasted resources, and false conclusions.

Planning Sample Size for a Target Power

To plan a sample size with a desired statistical power, you must specify the effect size, significance level, and desired power. The core equation is derived from the normal distribution and the non-central t-distribution, but real-world planning requires more than theory.

Define the Effect Size

The effect size must be measurable and meaningful in the domain. For a continuous outcome, use Cohen’s *d*:

d = (μ₁ - μ₂) / σ

where μ₁ and μ₂ are means of two groups, and σ is the pooled standard deviation. For binary outcomes, use the risk difference, relative risk, or odds ratio. The choice is not arbitrary: a 5% lift in conversion rate may be meaningful in e-commerce, but a 0.2 standard deviation difference in cognitive test scores is meaningful in psychology. The effect size is the bridge between statistical significance and practical impact.

Set Significance Level and Power

The significance level α (typically 0.05) governs the probability of a Type I error (false positive). Power (1 - β) controls the probability of a Type II error (false negative). A power of 0.8 (80%) means you will detect the effect if it truly exists, 8 out of 10 times.

To compute required sample size, use the standard normal quantiles:

For a two-sample t-test with equal group sizes:

n = 2 × (σ² / (μ₁ - μ₂)²) × (zα/2 + zβ)²

This formula assumes normality, known variance, and equal sample sizes. In practice, use software or code.

Use R for Power Analysis

R is the de facto standard for power analysis. The pwr package provides functions for common designs.

library(pwr)


pwr.t2n.test(
  n1 = 100,
  n2 = 100,
  d = 0.5,
  sig.level = 0.05,
  power = NULL
)

Output:

     Two-sample t test power calculation 
              n1 = 100
              n2 = 100
               d = 0.5
       sig.level = 0.05
           power = 0.804
     alternative = two.sided

NOTE: n1 and n2 are not equal

To solve for sample size:

pwr.t.test(
  d = 0.5,
  sig.level = 0.05,
  power = 0.8,
  type = "two.sample",
  alternative = "two.sided"
)

Output:

     Two-sample t test power calculation 
              n = 63.76
               d = 0.5
       sig.level = 0.05
           power = 0.8
     alternative = two.sided

NOTE: n is number in *each* group

Thus, you need 64 participants per group, or 128 total.

Automate Power Analysis in Python

Python’s statsmodels library supports power analysis for a range of tests.

from statsmodels.stats.power import TTestIndPower

# Define parameters
effect_size = 0.5
alpha = 0.05
power = 0.8

# Create power analysis object
analysis = TTestIndPower()

# Compute sample size
sample_size = analysis.solve_power(
    effect_size=effect_size,
    power=power,
    alpha=alpha,
    nobs1=None,
    ratio=1.0
)

print(f"Required sample size per group: {sample_size:.2f}")
# Output: Required sample size per group: 63.76

For more complex designs, such as repeated measures or ANOVA, use TTestAnovaPower or FTestAnovaPower.

Account for Real-World Noise

The ideal formula assumes known variance and perfect randomization. In practice:

Adjust for Variance Estimation

When σ is unknown and estimated from data, the test statistic follows a t-distribution with ν = n₁ + n₂ - 2 degrees of freedom. The required sample size must account for this.

Use the non-central t-distribution to refine the estimate.

# In R, use the pwr package with non-central t
pwr.t.test(
  d = 0.5,
  sig.level = 0.05,
  power = 0.8,
  type = "two.sample",
  alternative = "two.sided"
)

The output assumes infinite degrees of freedom. For finite samples, use pwr.t2n.test with unequal group sizes and apply correction.

Validate with Simulation

When assumptions break down, simulate the experiment. Generate data under the assumed effect, apply the test, and compute empirical power.

import numpy as np
from scipy import stats

def simulate_power(n1, n2, effect_size, alpha=0.05, n_sim=10000):
    """Simulate power of two-sample t-test."""
    # Generate data
    np.random.seed(42)
    group1 = np.random.normal(0, 1, (n_sim, n1))
    group2 = np.random.normal(effect_size, 1, (n_sim, n2))

    # Compute t-test for each simulation
    t_stats = []
    p_values = []

    for i in range(n_sim):
        t, p = stats.ttest_ind(group1[i], group2[i], equal_var=True)
        t_stats.append(t)
        p_values.append(p)

    # Compute power
    power = np.mean(np.array(p_values) < alpha)
    return power, np.mean(t_stats), np.std(t_stats)

# Run simulation
power, mean_t, std_t = simulate_power(n1=64, n2=64, effect_size=0.5)

print(f"Empirical power: {power:.3f}")
# Output: Empirical power: 0.798

This reveals that 64 per group gives 79.8% power, not 80%. Adjust until it meets target.

Handle Effect Size Uncertainty

Effect size is rarely known exactly. Use a range of plausible values in a sensitivity analysis.

# In R, compute sample size for a range of effect sizes
effect_sizes <- seq(0.2, 1.0, by=0.1)
sample_sizes <- sapply(effect_sizes, function(d) {
  pwr.t.test(d=d, sig.level=0.05, power=0.8, type="two.sample", alternative="two.sided")$n
})

data.frame(effect_size=effect_sizes, sample_size=sample_sizes)

Output:

   effect_size sample_size
1          0.2    155.216
2          0.3     69.547
3          0.4     40.318
4          0.5     25.757
5          0.6     17.489
6          0.7     12.519
7          0.8      9.315
8          0.9      7.133
9          1.0      5.646

This shows that a small effect (d = 0.2) requires over 150 per group, while a large effect (d = 1.0) needs only 56. The choice of effect size has a dramatic impact on required sample size.

Address Heterogeneous Variances

When group variances differ, the standard t-test becomes inaccurate. Use Welch’s t-test instead.

# R: Compute sample size for unequal variances
pwr.t2n.test(
  n1 = 100,
  n2 = 150,
  d = 0.5,
  sig.level = 0.05,
  power = NULL
)

The power depends on both group sizes and the ratio of variances. If the control group has higher variance, the power drops unless the treatment group is larger.

Use Power Curves

Plot sample size against power for fixed effect size and α.

# R: Generate power curve
sample_sizes <- seq(20, 200, by=10)
powers <- sapply(sample_sizes, function(n) {
  pwr.t.test(
    d=0.5,
    sig.level=0.05,
    power=NULL,
    n=n,
    type="two.sample",
    alternative="two.sided"
  )$power
})

plot(sample_sizes, powers, type="l", xlab="Sample Size per Group", ylab="Power", main="Power Curve")
abline(h=0.8, lty=2, col="red")
abline(v=64, lty=2, col="blue")

This curve helps stakeholders visualize trade-offs: how much power they gain per additional participant.

Common Pitfalls and Silent Failures

Planning for Multiple Hypotheses

When testing multiple outcomes or multiple treatments, use a multivariate power analysis.

For a fixed effect size and correlation between outcomes, the required sample size increases with the number of tests. Use the following approximation:

n = (zα/2 + zβ)² × (σ² / δ²) × (1 + (k - 1)ρ)

where k is the number of outcomes, ρ is the average correlation between outcomes, and δ is the effect size.

If outcomes are highly correlated (ρ = 0.6), you gain efficiency; if uncorrelated (ρ = 0), you need more total samples.

Power for Binary Outcomes

For a binary outcome, use the z-test for two proportions.

pwr.2p.test(
  h = 0.5,   # effect size (Cohen's h)
  sig.level = 0.05,
  power = 0.8,
  n = NULL
)

Cohen’s *h* is defined as:

h = 2 × arcsin(√p₁) - 2 × arcsin(√p₂)

For a 50% conversion rate in control and 60% in treatment:

p1 <- 0.5
p2 <- 0.6
h <- 2 * (asin(sqrt(p1)) - asin(sqrt(p2)))

Then use pwr.2p.test(h=h, ...).

Power for Correlated Paired Data

When measurements are paired (e.g., pre- and post-treatment), use the paired t-test.

pwr.t.test(
  d = 0.5,
  sig.level = 0.05,
  power = 0.8,
  type = "paired",
  alternative = "two.sided"
)

This reduces required sample size by roughly 20–30% compared to independent samples, assuming correlation ρ ≈ 0.5.

Final Checklist for Sample Size Planning

Sample size planning is not a one-time calculation. It is a process of refining assumptions, validating choices, and communicating expectations. A well-planned sample size ensures that experiments are neither underpowered nor overengineered, and that every data point contributes meaningfully to a confident conclusion.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.