Hypothesis Testing Framework

Production engineering guide for hypothesis testing framework covering patterns, implementation strategies, and operational best practices.

Hypothesis testing is the backbone of evidence-based decision-making in data science. It transforms raw measurements into actionable conclusions: a new model improves accuracy, a feature change increases engagement, or a system upgrade reduces latency. This framework provides a standardized, repeatable process for evaluating whether observed differences are statistically significant, not mere noise. Use it when you need to answer “Did this change do what we expected?” with confidence, and when the stakes demand more than a glance at a dashboard.

Designing a Hypothesis Test

The first step is defining the test’s core components: the null and alternative hypotheses, the test statistic, and the significance threshold.

For example, to test whether a new recommendation engine increases click-through rate (CTR), define:

The test statistic is often a *t*-score or *z*-score, computed from the sample mean, standard deviation, and sample size.

Choosing the Test Statistic

Selecting the right test statistic depends on data distribution, sample size, and variance assumptions.

To compute a *t*-statistic for a one-sample test:

import numpy as np

def compute_t_statistic(sample, mu_0):
    sample_mean = np.mean(sample)
    sample_std = np.std(sample, ddof=1)
    n = len(sample)
    se = sample_std / np.sqrt(n)
    t = (sample_mean - mu_0) / se
    return t

This function returns a single scalar: the *t*-statistic. The output is used to compute the *p*-value.

Setting the Significance Level and Power

The significance level α controls the false positive rate: a test with α = 0.05 will reject H₀ even when it is true in 5% of repeated experiments.

Define the power of the test (1 − β) as the probability of correctly rejecting H₀ when H₁ is true. Power depends on:

In practice, power is set to 0.8 (80%) or higher. A test with 80% power detects a true effect with 80% probability.

Use a standard *t*-test power calculation:

from scipy.stats import ttest_ind_from_stats


mean_control, std_control, n_control = 0.15, 0.04, 1000
mean_treatment, std_treatment, n_treatment = 0.165, 0.04, 1000

# Compute test
t_stat, p_value = ttest_ind_from_stats(
    mean_control, std_control, n_control,
    mean_treatment, std_treatment, n_treatment,
    equal_var=False
)

# Interpret result
if p_value < 0.05:
    print("Reject null hypothesis: the treatment effect is significant.")
else:
    print("Fail to reject null hypothesis.")

This code computes a two-sample *t*-test from summary statistics—common when data is aggregated and stored in databases.

Executing the Test

Once the design is complete, execute the test by collecting data and computing the test statistic.

Data Collection and Pipeline Integration

Data must be collected under controlled conditions. Use a consistent sampling strategy: random, stratified, or time-based.

For a web platform, collect events via a logging pipeline:

{
  "event": "click",
  "user_id": "u12345",
  "timestamp": "2024-04-05T12:34:56Z",
  "experiment_id": "rec-engine-v2",
  "variant": "treatment",
  "page_id": "home",
  "click_time": 1.2
}

Aggregate this data into a fact table:

CREATE TABLE experiment_events AS
SELECT
    experiment_id,
    variant,
    COUNT(*) AS total_impressions,
    SUM(CASE WHEN event = 'click' THEN 1 ELSE 0 END) AS total_clicks,
    SUM(CASE WHEN event = 'click' THEN 1 ELSE 0 END) / COUNT(*) AS cta_rate
FROM events
WHERE experiment_id = 'rec-engine-v2'
  AND timestamp BETWEEN '2024-04-01' AND '2024-04-30'
GROUP BY experiment_id, variant;

This table feeds into the hypothesis test.

Computing the p-Value

The *p*-value is the probability of observing a test statistic as extreme as, or more extreme than, the one computed, assuming H₀ is true.

Use the *t*-distribution to compute the *p*-value:

from scipy.stats import t

def compute_p_value(t_stat, df):
    # Two-tailed test
    p_value = 2 * (1 - t.cdf(abs(t_stat), df))
    return p_value

For a one-sample *t*-test with 1000 observations:

A *p*-value < α (e.g., 0.05) leads to rejection of H₀.

Reporting the Results

Output the test result in a standardized format, including:

{
  "test_type": "two-sample t-test",
  "test_statistic": 2.4,
  "degrees_of_freedom": 999,
  "p_value": 0.016,
  "confidence_interval": {
    "level": 0.95,
    "lower": 0.152,
    "upper": 0.178
  },
  "effect_size": {
    "name": "Cohen's d",
    "value": 0.375
  },
  "conclusion": "Reject null hypothesis: the treatment increases CTR."
}

This JSON is consumed by dashboards, APIs, and alerting systems.

Validating Assumptions

A test is only as good as its assumptions. Failing to validate them silently undermines the entire analysis.

Normality and Homoscedasticity

Use the Shapiro-Wilk test for normality:

from scipy.stats import shapiro

def check_normality(data, alpha=0.05):
    stat, p_value = shapiro(data)
    is_normal = p_value > alpha
    return {
        "shapiro_statistic": stat,
        "p_value": p_value,
        "assumption_met": is_normal
    }

If p_value < 0.05, reject normality.

For homoscedasticity, use Levene’s test:

from scipy.stats import levene

def check_homoscedasticity(group1, group2, alpha=0.05):
    stat, p_value = levene(group1, group2)
    is_homogeneous = p_value > alpha
    return {
        "levene_statistic": stat,
        "p_value": p_value,
        "assumption_met": is_homogeneous
    }

A common failure: a *t*-test assumes equal variance but the data is heteroscedastic. The result is a *p*-value that is too optimistic.

Handling Violations

When assumptions fail:

from scipy.stats import mannwhitneyu

def mann_whitney_test(group1, group2, alternative='two-sided'):
    stat, p_value = mannwhitneyu(
        group1, group2,
        alternative=alternative,
        method='auto'
    )
    return {
        "test": "Mann-Whitney U",
        "test_statistic": stat,
        "p_value": p_value
    }

This test does not assume normality or equal variance, making it robust.

Multiple Testing and Error Control

When running many tests (e.g., multiple metrics, multiple variants), the chance of false positives grows.

Family-Wise Error Rate (FWER)

The family-wise error rate (FWER) is the probability of at least one false positive across all tests.

Use Bonferroni correction to control FWER:

For 10 tests and α = 0.05:

def bonferroni_correction(p_values, alpha=0.05):
    k = len(p_values)
    corrected_alpha = alpha / k
    significant = [p < corrected_alpha for p in p_values]
    return {
        "corrected_alpha": corrected_alpha,
        "significant": significant
    }

False Discovery Rate (FDR)

The false discovery rate (FDR) is the expected proportion of false positives among all significant tests.

Use Benjamini-Hochberg procedure:

def benjamini_hochberg(p_values, alpha=0.05):
    # Sort p-values and assign ranks
    sorted_indices = sorted(range(len(p_values)), key=lambda i: p_values[i])
    sorted_p = [p_values[i] for i in sorted_indices]
    
    # Compute threshold for each rank
    m = len(p_values)
    thresholds = [alpha * (i + 1) / m for i in range(m)]
    
    # Find largest i such that p_i ≤ threshold_i
    significant = [False] * m
    for i in reversed(range(m)):
        if sorted_p[i] <= thresholds[i]:
            significant[i] = True
        else:
            break
    
    # Map back to original order
    result = [False] * m
    for i, sig in enumerate(significant):
        result[sorted_indices[i]] = sig
    
    return {
        "significant": result,
        "adjusted_p_values": [
            min(p, alpha * (i + 1) / m) for i, p in enumerate(sorted_p)
        ]
    }

This method is more powerful than Bonferroni and widely used in genomics and A/B testing.

Common Pitfalls and Silent Failures

Automating the Framework

Integrate the hypothesis testing pipeline into a data workflow:

# Run hypothesis test for a given experiment
./run-hypothesis-test \
    --experiment-id "rec-engine-v2" \
    --metric "click-through-rate" \
    --control-variant "original" \
    --treatment-variant "v2" \
    --alpha 0.05 \
    --output-format json \
    --validate-assumptions true \
    --adjust-fdr benjamini-hochberg

The script:

Failure modes:

This framework becomes a reusable, auditable process for every data-driven decision.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.