A/B Testing Statistical Framework

A/B testing is the backbone of data-driven product iteration. This statistical framework defines how to design, execute, and interpret experiments that measure the causal effect of a change—be it a UI tweak, a pricing model, or a new algorithm—on a key outcome metric. It matters when you need to distinguish signal from noise across millions of user interactions, and when your team must make decisions with confidence, not consensus.

Designing an A/B Test

The first step is to define the experiment’s core components: units of assignment, treatment groups, and the primary outcome. Each user, session, or event is assigned to one of two or more variants (A/B/C) via a randomization mechanism. The assignment is recorded in a experiment_assignments table with columns: user_id, experiment_id, variant, assignment_timestamp, and assignment_context.

The experiment is described in a experiments table with columns: experiment_id, name, description, start_time, end_time, primary_metric, treatment_group, control_group, assignment_type, and sampling_rate. The assignment_type is one of user, session, or event, and determines how units are assigned and how metrics are aggregated.

For example, to define a test that evaluates a new onboarding flow:

INSERT INTO experiments (
  experiment_id,
  name,
  description,
  start_time,
  end_time,
  primary_metric,
  treatment_group,
  control_group,
  assignment_type,
  sampling_rate
) VALUES (
  'onboarding-v2',
  'New Onboarding Flow',
  'Test redesign of first-time user onboarding flow',
  '2024-04-01T00:00:00Z',
  '2024-06-30T23:59:59Z',
  'signup_completion_rate',
  'v2_flow',
  'original_flow',
  'user',
  1.0
);

The experiment’s sampling_rate controls how often users are assigned. A value of 1.0 assigns every user. A value of 0.75 assigns 75% of users, with the assignment recorded using a uniform random sample at assignment_timestamp.

Running the Experiment

Once deployed, the experiment must be tracked across the product stack. Each user interaction is logged into an events table with fields: event_id, user_id, event_type, timestamp, properties, and experiment_id. The properties field is a JSON object containing metadata such as variant, experiment_id, session_id, and device_type.

To ensure assignment consistency, the experiment’s assignment_type dictates the logic:

• User-level assignment: A user is assigned once and remains in the same variant throughout their lifetime. The experiment_assignments table is consulted during every session.
• Session-level assignment: Each session is independently assigned, allowing users to experience different variants across sessions. A new assignment is generated per session_id.
• Event-level assignment: Every event (e.g., a button click) is independently assigned to a variant, allowing for within-session variation. This is useful for A/B tests on real-time dashboards or dynamic content.

A typical event log entry for a user in the onboarding test:

{
  "event_id": "evt-12345",
  "user_id": "usr-67890",
  "event_type": "onboarding_step_completed",
  "timestamp": "2024-04-05T12:30:45Z",
  "properties": {
    "variant": "v2_flow",
    "experiment_id": "onboarding-v2",
    "step": 3,
    "session_id": "sess-7777",
    "device_type": "mobile"
  }
}

Computing Metrics

The primary outcome is computed from the events table. For a binary metric like signup_completion_rate, the analysis proceeds as follows:

• Identify all users who started the onboarding flow (e.g., event_type = 'onboarding_started').
• For each user, determine whether they completed the flow (e.g., event_type = 'onboarding_completed').
• Aggregate by variant and compute the proportion of completions.

This is done via a SQL query:

WITH user_sessions AS (
  SELECT
    e1.user_id,
    e1.experiment_id,
    e1.properties->>'variant' AS variant,
    MIN(e1.timestamp) AS session_start,
    MAX(e2.timestamp) AS session_end
  FROM events e1
  INNER JOIN events e2 ON e1.user_id = e2.user_id
    AND e2.event_type = 'onboarding_completed'
    AND e2.timestamp >= e1.timestamp
  WHERE e1.event_type = 'onboarding_started'
  GROUP BY e1.user_id, e1.experiment_id, e1.properties->>'variant'
),
completion_status AS (
  SELECT
    us.user_id,
    us.variant,
    us.experiment_id,
    CASE
      WHEN MAX(e2.timestamp) > us.session_start THEN 1
      ELSE 0
    END AS completed
  FROM user_sessions us
  LEFT JOIN events e2 ON e2.user_id = us.user_id
    AND e2.event_type = 'onboarding_completed'
    AND e2.timestamp >= us.session_start
  GROUP BY us.user_id, us.variant, us.experiment_id
)
SELECT
  variant,
  COUNT(*) AS total_users,
  SUM(completed) AS completed_users,
  SUM(completed) * 1.0 / COUNT(*) AS signup_completion_rate
FROM completion_status
GROUP BY variant;

For a continuous metric like time_to_first_purchase, the query computes the average time from first event to purchase, grouping by variant.

Estimating Effect Size and Confidence

The observed difference between variants is the point estimate of the treatment effect. The framework uses a two-sample t-test for continuous metrics and a z-test for binary metrics, assuming independent samples and normality.

For a binary metric, the standard error (SE) of the difference in proportions is:

SE = √(p₁(1−p₁)/n₁ + p₂(1−p₂)/n₂)

where p₁, p₂ are the proportions in the control and treatment groups, and n₁, n₂ are the sample sizes.

The 95% confidence interval is:

difference ± 1.96 × SE

For a continuous metric, the pooled standard error is:

SE = √(s₁²/n₁ + s₂²/n₂)

where s₁, s₂ are the sample standard deviations.

The test statistic is:

t = (mean₁ − mean₂) / SE

and the p-value is derived from the t-distribution with degrees of freedom based on sample sizes.

In practice, the test is computed in a script. A Python function for computing significance:

def compute_ab_test_result(control, treatment, alpha=0.05):
    n1, p1 = control['n'], control['p']
    n2, p2 = treatment['n'], treatment['p']
    
    pooled_p = (control['successes'] + treatment['successes']) / (n1 + n2)
    se = (
        pooled_p * (1 - pooled_p) * (1/n1 + 1/n2)
    ) ** 0.5
    z = (p1 - p2) / se
    p_value = 2 * (1 - norm.cdf(abs(z)))
    
    margin_error = 1.96 * se
    ci_lower = (p1 - p2) - margin_error
    ci_upper = (p1 - p2) + margin_error
    
    return {
        'difference': p1 - p2,
        'p_value': p_value,
        'ci_lower': ci_lower,
        'ci_upper': ci_upper,
        'significant': p_value < alpha,
        'se': se
    }

Interpreting Results

The primary output is a ab_test_results table with columns: experiment_id, variant, metric, value, n, p_value, ci_lower, ci_upper, is_significant, and analysis_timestamp.

A result row for the onboarding test:

{
  "experiment_id": "onboarding-v2",
  "variant": "v2_flow",
  "metric": "signup_completion_rate",
  "value": 0.68,
  "n": 15234,
  "p_value": 0.0034,
  "ci_lower": 0.652,
  "ci_upper": 0.708,
  "is_significant": true,
  "analysis_timestamp": "2024-05-10T14:22:10Z"
}

The framework assumes that significance is declared when p_value < 0.05, but this threshold can vary. The analysis must also flag multiple testing. When evaluating multiple metrics or variants, the framework applies a correction such as Bonferroni or Holm’s method.

For example, if three metrics are tested in the onboarding experiment (completion rate, time to first purchase, and session duration), the adjusted significance threshold is:

alpha_adj = alpha / number_of_tests

The framework also tracks preference bias—users who are more likely to be assigned to a variant due to their behavior. This is detected by comparing the distribution of user features (e.g., age, device type, sign-up source) between groups. A Kolmogorov-Smirnov test compares the cumulative distribution functions of a feature across groups. A significant difference indicates imbalance, potentially introducing confounding.

Handling Common Failures

• Assignment leakage: Users are assigned to a variant but the experiment is not applied. This occurs when the experiment_id is not properly propagated from the assignment step to the event logging pipeline. A common error is mismatched experiment_id in events and experiment_assignments. The fix is to add a JOIN on experiment_id and verify consistency via a query:

  SELECT
    e.experiment_id,
    COUNT(*) AS event_count,
    COUNT(DISTINCT e.user_id) AS unique_users,
    COUNT(DISTINCT a.user_id) AS assigned_users
  FROM events e
  FULL OUTER JOIN experiment_assignments a
    ON e.user_id = a.user_id AND e.experiment_id = a.experiment_id
  GROUP BY e.experiment_id;

• Late arrivals: Users who join the experiment after the start time may be missing data. The framework includes a filtering step that excludes users who entered the experiment before start_time but had no events before that time.

• Spillover effects: A user in the treatment group influences others. This is detected by comparing the outcome for users who were assigned but not exposed (e.g., they never visited the onboarding flow). The framework computes a treatment exposure metric: whether a user actually experienced the treatment. This is done via event_type = 'onboarding_flow_visited'.

• Non-constant treatment effect: The effect size varies over time. The framework computes a time-to-event analysis using Kaplan-Meier estimators and Cox proportional hazards models. The key insight is that the treatment effect may grow over time (e.g., users take longer to complete onboarding but improve over weeks).

Validating the Framework

The framework includes a set of sanity checks that run after every analysis:

• Assignment completeness: All users who should be assigned are present.
• Event coverage: At least 95% of assigned users have at least one event during the experiment period.
• Metric consistency: The same metric computed on raw events and aggregated data match within 1%.
• p-value stability: Re-running the analysis on a rolling window of data (e.g., last 7 days) shows consistent results.

A failure in any check triggers an alert via a alerting system:

{
  "alert_id": "ab-test-validation-failure",
  "experiment_id": "onboarding-v2",
  "check": "assignment_completeness",
  "status": "failed",
  "timestamp": "2024-05-10T15:00:00Z",
  "details": {
    "expected_users": 16000,
    "assigned_users": 15234,
    "missing_users": 766,
    "missing_user_ids": [ "usr-1001", "usr-2045", "usr-3012" ]
  }
}

The framework is designed to be composable: each step can be run independently, and the results are versioned and reproducible. The analysis_timestamp ensures that the experiment’s interpretation is tied to the state of the data at the time of analysis.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.