A/B Testing at Scale | The Garnet Wiki

A/B testing at scale is the practice of simultaneously evaluating multiple variants of a product change across millions of users, with automated, end-to-end workflows that handle experiment creation, treatment assignment, data collection, and statistical analysis—each step tuned for throughput, reliability, and consistency across a large, dynamic user base. It matters when a product team must validate small changes with high confidence, and when the cost of a wrong decision—such as a misaligned feature release or a suboptimal user interface—outweighs the effort of maintaining a sophisticated testing infrastructure.

Experiment Lifecycle Management

Creating and managing experiments at scale requires a well-defined workflow that spans design, deployment, and operational monitoring. Each experiment is defined by a set of core parameters: experiment_id, name, description, status, start_time, end_time, variants, and recruitment_strategy. These are stored in a central database, typically a relational system with a experiments table.

CREATE TABLE experiments (
    experiment_id UUID PRIMARY KEY,
    name TEXT NOT NULL,
    description TEXT,
    status TEXT CHECK (status IN ('draft', 'active', 'paused', 'completed')),
    start_time TIMESTAMPTZ NOT NULL,
    end_time TIMESTAMPTZ,
    variants JSONB NOT NULL,
    recruitment_strategy JSONB NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

The variants field is a JSON object mapping variant names (e.g., "control", "variant-A") to their configuration, including treatment_id, weight, and activation_condition. The recruitment_strategy defines how users are assigned to the experiment. A common pattern is bucketed assignment, where users are assigned to an experiment based on a hash of their user_id or session_id.

{
  "recruitment_strategy": {
    "type": "bucketed",
    "key": "user_id",
    "bucket_count": 10000,
    "seed": 42,
    "assignment_window": "all-time"
  }
}

The assignment_window can be "all-time", "active-users-only", or "last-30-days", with the latter used to recruit users who have interacted with the product recently. When a user is recruited, an assignment record is inserted into a user_assignments table.

CREATE TABLE user_assignments (
    user_id UUID,
    experiment_id UUID,
    variant_name TEXT,
    assignment_time TIMESTAMPTZ NOT NULL,
    bucket_id INTEGER,
    PRIMARY KEY (user_id, experiment_id)
);

A common failure mode is assignment drift: users change variants over time due to inconsistent hashing, leading to contamination across experiment groups. This occurs when the seed or key changes without updating the assignment logic. To detect drift, a nightly job computes the variant distribution over time and flags inconsistencies.

Treatment Delivery and Feature Flags

Once a user is assigned to an experiment, the treatment is applied via feature flags. The treatment is configured in a feature_flags table, which supports hierarchical configuration.

CREATE TABLE feature_flags (
    flag_name TEXT PRIMARY KEY,
    enabled BOOLEAN NOT NULL,
    default_value JSONB,
    variant_overrides JSONB,
    rollout_strategy JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

For an A/B test, each variant maps to a distinct variant_overrides configuration. For example, variant-A might override theme to "dark" and font_size to 16.

{
  "flag_name": "feature_homepage_layout",
  "enabled": true,
  "default_value": {
    "layout_type": "grid"
  },
  "variant_overrides": {
    "control": {
      "layout_type": "grid"
    },
    "variant-A": {
      "layout_type": "cards",
      "show_trending": true
    },
    "variant-B": {
      "layout_type": "masonry",
      "show_related": false
    }
  }
}

The service that evaluates feature flags must resolve the variant at runtime. The evaluation pipeline typically follows this path:

Fetch the user’s experiment_id from session or user data.
Look up the experiment record to get variants and recruitment_strategy.
Compute the bucket ID using the key and seed.
Query user_assignments to get the current variant_name.
Fetch feature_flags for flag_name.
Apply variant_overrides to default_value.

This process is idempotent and stateless, enabling deployment across multiple microservices. A common failure is flag misalignment: the variant_overrides are not synchronized across services. One service might use variant-A but a different service uses variant-A with slightly different JSON. The root cause is often versioning of the experiment configuration without a migration step.

Data Collection and Event Schema

Each experiment relies on event data to measure outcomes. Events are logged in a user_events table with a consistent schema.

CREATE TABLE user_events (
    event_id UUID PRIMARY KEY,
    user_id UUID NOT NULL,
    experiment_id UUID,
    variant_name TEXT,
    event_type TEXT NOT NULL,
    event_data JSONB,
    timestamp TIMESTAMPTZ NOT NULL,
    processed_at TIMESTAMPTZ,
    FOREIGN KEY (experiment_id) REFERENCES experiments(experiment_id)
);

Standard event types include page_view, click, purchase, session_start, and experiment_started. The experiment_started event is emitted when a user first enters an experiment and includes the variant_name and assignment_time.

{
  "event_type": "experiment_started",
  "event_data": {
    "experiment_id": "e1a2b3c4-d5e6-f7g8-h9i0-j1k2l3m4n5o6",
    "variant_name": "variant-A",
    "assignment_time": "2023-11-01T10:00:00Z"
  },
  "timestamp": "2023-11-01T10:00:01Z"
}

For outcome measurement, a metric table defines the aggregations and definitions.

CREATE TABLE metrics (
    metric_id UUID PRIMARY KEY,
    experiment_id UUID,
    name TEXT NOT NULL,
    definition TEXT NOT NULL,
    aggregation_type TEXT CHECK (aggregation_type IN ('count', 'sum', 'avg', 'rate')),
    event_type TEXT,
    window_duration INTERVAL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

For example, the click-through-rate metric is defined as:

{
  "metric_id": "ctr-1",
  "experiment_id": "e1a2b3c4-d5e6-f7g8-h9i0-j1k2l3m4n5o6",
  "name": "click-through-rate",
  "definition": "Ratio of click events to page_view events",
  "aggregation_type": "rate",
  "event_type": "click",
  "window_duration": "7 days"
}

A frequent failure is event delay: events are logged with a lag of hours or days due to batch processing. This leads to inaccurate early-stage results. To mitigate, a metrics_materialization job runs every 15 minutes to compute daily aggregations and store them in a metric_results table.

Statistical Analysis and Result Reporting

Statistical analysis at scale begins with the metric_results table:

CREATE TABLE metric_results (
    experiment_id UUID,
    metric_id UUID,
    variant_name TEXT,
    value DOUBLE PRECISION,
    confidence_interval JSONB,
    n_users INTEGER,
    n_events INTEGER,
    analysis_time TIMESTAMPTZ,
    PRIMARY KEY (experiment_id, metric_id, variant_name)
);

The confidence_interval is stored as a JSON object with lower, upper, and method fields.

{
  "lower": 0.152,
  "upper": 0.168,
  "method": "t-test",
  "level": 0.95
}

A dedicated analysis_service runs the statistical pipeline. It fetches all metric_results, computes pairwise differences between variants, and applies a significance test.

def compute_significance(results: List[Dict]) -> List[Dict]:
    from scipy.stats import ttest_ind_from_stats
    comparisons = []
    for i, row1 in enumerate(results):
        for j, row2 in enumerate(results):
            if i >= j:
                continue
            variant1, variant2 = row1['variant_name'], row2['variant_name']
            # Extract stats from results
            stat1 = row1['value'], row1['n_users'], row1['n_events']
            stat2 = row2['value'], row2['n_users'], row2['n_events']
            t_stat, p_value = ttest_ind_from_stats(*stat1, *stat2)
            comparisons.append({
                "variant1": variant1,
                "variant2": variant2,
                "difference": row2['value'] - row1['value'],
                "p_value": p_value,
                "t_statistic": t_stat,
                "significant": p_value < 0.05
            })
    return comparisons

The results are stored in a experiment_results table.

CREATE TABLE experiment_results (
    experiment_id UUID,
    analysis_id UUID,
    status TEXT,
    summary JSONB,
    comparisons JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    FOREIGN KEY (experiment_id) REFERENCES experiments(experiment_id)
);

The summary field contains overall experiment metrics: total_users, conversion_rate, avg_time_on_page, and statistical_power.

Failure Modes and Operational Insights

Several silent failures are common in large-scale A/B testing:

Assignment inconsistency: Users are assigned to an experiment but their variant_name does not match the bucket_id. This happens when the hashing algorithm is not synchronized across services. A check query comparing user_assignments with a re-computed bucket map will reveal mismatches.

Missing events: Events are logged but not processed. The processed_at field in user_events is NULL for events older than 24 hours. A monitoring alert checks for event_count > 0 AND processed_at IS NULL and triggers a restart of the processing pipeline.

Delayed analysis: The analysis_service runs hourly but sometimes takes 45 minutes to complete. This leads to stale results in the reporting dashboard. The analysis_time is used to detect late completions and to trigger re-analysis.

Variant contamination: A user is assigned to variant-A but later sees variant-B due to a feature flag rollout. The root cause is often that the feature flag is configured for the experiment but not updated when a new variant is added. A variant_consistency_check job compares experiment.variants with actual user_events and flags mismatches.

Metric drift: A metric definition changes over time, but old data uses the old definition. The metric table includes version and definition_hash fields. A metric_versioning pipeline computes the hash of the definition and stores it. When a new analysis runs, it joins metric_results with metric_versions to ensure consistency.

High-Throughput Experiment Workflows

At scale, experiments are orchestrated via a workflow engine. A typical workflow consists of:

Experiment creation: A POST /experiments request with JSON payload.
Assignment job: A cron job runs assign_users(experiment_id) every 6 hours.
Event ingestion: Services emit events via POST /events with application/json payloads.
Materialization: A pipeline consumes events and updates user_events and metric_results.
Analysis: A service runs analyze_experiment(experiment_id) daily.
Reporting: A dashboard service queries experiment_results and metric_results.

This workflow is resilient to failure. Each step is idempotent, and error logs are stored with error_id, step_name, experiment_id, and details.

A critical edge is experiment versioning: when a new experiment is rolled out, it must coexist with older versions. This is achieved by versioning the experiment_id and using experiment_version in the user_assignments table. A user can be part of multiple experiments simultaneously, each with their own variant_name and assignment_time.

Summary

A/B testing at scale is not just about running experiments—it is about building a repeatable, observable, and reliable system for validating product changes across a large, evolving user base. It rests on consistent data schemas, idempotent workflows, and careful attention to failure modes. The core of the system is the loop from assignment to event collection to statistical analysis, each step designed to handle high volume and low latency. The success of a large-scale A/B program depends not on the elegance of the statistical model, but on the robustness of this end-to-end pipeline.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.