Build experimentation infrastructure that produces trustworthy results. Covers statistical foundations, feature flag integration, sample size calculations, metric selection, guardrail metrics, and the organizational patterns that prevent HiPPO-driven decisions.
A/B testing infrastructure enables teams to evaluate feature changes in production with confidence, reducing the risk of introducing regressions while measuring impact. It matters when teams deploy code frequently, need to validate hypotheses beyond unit tests, and want to ship decisions—not just features—across engineering, product, and data teams.
The first step is defining a test with clear treatment and control groups. Tests are described using a schema in a shared configuration store, such as a database table or a set of YAML files in version control. Each test has a unique test_id, a description, and a status field (draft, active, archived). The core configuration specifies which exposure rules determine which users see which variant.
test_id: checkout-flow-v2
description: |
Evaluate a new checkout flow with collapsed sections and inline validation.
status: active
start_date: 2024-05-10
end_date: 2024-06-10
variants:
- id: control
name: "Old Flow"
weight: 0.7
- id: treatment
name: "New Flow"
weight: 0.3
exposure_rules:
- type: user_id_hash
field: user_id
min_value: 0
max_value: 1000000
bucket: 10000
variant: control
- type: user_id_hash
field: user_id
min_value: 1000000
max_value: 2000000
bucket: 10000
variant: treatmentThis configuration is versioned in Git, and the active test is deployed via a CI pipeline that validates the schema and pushes changes to a central experiment registry. The registry is queried by the feature rollout service using a REST API:
GET /v1/experiments/checkout-flow-v2?user_id=123456Response:
{
"test_id": "checkout-flow-v2",
"variant": "treatment",
"variant_id": "new-flow",
"experiment_version": "v3",
"metadata": {
"assigned_at": "2024-05-10T14:23:01Z",
"bucket": 123456
}
}If a user has not been assigned to a test, the service assigns them using consistent hashing based on user_id, ensuring that the same user always sees the same variant across sessions.
Once deployed, experiments must be instrumented to collect event data. Every user interaction is logged with a timestamp, user_id, experiment_id, variant_id, and event_type. Events are streamed into a real-time data pipeline using Apache Kafka or Amazon Kinesis.
{
"user_id": 123456,
"experiment_id": "checkout-flow-v2",
"variant_id": "new-flow",
"event_type": "checkout_started",
"timestamp": "2024-05-10T14:23:01.456Z",
"metadata": {
"page_url": "/checkout",
"device_type": "mobile"
}
}These events are materialized into a feature store or a data warehouse (e.g., BigQuery, Redshift, or ClickHouse) where they are joined with user profiles and session data. Queries to analyze test outcomes use standard SQL with window functions and cohort filters.
SELECT
e.variant_id,
COUNT(*) AS sessions,
AVG(s.time_on_page) AS avg_time_on_page,
SUM(s.checkout_completed) AS conversions
FROM experiments e
JOIN sessions s ON e.user_id = s.user_id
WHERE e.experiment_id = 'checkout-flow-v2'
AND e.timestamp >= '2024-05-10'
AND e.timestamp < '2024-06-10'
GROUP BY e.variant_id;The pipeline is monitored via a dashboard showing real-time event ingestion rates, assignment latency, and experiment coverage. A drop in ingestion rate triggers an alert: kafka_consumer_lag > 10000 messages or event_ingestion_rate < 500 events/sec.
Before shipping a feature, teams validate that the experiment is correctly assigned and data is being collected. A data validation job runs nightly and checks for:
The validation job outputs a report in JSON format:
{
"test_id": "checkout-flow-v2",
"assignment_completeness": 0.98,
"assignment_consistency": 0.96,
"event_coverage": {
"control": 0.89,
"treatment": 0.94
},
"missing_events": [
{
"user_id": 345678,
"variant": "control",
"expected_events": ["checkout_started", "payment_submitted"],
"observed_events": ["checkout_started"]
}
]
}A failure in this job halts the CI/CD pipeline: data_validation_failed: checkout-flow-v2. The job is idempotent and can be re-run on demand using a CLI command.
./run-experiment-validation.sh \
--test-id checkout-flow-v2 \
--start-date 2024-05-10 \
--end-date 2024-06-10 \
--output-format json \
--retry-on-failureExperiments evolve over time. Teams may split a test to add a new variant, merge two experiments, or roll out a new feature based on a test’s results.
To split the checkout-flow-v2 experiment into three variants (old, new, A/B split), the team updates the experiment configuration and deploys it. The rollout service detects the change and begins assigning users to the new variants using the updated weight and exposure_rules.
The service also supports feature flags that can be used to toggle experiments on and off. When a feature flag is disabled, users are assigned to the experiment but the feature is not active. This allows for pre-assignment: users are assigned to variants before they interact with the feature.
# experiments/checkout-flow-v2.yaml
feature_flag: checkout_flow_v2_enabled
variant_weights:
control: 0.4
treatment: 0.3
variant_a: 0.3The experiment is active when the feature flag is true. When false, the experiment is in preview mode, collecting data but not serving the feature.
A key failure mode is assignment drift: users are assigned to a variant, but due to a bug in the assignment service, some users receive the wrong variant. This is detected by a variant consistency check job that compares the expected and actual assignments.
def verify_assignment_consistency(experiment_id: str, window_days: int = 7) -> dict:
query = f"""
SELECT
user_id,
variant_id,
MIN(timestamp) as assigned_at
FROM experiments
WHERE experiment_id = '{experiment_id}'
AND timestamp >= NOW() - INTERVAL {window_days} DAY
GROUP BY user_id, variant_id
HAVING COUNT(*) > 1
"""
assigned = run_query(query)
expected = load_expected_assignments(experiment_id)
drift = []
for user in assigned:
expected_variant = expected[user.user_id]
if user.variant_id != expected_variant:
drift.append({
"user_id": user.user_id,
"expected_variant": expected_variant,
"actual_variant": user.variant_id,
"assigned_at": user.assigned_at
})
return {"drifted_users": drift, "total_assigned": len(assigned)}The job outputs a list of users who changed variant during the window. A threshold of 5% drifted users triggers an alert.
Another silent failure is event loss: a service logs events but fails to publish them to Kafka. The pipeline is monitored using a heartbeat mechanism: each service emits a ping event every 30 seconds.
{
"service": "checkout-service",
"event_type": "heartbeat",
"timestamp": "2024-05-10T14:23:01Z",
"metadata": {
"uptime": 123456,
"memory_usage": 123456789
}
}An alert is triggered if heartbeat events are missing for more than 10 minutes. The root cause is often a dropped Kafka topic or a misconfigured consumer group.
When a test shows a negative result, teams roll back the feature. The rollback process is automated and reversible.
The rollout service maintains a rollout plan for each experiment:
rollout_plan:
id: checkout-flow-v2-rollout-20240510
experiment_id: checkout-flow-v2
status: active
steps:
- step: 1
action: update_variant_weights
variant_weights:
control: 0.5
treatment: 0.5
duration: 3600
status: pending
- step: 2
action: disable_feature_flag
feature_flag: checkout_flow_v2_enabled
duration: 1800
status: pending
- step: 3
action: rollback_to_control
variant: control
duration: 7200
status: scheduledRollback is initiated via a CLI command:
./rollback-experiment.sh \
--experiment-id checkout-flow-v2 \
--to-variant control \
--reason "Conversion rate dropped below 0.42 after 72 hours" \
--notify-teams trueThis command updates the experiment configuration, triggers a deployment, and sends an email to the product and engineering teams.
A failure mode is incomplete rollback: the feature flag is disabled, but the old code path is not yet active. This is detected by feature readiness checks, which query the service for the current variant and compare it to the expected one.
GET /v1/experiments/checkout-flow-v2?user_id=123456If the response contains variant: "treatment" but the service is configured to serve control, the check fails. A readiness probe checks for this state.
Multiple experiments may run simultaneously, and teams must coordinate to avoid conflicts. The experiment registry includes a dependencies field that lists other experiments that must be active.
test_id: checkout-flow-v2
dependencies:
- id: user-profile-updates
status: active
required_version: "v2.1"
- id: payment-gateway-upgrade
status: active
required_version: ">=1.5"The rollout service checks these dependencies before activating the experiment. If user-profile-updates is draft, the experiment is marked pending. If a dependency fails, the experiment is paused.
Teams also use experiment metadata to track decisions. A decision_log field in the experiment configuration records key decisions, such as:
decision_log:
- date: "2024-05-01"
decision: "Chose 'new flow' variant based on pilot data"
owner: "[email protected]"
rationale: "Higher engagement in pilot cohort with 42% more time on page"
- date: "2024-05-08"
decision: "Decided to split experiment after 48 hours of data"
owner: "[email protected]"
rationale: "Early data showed strong signal in treatment group"These logs are used in post-mortems and retrospectives, and are accessible via a REST API.
Experiment configurations are stored in a versioned config store, typically a Git repository. Each experiment has a config.yaml file, and changes are reviewed via pull requests.
The experiment schema is validated using a JSON Schema definition:
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["test_id", "variants", "exposure_rules"],
"properties": {
"test_id": { "type": "string", "pattern": "^[a-z][a-z0-9_]+$" },
"variants": {
"type": "array",
"items": {
"type": "object",
"required": ["id", "name"],
"properties": {
"id": { "type": "string" },
"name": { "type": "string" },
"weight": { "type": "number", "minimum": 0, "maximum": 1 }
}
}
},
"exposure_rules": {
"type": "array",
"items": {
"type": "object",
"properties": {
"type": { "enum": ["user_id_hash", "session_id_hash", "geo_region"] },
"field": { "type": "string" },
"variant": { "type": "string" }
}
}
}
}
}A CI pipeline runs jsonschema --strict config.yaml on every PR. If validation fails, the PR is blocked.
A/B testing infrastructure is not a single tool but a coordinated system of configuration, data collection, validation, and rollout. It enables teams to make decisions with confidence, knowing that users are consistently assigned, events are collected, and experiments are monitored. The key to success is consistency: the same assignment logic, the same event schema, the same validation process, and the same rollback protocol across all teams. When it breaks, the symptoms are clear: assignment drift, event loss, missing data, and inconsistent results. The system is resilient, but only when every part is understood, monitored, and maintained.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy