Production engineering guide for experiment design patterns covering patterns, implementation strategies, and operational best practices.
Experiments are the foundation of data-driven decision-making in engineering and product development. They answer *what if* questions by measuring the causal effect of a change. Experiment design patterns are reusable, well-structured approaches to setting up, running, and interpreting experiments—especially A/B tests—across teams and systems. These patterns matter when teams need to move fast, coordinate multiple changes, and ensure that every experiment is not only valid but also scalable and maintainable.
When the primary goal is to explore multiple interventions—especially when the optimal treatment is unknown—sequential and multi-armed bandit (MAB) designs provide a framework that balances exploration and exploitation. These patterns are used when the cost of experimentation is high, the effect sizes are uncertain, and decisions must be made in real time.
A sequential experiment uses repeated interim analyses to decide whether to stop early, continue, or adapt. The design is driven by a stopping rule: if the p-value crosses a threshold at any analysis point, the experiment may conclude. A common configuration uses the O’Brien-Fleming boundary for alpha spending, where a stricter threshold is applied at earlier looks.
from statsmodels.stats.power import TTestIndPower
# Assume we want 80% power at alpha = 0.05, effect size 0.4
analysis = TTestIndPower()
n_per_group = analysis.solve_power(effect_size=0.4, power=0.8, alpha=0.05)
# Define sequential looks at 25%, 50%, 75%, and 100% of total sample
looks = [0.25, 0.5, 0.75, 1.0]
alpha_spending = [0.01, 0.02, 0.035, 0.05]
# At each look, check whether the p-value is below the boundary
for t, p_value in enumerate(p_values):
expected_n = n_per_group * looks[t]
observed_n = cumulative_sample_size[t]
critical_z = norm.ppf(1 - alpha_spending[t])
z_score = (effect_estimate[t] - 0) / standard_error[t]
if z_score > critical_z:
print(f"Stop at look {t+1}: z = {z_score:.3f}, critical = {critical_z:.3f}")
breakThe MAB design, often implemented with Thompson sampling, dynamically allocates users to treatments based on their current estimated success rates. A treatment with higher uncertainty (low sample size) is more likely to be selected, promoting exploration.
In a production system, a bandit experiment is configured via a JSON specification:
{
"experiment_id": "mab-recommendation-2023",
"type": "multi-armed-bandit",
"strategy": "thompson-sampling",
"arm_configs": [
{
"arm_id": "default",
"initial_pulls": 100,
"prior_alpha": 1.0,
"prior_beta": 1.0
},
{
"arm_id": "content-based",
"initial_pulls": 50,
"prior_alpha": 1.5,
"prior_beta": 0.8
},
{
"arm_id": "collaborative-filtering",
"initial_pulls": 30,
"prior_alpha": 0.5,
"prior_beta": 1.5
}
],
"tracking": {
"event_stream": "experiment_events",
"user_id_field": "user_id",
"treatment_field": "treatment",
"outcome_field": "click"
}
}The system continuously updates arm probabilities and logs each user’s assignment and outcome. A failure mode arises when the prior parameters are misaligned: a large prior_alpha with few observed successes leads to overconfidence in early estimates, causing slow adaptation. Another silent failure is the assumption that the outcome metric (e.g., click) is independent across arms—when in fact, users are exposed to multiple treatments, leading to carryover effects.
When experiments span millions of users across diverse segments, stratified sampling ensures that each subgroup is well-represented. This pattern is essential when user behavior varies significantly by geography, device type, or cohort.
The design involves partitioning the population into strata—e.g., region, device_type, first_visit_month—and allocating treatments within each stratum independently. The allocation is typically done at the user level, with a deterministic or probabilistic assignment function.
-- Create a stratified sample for A/B testing
WITH strata AS (
SELECT
user_id,
region,
device_type,
first_visit_month,
-- Assign a stratum key
CONCAT(region, '_', device_type, '_', first_visit_month) AS stratum_key
FROM users
WHERE active_in_last_30_days = true
),
treatment_assignment AS (
SELECT
s.*,
-- Use hash-based assignment within each stratum
CASE
WHEN hash(s.stratum_key) % 100 < 70 THEN 'control'
ELSE 'treatment'
END AS treatment
FROM strata s
)
INSERT INTO experiment_assignments (user_id, experiment_id, treatment, stratum)
SELECT
user_id,
'revenue-optimization-v1',
treatment,
stratum_key
FROM treatment_assignment
WHERE experiment_id = 'revenue-optimization-v1';The key challenge is ensuring that the stratification is stable across time. If the stratum_key is computed from a snapshot of user data but the assignment is applied retroactively, the experiment may suffer from time-shifted effects. A common failure mode is the mismatch between the strata used in the assignment and those used in the analysis: one team uses region and device_type, while another uses country and os_version, leading to inconsistent estimates.
Another edge case: when the number of strata grows beyond a few hundred, the assignment query becomes slow and memory-intensive. The solution is to pre-compute stratum assignments and store them in a lookup table, which is updated daily via a batch job.
When multiple treatments are applied simultaneously and their interactions matter, multivariate experiments—often called MVTs—become necessary. This pattern is used when teams want to test combinations of features, such as a new UI layout (A/B), a new recommendation algorithm (C/D), and a revised onboarding flow (E/F).
The design involves a factorial structure, where each factor (e.g., layout, algorithm, onboarding) has multiple levels. The experiment is defined by a set of treatment combinations, and users are assigned to one full combination.
experiment:
id: "ui-recommendation-onboarding-mvt"
type: multivariate
factors:
- name: layout
levels:
- name: classic
description: "Original layout with sidebar"
- name: modern
description: "New card-based layout"
- name: recommendation_algorithm
levels:
- name: collaborative
description: "User-based collaborative filtering"
- name: content_based
description: "Content similarity-based recommendations"
- name: onboarding_flow
levels:
- name: simple
description: "Single-page onboarding"
- name: interactive
description: "Step-by-step interactive walkthrough"
treatments:
- id: "classic-collaborative-simple"
factor_assignments:
layout: classic
recommendation_algorithm: collaborative
onboarding_flow: simple
- id: "classic-collaborative-interactive"
factor_assignments:
layout: classic
recommendation_algorithm: collaborative
onboarding_flow: interactive
# ... other combinationsThe analysis involves modeling the main effects and interactions. A common mistake is treating a multivariate experiment as a series of independent A/B tests, leading to incorrect estimates of effect sizes and overconfident p-values. The model must account for the full factorial design.
A nested experiment pattern extends this idea: one experiment is run within the groups of another. For example, a global feature rollout (experiment A) is followed by a sub-experiment (B) within each treatment group of A. This is used to test how a new feature performs across different segments defined by a prior intervention.
In a nested design, the assignment flow is hierarchical:
A failure mode occurs when the inner experiment’s treatment is not properly isolated: users in the inner experiment may be exposed to the outer treatment, causing contamination. Another issue is the imbalance in the number of users per inner group: some outer groups have many users, while others are small, leading to variable precision in estimating the inner effects.
Pre-registration is the practice of specifying the experimental design—hypotheses, data collection, analysis, and statistical models—before collecting data. This pattern reduces researcher degrees of freedom and minimizes p-hacking.
A pre-registered experiment is described in a standard format, often as a JSON or YAML file:
{
"pre_registration_id": "2023-06-12-1400",
"study_title": "Effect of Dark Mode on User Engagement",
"hypotheses": [
{
"id": "h1",
"statement": "Users exposed to dark mode will spend more time on the app.",
"expected_direction": "positive",
"significance_level": 0.05
},
{
"id": "h2",
"statement": "Dark mode improves retention at 7 days.",
"expected_direction": "positive"
}
],
"data_collection": {
"source": "analytics_event_stream",
"event_types": ["session_start", "session_end", "user_engagement"],
"metrics": [
{
"name": "time_spent",
"aggregation": "sum",
"window": "all_time"
},
{
"name": "retention_7d",
"aggregation": "avg",
"window": "7_day"
}
],
"inclusion_criteria": [
"active_in_last_7_days",
"device_type IN ('mobile', 'tablet')"
]
},
"analysis_plan": {
"primary_analysis": {
"method": "linear_regression",
"dependent_variable": "time_spent",
"independent_variables": [
"treatment_dark_mode",
"age_group",
"device_type",
"first_visit_month"
],
"model_selection": "stepwise",
"adjustment": "bonferroni"
},
"secondary_analysis": [
"cox_proportional_hazards",
"mixed_effects_model"
]
}
}Once the experiment begins, the analysis plan is locked. Any deviation—adding a covariate, changing the aggregation window—must be documented and justified.
Blinding is the practice of keeping participants, implementers, and analysts unaware of the treatment assignment. This reduces bias and increases the credibility of the results.
In a double-blind experiment, both the user-facing team (e.g., product engineers) and the data analysis team are unaware of which users received which treatments. The assignment is performed by a third party, and the treatment is applied via a configuration pipeline.
# Experiment configuration with blinding metadata
experiment:
id: "dark-mode-blinded"
type: ab_test
blinding:
level: double,
implementers: ["analytics-engineers", "frontend-team"],
analysts: ["data-science-team"],
assignment_method: "randomized_block_allocation",
block_size: 500,
assignment_log: "s3://experiments/assignments/dark-mode-blinded.jsonl"A silent failure in blinding is the leakage of treatment information: a dashboard showing treatment status becomes public, and team members start tailoring their work to the treatment group. This leads to differential expectations and confounds the true effect.
When experiments are part of a continuous delivery pipeline, the design pattern of a pipeline-based experiment ensures that every change is tested, tracked, and evaluated systematically.
Each experiment is defined as a pipeline step in a CI/CD system. The process is:
main.# GitHub Actions workflow for a pipeline-based experiment
name: Run A/B Test
on:
push:
branches: [ main ]
workflow_dispatch:
jobs:
deploy-experiment:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: npm install
- run: npm run build
- run: |
# Deploy to staging
kubectl apply -f k8s/staging.yaml
# Trigger experiment
curl -X POST \
-H "Content-Type: application/json" \
-d @experiment-config.json \
https://api.experiment-service.com/v1/experiments
- name: Log experiment start
run: |
echo "Experiment started at $(date)" >> logs/experiment-start.txt
git add logs/experiment-start.txt
git commit -m "Start experiment"
git push
analyze-results:
needs: deploy-experiment
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: pip install -r requirements.txt
- run: python analyze_results.py --experiment-id dark-mode-v1
- run: |
# Upload report to S3
aws s3 cp report.html s3://experiment-reports/dark-mode-v1/
# Post to Slack
curl -X POST -H "Content-Type: application/json" \
-d '{"text": "Experiment completed: dark-mode-v1"}' \
https://hooks.slack.com/services/ABC123/XYZ456The key failure mode in pipeline-based experiments is the lack of versioning: the experiment configuration is not tied to the code version, so a change in the analysis script may not align with the deployment that produced the data. This leads to a mismatch between the experiment’s definition and its execution.
Another edge case: when multiple experiments run concurrently, the pipeline must handle dependencies—e.g., one experiment depends on the outcome of another. This is managed through metadata and event-driven triggers. A common failure is the “phantom experiment”: a pipeline runs, but the results are never discovered because the experiment ID is not published to a central registry.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy