Move beyond correlation to understand causation in product decisions. Covers A/B test limitations, difference-in-differences, instrumental variables, regression discontinuity, propensity score matching, and when to use each causal inference technique.
Causal inference for product answers the question: *Did this feature, change, or policy actually cause the observed outcome, and by how much?* It is essential when A/B tests are no longer sufficient—when confounding factors, time delays, or non-constant treatment effects obscure the true impact. Use it when product teams must decide between two policies, when rollout is costly, and when decisions are irreversible.
To quantify the causal impact of a product change, compute the Average Treatment Effect (ATE) across a population. The ATE is the difference in outcome between units assigned to treatment and those in control, under a stable unit treatment value assumption (SUTVA).
Given a dataset with experiment_id, unit_id, treatment, and outcome fields, use a linear regression model to estimate ATE with covariate adjustment.
SELECT
AVG(CASE WHEN treatment = TRUE THEN outcome ELSE 0 END) -
AVG(CASE WHEN treatment = FALSE THEN outcome ELSE 0 END) AS ate
FROM experiments
WHERE experiment_id = 'feature_x_rollout'
AND timestamp >= '2024-01-01'
AND timestamp < '2024-04-01'
GROUP BY experiment_id;This yields a point estimate, but to improve precision, include pre-treatment covariates (e.g., user_lifetime_value, last_login_days_ago, device_type) to reduce residual variance.
When treatment assignment is not randomized—e.g., users with higher engagement are more likely to receive a new feature—use propensity score matching to balance covariates between groups.
First, estimate the propensity score: the probability of receiving treatment given observed covariates.
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
X = df[['user_lifetime_value', 'last_login_days_ago', 'device_type']].values
y = df['treatment'].values
# Fit model
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
model = LogisticRegression()
model.fit(X_scaled, y)
# Predict propensity scores
df['propensity_score'] = model.predict_proba(X_scaled)[:, 1]Then, match each treated unit to one or more control units with similar propensity scores using nearest-neighbor matching.
WITH matched AS (
SELECT
t.unit_id AS treated_id,
c.unit_id AS control_id,
t.outcome AS treated_outcome,
c.outcome AS control_outcome,
ABS(t.propensity_score - c.propensity_score) AS distance
FROM
treatments t
INNER JOIN controls c ON ABS(t.propensity_score - c.propensity_score) < 0.05
)
SELECT AVG(treated_outcome - control_outcome) AS ate
FROM matched;This method reduces bias from observed confounders and is especially effective when randomization is imperfect or impossible.
When a product change takes time to influence behavior—e.g., a new onboarding flow that improves retention over weeks—use time-series causal models.
The key is to define the treatment start time and model outcome over a lagged window.
-- Define treatment periods and outcome windows
SELECT
e.user_id,
e.treatment_start_date,
AVG(m.metric_value) AS outcome,
COUNT(m.metric_value) AS n_measurements
FROM experiments e
JOIN metrics m
ON m.user_id = e.user_id
AND m.timestamp >= e.treatment_start_date + INTERVAL '1 week'
AND m.timestamp < e.treatment_start_date + INTERVAL '4 weeks'
GROUP BY e.user_id, e.treatment_start_date;This captures the cumulative effect of treatment over a four-week window, allowing for gradual adoption. The outcome metric must be time-locked to treatment, not just to user sign-up.
Failure mode: *users start treatment in week 1 but are measured only from week 3, missing early adoption.* The solution is to create a treatment_window table that tracks the effective duration of treatment per user.
When users do not adhere to treatment—e.g., a feature rollout that users enable manually—the instrument variable (IV) method isolates the causal effect of *actual treatment* from *intended treatment*.
An instrument is a variable that affects treatment assignment but not outcome directly. For example, a rollout strategy that sends a notification to all users, but only 60% follow it.
The IV estimator is derived via two-stage least squares (2SLS):
# First stage: instrument → treatment
model1 = LogisticRegression()
model1.fit(instrument_data, actual_treatment)
# Predict treatment
predicted_treatment = model1.predict(instrument_data)
# Second stage: predicted treatment → outcome
model2 = LinearRegression()
model2.fit(predicted_treatment.reshape(-1, 1), outcome)
# IV estimate
iv_estimate = model2.coef_[0]Use this when compliance is variable and the instrument is strong—e.g., a push notification that increases adoption but doesn’t directly affect user engagement.
Causal inference is not a one-off analysis. It must be part of the product data pipeline, with causal metrics updated daily.
Use a causal_metrics table with a materialized view that aggregates causal estimates over time.
CREATE MATERIALIZED VIEW causal_metrics AS
SELECT
experiment_id,
DATE_TRUNC('day', treatment_start_date) AS day,
COUNT(*) AS n_units,
AVG(outcome) AS mean_outcome,
STDDEV(outcome) AS stddev_outcome,
AVG(treatment_effect) AS mean_treatment_effect
FROM (
SELECT
e.experiment_id,
e.treatment_start_date,
u.user_id,
u.outcome,
u.treatment_effect
FROM experiments e
JOIN (
SELECT
user_id,
outcome,
treatment_effect,
treatment_start_date
FROM user_level_causal_metrics
WHERE outcome_timestamp >= treatment_start_date + INTERVAL '7 days'
) u
ON e.user_id = u.user_id
WHERE e.status = 'active'
) q
GROUP BY experiment_id, day
ORDER BY day DESC;This view is refreshed nightly via a cron job.
# Run every night at 02:00
0 2 * * * /usr/local/bin/refresh-causal-metrics.shThe script calls a Python function that computes ATE, propensity scores, and matches, then inserts results into the materialized view.
Failure mode: *the script runs but fails silently when a new experiment ID is added and the user_level_causal_metrics table lacks the treatment_start_date column.* The fix is to add a schema version table and validate schema before each run.
Causal inference relies on untestable assumptions. Validate them in production.
Common assumptions:
To validate SUTVA, check for spillover effects.
-- Check if treated users affect their neighbors' outcomes
WITH treated_neighbors AS (
SELECT
t.user_id AS treated_user,
t.outcome AS treated_outcome,
n.user_id AS neighbor_user,
n.outcome AS neighbor_outcome,
COUNT(*) AS n_neighbors
FROM treatments t
JOIN user_connections uc ON t.user_id = uc.user_id
JOIN users n ON uc.connected_user_id = n.user_id
WHERE t.treatment = TRUE
GROUP BY t.user_id, t.outcome, n.user_id, n.outcome
)
SELECT
AVG(neighbor_outcome) AS avg_neighbor_outcome,
AVG(treated_outcome) AS avg_treated_outcome,
AVG(neighbor_outcome - treated_outcome) AS spillover_effect
FROM treated_neighbors;If spillover is large, the estimate of ATE is biased. The solution is to use a clustered treatment design, where entire groups (e.g., teams, cohorts) receive treatment together.
When covariates change over time—e.g., user engagement, app version, or seasonality—the causal model must account for time-varying confounders.
Use doubly robust estimators or G-computation.
# G-computation: simulate counterfactual outcomes
def g_computation(data, treatment_col, outcome_col, covariates):
# Forward pass: predict outcome given history
model = LinearRegression()
model.fit(data[covariates], data[outcome_col])
# Simulate counterfactuals
data['predicted_outcome'] = model.predict(data[covariates])
# Adjust for treatment
data['treatment_effect'] = data[outcome_col] - data['predicted_outcome']
return data.groupby(treatment_col).mean()['treatment_effect']This method is robust to misspecification in either the outcome model or the treatment model.
Failure mode: *the model assumes time-invariant confounders, but user activity spikes on weekends, causing overestimation of treatment effect during weekends.* The fix is to include day_of_week and hour_of_day as covariates and to compute treatment effect separately for weekdays vs. weekends.
A common silent failure: treatment is defined at the user level, but outcome is measured at the session level.
-- Mismatch: treatment assigned once, but outcome per session
SELECT
u.user_id,
u.treatment,
s.session_id,
s.timestamp,
s.metric_value,
s.timestamp >= u.treatment_start_date AS is_treated_session
FROM users u
JOIN sessions s ON u.user_id = s.user_id
WHERE s.timestamp BETWEEN '2024-01-01' AND '2024-01-31';If treatment starts on January 5, but a session on January 2 is counted as "treated," the causal effect is biased.
Solution: define a treatment_window for each user and join sessions to the correct treatment period.
-- Ensure outcome is measured during treatment window
WITH treatment_periods AS (
SELECT
user_id,
treatment_start_date,
treatment_start_date + INTERVAL '30 days' AS treatment_end_date
FROM experiments
)
SELECT
s.user_id,
s.session_id,
s.timestamp,
tp.treatment_start_date,
tp.treatment_end_date,
s.timestamp BETWEEN tp.treatment_start_date AND tp.treatment_end_date AS in_treatment_window
FROM sessions s
JOIN treatment_periods tp ON s.user_id = tp.user_id;This ensures that only sessions within the effective treatment period are included in outcome calculation.
Causal models degrade over time as user behavior shifts. Monitor drift via a causal_model_drift table.
CREATE TABLE causal_model_drift (
model_name TEXT,
experiment_id TEXT,
timestamp TIMESTAMPTZ,
ate DOUBLE PRECISION,
ate_std_error DOUBLE PRECISION,
sample_size INTEGER,
mean_propensity_score DOUBLE PRECISION,
treatment_coverage DOUBLE PRECISION,
drift_score DOUBLE PRECISION
);Each day, compute the ATE and update the table.
Failure mode: *the model drifts, but the drift score is never visualized. Teams miss changes in treatment effectiveness until a product review.* The fix is to create a dashboard that plots ate and drift_score over time, with alerts when drift exceeds a threshold.
Use diagnostic plots to validate assumptions.
Use matplotlib or plotly to generate these.
import matplotlib.pyplot as plt
def plot_causal_diagnostics(df, treatment_col, outcome_col, covariates):
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
# Balance plot
df.groupby(treatment_col)[covariates].mean().plot(kind='bar', ax=axes[0,0])
axes[0,0].set_title('Covariate Balance')
# Density plot
for group in df[treatment_col].unique():
subset = df[df[treatment_col] == group]
plt.hist(subset[outcome_col], alpha=0.5, label=f'{group}')
plt.legend()
plt.title('Outcome Distribution')
plt.xlabel('Outcome')
plt.ylabel('Frequency')
# Residuals
model = LinearRegression().fit(df[covariates], df[outcome_col])
residuals = df[outcome_col] - model.predict(df[covariates])
axes[1,0].scatter(model.predict(df[covariates]), residuals)
axes[1,0].set_xlabel('Fitted Values')
axes[1,0].set_ylabel('Residuals')
axes[1,0].set_title('Residual Plot')
# QQ plot
from scipy.stats import probplot
probplot(residuals, dist="norm", plot=axes[1,1])
axes[1,1].set_title('QQ Plot of Residuals')
plt.tight_layout()
plt.savefig('/tmp/causal_diagnostics.png')This function is run nightly and outputs a diagnostic image for each experiment.
The most common failure: *computing ATE, but not checking whether the treatment was actually applied when it mattered.* The solution: define a treatment window and enforce it in the data pipeline.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy