Survival Analysis

Production engineering guide for survival analysis covering patterns, implementation strategies, and operational best practices.

Survival analysis models the time until an event of interest occurs, particularly when some observations are censored—meaning the event hasn’t happened by the end of the study or the subject was lost to follow-up. It is essential in engineering systems where failure, user churn, or system degradation is measured over time: predicting when a disk will fail, estimating user lifetime after signup, or monitoring server uptime before deployment rollback. It matters when the outcome is not just binary (event occurred or not) but *when* it occurred, and when not all subjects are observed for the entire duration.

Estimating the Survival Function

To estimate the probability that an event has not occurred by time *t*, use the Kaplan-Meier estimator. This non-parametric method computes survival probability at each observed event time, adjusting for censored observations.

Given a dataset with id, start_time, event_time, and censored flags, the survival function is constructed by:

$$ S(t_i) = S(t_{i-1}) \cdot \left(1 - \frac{d_i}{n_i}\right) $$ where *dᵢ* is the number of events at *tᵢ*, and *nᵢ* is the number of subjects at risk just before *tᵢ*.

In SQL, compute the survival curve as:

WITH events AS (
  SELECT
    id,
    start_time,
    event_time,
    CASE WHEN event_time IS NOT NULL THEN 1 ELSE 0 END AS event,
    COALESCE(event_time, 1e9) AS observed_time
  FROM user_engagement
),
at_risk AS (
  SELECT
    e1.id,
    e1.start_time,
    e1.event_time,
    e1.event,
    e1.observed_time,
    COUNT(e2.id) AS at_risk_count,
    SUM(e2.event) AS cumulative_events
  FROM events e1
  JOIN events e2 ON e2.start_time <= e1.start_time
                 AND (e2.event_time IS NULL OR e2.event_time >= e1.start_time)
  GROUP BY e1.id, e1.start_time, e1.event_time, e1.event, e1.observed_time
),
survival AS (
  SELECT
    e.start_time AS time,
    e.at_risk_count AS n,
    SUM(e.event) AS d,
    AVG(
      CASE
        WHEN e.event = 1 THEN 1.0
        ELSE 0.0
      END
    ) AS event_rate,
    SUM(
      CASE
        WHEN e.event = 1 THEN 1.0
        ELSE 0.0
      END
    ) / e.at_risk_count AS event_probability
  FROM at_risk e
  GROUP BY e.start_time, e.at_risk_count
  ORDER BY e.start_time
)
SELECT
  time,
  1.0 AS survival,
  SUM(
    (1.0 - event_probability) * LAG(survival, 1) OVER (ORDER BY time)
  ) AS survival
FROM survival
ORDER BY time;

The result is a time-series of survival probabilities. The curve drops at each event time, and the drop size depends on the proportion of subjects experiencing the event.

Common Pitfalls

Fitting Parametric Survival Models

When the hazard rate follows a known functional form, fit a parametric model to improve extrapolation and inference. Common models include Weibull, exponential, and log-normal.

To fit a Weibull model, minimize the negative log-likelihood:

$$ \mathcal{L}(\alpha, \beta) = \sum_{i=1}^{n} \left[ \log(\beta) + \log(\alpha) + (\alpha - 1) \log(t_i) + (t_i)^\alpha \right] \cdot \delta_i

(t_i)^\alpha \right] \cdot (1 - \delta_i) $$

where *α* is the shape parameter, *β* is the scale parameter, *tᵢ* is the observed time, and *δᵢ* is the event indicator.

In Python using lifelines (a real library), fitting a Weibull model is straightforward:

from lifelines import WeibullFitter
import pandas as pd

data = pd.read_csv("user_survival.csv")
data['event'] = data['purchase_date'].notnull()

wf = WeibullFitter()
wf.fit(data, duration_col='days_since_signup', event_col='event')


print(wf.alpha_)  # shape parameter
print(wf.beta_)   # scale parameter
print(wf.summary) # full parameter table with confidence intervals

# Predict survival at specific times
survival_at_30_days = wf.survival_at_times(30)
print(survival_at_30_days)

The WeibullFitter assumes the baseline hazard is Weibull-shaped and estimates parameters via maximum likelihood. The model produces a closed-form survival function:

$$ S(t) = \exp\left( -\left(\frac{t}{\beta}\right)^\alpha \right) $$

Key Configuration and Failure Modes

Hazard and Cumulative Hazard Functions

The hazard function *h(t)* gives the instantaneous rate of event occurrence at time *t*, given survival up to *t*. The cumulative hazard function *H(t)* is the integral of the hazard function.

To compute the cumulative hazard:

$$ \hat{H}(t) = \sum_{t_i \leq t} \frac{d_i}{n_i} $$ where *dᵢ* is the number of events at *tᵢ*, and *nᵢ* is the number at risk.

In Python:

from lifelines import NelsonAalenFitter

naf = NelsonAalenFitter()
naf.fit(data, duration_col='days_since_signup', event_col='event')

# Plot cumulative hazard
naf.plot_cumulative_hazard()

# Extract hazard at specific points
hazard_at_14_days = naf.cumulative_hazard_at_times([14])

The NelsonAalenFitter returns a cumulative_hazard_ attribute with time-indexed values. Plotting it shows how the risk of event increases over time.

Critical Observations

Cox Proportional Hazards Regression

When covariates (features) influence survival, use the Cox model to estimate how each variable affects the hazard rate.

The Cox model assumes:

$$ h(t) = h_0(t) \cdot \exp(\mathbf{x}^T \boldsymbol{\beta}) $$

where *h₀(t)* is the baseline hazard, *x* is the vector of covariates, and *β* is the vector of coefficients.

In lifelines, fit a Cox model as:

from lifelines import CoxPHFitter

cph = CoxPHFitter()
cph.fit(data, duration_col='days_since_signup', event_col='event',
        formula='age + is_mobile + last_login_days_ago')

# View estimated coefficients
print(cph.summary)
print(cph.coef_)

# Predict hazard for a new user
new_user = pd.DataFrame({
    'age': [28],
    'is_mobile': [True],
    'last_login_days_ago': [7]
})
hazard_ratio = cph.predict_partial_hazard(new_user)

The formula argument supports standard R-style syntax: log(age), age + age:is_mobile, np.log(last_login_days_ago).

Common Configuration and Failure Modes

Validating and Deploying Survival Models

To ensure survival models are production-ready:

from lifelines import cross_validation
from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)
scores = cross_validation(cph, data, duration_col='days_since_signup', event_col='event',
                          cv=tscv, scoring_method='concordance_index')

Summary

Survival analysis is not a single algorithm but a suite of techniques for modeling event timing, with Kaplan-Meier, parametric models, and Cox regression forming the core. The key insight is that time-to-event data cannot be reduced to a simple binary outcome; censoring must be explicitly modeled, and the hazard function reveals dynamic risk over time. The choice of model depends on data structure, covariates, and the need for extrapolation. The real challenge lies not in fitting a model but in correctly encoding the event, aligning time, and validating assumptions—particularly the proportional hazards assumption in Cox models. Failure in any of these leads to models that under- or over-predict survival, with cascading effects in system design, user experience, and maintenance planning.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.