Burn Rate Alerting | The Garnet Wiki

Burn rate alerting is a real-time monitoring practice that tracks how quickly a system consumes its operational budget—typically measured in cost per time unit, such as dollars per minute or compute-hours per day—against a forecasted or target burn rate. It matters when a system’s financial health must be visible not just at month-end, but in real time: during a deployment, after a traffic spike, or when a new service rolls out. This is not about total cost, nor about average resource usage. It is about *velocity* of cost consumption. A burn rate alert triggers when actual cost is diverging from expected cost faster than anticipated—whether that cost is in cloud compute, database I/O, or message queue processing.

Detecting Deviation from Forecast

The core task is to compute, at regular intervals, the actual cost consumed over a recent window and compare it to the forecasted cost for the same window. A deviation beyond a threshold triggers an alert.

The simplest implementation uses a time-series database—such as InfluxDB or Prometheus—with a metric named system.burn_rate.actual and another named system.burn_rate.forecast. These metrics are updated every 5 minutes, and the alerting system checks the difference between actual and forecasted cost over the last 1 hour.

To compute actual cost, a periodic job aggregates cost data from cloud billing APIs, database logs, and service-level monitoring:


$ systemctl enable --now burn-rate-aggregator.timer

The aggregation script reads billing events from a queue:

# Process events from a Kafka topic named 'billing.events'
$ kafka-console-consumer.sh \
    --bootstrap-server kafka.example.com:9092 \
    --topic billing.events \
    --partition 0 \
    --offset latest \
    --max-messages 1000 \
    --property print.key=true \
    --property print.value=true

Each message is a JSON object with fields: resource_type, cost, timestamp, region, service_id. The script parses these and updates a rolling sum over 1-hour windows.

A PostgreSQL table stores the forecasted burn rate:

CREATE TABLE burn_rate_forecasts (
    id SERIAL PRIMARY KEY,
    service_id TEXT NOT NULL,
    window_start TIMESTAMPTZ NOT NULL,
    window_end TIMESTAMPTZ NOT NULL,
    expected_cost NUMERIC(12,4) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Example data
INSERT INTO burn_rate_forecasts (service_id, window_start, window_end, expected_cost)
VALUES ('api-gateway', '2024-04-05T08:00:00Z', '2024-04-05T09:00:00Z', 1872.41);

The aggregation job updates the actual burn rate every 5 minutes:

UPDATE burn_rate_actual
SET actual_cost = $1, last_updated = NOW()
WHERE service_id = $2 AND window_start = $3;

The alerting engine queries both tables and computes the deviation:

SELECT 
    f.service_id,
    f.window_start,
    f.expected_cost,
    a.actual_cost,
    (a.actual_cost - f.expected_cost) AS deviation,
    (a.actual_cost / f.expected_cost) AS ratio
FROM burn_rate_forecasts f
JOIN burn_rate_actual a ON f.service_id = a.service_id AND f.window_start = a.window_start
WHERE a.last_updated >= NOW() - INTERVAL '10 minutes'
  AND (a.actual_cost - f.expected_cost) > (f.expected_cost * 0.10)  -- 10% threshold
ORDER BY deviation DESC;

When this query returns one or more rows, an alert is triggered.

Configuring Alert Thresholds and Sensitivity

Thresholds are not static. A 10% deviation is meaningful for a high-cost service like a real-time analytics pipeline but too loose for a low-cost, high-frequency event processor.

The alerting configuration is defined in a YAML file, used by the alert engine:

# alerting-config.yaml
alerting:
  default_threshold: 0.10  # 10% deviation
  rules:
    - name: "High Burn Rate"
      service_id: "api-gateway"
      threshold: 0.15  # 15% deviation
      window: "1h"
      severity: "warning"
      duration: "30m"
      condition: "actual_cost > forecasted_cost * 1.15"
      metadata:
        alert_type: "burn_rate"
        metric: "system.burn_rate.actual"
        source: "kafka-billing-processor"
        window: "60m"
        window_start: "2024-04-05T08:00:00Z"
    - name: "Critical Burn Rate"
      service_id: "realtime-processor"
      threshold: 0.25
      window: "30m"
      severity: "critical"
      duration: "15m"
      condition: "actual_cost > forecasted_cost * 1.25"
      metadata:
        alert_type: "burn_rate"
        metric: "system.burn_rate.actual"
        source: "stream-processor"
        window: "30m"

The alert engine applies rules in order. If a service matches multiple rules, it combines them into a single alert with metadata.

The duration field is critical: it ensures alerts are not triggered by transient spikes. A rule with duration: "15m" requires that the deviation persists for at least 15 minutes before raising an alert. This prevents flapping alerts due to short-lived load bursts.

Handling Out-of-Order and Delayed Data

A common failure is delayed billing data. Events arrive hours after they were generated, causing the burn rate to be underestimated in real time.

To handle this, the aggregation job includes a processing_lag metric:

CREATE TABLE system_metrics (
    metric_name TEXT NOT NULL,
    value NUMERIC(12,4),
    recorded_at TIMESTAMPTZ NOT NULL
);

-- Update lag metric every 5 minutes
INSERT INTO system_metrics (metric_name, value, recorded_at)
VALUES ('system.burn_rate.processing_lag', 2.34, NOW());

The alerting engine checks this metric and adjusts thresholds dynamically.

When processing lag exceeds 60 minutes, the system increases the threshold by 0.05 for all burn rate alerts:

SELECT 
    r.name,
    r.threshold + (0.05 * (SELECT value FROM system_metrics WHERE metric_name = 'system.burn_rate.processing_lag'))
        AS adjusted_threshold
FROM alerting_rules r
WHERE r.service_id = 'api-gateway';

The alerting engine also marks alerts as delayed when the window start time is earlier than the actual data arrival window.

A critical failure mode: if the aggregation job fails to process all events in a window, the actual cost is computed from only a subset of the data. This causes *underreporting* of cost, which leads to false alerts.

To detect this, a missing_events metric is maintained:

-- Count expected vs. processed events per window
CREATE TABLE burn_rate_window_stats (
    window_start TIMESTAMPTZ PRIMARY KEY,
    expected_count INTEGER,
    processed_count INTEGER,
    completion_rate NUMERIC(5,4),
    alert_on_completion: BOOLEAN DEFAULT TRUE
);

The alert engine queries for windows where completion_rate < 0.8 and raises a window_completeness alert.

Alert Delivery and Escalation

An alert is not just a notification. It is a state machine with status transitions: pending, firing, resolved, acknowledged.

The alert state is stored in a database table:

CREATE TABLE alerts (
    id UUID PRIMARY KEY,
    rule_id UUID,
    service_id TEXT,
    window_start TIMESTAMPTZ,
    window_end TIMESTAMPTZ,
    severity TEXT,
    status TEXT DEFAULT 'pending',
    triggered_at TIMESTAMPTZ,
    resolved_at TIMESTAMPTZ,
    last_updated TIMESTAMPTZ,
    metadata JSONB,
    annotations JSONB
);

The alerting engine sends alerts via multiple channels:

Email: using sendmail with a template:

  # sendmail -t
  To: [email protected]
  Subject: [Burn Rate Alert] High Burn Rate on api-gateway (2024-04-05T08:00:00Z)
  From: [email protected]
  MIME-Version: 1.0
  Content-Type: text/html; charset=utf-8

  <h1>High Burn Rate Alert</h1>
  <p><strong>Service:</strong> api-gateway</p>
  <p><strong>Window:</strong> 2024-04-05T08:00:00Z – 2024-04-05T09:00:00Z</p>
  <p><strong>Expected Cost:</strong> $1,872.41</p>
  <p><strong>Actual Cost:</strong> $2,153.20</p>
  <p><strong>Deviation:</strong> +15.0%</p>
  <p><strong>Status:</strong> <span style="color: red;">Firing</span></p>
  <p><strong>Source:</strong> kafka-billing-processor</p>
  <p><strong>Alert Rule:</strong> High Burn Rate</p>

Slack: using curl to call the Slack webhook:

  curl -X POST \
    -H "Content-Type: application/json" \
    -d '{
      "text": "🔥 High Burn Rate Alert: api-gateway",
      "blocks": [
        {
          "type": "section",
          "text": {
            "type": "mrkdwn",
            "text": "*Service:* api-gateway\n*Window:* 2024-04-05T08:00:00Z – 2024-04-05T09:00:00Z\n*Expected:* $1,872.41\n*Actual:* $2,153.20\n*Deviation:* +15.0%"
          }
        },
        {
          "type": "context",
          "elements": [
            {
              "type": "mrkdwn",
              "text": "Source: kafka-billing-processor"
            }
          ]
        }
      ]
    }' \
    https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

PagerDuty: via curl to the PagerDuty Events API:

  POST /incidents
  {
    "service": {
      "id": "P123456789",
      "type": "service"
    },
    "incident": {
      "title": "High Burn Rate on api-gateway",
      "body": "Actual cost ($2,153.20) exceeds forecast ($1,872.41) by 15.0%.",
      "custom_fields": [
        { "name": "Service ID", "value": "api-gateway" },
        { "name": "Window Start", "value": "2024-04-05T08:00:00Z" },
        { "name": "Source", "value": "kafka-billing-processor" }
      ],
      "priority": "high"
    }
  }

Common Pitfalls and Silent Failures

Window boundary mismatches: The aggregation job computes 1-hour windows but the forecast table uses 60-minute windows with 30-minute overlaps. This causes alerts to be raised on partial or overlapping windows without clear correlation.

Time zone confusion: The system uses UTC internally, but the alert engine assumes local time for the window start. A service in Europe/Berlin shows alerts starting at 08:00 UTC but the actual window spans 08:00–09:00 Berlin time, which is 07:00–08:00 UTC. The window is misaligned.

Thresholds applied too early: The alert engine evaluates thresholds every 5 minutes, but the aggregation job takes 10 minutes to process events. The first alert is computed with data that is already 15 minutes old.

Missing window metadata: When the aggregation job fails to update window_start in the burn_rate_actual table, alerts appear with incorrect window boundaries, and the last_updated timestamp is stale.

Alert deduplication: Multiple rules can trigger for the same window. Without deduplication, a single event can generate 10 alerts. The alert engine must merge alerts with the same service_id and window_start.

Failure to resolve alerts: An alert goes from firing to resolved when the actual cost drops below the threshold. But if the cost is stable, the alert remains resolved even when a new window begins. The system must explicitly check for new window starts and re-evaluate.

Incomplete alert state transitions: An alert is firing but the resolved_at field is NULL. The alerting engine assumes resolution when status = 'resolved', but a pending alert never becomes firing due to a missing triggered_at timestamp.

Monitoring the Alerting System Itself

The alerting engine must monitor itself. A failure in the burn rate alerting system itself is a critical alert.

The engine logs its own health:

{
  "timestamp": "2024-04-05T08:30:00Z",
  "alerting_system": {
    "status": "running",
    "active_rules": 23,
    "pending_alerts": 14,
    "firing_alerts": 6,
    "last_run": "2024-04-05T08:25:00Z",
    "duration_seconds": 12.3,
    "errors": [
      {
        "source": "kafka-processor",
        "message": "Failed to consume 500+ messages from 'billing.events'",
        "timestamp": "2024-04-05T08:25:01Z"
      }
    ]
  }
}

This health report is pushed to a metrics endpoint:

curl -X POST \
  -H "Content-Type: application/json" \
  -d @health.json \
  http://prometheus.example.com/metrics

A separate alert rule watches for alerting_system.status = 'failed' and raises a system_alerting_health alert when the alerting system is down.

Summary

Burn rate alerting is not about measuring cost. It is about measuring *velocity* of cost. It requires precise windowing, dynamic thresholds, and robust handling of delayed data. The alerting system must be self-monitoring, and its configuration must be explicit and versioned. The key insight is that a 15% deviation in burn rate is not just a number—it is a signal that a system is consuming its budget faster than planned, and that action is needed *now*.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.