Finops Anomaly Detection | The Garnet Wiki

Finops anomaly detection identifies unexpected changes in cloud spending patterns—spikes, dips, or shifts in cost distribution—across services, regions, teams, or cost categories. It matters when budgets are fixed, commitments are due, or teams need to respond rapidly to unexpected overhead. The system runs continuously, comparing actual spend against historical baselines, and alerts when deviations exceed thresholds, enabling proactive cost governance.

Detecting Anomalies in Cloud Spend

Anomalies are detected by comparing actual cost data over a time window against one or more reference profiles. The reference is typically a time series of historical costs, computed per dimension—e.g., by service, account, project, or tag—and stored in a database or data warehouse. The core mechanism involves computing a deviation score for each data point using a rolling window of past values.

The reference profile is built by aggregating daily or hourly cost data. For example, a daily cost per service is derived from a cost export table:

SELECT
  DATE(tracked_at) AS day,
  service,
  SUM(cost) AS daily_cost
FROM cost_events
WHERE tracked_at >= '2024-01-01'
GROUP BY day, service
ORDER BY day, service;

This table is used to compute a baseline for each service: the average cost per day over the past 28 days, with a standard deviation. Deviation is measured in standard deviations (σ), and anomalies are flagged when the observed cost exceeds the mean by more than a threshold, such as 2σ.

The detection logic is implemented in a Python script using pandas and scikit-learn, but any language or tool can use this pattern.

import pandas as pd
from datetime import timedelta

def detect_anomalies(df, window_days=28, threshold_sigma=2.0, min_records=10):
    """
    Detect anomalies in a cost time series using rolling mean and standard deviation.
    df: DataFrame with columns `day`, `service`, `daily_cost`
    """
    df = df.sort_values('day')
    df['rolling_mean'] = df.groupby('service')['daily_cost'].transform(
        lambda x: x.rolling(window=window_days, min_periods=min_records).mean()
    )
    df['rolling_std'] = df.groupby('service')['daily_cost'].transform(
        lambda x: x.rolling(window=window_days, min_periods=min_records).std()
    )
    df['anomaly_score'] = (
        (df['daily_cost'] - df['rolling_mean']) / df['rolling_std']
    )
    df['is_anomaly'] = df['anomaly_score'] > threshold_sigma
    return df

Anomaly detection runs daily, triggered by a systemctl service:

# /etc/systemd/system/finops-anomaly-detection.service
[Unit]
Description=Finops Anomaly Detection Pipeline
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/run-anomaly-detection.py
User=finops
WorkingDirectory=/opt/finops/anomaly-detection
Environment=PYTHONPATH=/opt/finops/anomaly-detection/lib
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

The system is configured to run the detection job every 24 hours, with a fixed window size and threshold.

Configuring Baseline Models

A baseline model defines how the reference profile is constructed. It is stored as a JSON configuration file, used by multiple detection jobs.

{
  "model": "rolling-sigma",
  "window_days": 28,
  "threshold_sigma": 2.0,
  "min_records": 10,
  "dimensions": [
    "service",
    "region",
    "project",
    "tag:team"
  ],
  "aggregation": {
    "granularity": "daily",
    "metric": "cost",
    "time_column": "tracked_at",
    "sum_column": "cost"
  },
  "output": {
    "table": "anomalies",
    "schema": "finops"
  }
}

This file is shared across teams and applied to different cost sources. The dimensions list determines the grouping level: a model with ["service", "tag:team"] computes separate baselines for each team within each service.

The system applies the model by joining the raw cost data with the model configuration and computing the anomaly score across all dimensions. The output is written to a finops.anomalies table.

Alerting on Detected Anomalies

When an anomaly is detected, the system produces a structured alert. Alerts are emitted as JSON messages to a message queue like nats or kafka.

{
  "event_id": "anomaly-2024-06-15-1023",
  "timestamp": "2024-06-15T12:00:00Z",
  "service": "EC2",
  "team": "Data Engineering",
  "region": "us-west-2",
  "date": "2024-06-14",
  "actual_cost": 8742.31,
  "baseline_mean": 6421.50,
  "baseline_std": 1033.42,
  "anomaly_score": 2.24,
  "is_anomaly": true,
  "confidence": 0.95,
  "context": {
    "previous_7_days": [
      { "date": "2024-06-07", "cost": 6120.10 },
      { "date": "2024-06-08", "cost": 6340.23 },
      { "date": "2024-06-09", "cost": 6712.05 },
      { "date": "2024-06-10", "cost": 6581.12 },
      { "date": "2024-06-11", "cost": 6921.44 },
      { "date": "2024-06-12", "cost": 7143.08 },
      { "date": "2024-06-13", "cost": 7420.11 }
    ],
    "trending": "upward",
    "seasonality": "weekly"
  }
}

These alerts are consumed by a monitoring system that triggers notifications via email, Slack, or PagerDuty.

Handling Edge Cases and Failure Modes

The most common failure is a silent gap in the cost data. The system assumes a complete time series, but if a day is missing—because the cost export failed or was delayed—then the rolling window calculation is skewed.

For example, if the cost_events table is missing data for June 10, 2024, the rolling_mean computed for June 14 will include only 27 days instead of 28, leading to a lower mean and higher anomaly scores. The system detects this as a false positive unless it explicitly accounts for missing days.

To handle this, the detection script computes the effective window size by counting actual days in the rolling window:

def effective_window_size(series, window_days):
    """Return the number of non-null days in the last window_days."""
    return series.rolling(window=window_days).count().iloc[-1]

The anomaly score is then adjusted by multiplying the raw score by a window_coverage factor:

df['window_coverage'] = df.groupby('service')['day'].transform(
    lambda x: x.rolling(window=window_days).count()
)
df['adjusted_anomaly_score'] = (
    df['anomaly_score'] * (df['window_coverage'] / window_days)
)

This correction prevents false alerts on systems with irregular data ingestion.

Another failure mode is dimension drift: a new team or region appears mid-month. The baseline model assumes that all dimensions have been stable, but if a new region eu-central-1 appears on June 12, the system will treat its first few days as anomalies even if the cost is normal.

To address this, the system enriches the anomaly detection with a dimension_stability check:

WITH stability AS (
  SELECT
    service,
    region,
    MIN(day) AS first_seen,
    COUNT(*) AS total_days
  FROM cost_events
  GROUP BY service, region
),
thresholds AS (
  SELECT
    service,
    region,
    first_seen,
    total_days,
    CASE
      WHEN total_days < 7 THEN 'new'
      WHEN total_days < 28 THEN 'emerging'
      ELSE 'stable'
    END AS stability_level
  FROM stability
)
SELECT
  e.day,
  e.service,
  e.region,
  e.daily_cost,
  a.anomaly_score,
  t.stability_level,
  CASE
    WHEN t.stability_level = 'new' THEN a.anomaly_score * 0.8
    WHEN t.stability_level = 'emerging' THEN a.anomaly_score * 0.9
    ELSE a.anomaly_score
  END AS final_score
FROM cost_events e
JOIN anomalies a ON e.day = a.day AND e.service = a.service AND e.region = a.region
JOIN thresholds t ON e.service = t.service AND e.region = t.region;

This ensures that new dimensions are not penalized for being “anomalous” simply because they are new.

Integrating with Cost Allocation and Showback

Anomalies are not standalone events. They are integrated into the broader cost allocation and showback workflow.

When an anomaly is detected, the system enriches the alert with allocation data from the cost_allocation table:

SELECT
  allocation_id,
  project,
  team,
  cost,
  allocation_type,
  allocation_date
FROM cost_allocation
WHERE allocation_date = '2024-06-14';

The anomaly detection pipeline joins this with the raw cost data and includes the allocation breakdown in the alert.

{
  "context": {
    "allocation": [
      { "project": "Analytics", "team": "Data Engineering", "cost": 3200.00 },
      { "project": "ML Platform", "team": "AI", "cost": 2500.00 },
      { "project": "Billing", "team": "Finance", "cost": 1500.00 }
    ],
    "allocation_method": "tag-based",
    "allocation_date": "2024-06-14"
  }
}

This allows teams to see not only that a cost spike occurred but also which projects and teams are responsible.

Monitoring and Maintenance

The system is monitored using systemctl and journalctl to track job execution:

# Check the status of the anomaly detection job
sudo systemctl status finops-anomaly-detection

# View recent logs
sudo journalctl -u finops-anomaly-detection --since "2024-06-14" -f

# Check for missing days in the cost data
psql -c "
SELECT
  DATE(tracked_at) AS day,
  COUNT(*) AS event_count
FROM cost_events
GROUP BY day
ORDER BY day
HAVING COUNT(*) = 0
"

A health check script verifies that the anomaly detection pipeline is running and producing output:

#!/bin/bash
# /opt/finops/health-check.sh

set -euo pipefail

export PGPASSWORD=secret

# Check if the last 30 days of anomalies exist
check_anomalies() {
  psql -c "
    SELECT
      COUNT(*) AS num_days
    FROM finops.anomalies
    WHERE date >= CURRENT_DATE - INTERVAL '30 days'
    " | grep -q '30'
}

# Check if the cost data has been ingested for today
check_cost_ingestion() {
  psql -c "
    SELECT
      COUNT(*) AS count
    FROM cost_events
    WHERE DATE(tracked_at) = CURRENT_DATE
    " | grep -q '1'
}

# Run checks and report status
if check_anomalies && check_cost_ingestion; then
  echo "OK: Anomalies and cost data present"
  exit 0
else
  echo "ERROR: Anomaly detection pipeline failed"
  exit 1
fi

This script is run every 15 minutes by cron and alerts on failure.

Tuning for Different Use Cases

Different teams tune the anomaly detection for their needs.

Finance team: uses a 60-day window and a 1.5σ threshold to detect gradual changes in spending.
Engineering team: uses a 14-day window and a 3σ threshold to catch sudden spikes, such as from a new deployment.
Cloud Operations team: enables context-aware thresholds—higher sensitivity during business hours, lower during weekends.

All tuning is done through the model configuration file. No ad-hoc scripts or dashboard settings are required.

The system supports multiple models, each with its own configuration, and can run them in parallel. The output from each model is merged and enriched with a common metadata layer, enabling cross-model correlation.

Summary

Finops anomaly detection is a continuous, data-driven process that surfaces unexpected cloud costs. It relies on a consistent time series of cost data, a well-defined baseline model, and a robust alerting mechanism. The core pattern—rolling window, standard deviation, dimensioned aggregation—is used across all teams and platforms, with configuration as the primary mechanism for variation. The system handles real-world issues: missing data, dimension drift, and context shifts—without requiring deep integration with a specific product. It is not tied to a UI, a dashboard, or a proprietary API. It is a pattern that can be implemented in any engineering environment, using only standard tools and practices.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.