Anomaly Detection with ML | The Garnet Wiki

Anomaly detection using machine learning identifies unusual patterns in data that deviate significantly from expected behavior. This technique is essential when systems generate large volumes of telemetry or operational data, and manual thresholds are insufficient to catch subtle or novel deviations.

Detecting Anomalies in Time Series Data

Training a Univariate Anomaly Detection Model

Use scikit-learn's IsolationForest for unsupervised anomaly detection in time series data:

from sklearn.ensemble import IsolationForest
import numpy as np

# Sample training data: hourly system CPU utilization percentages
X_train = np.array([[0.8], [0.75], [0.9], [0.85], [0.6], [0.7], [0.88], [0.92]])

# Initialize and fit model
model = IsolationForest(contamination=0.1, random_state=42)
model.fit(X_train)

# Predict anomalies in new data
X_test = np.array([[0.7], [0.5], [0.95]])
predictions = model.predict(X_test)

The contamination parameter controls the expected proportion of anomalous instances. Setting it to 0.1 assumes 10% of data points are anomalies. If you set this too high, normal data points will be flagged as anomalies; if too low, real anomalies may go undetected.

Using Local Outlier Factor for Multivariate Series

For datasets with multiple correlated features:

from sklearn.neighbors import LocalOutlierFactor

X_train = np.array([[1.0, 2.0], [1.1, 2.1], [0.9, 1.9], [10.0, 15.0]])

# Fit model with n_neighbors=5
model = LocalOutlierFactor(n_neighbors=5, contamination=0.1)
model.fit(X_train)

# Get anomaly scores (lower is more anomalous)
scores = model.negative_outlier_factor_

n_neighbors controls how many neighbors to consider when calculating local density. If set too low, the algorithm becomes sensitive to noise; if too high, it may fail to detect subtle anomalies.

Model Persistence and Deployment

Save trained models using joblib:

pip install joblib

import joblib

# Save model
joblib.dump(model, 'anomaly_model.pkl')

# Load model
loaded_model = joblib.load('anomaly_model.pkl')

If you reload a model in a different Python environment, ensure the scikit-learn version matches. Version mismatches can silently produce incorrect predictions.

Implementing Real-Time Anomaly Scoring

Streaming Data Pipeline

Use Apache Kafka with a Python consumer to process streaming metrics:

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'metrics-topic',
    bootstrap_servers=['localhost:9092'],
    value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)

for message in consumer:
    data_point = message.value
    # Apply trained model to new data point
    prediction = loaded_model.predict([data_point['value']])

Kafka's max.poll.interval.ms setting must be longer than the time required for your anomaly detection logic, or you risk consumer rebalancing failures.

Batch Scoring with Pandas

For batch processing of historical data:

import pandas as pd
from sklearn.preprocessing import StandardScaler

df = pd.read_csv('metrics.csv')
scaler = StandardScaler()
df['scaled_value'] = scaler.fit_transform(df[['value']])

# Apply model
df['anomaly'] = model.predict(df[['scaled_value']])

If the data has different scales, scaling before prediction is essential. Without it, features with larger magnitudes will dominate the anomaly detection algorithm.

Configuring Anomaly Detection Thresholds

Setting Dynamic Thresholds

Use quantile-based thresholds instead of fixed values:

import numpy as np

# Calculate 95th percentile for normal range
threshold = np.percentile(X_train.flatten(), 95)
anomaly_scores = model.decision_function(X_test)

# Points with scores below threshold are anomalies
is_anomaly = anomaly_scores < threshold

The decision_function returns negative scores, where more negative values indicate stronger anomalies. If you use predict() instead of decision_function(), you lose the ability to adjust thresholds dynamically.

Using Precision-Recall Trade-offs

When tuning for precision vs recall:

# Adjust contamination parameter to control false positive rate
model = IsolationForest(contamination=0.05)  # Lower contamination = higher precision
model.fit(X_train)

Setting contamination too low causes the model to miss anomalies, while setting it too high results in many false positives.

Handling Data Drift

Monitoring Model Performance Over Time

Track prediction consistency:

# Compare recent predictions with historical baseline
recent_predictions = model.predict(new_data)
historical_performance = calculate_accuracy(baseline_predictions, recent_predictions)

If performance drops significantly, retraining may be necessary. Monitor the score_samples method for evidence of changing data distributions:

scores = model.score_samples(X_test)
mean_score = np.mean(scores)
std_score = np.std(scores)

A large increase in standard deviation of scores suggests data distribution has shifted.

Retraining Strategy

Retrain when performance drops below a threshold:

# Simple drift detection
if mean_score < -0.5 or std_score > 0.3:
    model.fit(X_train_new)  # Retrain with recent data

If you do not retrain, the model may become obsolete as data characteristics change.

Common Failure Modes

Silent Data Format Mismatches

Ensure input data matches training format exactly:

# Training: expects single feature
X_train = np.array([[0.8], [0.75]])  # Shape: (2, 1)

# Testing: wrong shape
X_test = np.array([0.9, 0.85])  # Shape: (2,) - not (2, 1)

This mismatch produces incorrect predictions without raising an error.

Inconsistent Scaling Across Environments

Always use the same scaler for training and inference:

# Training
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)

# Inference - must use same scaler
X_test_scaled = scaler.transform(X_test)  # NOT fit_transform

Using fit_transform during inference introduces bias and leads to incorrect anomaly detection.

Model Versioning Issues

When deploying to production, ensure model version matches training data:

# Store model with metadata
model_info = {
    "version": "1.0",
    "training_date": "2023-06-15",
    "features": ["cpu_utilization"],
    "contamination": 0.1
}

If a new version of the model uses different features, but the deployment pipeline does not enforce consistency, predictions may be made on incompatible data.

Missing Input Validation

Always validate incoming data before scoring:

def validate_input(data):
    if not isinstance(data, (list, np.ndarray)):
        raise ValueError("Input must be list or numpy array")
    
    # Check for NaN values
    if np.isnan(data).any():
        raise ValueError("Input contains NaN values")

validate_input(X_test)

NaN values in input cause silent failures in many scikit-learn models.

Production Deployment Considerations

Model Serving Architecture

Deploy models using standard REST APIs with JSON payloads:

# Example API endpoint for anomaly detection
curl -X POST http://localhost:8080/anomaly \
  -H "Content-Type: application/json" \
  -d '{"value": 0.95}'

Handling Large Volumes

For high-throughput systems, batch processing is more efficient:

# Batch predictions
batch_size = 1000
for i in range(0, len(X_test), batch_size):
    batch = X_test[i:i+batch_size]
    predictions = model.predict(batch)

If batch size is too large, memory issues may occur. If too small, processing overhead increases.

Monitoring and Alerting

Monitor prediction performance through metrics:

# Track number of anomalies detected
anomaly_count = np.sum(predictions == -1)

Set up alerts when anomaly count exceeds thresholds or when prediction scores show unusual patterns.

Error Handling in Production

Always wrap model calls in error handling:

try:
    result = model.predict(input_data)
except Exception as e:
    # Log error and return default response
    logger.error(f"Anomaly detection failed: {e}")
    return {"anomaly": False, "error": str(e)}

Missing error handling leads to silent failures that may go unnoticed for weeks.

The key to successful anomaly detection with ML lies in maintaining consistent data formats, preserving model integrity across environments, and establishing clear monitoring for performance degradation.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.