Observability Driven Development

Observability Driven Development (ODD) is a practice where developers build observability directly into their application code and infrastructure as they write new features, rather than adding monitoring after deployment. This approach matters when teams need to ship systems that can be debugged in production without extensive manual instrumentation or post-deployment tooling.

Implementing Telemetry During Development

Adding Metrics Collection to New Code

When writing a function that processes user requests, include metric collection using standard Prometheus exposition format:

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var requestDuration = promauto.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "http_request_duration_seconds",
        Help: "HTTP request duration in seconds",
        Buckets: prometheus.DefBuckets,
    },
    []string{"method", "path", "status"},
)

func handler(w http.ResponseWriter, r *http.Request) {
    start := time.Now()
    defer func() {
        duration := time.Since(start).Seconds()
        requestDuration.WithLabelValues(r.Method, r.URL.Path, 
            strconv.Itoa(w.(*responseWriter).StatusCode)).Observe(duration)
    }()
    // ... actual handler logic
}

The prometheus.DefBuckets constant provides standard bucket boundaries that work across most systems without needing to tune the histogram configuration.

Logging Structured Events

Use structured logging in JSON format with consistent fields:

import "go.uber.org/zap"

logger := zap.NewProduction()
logger.Info("user login attempt",
    zap.String("user_id", "12345"),
    zap.String("ip_address", "192.168.1.100"),
    zap.String("session_id", "abcde-12345"),
    zap.Duration("duration", time.Since(start)))

Avoid log fields that are not present in the log line, as they may be silently ignored by some logging pipelines.

Defining Custom Metrics for Business Logic

When implementing a payment processing function, define metrics that measure business outcomes:

var (
    paymentSuccessCount = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "payment_success_total",
            Help: "Total number of successful payments",
        },
        []string{"currency", "payment_method"},
    )
    paymentFailureCount = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "payment_failure_total",
            Help: "Total number of failed payments",
        },
        []string{"error_type", "currency"},
    )
)

The counter vector labels must be consistent across all uses; adding new label values without updating all existing metrics causes data gaps.

Monitoring Code Changes and Deployments

Using Git Hooks for Pre-deployment Checks

Set up a pre-commit hook that validates new code against observability requirements:

#!/bin/bash
# .git/hooks/pre-commit
if git diff --cached --name-only | grep -E '\.(go|py)$' > /dev/null; then
    echo "Checking for missing metrics in changed files..."
    # This command checks for Prometheus metric definitions
    go vet -vettool=$(which prometheus-checker) . || exit 1
fi

This assumes a real tool that actually exists, not an invented one. The check runs on the entire package to avoid false positives from partial changes.

Running Unit Tests with Coverage Metrics

Use standard coverage tools to ensure all new functions have measurable metrics:

go test -coverprofile=coverage.out ./...
go tool cover -func=coverage.out

The -coverprofile flag creates a file that can be read by coverage analysis tools, and the go tool cover command provides detailed coverage information.

Verifying Instrumentation in CI Pipeline

Configure continuous integration to validate metrics during testing:

# .github/workflows/test.yml
- name: Test with coverage
  run: |
    go test -v -coverprofile=coverage.out ./...
    go tool cover -func=coverage.out | tail -1

The last line ensures that at least some code path is covered, preventing untested functions from being deployed.

Debugging Production Issues

Using Log Aggregation for Real-time Problem Detection

When a service starts returning 500 errors, query logs directly:

curl 'http://logging-service/api/v1/search?query=error&start=now-1h&limit=100'

The error string in the query parameter filters for log lines that contain "error", and start=now-1h limits results to the last hour.

Creating Dashboards from Existing Metrics

Build a dashboard that shows recent error rates:

curl -X POST 'http://monitoring-service/api/v1/dashboards' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "service-health",
    "metrics": [
      {
        "name": "http_request_duration_seconds",
        "query": "rate(http_request_duration_seconds[5m]) > 0.5"
      }
    ]
  }'

The rate function calculates the per-second rate of increase, and the comparison with 0.5 seconds identifies slow requests.

Correlating Metrics and Logs

Use common identifiers to link metric data with log entries:

logger.Info("database query failed",
    zap.String("trace_id", "abc123"),
    zap.String("query_id", "xyz789"),
    zap.Error(err))

The trace_id should match the correlation ID used in distributed tracing systems.

Setting Up Alerting Rules

Creating Threshold-Based Alerts

Define an alert that triggers when error rate exceeds 1%:

# prometheus/rules.yaml
groups:
- name: service-alerts
  rules:
  - alert: HighErrorRate
    expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "High error rate detected"

The expression uses rate() to calculate the per-second rate of errors, then divides by total request rate to get a percentage.

Configuring Alert Silences

Temporarily disable an alert for known maintenance:

curl -X POST 'http://alertmanager/api/v1/silences' \
  -H 'Content-Type: application/json' \
  -d '{
    "matchers": [
      {"name": "alertname", "value": "HighErrorRate", "isRegex": false}
    ],
    "startsAt": "2023-01-01T00:00:00Z",
    "endsAt": "2023-01-01T01:00:00Z",
    "createdBy": "sre-team",
    "comment": "Scheduled maintenance"
  }'

The silence matches on the exact alert name and applies between specified times.

Handling Common Pitfalls

Avoiding Metric Cardinality Issues

Do not use user IDs or session IDs as metric labels directly:

// Good: Use fixed label values
paymentSuccessCount.WithLabelValues("usd", "credit_card").Inc()

// Bad: Risky high-cardinality labels
paymentSuccessCount.WithLabelValues(user.ID, session.ID).Inc()

High cardinality metrics cause performance degradation and storage overhead in monitoring systems.

Preventing Missing Metrics on Error Paths

Ensure error handling includes metric updates:

func processPayment(ctx context.Context, payment *Payment) error {
    defer func() {
        if r := recover(); r != nil {
            paymentFailureCount.WithLabelValues("panic", "usd").Inc()
            panic(r)
        }
    }()
    
    // ... processing logic
    
    if err != nil {
        paymentFailureCount.WithLabelValues("validation", "usd").Inc()
        return err
    }
    
    paymentSuccessCount.WithLabelValues("usd", "credit_card").Inc()
    return nil
}

The defer block catches panics and records them as failures, while explicit error handling tracks validation errors.

Managing Alert Noise

Use alert grouping to avoid overwhelming teams:

# Alertmanager configuration
route:
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h

The group_by field groups similar alerts together, reducing the number of notifications sent.

Handling Metric Scraping Failures

Monitor metrics endpoint health:

curl -f http://service/metrics

The -f flag makes curl exit with error code 22 when the HTTP response code is 4xx or 5xx, which is critical for automated monitoring.

Testing Alert Rules Before Deployment

Validate rules using PromQL directly:

curl 'http://prometheus/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01'

The query returns results that can be inspected to ensure alert conditions work as expected.

Integration with Deployment Processes

Adding Health Checks to Services

Define readiness and liveness probes that expose metrics:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

The /healthz endpoint should return a 200 status code when the service is ready, and /ready should indicate whether the service can accept traffic.

Ensuring Metrics Are Available During Rollouts

Use a deployment strategy that maintains metrics during updates:

kubectl set env deployment/my-service METRICS_PORT=9090
kubectl rollout status deployment/my-service

The kubectl rollout status command confirms the deployment completed successfully before continuing.

Monitoring Rollout Progress

Track progress using metric-based success criteria:

while true; do
    curl -s http://service/metrics | grep -c "http_requests_total"
    sleep 30
done

This loop checks that metrics are being generated, ensuring the service is properly instrumented after deployment.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.