Error Handling Philosophy

Production engineering guide for error handling philosophy covering patterns, implementation strategies, and operational best practices.

This document describes how to structure error handling in software systems to ensure consistent, actionable failures that support both development workflows and production reliability. It applies when designing or reviewing any component that may encounter exceptional conditions: HTTP APIs, database operations, external service calls, or internal business logic.

Understanding the Failure Modes

Distinguishing between recoverable and fatal errors

Recoverable errors should be returned with sufficient information for callers to attempt retries or fallbacks. Fatal errors indicate conditions that require human intervention or system restarts.

Example HTTP response for a recoverable error:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 60

{
  "error": {
    "code": "RATE_LIMITED",
    "message": "Request rate exceeded",
    "details": {
      "limit": 100,
      "period": "minute",
      "retry_after_seconds": 60
    }
  }
}

Example HTTP response for a fatal error:

HTTP/1.1 500 Internal Server Error
Content-Type: application/json

{
  "error": {
    "code": "INTERNAL_ERROR",
    "message": "Database connection pool exhausted",
    "details": {
      "service": "user-service",
      "timestamp": "2023-10-15T14:30:00Z",
      "request_id": "req-12345"
    }
  }
}

Handling errors without loss of context

When wrapping or translating errors, preserve the original error type and any relevant metadata. Never silently discard failure information.

In Go:

func processUser(id int) error {
    user, err := fetchUser(id)
    if err != nil {
        return fmt.Errorf("failed to fetch user %d: %w", id, err)
    }
    
    if user.Status == "inactive" {
        return fmt.Errorf("user %d is inactive: %w", id, ErrUserInactive)
    }
    
    return nil
}

In Python:

def process_user(user_id):
    try:
        user = fetch_user(user_id)
    except DatabaseError as e:
        raise UserProcessingError(f"Failed to fetch user {user_id}") from e
    
    if user.status == "inactive":
        raise UserInactiveError(f"User {user_id} is inactive")

Defining Error Responses

Use consistent response structures

All error responses must follow a predictable format including error code, human-readable message, and optional structured details.

Example structure:

{
  "error": {
    "code": "string",
    "message": "string",
    "details": {
      "string": "any"
    }
  }
}

Map internal errors to external codes

Use consistent error codes that align with business concepts rather than implementation details.

Valid codes:

Invalid codes:

Handling External Dependencies

Propagate timeouts appropriately

When calling external services, use timeout configurations that reflect both the expected service response time and acceptable failure handling.

Example using Python requests:

import requests

response = requests.get(
    "https://api.example.com/users/123",
    timeout=(5, 10)  # connect timeout 5s, read timeout 10s
)

Implement circuit breaker behavior

When external services are unavailable, fail fast with appropriate error codes rather than waiting indefinitely.

Example retry logic:

import time
import requests

def call_external_api(url, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = requests.get(url, timeout=5)
            if response.status_code == 200:
                return response.json()
            elif response.status_code == 429:
                time.sleep(2 ** attempt)  # exponential backoff
                continue
            else:
                raise ExternalServiceError(f"HTTP {response.status_code}")
        except requests.Timeout:
            if attempt == max_retries - 1:
                raise ExternalServiceTimeout("External service timed out")
            time.sleep(2 ** attempt)
    raise ExternalServiceError("Max retries exceeded")

Database Error Handling

Distinguish between constraint violations and system errors

Database constraint violations should return specific error codes, not generic internal errors.

Example Postgres constraint violation:

-- This query might return a constraint violation error
INSERT INTO users (email, username) VALUES ('[email protected]', 'existing');
-- Error: duplicate key value violates unique constraint "users_email_key"

Handle transaction failures gracefully

When database transactions fail, ensure rollback behavior is consistent and errors are propagated appropriately.

In PostgreSQL:

BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;
-- If either update fails, the entire transaction rolls back

Testing Error Conditions

Test error paths explicitly

Every error condition must be tested with specific assertions about response codes, messages, and structure.

Example using pytest:

def test_user_not_found():
    response = client.get("/users/999")
    assert response.status_code == 404
    assert response.json()["error"]["code"] == "RESOURCE_NOT_FOUND"

Validate error responses match documented schemas

Ensure all error response bodies conform to the expected JSON schema.

Common Failure Patterns to Avoid

Silent failures in background processes

Background tasks must handle errors and either retry or report them explicitly.

Example worker process:

def process_job(job_id):
    try:
        result = perform_work(job_id)
        mark_job_complete(job_id, result)
    except WorkError as e:
        # This error must be logged and reported
        log_error(f"Failed to process job {job_id}: {e}")
        report_failure(job_id, str(e))
        raise  # Re-raise to indicate failure to calling system

Confusing error codes with HTTP status codes

Never use a 500 Internal Server Error for a validation problem.

Correct:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid email format"
  }
}

Incorrect:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid email format"
  }
}

Status code: 500 (should be 400)

Inconsistent error message formats

All error messages should be in consistent natural language and avoid internal implementation details.

Good:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Email must be a valid format"
  }
}

Poor:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Email is invalid"
  }
}

Production Monitoring and Alerting

Define error thresholds for alerting

Critical errors should trigger immediate alerts, while recoverable ones may be logged with less urgency.

Example alert configuration:

Include request context in error logs

All errors must include trace identifiers and relevant request parameters for debugging.

Example log format:

ERROR [req-12345] User processing failed: user_id=123, error=USER_NOT_FOUND

Configuration and Deployment

Set appropriate timeout values

Timeouts should be based on expected service behavior and acceptable failure modes.

Example configuration:

external_services:
  api_gateway:
    connect_timeout: 5s
    read_timeout: 10s
    max_retries: 3

Define error handling policies per service

Each service must define its own error handling strategy for different types of failures.

Example policy:

Implementation Patterns

Use error types for structured handling

Implement distinct error types that carry metadata useful for consumers.

In Go:

type ValidationError struct {
    Field   string
    Message string
}

func (e ValidationError) Error() string {
    return fmt.Sprintf("validation error on %s: %s", e.Field, e.Message)
}

Implement error logging middleware

Ensure all errors are logged with appropriate context and severity.

Example middleware:

def error_middleware(func):
    def wrapper(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except Exception as e:
            logger.error(f"Error in {func.__name__}: {e}", extra={
                "error_type": type(e).__name__,
                "request_id": get_current_request_id()
            })
            raise
    return wrapper

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.