Error Handling Strategies

Production engineering guide for error handling strategies covering patterns, implementation strategies, and operational best practices.

Error handling is the backbone of resilient backend systems. It determines how services respond to failures—network glitches, database timeouts, invalid inputs, and transient errors—transforming a fragile system into a robust one. This page details practical strategies for designing error handling that works under pressure, from the moment a request enters a service to the point where a user sees a meaningful response. These strategies apply at every layer: API endpoints, database interactions, background jobs, and distributed workflows.

Handling Errors in HTTP APIs

When a client makes an HTTP request to a backend service, the server must communicate not only success data but also failure information. The key is to use standardized error responses that clients can parse and act upon.

A well-structured error response includes at least:

For example, a POST /users request that fails due to a missing required field returns:

{
  "status": 400,
  "code": "validation_error",
  "message": "One or more fields are invalid.",
  "details": [
    {
      "field": "email",
      "message": "Email is required."
    },
    {
      "field": "age",
      "message": "Age must be an integer."
    }
  ]
}

Clients can use the code to trigger specific UI behaviors—e.g., highlighting fields with validation_error, or showing localized messages based on code values.

Use consistent error codes across services. Common patterns include:

Clients should expect 5xx responses to be retryable. Services should include a Retry-After header when a response is retryable, with the number of seconds until the client may retry:

HTTP/1.1 503 Service Unavailable
Retry-After: 15
Content-Type: application/json

{
  "status": 503,
  "code": "service_unavailable",
  "message": "The service is temporarily overloaded."
}

Managing Database Errors

Database operations are a primary source of failure. A query may time out, a transaction may fail, or a foreign key constraint may be violated. The key is to handle these errors with precision and clarity.

Use psql to demonstrate common failure scenarios and their recovery:

psql -c "SELECT * FROM users WHERE id = 123;"

This may return:

psql: error: could not connect to server: Connection refused
        Is the server running on host "db.example.com" (192.168.1.10) and accepting
        TCP/IP connections on port 5432?

This error indicates a network or service-level issue. The application must detect this and fall back to a cached or degraded state.

For query-level failures, use EXCEPTION blocks in PostgreSQL:

DO $$
BEGIN
  INSERT INTO users (id, name, email) VALUES (123, 'Alice', '[email protected]');
EXCEPTION
  WHEN UNIQUE_VIOLATION THEN
    RAISE NOTICE 'User already exists: %', SQLERRM;
    UPDATE users SET name = 'Alice (Updated)' WHERE id = 123;
  WHEN FOREIGN_KEY_VIOLATION THEN
    RAISE NOTICE 'Invalid foreign key: %', SQLERRM;
    INSERT INTO user_logs (user_id, action, message) VALUES (123, 'failed_insert', SQLERRM);
END $$;

The application can catch these exceptions and map them to higher-level error codes. For example, UNIQUE_VIOLATION becomes user_already_exists, and FOREIGN_KEY_VIOLATION becomes invalid_user_reference.

When a transaction fails, include a transaction_id in the error response so clients can correlate logs and traces. Use SAVEPOINT to enable partial rollbacks:

BEGIN;
  INSERT INTO orders (id, user_id) VALUES (1001, 123);
  SAVEPOINT order_created;

  INSERT INTO order_items (order_id, product_id, quantity)
    VALUES (1001, 456, 10);

  INSERT INTO inventory (product_id, quantity)
    VALUES (456, 50);

  -- If inventory update fails, rollback to order_created
  EXCEPTION
    WHEN OTHERS THEN
      ROLLBACK TO SAVEPOINT order_created;
      RAISE NOTICE 'Order created, but inventory update failed: %', SQLERRM;
END;

Error Handling in Background Jobs

Background jobs often run asynchronously, making error visibility and recovery more challenging. Use systemctl to manage job workers and curl to monitor their state.

A worker process might be started with:

systemctl start user-processor.service

The service file (/etc/systemd/system/user-processor.service) includes:

[Unit]
Description=User Processing Worker
After=network.target

[Service]
Type=simple
User=app
WorkingDirectory=/app
ExecStart=/usr/bin/python3 -m user_processor
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
Environment=DATABASE_URL=postgresql://db.example.com/users
Environment=REDIS_URL=redis://cache.example.com:6379

[Install]
WantedBy=multi-user.target

When a job fails, the worker logs to the system journal:

journalctl -u user-processor.service -f

A typical failure log:

Jul 10 14:23:11 app-server user-processor[1234]: ERROR: Failed to process user 456: database connection lost
Jul 10 14:23:11 app-server user-processor[1234]: Traceback (most recent call last):
Jul 10 14:23:11 app-server user-processor[1234]:   File "/app/user_processor.py", line 42, in process_user
Jul 10 14:23:11 app-server user-processor[1234]:     cursor.execute("UPDATE users SET status = 'active' WHERE id = %s", (user_id,))
Jul 10 14:23:11 app-server user-processor[1234]: psycopg2.OperationalError: could not connect to server: Connection timed out
Jul 10 14:23:11 app-server user-processor[1234]:     Is the server running on host "db.example.com" and accepting
Jul 10 14:23:11 app-server user-processor[1234]:     TCP/IP connections on port 5432?

The worker must handle transient errors by retrying failed jobs. Use a retry queue with exponential backoff.

When a job fails, the service should:

For example, a jobs table with:

CREATE TABLE jobs (
  id UUID PRIMARY KEY,
  name TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT CHECK (status IN ('pending', 'processing', 'failed', 'completed')),
  attempts INTEGER DEFAULT 0,
  error TEXT,
  retry_at TIMESTAMPTZ
);

The worker updates the job on failure:

UPDATE jobs
SET status = 'failed',
    error = 'Database connection lost',
    attempts = attempts + 1,
    retry_at = NOW() + (30 * (2 ^ (attempts + 1))) * '1 second'
WHERE id = 'a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8';

Error Propagation Across Services

In a microservices architecture, a single client request may traverse multiple services. Errors must propagate across service boundaries with fidelity.

Use curl to simulate a request that fails in a downstream service:

curl -X POST http://api.example.com/v1/orders \
  -H "Content-Type: application/json" \
  -d '{
    "user_id": 123,
    "items": [
      {"product_id": 456, "quantity": 10}
    ]
  }'

The orders service calls the catalog service:

curl -X GET http://catalog.example.com/v1/products/456

If the catalog service is down, the orders service receives a 503 response with:

{
  "status": 503,
  "code": "catalog_unavailable",
  "message": "Catalog service is down or unreachable.",
  "details": {
    "url": "http://catalog.example.com/v1/products/456",
    "attempt": 1,
    "retry_after": 10
  }
}

The orders service should:

The final response to the client includes the original error and any propagated details:

{
  "status": 400,
  "code": "order_creation_failed",
  "message": "Failed to create order due to catalog service failure.",
  "details": {
    "catalog_error": {
      "status": 503,
      "code": "catalog_unavailable",
      "message": "Catalog service is down or unreachable.",
      "details": {
        "url": "http://catalog.example.com/v1/products/456",
        "attempt": 1,
        "retry_after": 10
      }
    },
    "input": {
      "user_id": 123,
      "items": [
        {"product_id": 456, "quantity": 10}
      ]
    }
  }
}

Use X-Trace-ID headers to correlate logs across services. The orders service sets:

X-Trace-ID: 9a8b7c6d-5e4f-3a2b-1c0d-9a8b7c6d5e4f

Each service appends to the trace ID, and logs are enriched with the full chain.

Error Handling in Configuration and Deployment

Configuration errors are silent but common. A service may start up, but fail to connect to a database because of a typo in the DATABASE_URL.

Use systemctl to inspect service state:

systemctl status user-processor.service

Output:

● user-processor.service - User Processing Worker
   Loaded: loaded (/etc/systemd/system/user-processor.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2024-07-10 14:23:11 UTC; 2min 15s ago
  Process: 1234 ExecStart=/usr/bin/python3 -m user_processor (code=exited, status=1/FAILURE)
 Main PID: 1234 (code=exited, status=1/FAILURE)

Jul 10 14:23:11 app-server user-processor[1234]: ERROR: Failed to connect to database: Connection timed out
Jul 10 14:23:11 app-server user-processor[1234]: ERROR: Database URL: postgresql://db.example.com:5433/users
Jul 10 14:23:11 app-server user-processor[1234]: ERROR: Expected port 5432, but got 5433.

The error reveals a configuration mismatch: the service expects 5432, but the database runs on 5433.

Use docker-compose.yml to define and manage multi-service deployments. A service may fail to start due to a missing environment variable:

version: '3.8'
services:
  app:
    image: myapp:latest
    environment:
      - DATABASE_URL=postgresql://db:5432/users
      - REDIS_URL=redis://cache:6379
      - LOG_LEVEL=info
    depends_on:
      - db
      - cache
    restart: unless-stopped

If db fails to start, app logs:

ERROR: Failed to connect to database: could not connect to server: Connection refused
  Is the server running on host "db" (172.18.0.2) and accepting TCP/IP connections on port 5432?

The app service must retry connection setup with exponential backoff, or fail fast with a healthcheck:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 30s

Error Handling in Code: Try-Catch and Error Types

Use try-catch blocks in Python to handle multiple error types in a single function:

def create_user(email, name, age):
    try:
        db = get_database_connection()
        db.execute(
            "INSERT INTO users (email, name, age) VALUES (%s, %s, %s)",
            (email, name, age)
        )
        db.commit()
    except psycopg2.DatabaseError as e:
        if e.pgcode == '23505':  # unique violation
            return {
                'status': 409,
                'code': 'user_already_exists',
                'message': f'User with email {email} already exists.',
                'details': {'email': email}
            }
        elif e.pgcode == '23503':  # foreign key violation
            return {
                'status': 400,
                'code': 'invalid_user_reference',
                'message': 'Invalid foreign key in user creation.',
                'details': {'constraint': e.diag.constraint_name}
            }
        else:
            return {
                'status': 500,
                'code': 'database_error',
                'message': 'Failed to create user.',
                'details': {'error': str(e)}
            }
    except ValueError as e:
        return {
            'status': 400,
            'code': 'invalid_input',
            'message': 'Invalid input provided.',
            'details': {'error': str(e)}
        }
    except Exception as e:
        return {
            'status': 500,
            'code': 'internal_error',
            'message': 'An unexpected error occurred.',
            'details': {'error': str(e)}
        }

The pgcode field allows precise error classification. The service returns structured error responses that clients can use to adapt their behavior.

Common Pitfalls

Error handling is not a one-time task but an ongoing discipline. The goal is not just to detect failures, but to communicate them clearly, recover from them efficiently, and make the entire system feel resilient to users and operators alike.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.