Background Job Processing

Background job processing is the practice of decoupling long-running, asynchronous, or resource-intensive tasks from the immediate request-response cycle of an API or web service. It matters when a user request must trigger work that takes longer than a few hundred milliseconds—such as sending email, processing uploaded images, generating reports, or synchronizing data across systems—without blocking the caller. The job is placed into a queue, processed independently by one or more workers, and completed with status, logs, and results.

Job Submission and Scheduling

Submitting a job begins with an API endpoint that accepts the job’s parameters and enqueues it. A typical request uses POST to /jobs with a JSON body:

{
  "type": "image_resize",
  "payload": {
    "image_id": 12345,
    "width": 800,
    "height": 600,
    "format": "webp"
  },
  "priority": "high",
  "delay": 30000
}

The type field identifies the job handler, payload contains input data, priority sets the job’s placement in the queue (e.g., low, normal, high), and delay schedules the job to run after a given number of milliseconds.

The job is persisted to a message queue, typically backed by a database or a dedicated message broker. A common pattern is to store jobs in a jobs table with columns: id, type, payload, status, priority, delayed_until, attempt, error, created_at, updated_at.

On receipt, the system returns a 201 Created response with the job’s ID:

{
  "job_id": "j-1a2b3c4d",
  "url": "/jobs/j-1a2b3c4d"
}

The client may poll this endpoint or use a callback URL to receive updates.

Worker Architecture and Execution

Workers are background processes that continuously pull jobs from the queue and execute them. A worker process runs a loop that:

1. Queries for jobs with status = 'pending' and delayed_until <= now()
2. Locks selected jobs using a SELECT ... FOR UPDATE SKIP LOCKED statement to prevent race conditions
3. Processes each job using its type to dispatch to a handler function
4. Updates the job’s status, attempt, and updated_at fields
5. Commits the transaction

A typical worker script in Python using psycopg2:

import psycopg2
import json
import time

def process_jobs():
    conn = psycopg2.connect("host=localhost dbname=app user=worker password=secret")
    cursor = conn.cursor()

    while True:
        cursor.execute("""
            SELECT id, type, payload, attempt
            FROM jobs
            WHERE status = 'pending'
              AND delayed_until <= NOW()
            FOR UPDATE SKIP LOCKED
            ORDER BY priority DESC, created_at ASC
            LIMIT 10
        """)

        jobs = cursor.fetchall()
        if not jobs:
            time.sleep(1)
            continue

        for job_id, job_type, payload_json, attempt in jobs:
            payload = json.loads(payload_json)
            try:
                result = dispatch_job(job_type, payload)
                cursor.execute(
                    "UPDATE jobs SET status = 'completed', result = %s, updated_at = NOW(), attempt = %s WHERE id = %s",
                    (json.dumps(result), attempt + 1, job_id)
                )
            except Exception as e:
                error = str(e)
                cursor.execute(
                    "UPDATE jobs SET status = 'failed', error = %s, updated_at = NOW(), attempt = %s WHERE id = %s",
                    (error, attempt + 1, job_id)
                )
                # Optionally, log to external system
                log_error(job_id, error)

        conn.commit()

def dispatch_job(job_type, payload):
    handlers = {
        "image_resize": resize_image,
        "email_send": send_email,
        "report_generate": generate_report,
    }
    handler = handlers.get(job_type)
    if not handler:
        raise ValueError(f"Unknown job type: {job_type}")
    return handler(payload)

def resize_image(payload):
    # Example implementation
    image_id = payload["image_id"]
    width = payload["width"]
    height = payload["height"]
    format = payload["format"]

    # Simulate processing
    time.sleep(1.5)

    return {
        "original_size": 1024000,
        "new_size": 345000,
        "resized_at": "2024-06-14T12:34:56Z",
        "format": format,
        "width": width,
        "height": height
    }


process_jobs()

The FOR UPDATE SKIP LOCKED pattern ensures that multiple workers do not pick the same job, and only one worker processes each job. Jobs that fail are retried, with backoff and exponential delay strategies.

Retry and Failure Handling

Jobs fail when an exception is raised during processing. The failure is recorded in the error column of the jobs table, and the job is updated to status = 'failed'. Retries are managed by incrementing the attempt counter and optionally setting a retry_delay field.

A common retry strategy is to schedule the job to run again after a delay, based on the attempt number:

• Attempt 1: delay 10s
• Attempt 2: delay 30s
• Attempt 3: delay 120s
• Attempt 4+: delay 300s

This is implemented by setting delayed_until to NOW() + delay after a failed job. A worker that processes failed jobs can also periodically requeue jobs that have been in failed state for too long.

Jobs may be marked as failed with a structured error object:

{
  "job_id": "j-1a2b3c4d",
  "status": "failed",
  "error": {
    "type": "ImageProcessingError",
    "message": "Failed to decode image: unsupported format",
    "traceback": [
      "File \"/app/jobs.py\", line 42, in resize_image",
      "    image = Image.open(io.BytesIO(data))",
      "  File \"/usr/lib/python3.11/PIL/Image.py\", line 2953, in open",
      "    raise IOError(\"cannot identify image file\")",
      "OSError: cannot identify image file"
    ],
    "details": {
      "image_id": 12345,
      "input_format": "tiff",
      "expected_formats": ["jpeg", "png", "webp"]
    }
  },
  "attempt": 3,
  "updated_at": "2024-06-14T12:35:00Z"
}

The traceback field captures the full stack, and details holds context from the job.

Job Lifecycle and Status Tracking

Jobs progress through a well-defined lifecycle:

• pending: job is ready to run, not yet assigned to a worker
• processing: job is being executed by a worker
• completed: job finished successfully
• failed: job failed and may be retried
• canceled: job was explicitly canceled before completion

Status transitions are recorded in the updated_at timestamp, and clients can query for jobs with filters:

GET /jobs?status=failed&since=2024-06-14T12:00:00Z

A job’s state is visible via a read-only endpoint:

{
  "job_id": "j-1a2b3c4d",
  "type": "image_resize",
  "status": "completed",
  "priority": "high",
  "attempt": 1,
  "payload": {
    "image_id": 12345,
    "width": 800,
    "height": 600,
    "format": "webp"
  },
  "result": {
    "original_size": 1024000,
    "new_size": 345000,
    "resized_at": "2024-06-14T12:34:56Z",
    "format": "webp",
    "width": 800,
    "height": 600
  },
  "error": null,
  "created_at": "2024-06-14T12:30:00Z",
  "updated_at": "2024-06-14T12:34:56Z"
}

When a job fails, its error field is populated with the exception details, and the result field may be populated even on success.

Delayed and Scheduled Jobs

Jobs may be scheduled to run at a future time using the delay field. A worker processes the queue and checks delayed_until on each job. Jobs with delayed_until <= now() are eligible for processing.

The delay can be used for:

• Backoff retries: after failure, delayed_until = NOW() + base_delay * (2^attempt)
• Scheduled batch processing: run a job every 15 minutes, or on a daily schedule
• Time-based workflows: e.g., send a reminder email 48 hours after a user signs up

For example, to schedule a job 1 hour from now:

POST /jobs
{
  "type": "send_reminder_email",
  "payload": {
    "user_id": 1024,
    "template": "welcome"
  },
  "delay": 3600000
}

A dedicated scheduler service can periodically insert jobs with delayed_until set, or a worker can emit new jobs based on the outcome of a previous job.

Monitoring and Observability

A key challenge is visibility into job execution. The following metrics are essential:

• Number of jobs in each status (pending, processing, completed, failed)
• Average processing time per job type
• Error rates per job type
• Time from submission to completion

These are collected through:

• updated_at timestamps for status tracking
• result fields that include timing and metadata
• Logs from worker processes, including:
• Job start time
• Job completion time
• Worker instance ID
• CPU and memory usage at start and end

A monitoring endpoint like GET /metrics/jobs returns:

{
  "total_jobs": 1247,
  "pending": 123,
  "processing": 17,
  "completed": 1012,
  "failed": 95,
  "avg_processing_time_ms": 1842,
  "job_types": [
    {
      "type": "image_resize",
      "count": 750,
      "avg_time": 1940,
      "error_rate": 0.04
    },
    {
      "type": "email_send",
      "count": 320,
      "avg_time": 1120,
      "error_rate": 0.02
    }
  ]
}

Workers can emit structured logs using JSON format:

{
  "timestamp": "2024-06-14T12:30:00.123Z",
  "level": "info",
  "worker_id": "worker-01",
  "job_id": "j-1a2b3c4d",
  "type": "image_resize",
  "message": "Started processing job",
  "payload": {
    "image_id": 12345,
    "width": 800,
    "height": 600
  }
}

Sharp Edges and Silent Failures

• Job duplication: when multiple workers pick the same job, FOR UPDATE SKIP LOCKED prevents this, but if the worker crashes mid-processing, the job may be processed twice. To avoid this, jobs should be idempotent or use a lock_token field.
• Stale jobs: jobs may remain processing for a long time, but the worker process dies. A heartbeat or updated_at threshold can detect stalled jobs and reassign them.
• Unreliable message delivery: if the database crashes after job creation but before worker pickup, a job may be lost. Using a transactional queue (e.g., a jobs table with status and lock_token) ensures durability.
• Incorrect retry delay: developers often confuse delay (when to run) with retry_delay (how long to wait before retry), leading to jobs that are delayed but not retried.
• Failure to update attempt: if a job fails but does not increment attempt, retries are not tracked, and the job may be retried indefinitely without progress.
• Error data is not persisted: a job fails, but the error field is not updated, making debugging difficult. The job’s error field must be populated in all failure paths.

Deployment and Operational Patterns

Workers are deployed as daemons using systemctl or supervisord. A typical systemd service file:

[Unit]
Description=Background Job Worker
After=network.target

[Service]
User=worker
WorkingDirectory=/app
ExecStart=/usr/bin/python3 /app/worker.py
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Workers can be scaled horizontally by running multiple instances. Each instance polls the queue independently, and jobs are distributed across them.

To pause job processing, set status = 'paused' on the worker, or add a paused flag to the workers table. A worker can also be configured with a max_jobs_per_run setting to limit the number of jobs pulled per iteration.

Common Job Types and Their Patterns

• image_resize: input is image_id, output includes dimensions, size, and format. Often idempotent.
• email_send: uses a template and recipient list. Can be batched and retried.
• report_generate: long-running, may emit multiple files. Result includes file paths and metadata.
• data_sync: pulls data from an external API and updates local database. Uses payload to store sync state.

These patterns are used across services, and the same job submission and processing logic applies regardless of the underlying technology stack.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.