Serverless Architecture Patterns

Serverless architecture patterns enable systems to scale automatically, pay only for execution time, and decouple components into independent, event-driven functions. These patterns matter when you need to react to external events—like file uploads, API requests, or IoT data—without managing servers, and when workloads vary unpredictably in frequency and duration.

Event-Driven Workflow Orchestration

Orchestrate a sequence of serverless functions triggered by a single event, where each step processes data and passes state to the next.

Use a workflow engine that supports step definitions, state persistence, and error handling. The workflow is defined as a JSON document with a steps array, each step specifying a function, input mapping, and error handling.

{
  "version": "1.0",
  "startAt": "processImage",
  "steps": {
    "processImage": {
      "type": "task",
      "resource": "arn:aws:lambda:us-east-1:123456789012:function:resizeImage",
      "input": {
        "s3Key": "$.event.s3.key",
        "s3Bucket": "$.event.s3.bucket"
      },
      "output": {
        "resizedKey": "$.output.key",
        "resizedSize": "$.output.size"
      },
      "catch": [
        {
          "errorEquals": ["ImageProcessingError"],
          "resultPath": "$.error",
          "next": "retryProcessImage"
        }
      ],
      "next": "generateThumbnail"
    },
    "generateThumbnail": {
      "type": "task",
      "resource": "arn:aws:lambda:us-east-1:123456789012:function:makeThumbnail",
      "input": {
        "imageKey": "$.processImage.output.resizedKey"
      },
      "next": "saveToDatabase"
    },
    "saveToDatabase": {
      "type": "task",
      "resource": "arn:aws:lambda:us-east-1:123456789012:function:storeMetadata",
      "input": {
        "imageKey": "$.processImage.output.resizedKey",
        "thumbnailKey": "$.generateThumbnail.output.thumbnailKey",
        "size": "$.processImage.output.resizedSize"
      },
      "end": true
    },
    "retryProcessImage": {
      "type": "task",
      "resource": "arn:aws:lambda:us-east-1:123456789012:function:resizeImage",
      "input": {
        "s3Key": "$.event.s3.key",
        "s3Bucket": "$.event.s3.bucket",
        "retryCount": "$.error.retryCount"
      },
      "next": "generateThumbnail"
    }
  }
}

Each function receives a context object with event, input, output, and state fields. The workflow engine maintains state in a durable storage—typically a database or key-value store—so that if a function fails, the workflow resumes from the last checkpoint.

Failure modes:

• A function returns {"error": "InvalidImage", "cause": "Corrupted JPEG"} but the workflow engine expects error and cause as top-level keys. If the function returns error: "InvalidImage" without cause, the catch block misses the error.
• If a function takes longer than the maximum execution time (e.g., 15 minutes in AWS Lambda), the workflow engine may timeout and retry, but the function itself must be idempotent to avoid duplicate processing.
• The input and output fields use JSONPath expressions. A typo like $.event.s3.keys (plural) instead of $.event.s3.key silently fails, leading to null values.

Asynchronous Batch Processing

Process large volumes of data in batches, where each record is processed independently by a function, and results are aggregated.

Trigger a function for each item in a queue—typically an event stream or message queue—using a single event source.

For example, when a user uploads a CSV file to an S3 bucket, the system triggers a processBatch function for each row in the file.

{
  "version": "1.0",
  "id": "batch-12345",
  "source": "s3://logs/2023-10-01.csv",
  "items": [
    { "id": 1, "name": "Alice", "amount": 100 },
    { "id": 2, "name": "Bob", "amount": 200 },
    { "id": 3, "name": "Charlie", "amount": 150 }
  ]
}

Each item is enqueued as a message on a queue such as AWS SQS, Azure Service Bus, or Kafka.

A batch processor function reads from the queue, processes one item per invocation, and writes results to a result store.

import json
import boto3

def lambda_handler(event, context):
    sqs = boto3.client("sqs")
    dynamodb = boto3.resource("dynamodb")
    table = dynamodb.Table("batch_results")

    for record in event["Records"]:
        body = json.loads(record["body"])
        item_id = body["id"]
        name = body["name"]
        amount = body["amount"]

        # Simulate processing delay
        import time
        time.sleep(0.5)

        # Save result
        table.put_item(
            Item={
                "item_id": item_id,
                "processed": True,
                "total": amount * 1.1,
                "timestamp": context.aws_request_id
            }
        )

        # Acknowledge completion
        sqs.delete_message(
            QueueUrl="https://sqs.us-east-1.amazonaws.com/123456789012/batch-queue",
            ReceiptHandle=record["receiptHandle"]
        )

    return {
        "processed": len(event["Records"]),
        "batchId": event["batchId"]
    }

Key failure modes:

• If the queue message is not deleted after processing, a message can be processed multiple times—especially when the function fails partway through.
• The function must be idempotent: processing the same message twice should not produce duplicate results.
• The batch processor can be triggered by a s3:ObjectCreated:* event, but the event payload is not guaranteed to include s3.object.key—it may use s3.key or object.key. Use $.s3.object.key to avoid silent failures.
• When using a queue with visibility timeouts (e.g., SQS), messages become invisible during processing. If a function takes longer than the visibility timeout, the message becomes available again and may be processed by another instance.

Event-Driven Data Pipeline with Stream Processing

Ingest and transform data in real time from a stream, such as IoT telemetry, logs, or user activity.

Use a stream processor that reads events continuously and applies transformations or aggregations.

A Kafka stream feeds sensor data into a serverless function that computes moving averages.

{
  "sensorId": "s100",
  "timestamp": "2023-10-01T12:00:00Z",
  "temperature": 23.4,
  "humidity": 65.1
}

A function listens to the stream and maintains a window of recent data points using a stateful store.

import json
import redis

def lambda_handler(event, context):
    r = redis.Redis(host="cache.example.com", port=6379, db=0)

    for record in event["Records"]:
        data = json.loads(record["body"])
        sensor_id = data["sensorId"]

        # Add to time-series window
        r.zadd(f"sensor:{sensor_id}:data", {json.dumps(data): data["timestamp"]})
        r.zremrangebyscore(f"sensor:{sensor_id}:data", "-inf", f"{data['timestamp'] - 300}")

        # Compute average over last 5 minutes
        recent = r.zrangebyscore(f"sensor:{sensor_id}:data", "-inf", f"{data['timestamp']}")
        avg_temp = sum(d["temperature"] for d in map(json.loads, recent)) / len(recent)

        # Store result
        r.set(f"sensor:{sensor_id}:avg_temp", avg_temp)

        # Emit alert if temperature exceeds threshold
        if data["temperature"] > 30:
            r.publish("alerts", json.dumps({
                "sensor": sensor_id,
                "type": "temperature_alert",
                "value": data["temperature"],
                "timestamp": data["timestamp"]
            }))

    return {
        "processed": len(event["Records"]),
        "window": "5 minutes"
    }

Critical failure modes:

• The stream processor must handle out-of-order events. A message with timestamp 2023-10-01T12:00:00Z arrives after 2023-10-01T12:01:00Z, but the function processes it first. The windowing logic must account for this.
• If the function uses a Redis cluster, the zadd command may fail due to network partition. The function must retry or handle partial updates.
• The function must be stateful: it may be invoked on a new instance, but the Redis connection pool is not shared across invocations. Use a connection pool or create the client inside the handler.
• The stream may emit duplicate events. The function must deduplicate using message IDs or timestamps. Without deduplication, alerting becomes noisy.

Function Chaining with Payload Transformation

Chain multiple functions where each function transforms the payload and passes it to the next, forming a pipeline.

Each function consumes a message, performs a transformation, and emits a new message to the next stage.

For example, a user registration flow:

1. validateUser → validates input and adds metadata
2. sendWelcomeEmail → sends email and logs delivery
3. updateAnalytics → updates user statistics

{
  "email": "[email protected]",
  "name": "Alice",
  "signupSource": "web"
}

Each function reads from a queue, processes the message, and writes to the next queue.

def validateUser(event, context):
    data = event["data"]
    errors = []

    if not data.get("email") or "@" not in data["email"]:
        errors.append("Invalid email address")
    if not data.get("name") or len(data["name"]) < 2:
        errors.append("Name too short")

    return {
        "validated": True,
        "errors": errors,
        "timestamp": context.aws_request_id,
        "user": {
            "email": data["email"],
            "name": data["name"].title(),
            "id": f"user_{context.aws_request_id}"
        }
    }

The next function receives this output and can use it directly.

Silent pitfalls:

• The function outputs user as a nested object, but the next function expects user.email at the top level. The sendWelcomeEmail function fails to extract the email address.
• The payload format is not standardized across functions. One function returns {"data": { ... }} while another returns { "data": ... }. The pipeline breaks when the structure changes.
• The function must handle both single messages and batched messages. A function designed for one message may fail when given an array of messages.
• If the function emits a message to a queue with a body field, the message body must be a JSON string. If it returns a dictionary without json.dumps, the queue consumer reads a nested object instead of a flat string.

State Management with Persistent Storage

Maintain state across function invocations, where the state is shared, durable, and accessible to any function.

Use a database or key-value store to persist state, and define a pattern where functions read from and write to it.

For example, a shopping cart system where each function—add item, update quantity, checkout—modifies a cart state.

{
  "cartId": "c123",
  "userId": "u456",
  "items": [
    { "productId": "p1", "name": "Widget", "quantity": 2, "price": 15.99 }
  ],
  "total": 31.98,
  "updatedAt": "2023-10-01T12:00:00Z"
}

A function reads the cart, updates it, and writes it back.

import json
import boto3

def addCartItem(event, context):
    dynamodb = boto3.resource("dynamodb")
    table = dynamodb.Table("shopping_carts")

    cart_id = event["cartId"]
    item = event["item"]

    # Get current cart
    response = table.get_item(Key={"cartId": cart_id})
    cart = response.get("Item", {"items": [], "total": 0, "updatedAt": "1970-01-01T00:00:00Z"})

    # Find existing item or add new
    for i, existing in enumerate(cart["items"]):
        if existing["productId"] == item["productId"]:
            cart["items"][i]["quantity"] += item["quantity"]
            break
    else:
        cart["items"].append(item)

    # Update total
    cart["total"] = sum(
        i["quantity"] * i["price"]
        for i in cart["items"]
    )
    cart["updatedAt"] = event["timestamp"]

    # Write back
    table.put_item(Item=cart)

    return {
        "status": "success",
        "cartId": cart_id,
        "updatedTotal": cart["total"]
    }

Common failure modes:

• The function assumes the cart exists, but get_item returns no result. The cart object is initialized with defaults, but if updatedAt is missing, the timestamp field is not updated correctly.
• Multiple functions update the same cart concurrently. A race condition occurs when two addCartItem functions read the same cart, modify it, and write it back, overwriting each other’s changes.
• The function uses put_item with Item and ConditionExpression to ensure updates are atomic. But if the condition fails (e.g., cartId = c123), the function must handle the ConditionCheckFailedException by retrying or logging.
• The state is not versioned. When the function schema changes—e.g., adding currency to the cart—older functions may read the state incorrectly, leading to silent errors.

Error Handling and Retry Patterns

Design functions to handle failures gracefully, with retry logic, dead-letter queues, and error logging.

Functions should return structured error responses and use metadata to guide retry strategies.

{
  "error": "FailedToProcessImage",
  "message": "Could not decode JPEG file",
  "details": {
    "path": "/images/2023/10/01/photo.jpg",
    "size": 2048,
    "format": "JPEG"
  },
  "retryable": true,
  "timestamp": "2023-10-01T12:00:00Z"
}

A function may retry with exponential backoff. It logs errors to a central store, such as CloudWatch Logs or a database.

A dead-letter queue (DLQ) receives messages that fail after multiple retries.

import json
import boto3
import time

def processImage(event, context):
    try:
        # Simulate processing
        time.sleep(2)

        # Validate input
        if not event.get("imageKey"):
            raise ValueError("Missing imageKey")

        # Simulate failure
        if event.get("imageKey").endswith(".png"):
            raise RuntimeError("Image format not supported")

        return {
            "processed": True,
            "imageKey": event["imageKey"],
            "output": f"processed/{event['imageKey']}"
        }

    except Exception as e:
        error = {
            "error": type(e).__name__,
            "message": str(e),
            "details": event,
            "retryable": isinstance(e, (ConnectionError, TimeoutError, RuntimeError)),
            "timestamp": context.aws_request_id
        }

        # Write to error log
        logs_client = boto3.client("logs")
        logs_client.put_log_events(
            logGroupName="/functions/image-processor",
            logStreamName="errors",
            logEvents=[
                {
                    "timestamp": int(time.time() * 1000),
                    "message": json.dumps(error)
                }
            ]
        )

        # Return error for queue
        return {
            "error": error,
            "retryAfter": 30
        }

Key insights:

• A non-retryable error (e.g., InvalidInputError) should be sent to a DLQ and not retried.
• The function must define retry delays and maximum attempts. If the retry strategy is not configured, a function may retry a message once and then give up.
• The error format must be consistent: a retryable field should be a boolean, not a string or integer.
• When using a DLQ, the function must acknowledge the message only after processing it successfully. If the function fails after acknowledging, the message is lost.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.