Event-Driven Automation: Reacting to Changes Instead of Polling

Design automation systems that respond to events in real time rather than running on fixed schedules. Covers event sources, webhook patterns, event buses, CloudWatch Events, GitHub Actions triggers, and building reactive pipelines that scale without cron job sprawl.

Event-driven automation reacts to changes as they occur, rather than periodically checking for them. It is essential when latency, resource efficiency, and responsiveness are critical—such as in real-time monitoring, CI/CD pipelines, incident response, and infrastructure orchestration. The core idea is simple: instead of polling for state changes (e.g., every 5 minutes checking if a file has been uploaded), systems emit events upon change, and automation workflows listen and respond immediately.

Triggering Automation from Events

An event is a discrete, time-stamped signal that something has changed. It is typically a JSON object containing metadata and payload. The automation system listens for events on a message bus, topic, or queue, and executes a workflow when an event arrives.

Events are produced by sources such as:

Each event must carry at least:

Example event:

{
  "event_type": "file.uploaded",
  "timestamp": "2023-10-05T14:23:11Z",
  "source": "s3://my-bucket/uploads",
  "payload": {
    "file_id": "f1a2b3c4d5e6",
    "path": "/2023/10/05/photo.jpg",
    "size_bytes": 2097152,
    "mime_type": "image/jpeg",
    "uploader": "[email protected]"
  }
}

The automation system consumes events via a subscription mechanism. It may use a long-lived connection (e.g., Kafka consumer group, RabbitMQ durable queue) or a polling-based listener (e.g., curl loop with ?since=123456).

Event Sources and Producers

A file watcher using inotify emits IN_CREATE and IN_MOVED_TO events. These are captured by a daemon that publishes to a Kafka topic:

A script reads events.log and sends each event to Kafka:

#!/bin/bash
while IFS= read -r line; do
  event_type=$(echo "$line" | cut -d' ' -f1)
  file_path=$(echo "$line" | cut -d' ' -f4-)

  kafka-console-producer.sh --broker-list kafka:9092 --topic events \
    <<< '{
      "event_type": "file.uploaded",
      "timestamp": "'"$(date -Iseconds)"'",
      "source": "inotify://localhost/data/uploads",
      "payload": {
        "path": "'"${file_path}"'",
        "size_bytes": $(stat -c %s "${file_path}"),
        "mime_type": $(file -b --mime-type "${file_path}")
      }
    }'
done < /tmp/events.log

When a new file appears in /data/uploads, the event is emitted and consumed immediately.

Consuming Events in Automation Workflows

Automation workflows respond to events by executing a sequence of actions. The workflow engine listens on one or more event sources, matches events to triggers, and runs the associated logic.

A common workflow pattern uses a configuration file that maps event types to jobs. The configuration is a JSON object with a top-level triggers key:

{
  "triggers": [
    {
      "event_type": "file.uploaded",
      "conditions": [
        {
          "field": "payload.mime_type",
          "operator": "eq",
          "value": "image/jpeg"
        },
        {
          "field": "payload.size_bytes",
          "operator": "gt",
          "value": 1048576
        }
      ],
      "actions": [
        {
          "type": "run_job",
          "job": "resize-image",
          "input": {
            "source_path": "{{payload.path}}",
            "target_size": "1024x768"
          }
        },
        {
          "type": "send_email",
          "to": "{{payload.uploader}}",
          "subject": "Your image has been processed",
          "body": "Your file {{payload.path}} was successfully resized."
        }
      ]
    }
  ]
}

The workflow engine evaluates conditions on each incoming event. Conditions use a simple expression language:

When all conditions pass, the engine executes the actions in sequence. Actions may be:

Workflow Execution and State Management

Each workflow execution has a unique execution_id and is tracked through state transitions:

State is persisted in a database or file system. The workflow engine maintains a registry of active executions, and each action may emit a new event.

Example: resize-image job is a docker run command:

#!/bin/bash
set -euo pipefail

source_path="$1"
target_size="$2"
output_path="/var/results/$(basename "$source_path")"

mkdir -p "$(dirname "$output_path")"

ffmpeg -i "$source_path" \
  -vf "scale=${target_size}" \
  -c:v libx264 \
  -preset fast \
  -crf 23 \
  "$output_path"

# Emit completion event
curl -X POST http://events-service:8080/events \
  -H "Content-Type: application/json" \
  -d '{
    "event_type": "image.resized",
    "timestamp": "'"$(date -Iseconds)"'",
    "source": "resize-image-job",
    "payload": {
      "input_path": "'"${source_path}"'",
      "output_path": "'"${output_path}"'",
      "size": "'$(du -b "$output_path" | cut -f1)'"
    }
  }'

This job runs on a worker node and emits image.resized events when complete. The automation system listens for these and can trigger downstream actions.

Event-Driven Automation in Practice

Event Duplication and Idempotency

Event producers may emit duplicate events. A Kafka producer using acks=all may replay a message after a broker failure. The automation system must handle idempotency.

Each event must include a event_id (a UUID), and the workflow engine tracks processed event IDs in a database table:

CREATE TABLE event_log (
  event_id UUID PRIMARY KEY,
  event_type TEXT,
  processed_at TIMESTAMPTZ,
  execution_id UUID,
  status TEXT
);

When a new event arrives, the system checks if event_id already exists. If so, it skips processing. This prevents double-execution of actions.

A common failure: missing event_id in the payload. The event appears to be processed, but the workflow engine fails to deduplicate, leading to duplicate image resize jobs for the same file.

Event Ordering and Delivery Guarantees

Events are not guaranteed to arrive in order. A workflow engine may receive file.uploaded events out of sequence. To handle this, events should include sequence_number or event_id and the engine can:

For example, a system expects one file.uploaded per minute. It tracks the last event_id seen and logs gaps:

# Check for missing events every 5 minutes
last_event_id=$(psql -c "SELECT event_id FROM event_log WHERE event_type = 'file.uploaded' ORDER BY processed_at DESC LIMIT 1" -t)
current_event_id=$(curl -s http://events-service:8080/events?since=${last_event_id} | jq -r '.[0].event_id')

if [[ "$current_event_id" != "$last_event_id" ]]; then
  echo "Missing events between $last_event_id and $current_event_id"
fi

Handling Event Failures and Backpressure

When an action fails, the workflow engine logs the error and may retry the execution. Common failure modes:

The system should:

Example error log entry:

{
  "execution_id": "e1a2b3c4-d5e6-f7g8-h9i0",
  "event_id": "f1a2b3c4-d5e6-f7g8-h9i0",
  "status": "failed",
  "error": {
    "type": "runtime",
    "message": "Command 'ffmpeg' failed with exit code 1",
    "details": "ffmpeg: error: Cannot open input file: /data/uploads/2023/10/05/photo.jpg",
    "stack_trace": [
      "at main (resize-image.sh:25)",
      "at run_job (workflow-engine:43)"
    ]
  },
  "retries": 3,
  "last_retry": "2023-10-05T14:35:00Z"
}

The automation system can also handle backpressure:

Common Pitfalls and Sharp Edges

Synchronous vs. Asynchronous Event Flow

In some workflows, the event producer waits for the response from the automation system.

For example, an HTTP API endpoint accepts file uploads and returns a 202 Accepted immediately, but the actual processing is event-driven:

POST /upload HTTP/1.1
Host: api.example.com
Content-Type: application/json

{
  "file": "photo.jpg",
  "path": "/2023/10/05/photo.jpg",
  "size": 2097152
}

Response:

HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "event_id": "f1a2b3c4-d5e6-f7g8-h9i0",
  "status_url": "/events/f1a2b3c4-d5e6-f7g8-h9i0"
}

The client can poll /events/f1a2b3c4-d5e6-f7g8-h9i0 to track progress. The automation system updates the event record as it proceeds through the workflow.

This pattern enables synchronous event-driven processing—a blend of immediate feedback and eventual consistency.

Summary

Event-driven automation shifts the focus from polling to reacting. It reduces latency, saves resources, and scales naturally with load. The key is to:

The most common mistake is assuming events are delivered once and in order. In reality, events are duplicated, reordered, delayed, and lost. The automation system must be built to handle these realities—each failure mode shaping the design.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.