Workflow Orchestration Patterns

Build reliable multi-step automated workflows. Covers orchestration vs. choreography, workflow engines, retry and compensation patterns, and the patterns that make complex automation reliable and observable.

Workflow orchestration patterns are the structured, repeatable ways to coordinate sequences of automated tasks across systems, services, and teams. They matter when a single action requires multiple steps—each potentially idempotent, asynchronous, or failure-prone—and when coordination across boundaries (e.g., dev, ops, security, compliance) is required. This page describes the core patterns used to build reliable, observable, and maintainable workflows, focusing on *how* to design and implement them, not just *what* they are.

Sequential Execution with Conditional Branching

The most common pattern: a linear sequence of steps where each step runs after the previous one, but with decisions based on outputs, status, or time.

Use workflow.yml with a steps array, each step defining a run command, depends_on, and condition.

steps:
  - name: Validate PR
    run: |
      git fetch origin
      git checkout ${{ github.head_ref }}
      npm test -- --coverage
    depends_on: []
    condition: ${{ github.event_name == 'pull_request' }}

  - name: Build Artifact
    run: |
      docker build -t app:${{ github.sha }} .
      docker tag app:${{ github.sha }} registry.example.com/app:${{ github.sha }}
    depends_on:
      - Validate PR
    condition: ${{ success() }}

  - name: Deploy to Staging
    run: |
      kubectl apply -f k8s/staging.yaml
      kubectl rollout status deployment/app-staging
    depends_on:
      - Build Artifact
    condition: ${{ github.ref == 'refs/heads/main' }}

The condition field evaluates to a boolean using a simple expression language: success(), failure(), github.event_name == 'push', steps.build_artifact.outputs.image_digest, and contains(github.event.base, 'release/').

Common failure modes:

Use condition: ${{ always() }} to ensure a step runs regardless of upstream success or failure.

Parallel Execution with Fan-In and Fan-Out

When multiple independent tasks need to run concurrently, and the workflow must wait for all to complete before continuing.

steps:
  - name: Run Tests
    run: |
      npm test -- --parallel
    depends_on: []

  - name: Lint Code
    run: |
      npx eslint src/
      npx prettier --check src/
    depends_on: []

  - name: Build Docs
    run: |
      mkdocs build
    depends_on: []

  - name: Deploy to Production
    run: |
      kubectl apply -f k8s/prod.yaml
    depends_on:
      - Run Tests
      - Lint Code
      - Build Docs
    condition: ${{ success() }}

The Run Tests, Lint Code, and Build Docs steps run in parallel. The Deploy to Production step waits for all three.

Key insight: fan-out happens implicitly when multiple steps list the same parent via depends_on. Fan-in happens when a step depends on multiple earlier steps.

Sharp edges:

- name: Run Tests
  run: |
    npm test -- --coverage
    echo "coverage=87.2" >> $GITHUB_OUTPUT
  outputs:
    coverage: ${{ steps.run_tests.outputs.coverage }}

Then reference steps.run_tests.outputs.coverage in a later step.

Error Handling and Retry Policies

Workflows fail. Errors must be caught, handled, and retried with precision.

Define a retry block per step:

steps:
  - name: Deploy to Staging
    run: |
      kubectl apply -f k8s/staging.yaml
      kubectl rollout status deployment/app-staging
    retry:
      count: 3
      delay: 30s
      backoff: 2
      conditions:
        - ${{ failure() }}
        - ${{ contains(steps.deploy.outputs.error, 'timeout') }}
        - ${{ steps.deploy.outputs.status_code == 503 }}

The retry block specifies:

Failure modes:

For complex error handling, define a on_failure block:

on_failure:
  run: |
    echo "Workflow failed at step: ${{ github.job }}"
    echo "Error: ${{ github.event.workflow.failed_step }}"
    curl -X POST https://alert.example.com/webhook \
      -H "Content-Type: application/json" \
      -d '{
        "workflow": "${{ github.workflow }}",
        "job": "${{ github.job }}",
        "error": ${{ github.event.workflow.failed_step }},
        "timestamp": "'"$(date -Iseconds)"'"
      }'
  depends_on: []

This runs after any step fails, regardless of retry.

Data-Driven Workflows with Input and Output Contracts

When workflows consume structured input and produce structured output, enabling reuse and composition.

Define inputs and outputs at the workflow level.

inputs:
  environment:
    description: 'Target environment to deploy to'
    required: true
    type: string
    default: 'staging'
  image_tag:
    description: 'Docker image tag to deploy'
    required: false
    type: string
  rollout_delay:
    description: 'Delay between rollout steps'
    required: false
    type: number
    default: 30

outputs:
  deployment_id:
    description: 'Unique ID for this deployment'
    value: ${{ steps.deploy.outputs.deployment_id }}
  status:
    description: 'Deployment status'
    value: ${{ steps.deploy.outputs.status }}

The workflow receives inputs from a POST to /api/workflows/{id}/run with a JSON body:

{
  "environment": "production",
  "image_tag": "v1.2.3",
  "rollout_delay": 60
}

The workflow returns outputs in a 200 OK response:

{
  "deployment_id": "d-abc123",
  "status": "success",
  "timestamp": "2023-09-15T10:30:00Z"
}

Key patterns:

env:
  KUBE_CONTEXT: ${{ inputs.environment }}
  DEPLOYMENT_DELAY: ${{ inputs.rollout_delay }}

Sharp edges:

Stateful Workflows with Checkpointing and Resume

When workflows run for hours or days, and must survive restarts, crashes, and partial execution.

Use checkpoint to persist state between runs.

steps:
  - name: Fetch Data
    run: |
      curl -o data.json https://api.example.com/data
      echo "last_updated=$(date -Iseconds)" >> $GITHUB_OUTPUT
    outputs:
      data_file: data.json
      last_updated: ${{ steps.fetch_data.outputs.last_updated }}

  - name: Process Data
    run: |
      python process.py --input data.json --output results.csv
      echo "processed=true" >> $GITHUB_OUTPUT
    depends_on:
      - Fetch Data
    checkpoint: true

The checkpoint: true flag causes the workflow engine to:

On restart:

Failure modes:

Composable Workflows with Reusable Templates

When multiple workflows share structure: a common build, test, deploy pipeline.

Define a workflow-template.yml:

name: Deploy App
inputs:
  app_name: { type: string, required: true }
  image_tag: { type: string, default: 'latest' }
  environment: { type: string, default: 'staging' }

steps:
  - name: Build Image
    run: |
      docker build -t ${{ inputs.app_name }}:${{ inputs.image_tag }} .
      docker push ${{ inputs.app_name }}:${{ inputs.image_tag }}
    outputs:
      image: ${{ inputs.app_name }}:${{ inputs.image_tag }}

  - name: Deploy to Environment
    run: |
      kubectl apply -f k8s/${{ inputs.environment }}.yaml
      kubectl rollout status deployment/${{ inputs.app_name }}-${{ inputs.environment }}
    depends_on:
      - Build Image

Then invoke it in another workflow:

jobs:
  deploy:
    uses: ./.github/workflows/deploy-template.yml
    with:
      app_name: app
      image_tag: v1.0.0
      environment: production

Key patterns:

Sharp edges:

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.