Automation Migration Patterns

Automation Migration Patterns

Automation migration patterns capture the repeatable, tool-agnostic strategies for moving automation assets from one execution model, environment, or governance regime to another. They matter whenever a team outgrows its current scaffolding—whether that's a batch of shell scripts wedged into cron, a homegrown pipeline that can't express dependencies, or a set of ad‑hoc playbooks that need to live inside a governed inventory. This page is not a taxonomy of tools; it is a goal‑oriented guide: if you are trying to restructure, refactor, or rehost existing automation without losing state, semantics, or operational continuity, the patterns below describe what to do, what to watch for, and how to verify the move.

1 Migrate Cron‑Driven Scripts to Declarative Pipeline Runners

Reader goal: Replace a crontab of shell scripts with a structured, version‑controlled execution model (Makefile, simple CI matrix, or workflow definition) while preserving run semantics.

1.1 Export existing cron entries to a versioned manifest

crontab -l > automation_manifest.txt

This dumps current cron entries into a file tracked in Git. The manifest format is free‑form but should adopt a consistent layout (e.g., TIME CMD ARGS) so downstream tooling can parse it reliably.

1.2 Define a declarative entry point (Makefile example)

.PHONY: run-nightly
run-nightly:
	@./scripts/nightly-job.sh

Exact command to replace a cron line: locate the original 0 2 * * * /path/to/script.sh entry and substitute 0 2 * * * make run-nightly. The make utility reads the Makefile from the repo root, ensuring the same binary and environment are used every time.

1.3 Validate the transition before disabling the old crontab

make run-nightly   # verify exit code, output, and side‑effects
crontab -r         # only after the Makefile target runs cleanly

Many teams forget to verify the old crontab is disabled before the new pipeline fires, leading to duplicate executions and unclear ownership of results.

Sharp edges

• PATH leakage: Cron invokes scripts with a minimal default PATH. make inherits that environment, so a script that works from an interactive shell may fail with command not found. Fix: use full paths in the Makefile recipe (/usr/local/bin/foo) or export PATH at the top of the recipe.
• Shebang and interpreter version: If the script uses #!/usr/bin/env python3, the make runner on some CI hosts may lack python3 in the base path. Pin the interpreter explicitly (#!/usr/bin/python3.11) or invoke via make run-nightly: python3 ./scripts/nightly-job.sh.
• Signal handling: Cron sends SIGTERM on system shutdown; make jobs are killed after the step times out. Scripts that trap SIGTERM or rely on cleanup on exit may behave differently. Add trap 'cleanup' EXIT to scripts being migrated, or restructure them as idempotent units that can be safely re‑run.

2 Split a Monolithic Automation Script into Modular, Reusable Units

Reader goal: Break a single, large script that orchestrates multiple services into independent components that can be versioned, tested, and composed without copy‑paste duplication.

2.1 Establish a component directory layout

automation/
├── components/
│   ├── db-backup/
│   │   ├── entrypoint.sh
│   │   └── params.json
│   └── api-health/
│       └── check.sh
└── compose/
    └── pipeline.yml

No exotic config keys here—just a directory structure that any CI system or docker compose invocation can discover.

2.2 Compose modular units with a top‑level entrypoint

# automation/compose/pipeline.yml
services:
  db-backup:
    build: ./components/db-backup
    entrypoint: ["/entrypoint.sh"]
  api-health:
    build: ./components/api-health
    entrypoint: ["/check.sh"]

Exact flag: build: points to a Dockerfile context; if the runner is not Docker, replace with script: or role: equivalents—the pattern of referencing sub‑components remains.

2.3 Wire composition to a single trigger point

docker-compose -f automation/compose/pipeline.yml up --build

Or, in a shell‑only environment:

for component in components/*; do
  (cd "$component" && ./run-tests.sh)
done

Sharp edges

• Import cycles: If component A sources a function from component B, and B later calls A's entrypoint, you get a loop that only surfaces at runtime. Resolve by extracting shared logic into a third “lib” component with no outward dependencies.
• State leakage: Scripts often rely on a shared working directory or temp files at /tmp/. When split into containers or separate CI jobs, /tmp/ is isolated. Migrate explicit paths to volume mounts or explicit artifact passing (docker cp, actions/upload-artifact).
• Version skew: Each component may be pinned to different base images or library versions. Pin versions in a top‑level docker-compose.yml using image: tags, and run docker compose pull before upgrades to catch mismatches early.

3 Transition Scheduling Runners (Cron → systemd Timer → CI Scheduled Jobs)

Reader goal: Move automation from one time‑based trigger to another, typically during infrastructure modernisation or when migrating from a bare‑metal host to a containerised environment.

3.1 Convert a crontab entry to a systemd timer unit

Write the following to /etc/systemd/system/automation.timer:

[Unit]
Description=Run nightly automation after system boot

[Timer]
OnBootSec=5min
OnUnitActiveSec=24h
Persistent=true

[Install]
WantedBy=multi-user.target

Exact command to enable:

systemctl enable --now automation.timer

To keep the original cron as a reference, dump it first: crontab -l > /tmp/old-cron.txt.

3.2 Convert the same automation to a CI scheduled pipeline (generic pattern)

The cron expression syntax is identical across schedulers, but timezone handling differs. A generic pipeline entry that works in most systems:

# .pipeline/schedule.yml  (syntax varies by runner; this is the conceptual shape)
on:
  schedule:
    - cron: '0 2 * * *'
jobs:
  automation:
    runs-on: ubuntu-latest
    steps:
      - run: |
          ./scripts/nightly-job.sh
        env:
          TZ: UTC

Exact flag: cron: under on: schedule:. The expression format (minute hour day month weekday) is POSIX‑standard.

3.3 Validate the transition with a one‑off run

systemctl start automation.timer
journalctl -u automation.timer --no-pager -n 30

If the job didn't fire, check Timers.active with systemctl list-timers --all.

Sharp edges

• Overlapping executions: systemd timers with Persistent=true and a long‑running job can accumulate missed runs if the host was off. The timer fires OnBootSec after boot, potentially launching multiple instances. Use JobMode=replace (systemd 254+) or implement a lock file inside the automation script: flock -n /var/lock/automation-lock ./run.sh.
• Environment leakage: systemd timers inherit the service's environment, which may lack variables set in user crontab (PATH, HOME). Define EnvironmentFile=/etc/default/automation or export variables explicitly in the timer's [Unit] section via Environment=.
• CI timezone mismatch: As shown above, CI schedulers often default to UTC. If the automation depends on local time (e.g., “run after the business day ends in New York”), annotate the cron expression with the target timezone or set TZ in the job environment. Forgetting this is the most common post‑migration bug report.

4 Port Automation Across Environments (dev → staging → prod)

Reader goal: Move an automation artifact—script, container, or pipeline definition—through successive environments while keeping configuration drift under control.

4.1 Use environment substitution to avoid hard‑coded paths

Exact technique: envsubst with a .env file.

# .env.prod
DB_HOST=prod-db.internal
API_TOKEN=${CI_PROD_TOKEN}

# render the script at deploy time
envsubst < scripts/nightly-job.sh.template > scripts/nightly-job.sh

Exact code in the template (nightly-job.sh.template):

#!/usr/bin/env bash
set -euo pipefail
DB="${DB_HOST}"
TOKEN="${API_TOKEN}"
curl -s -H "Authorization: Bearer $TOKEN" "$DB/metrics"

4.2 Promote a Docker image through registries

Exact commands:

# Build and tag
docker build -t my-automation:dev .
docker tag my-automation:dev registry.example.com/my-automation:staging
docker push registry.example.com/my-automation:staging
# Promote to prod
docker tag my-automation:dev registry.example.com/my-automation:prod
docker push registry.example.com/my-automation:prod

No vendor‑specific registry API is invoked; the pattern is tag → push → deploy.

4.3 Deploy with kubectl, respecting context

Command:

kubectl apply -f deployment.yaml -n production

Before running, switch context:

kubectl config use-context prod-cluster

Exact flag: -n namespaces the resource; omitting it defaults to default, which is a frequent source of “deployed to wrong namespace” errors.

Sharp edges

• Secret propagation lag: When kubectl apply introduces a new Secret, the kubelet caches it for up to a minute (default secret-minimum-duration). A pod restart may not see the new value immediately. Work around by kubectl delete pod -l app=automation after applying the secret, or use kubectl rollout restart.
• Resource quota drift: Each environment may have different LimitRange defaults (CPU limits, memory requests). A script that works in dev may be evicted in prod because the pod exceeds the default request. Before migrating, run kubectl describe limitrange in the target namespace and adjust the pod spec (resources: requests: cpu: "100m" memory: "128Mi").
• Ingress/route changes: If the automation exposes a local port or service endpoint, the network base URL may change across environments. Parameterise the base URL via the substitution mechanism in 4.1 rather than hard‑coding http://localhost:8080.

5 Roll Back a Migration Safely

Reader goal: Undo an automation migration without leaving the system in an inconsistent state, minimizing downtime and data loss.

5.1 Feature‑flagged dual‑run pattern

During migration, run the old and new automation side‑by‑side behind a feature flag. The pattern is universal regardless of runner:

• Old path: if [ "$FLAG" = "legacy" ]; then ./scripts/old-job.sh; fi
• New path: if [ "$FLAG" = "new" ]; then ./scripts/new-job.sh; fi
• Deploy the flag as an environment variable or CI secret.

5.2 One‑command rollback for containerised automation

Command:

docker tag registry.example.com/my-automation:prod registry.example.com/my-automation:prod-backup-$(date +%Y%m%d)
docker pull registry.example.com/my-automation:prev-tag
docker compose up --force-recreate

Or, in Kubernetes:

kubectl rollout undo deployment/automation

Exact flag: undo reverts the deployment to the previous successful revision.

5.3 Database or state migration rollback

If the automation migrates database schema or state files, never rely on “just revert the script”. Maintain an explicit migration log:

# In the repo, under docs/migrations/
001-initial-schema.sql
002-add-index-on-requests.sql
003-rename-column-to-owner.sql

Rollback command (psql example):

psql -d mydb -f docs/migrations/002-add-index-on-requests.sql --command "DROP INDEX idx_requests_owner;"

The exact SQL syntax varies by dialect, but the pattern—recorded, versioned, executable migration files—is the safeguard.

Sharp edges

• In‑flight task loss: If the old automation was processing a queue (e.g., pulling support tickets) and you cut it over without draining, tickets assigned to the old worker may be stranded. Always implement a graceful shutdown sequence: pkill -f old-automation; sleep 5; pkill -9 -f old-automation or, in CI, a pre-exit hook that drains the queue.
• Idempotency gaps: The new automation may not be idempotent where the old one was. Running the new job twice can create duplicate resources (rows, firewall rules, etc.). Write idempotency checks (if not exists in SQL, create or replace in configs) before declaring the migration complete.
• Config drift after rollback: If the new automation wrote configuration files (e.g., updated /etc/auto-settings.yml), reverting the deployment does not restore those files. Keep a backup step in the rollback script: cp /etc/auto-settings.yml /etc/auto-settings.yml.bak-migration before the new job runs, and restore from the backup on undo.

Closing note

These patterns are not tied to any one scheduler, container platform, or configuration language. They describe the structural moves—export, substitute, compose, trigger, revert—that apply whether you are moving from cron to systemd, from shell scripts to CI pipelines, or from a single‑repo monolith to a multi‑component workflow. Apply the concrete commands verbatim, adapt the flags to your runner’s syntax, and always verify the sharp edges in a staging environment before the first production run.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.