Spot Instance Engineering: Architecting for Interruption

Leverage spot and preemptible instances for 60-90% cost savings by designing workloads that handle interruption gracefully. Covers workload selection, interruption handling, diversification strategies, and operational patterns for reliable spot-based infrastructure.

Spot or preemptible compute offers dramatically lower hourly rates, but it comes with a fundamental contract: the host can reclaim the instance with short notice, often 30 seconds to a few minutes. Engineering for this reality means designing workloads that treat every compute slot as potentially ephemeral. This page is about the patterns, tools, and failure modes that let you harvest cost savings without burning production uptime. It matters whenever you run batch jobs, stateless services, or any workload that can tolerate a restart, provided state is preserved and resumption is idempotent.

Detect the signal early

The host typically announces impending termination before the SIGTERM is delivered. In many environments this appears as a metadata entry, a file at a well‑known path, or a change in the power management state. The exact mechanism varies by provider, but the pattern is consistent: a signal arrives, a grace period begins, and if the process does not yield, the host force‑kills the VM.

A portable way to observe the signal is to poll a host‑provided endpoint or watch for the SIGTERM that the OS injects. The following curl request illustrates probing a metadata endpoint that many clouds expose; replace the path with the pattern your platform provides:

curl -s http://169.254.169.254/latest/meta-data/spot/termination-time 2>/dev/null || echo "no spot signal endpoint"

If a metadata endpoint is not available, you can rely on the kernel delivering SIGTERM after the configured grace period. A bash script can trap that signal:

#!/usr/bin/env bash
cleanup() {
  echo "[$(date)] received termination signal, performing cleanup" >> /var/log/spot-cleanup.log
  # enqueue work, flush buffers, etc.
}
trap cleanup SIGTERM

# long‑running work loop
while true; do
  sleep 60
  # application logic
done

In containerized workloads the same principle applies: the runtime forwards the host‑generated SIGTERM to the main process. Verifying that your process reacts within the grace period is the first line of defense.

Drain workloads gracefully

When the signal is caught, the next priority is to stop accepting new work and finish in‑flight units. In Kubernetes, the kubelet handles this during node shutdown, but the behavior depends heavily on pod configuration.

A pod that should respect the termination clock includes a terminationGracePeriodSeconds field and a preStop hook:

apiVersion: v1
kind: Pod
metadata:
  name: example-drainable
spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: app
    image: myapp:latest
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "echo draining >&2; sleep 5"]
    ports: [...]

Running kubectl drain with explicit flags ensures that pods respect the grace period and that daemonsets or critical-system pods are excluded:

kubectl drain node-1234 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=90s \
  --grace-period=60s

The --timeout flag caps how long kubectl will wait before force‑deleting pods that do not exit. If a pod exceeds the grace period, kubelet sends SIGKILL, which bypasses application‑level cleanup. Always size the grace period to your longest reasonable request‑completion time, plus a buffer for network propagation.

For non‑Kubernetes environments, an equivalent pattern is to signal the supervisor (systemd, supervisord, or a custom watchdog) to stop spawning new workers while allowing existing workers to finish their current unit of work. A systemd service can be written to accept SIGTERM and run a ExecStop script that mirrors the application‑level cleanup:

[Service]
ExecStop=/usr/local/bin/spot-stop.sh
Restart=on-failure

Preserve in‑flight state

Graceful shutdown is useless if the work being done is lost. The pattern is to checkpoint or flush state before the process exits. The exact mechanism depends on your stack, but the goal is always the same: durable storage before SIGTERM completes.

For database‑backed services, a common approach is to issue a FLUSH TABLES WITH READ LOCK or a WAL‑safe shutdown, followed by a controlled exit. A real pg_dump invocation triggered from a termination handler illustrates the idea:

import os, signal, subprocess, time

DB_URL = os.getenv("DATABASE_URL")

def handle_termin signum, frame):
    # flush pending transactions
    subprocess.run(["pg_dump", DB_URL, f">backup_{int(time.time())}.sql"], check=False)
    # exit cleanly after dump completes
    os._exit(0)

signal.signal(signal.SIGTERM, handle_termin)

# block until signal arrives
signal.pause()

If your service writes to an object store, flush buffers to the local filesystem and then aws s3 cp (or the equivalent for your provider) before exiting. The key is that the write to durable storage is the *last* synchronous operation before exit; anything after is non‑blocking housekeeping.

For pure compute workloads (e.g., image rendering, CI jobs), write a “checkpoint” file to a persistent volume or attached disk before the grace period expires. The file can contain the last completed task ID, partial results, or a resume token. On restart, the job reads this file and continues from the recorded point.

Reroute and resume on fresh capacity

Once state is durably persisted, the workload can be rescheduled onto a new instance. In Kubernetes, this is usually automatic: after a node is drained and terminated, the control plane schedules the pods onto a new node. The critical invariant is that the pod’s restartPolicy is set to Always (the default) and that the container image or code reads the checkpoint on startup.

If you manage workloads outside of k8s, the resume logic is explicit. A typical loop might look like:

#!/usr/bin/env bash
while true; do
  # submit or launch the job
  ./run-job.sh

  # observe exit code
  rc=$?
  if [[ $rc -eq 137 ]]; then
    # 137 = SIGKILL, likely host reclamation without grace
    echo "killed by host, waiting before retry"
    sleep 30
  elif [[ $rc -ne 0 ]]; then
    echo "job failed with code $rc, backing off"
    sleep $((rc * 5))
  else
    # normal completion, short pause before next iteration
    sleep 10
  fi
done

The loop detects termination (often via exit code 137 or a watchdog timeout) and re‑queues the work. Because state was checkpointed, the re‑run is idempotent.

Verify and test the flow

Never assume the shutdown‑to‑resume pipeline works until you’ve broken it in a controlled setting. A standard test is to simulate the signal and watch the full path:

  1. Start a long‑running process that writes a checkpoint file on each iteration.
  2. Send SIGTERM and time how long cleanup takes.
  3. Verify the checkpoint file exists and is valid after the process exits.
  4. Restart the process and confirm it reads the checkpoint and resumes from the saved point.

You can simulate the signal with:

PID=$(pgrep -f "my‑app")
kill -SIGTERM "$PID"

Watch dmesg, the process log, and the checkpoint file location. If the file is missing, adjust the handler to flush earlier or increase the grace period.

In Kubernetes, you can force a node drain and watch the pod lifecycle:

kubectl patch node node-1234 -p '{"spec": {"unschedulable": true}}'
kubectl drain node-1234 --ignore-daemonsets --delete-emptydir-data --timeout=60s
kubectl get events --field-selector involvedObject.name=node-1234 --sort-by=.metadata.creationTimestamp

Observe preStop hook execution, container state transitions, and the moment the pod is recreated on a new node.

Sharp edges you only learn in production

TL;DR checklist

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.