Spot Instance Strategies | The Garnet Wiki

Spot instance strategies are the disciplined approach to using interruptible compute resources to reduce cloud infrastructure costs, particularly for workloads that tolerate transient failures. They matter when a system must balance cost efficiency against availability, especially for batch processing, data pipelines, CI/CD jobs, and stateless services where node failures can be absorbed gracefully.

Selecting the Right Spot Instance Mix

To maximize value, teams must decide which workloads to run on spot instances and how many instances to use. The choice is not merely "use spot instances" but "which mix of instance types, durations, and pricing models."

The core decision is the instance type selection, based on the workload’s CPU, memory, and I/O profile. For example, a data processing job with heavy memory pressure may benefit from r5.4xlarge instances, while a video transcoding pipeline might prefer c5.8xlarge with high throughput and network bandwidth.

To orchestrate this, a typical deployment uses a launch template in a cloud provider’s EC2 service, with the following key configuration:

{
  "LaunchTemplateData": {
    "InstanceType": "r5.4xlarge",
    "ImageId": "ami-12345678",
    "KeyName": "prod-keypair",
    "SecurityGroups": [
      { "GroupId": "sg-87654321" }
    ],
    "BlockDeviceMappings": [
      {
        "DeviceName": "/dev/sda1",
        "Ebs": {
          "VolumeSize": 100,
          "VolumeType": "gp3",
          "DeleteOnTermination": true
        }
      }
    ],
    "UserData": "#!/bin/bash\napt update && apt install -y python3-pip\npip3 install pandas\n",
    "Monitoring": {
      "Enabled": true
    },
    "InstanceMarketOptions": {
      "MarketType": "spot",
      "SpotOptions": {
        "SpotInstanceType": "persistent",
        "SpotMaintenanceStrategies": {
          "MaxSpotInstanceCount": 50,
          "MinSpotInstanceCount": 10,
          "SpotInstanceRecovery": "true"
        }
      }
    }
  }
}

The SpotInstanceRecovery setting enables automatic recovery of spot instances when they are interrupted: the instance is launched again from the same launch template, with the same configuration, and the volume data preserved. This is critical for stateful jobs.

Managing Spot Instance Interruption

When a spot instance is interrupted, it receives a termination notice via the EC2 instance metadata service at http://169.254.169.254/latest/meta-data/spot/instance-action. The response is a JSON object:

{
  "instanceAction": "terminate",
  "instanceId": "i-12345678",
  "startTime": "2023-04-15T10:00:00Z",
  "reason": "price",
  "lifetime": 3600,
  "currentPrice": 0.1234
}

A consumer process must poll this endpoint every 30 seconds to detect the notice. The reason field indicates whether the interruption was due to price (spot price exceeded) or capacity (capacity shortage), and lifetime gives the expected time until termination.

To handle this, a service should register a termination handler script in the system’s systemd configuration:


[Unit]
Description=Spot Instance Termination Handler
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/spot-termination-handler.sh
Restart=always
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

The script itself polls metadata and triggers a graceful shutdown:

#!/bin/bash
# /usr/local/bin/spot-termination-handler.sh

set -euo pipefail

while true; do
  if curl -s -f http://169.254.169.254/latest/meta-data/spot/instance-action; then
    # Termination notice received
    echo "$(date): Spot instance interruption detected"
    # Signal application to stop accepting new work
    systemctl stop my-app.service
    # Wait for active jobs to complete
    sleep 60
    # Finalize state
    /opt/my-app/bin/backup-state.sh
    # Terminate instance
    shutdown -h now "Spot instance termination"
  fi
  sleep 30
done

This script ensures that the service can complete its current work, persist state, and shut down cleanly before the instance is terminated.

Scheduling and Preemptive Workload Placement

For workloads that are predictable, teams can pre-schedule spot instance usage to reduce the risk of interruptions during critical phases.

A common strategy is to use spot instance pools with dynamic capacity adjustments. This requires a scheduler that monitors spot prices and adjusts the number of instances accordingly.

A typical setup uses curl to query the EC2 Spot Instance Request API:

curl -X GET \
  "https://ec2.amazonaws.com/?Action=DescribeSpotInstanceRequests" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -H "X-Amz-Target: AmazonEC2V20160415.DescribeSpotInstanceRequests" \
  -H "X-Amz-Date: 20230415T120000Z" \
  -H "Authorization: AWS4-HMAC-SHA256 Credential=AKIAEXAMPLE/20230415/us-east-1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=abcdef1234567890" \
  --data-urlencode "SpotInstanceRequestIds.member.1=sir-12345678"

The response includes status updates for each request, including active, open, closed, and cancelled. A state machine monitors these and triggers actions:

When a request is active, the instance is running and can accept work.
When closed, the instance has been interrupted and is being terminated.
When cancelled, the request was explicitly cancelled, often due to budgeting or priority changes.

To automate this, a scheduler daemon runs every 15 minutes and checks current spot prices against a historical baseline:

#!/bin/bash
# /usr/local/bin/spot-scheduler.sh

set -euo pipefail

# Fetch current spot prices for r5.4xlarge in us-east-1
SPOT_PRICE=$(curl -s \
  "https://ec2.amazonaws.com/?Action=DescribeSpotPriceHistory" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -H "X-Amz-Target: AmazonEC2V20160415.DescribeSpotPriceHistory" \
  --data-urlencode "InstanceTypes.member.1=r5.4xlarge" \
  --data-urlencode "AvailabilityZone=us-east-1a" \
  --data-urlencode "StartTime=2023-04-15T10:00:00Z" \
  --data-urlencode "EndTime=2023-04-15T16:00:00Z" \
  | jq -r '.SpotPriceHistory[0].SpotPrice')

# Compare with 24-hour average
AVG_PRICE=$(psql -c "SELECT avg(spot_price) FROM spot_price_history WHERE date >= current_date - interval '1 day'" -t)

if (( $(echo "$SPOT_PRICE > $AVG_PRICE * 1.1" | bc -l) )); then
  # Price is high, scale down
  aws ec2 modify-spot-fleet-request --spot-fleet-request-id sf-12345678 --target-capacity 10
elif (( $(echo "$SPOT_PRICE < $AVG_PRICE * 0.9" | bc -l) )); then
  # Price is low, scale up
  aws ec2 modify-spot-fleet-request --spot-fleet-request-id sf-12345678 --target-capacity 30
fi

This script uses jq to parse JSON, psql to query a PostgreSQL database, and aws CLI with real AWS command line options and flags.

State Management and Recovery

Spot instances are inherently stateless, so managing state across interruptions is critical.

The best strategy is to externalize state to durable storage, such as Amazon S3, EFS, or a managed database.

For a batch job, the input and output data are stored in S3:

# Example: Submit a job to S3
aws s3 cp job-input.json s3://my-batch-jobs/input/$(date +%Y%m%d)/job-$(uuidgen).json

# Process job with spot instance
aws ec2 run-instances \
  --launch-template-name batch-job-template \
  --count 1 \
  --spot-fleet-request-id sf-12345678 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=JobId,Value=job-123456}]'

When the spot instance completes, it uploads output to a dedicated S3 prefix:

aws s3 cp /app/output/results.json s3://my-batch-jobs/output/2023/04/15/job-123456/

The job is now idempotent: it can be rerun from any spot instance, and the output is consistent.

For stateful services, such as a web application, the instance stores session data in Redis, and the database is hosted in Amazon RDS:

# In startup script
redis-cli -h redis-prod.internal --connect-timeout 10 \
  --eval "SCRIPT LOAD 'SET session:$(uuidgen):user $1' 1 'john'"

If an instance is interrupted, the session data remains in Redis, and the next instance picks up the session seamlessly.

Failure Modes and Silent Pitfalls

Several common failures silently degrade performance:

Missing termination notice: if the metadata endpoint is unavailable (e.g., due to network partition), the instance may miss the interruption notice entirely. This leads to aborted jobs and lost state.

Inconsistent instance types across spot fleets: a fleet may grow instances with m5.large and c5.xlarge types, but the application assumes a fixed memory footprint. This leads to underutilization or out-of-memory crashes.

Spot instance reboots not triggering a full startup: the UserData script runs only on first boot, but a spot instance can be reused across multiple lifetimes. If the startup script does not check for a previous state, it may reinitialize the environment every time, leading to redundant setup and increased boot time.

Incorrect spot price thresholds: if the scheduler uses a fixed threshold (e.g., 10% above average), it may overbuy during transient price spikes and underbuy during stable periods.

Missing instance metadata refresh: when an instance is interrupted, it may be launched on a different physical host. The instance-type metadata may not reflect the actual hardware, causing misconfiguration and performance drift.

Spot instance lifecycle confusion: a spot instance may be launched, running, interrupted, stopped, and terminated in a complex sequence. Teams often assume that "spot instance" means "always running", but the instance may be replaced multiple times, each with its own lifecycle.

Summary: The Spot Instance Strategy Pattern

A mature spot instance strategy is not a single command or configuration, but a coordinated system of:

Launch templates with consistent instance configurations
Termination handlers that react to metadata notices
Dynamic scaling based on spot price trends
Externalized state in durable storage
Idempotent workloads that tolerate interruptions

It is not enough to launch spot instances. It is essential to monitor their interruptions, recover from them, and optimize their usage over time. This pattern is general enough to apply across cloud providers and platforms, yet precise enough to guide engineering teams in building reliable, cost-efficient systems.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.