Storage Cost Optimization

Production engineering guide for storage cost optimization covering patterns, implementation strategies, and operational best practices.

Storage cost optimization is the process of reducing the total cost of storing data across cloud and on-premises systems by aligning storage tiers, access patterns, retention policies, and data lifecycle rules with actual usage and access frequency. It matters when data grows faster than budget, when cold data sits idle in expensive storage, or when backup and replication costs outpace innovation. This page details how to implement and maintain a storage cost optimization practice across infrastructure, applications, and data platforms.

Evaluate Current Storage Usage and Access Patterns

Begin by collecting metadata on storage consumption and access frequency. Use standard tools to interrogate file systems, object stores, and databases. For object storage, list all buckets and their total size, last access time, and storage class.

aws s3api list-buckets --query 'Buckets[].{Name: Name, Size: Size, LastModified: CreationDate}' --output table

For a single bucket, retrieve object-level metadata, including last modified and last accessed timestamps.

aws s3api list-objects-v2 \
  --bucket my-data-lake \
  --prefix raw/ \
  --query 'Contents[].{Key: Key, Size: Size, LastModified: LastModified, LastAccessed: LastAccessed}' \
  --output json

To determine access patterns, extract access logs from the S3 bucket and correlate them with object metadata.


aws s3api get-object \
  --bucket my-data-lake-logs \
  --key access-log-2024-03-15.csv \
  --query 'Body | read() | split("\n") | [1:] | map(. | split(","))' \
  --output text

The LastAccessed field from S3 is populated by CloudFront or by client applications using the GET operation with x-amz-expected-seconds header. If this field is missing, access patterns are underestimated.

Failure Mode: Last-Accessed Timestamps Not Updated

When an object is accessed via a client library (e.g., boto3 or AWS SDK), the LastAccessed timestamp is not updated unless explicitly set using the PutObject API with x-amz-expected-seconds or x-amz-storage-class headers. Without this, cold data appears accessed even when it has not been read.

Define and Apply Tiering Policies

Assign data to storage tiers based on access frequency and cost. Use a tiering policy that moves data between tiers automatically. Define tiers as follows:

Use a policy format that defines rules based on age and access frequency.

{
  "tiering": [
    {
      "name": "hot",
      "condition": {
        "min_age_days": 0,
        "max_age_days": 30,
        "min_accesses": 10
      },
      "target_storage_class": "STANDARD"
    },
    {
      "name": "warm",
      "condition": {
        "min_age_days": 31,
        "max_age_days": 180,
        "min_accesses": 3
      },
      "target_storage_class": "STANDARD_IA"
    },
    {
      "name": "cold",
      "condition": {
        "min_age_days": 181,
        "max_age_days": 365,
        "min_accesses": 1
      },
      "target_storage_class": "GLACIER"
    },
    {
      "name": "archive",
      "condition": {
        "min_age_days": 366,
        "min_accesses": 0
      },
      "target_storage_class": "GLACIER_DEEP_ARCHIVE"
    }
  ]
}

Apply this policy using a script that iterates over all objects and updates their storage class.

#!/bin/bash
# apply-tiering-policy.sh
set -euo pipefail

BUCKET="my-data-lake"
POLICY="tiering-policy.json"

while IFS= read -r -d '' key; do
  # Get object metadata
  meta=$(aws s3api get-object-attributes \
    --bucket "$BUCKET" \
    --key "$key" \
    --attributes "LastModified,StorageClass,ETag" \
    --output json)

  # Parse metadata
  last_modified=$(echo "$meta" | jq -r '.LastModified')
  age_days=$(( ( $(date -u +%s) - $(date -u -d "$last_modified" +%s) ) / 86400 ))

  # Determine access count via external logs (e.g., CloudWatch Logs)
  access_count=$(aws logs filter-log-events \
    --log-group-name "/aws/s3/$BUCKET" \
    --start-time $(( $(date -u +%s) - 30*86400 )) \
    --filter-pattern "S3 GET $key" \
    --query 'events | length' \
    --output text)

  # Match against policy
  for rule in $(jq -c '.tiering[]' "$POLICY"); do
    name=$(echo "$rule" | jq -r '.name')
    min_age=$(echo "$rule" | jq -r '.condition.min_age_days')
    max_age=$(echo "$rule" | jq -r '.condition.max_age_days')
    min_accesses=$(echo "$rule" | jq -r '.condition.min_accesses')
    target_class=$(echo "$rule" | jq -r '.target_storage_class')

    if [[ $age_days -ge $min_age ]] && 
       [[ $age_days -lt $max_age ]] &&
       [[ $access_count -ge $min_accesses ]]; then
      aws s3api put-object-tagging \
        --bucket "$BUCKET" \
        --key "$key" \
        --tagging "TagSet=[{Key=StorageTier,Value=$name}]"

      aws s3api copy-object \
        --bucket "$BUCKET" \
        --copy-source "$BUCKET/$key" \
        --key "$key" \
        --storage-class "$target_class" \
        --metadata-directive "REPLACE"
      break
    fi
  done
done < <(aws s3api list-objects-v2 --bucket "$BUCKET" --query 'Contents[].Key' --output text --pagination-token "")

Failure Mode: Storage Class Mismatch on Copy

When copying an object to a new storage class, the StorageClass field in the object metadata is not updated if CopyObject is not used with MetadataDirective=REPLACE. Without this, the object retains its original class, and the tiering policy relies on metadata that is out of sync.

Optimize Data Retention and Lifecycle

Define retention policies that delete or archive data after a defined period. Use lifecycle rules to automatically transition and expire objects. A retention policy is a list of rules that define when to transition and delete.

{
  "lifecycle_rules": [
    {
      "name": "raw-data-1-year",
      "prefix": "raw/",
      "transition": [
        {
          "days": 90,
          "storage_class": "STANDARD_IA"
        },
        {
          "days": 180,
          "storage_class": "GLACIER"
        }
      ],
      "expiration": {
        "days": 365,
        "expired_object_delete_marker": true
      }
    },
    {
      "name": "processed-data-5-year",
      "prefix": "processed/",
      "transition": [
        {
          "days": 365,
          "storage_class": "GLACIER"
        }
      ],
      "expiration": {
        "days": 1825,
        "expired_object_delete_marker": true
      }
    }
  ]
}

Apply these rules via the S3 lifecycle configuration.

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-data-lake \
  --lifecycle-configuration file://lifecycle-rules.json

Failure Mode: Expired Objects Not Deleted

When using ExpiredObjectDeleteMarker: true, the Expiration rule creates a delete marker, but the actual deletion of the object is not immediate. The delete marker remains until a DeleteMarker is explicitly processed, usually via a nightly job. Without this, the object is marked as expired but still present and accessible.

Reduce Storage Footprint through Data Compression and Deduplication

Apply compression and deduplication at the storage layer. Use standard tools to compress data before upload and enable compression on the storage system.

For Parquet files, apply compression using snappy or zstd.

# Compress a Parquet file with zstd
parquet-tools compress \
  --compression zstd \
  --level 10 \
  input.parquet \
  output.parquet

For raw files, use zstd or gzip with --fast or --best.

zstd --best --rm input.log
gzip -9 --keep input.csv

Enable deduplication in the storage system. For S3, use Etag or Content-MD5 to detect duplicate objects. Use a script to find duplicates.

#!/bin/bash
# find-duplicates.sh
set -euo pipefail

BUCKET="my-data-lake"
TEMP_DIR=$(mktemp -d)

# Generate ETag for each object
aws s3api list-objects-v2 \
  --bucket "$BUCKET" \
  --query 'Contents[].Key' \
  --output text \
  | while read key; do
    etag=$(aws s3api get-object-attributes \
      --bucket "$BUCKET" \
      --key "$key" \
      --attributes ETag \
      --output text)
    echo "$etag $key"
  done > "$TEMP_DIR/etags.txt"

# Group by ETag
sort "$TEMP_DIR/etags.txt" | uniq -d > "$TEMP_DIR/duplicate-etags.txt"

# Output duplicate keys
while read etag; do
  grep "^$etag" "$TEMP_DIR/etags.txt" | cut -d' ' -f2
done < "$TEMP_DIR/duplicate-etags.txt"

Failure Mode: ETag Mismatch Due to Chunked Uploads

When uploading large files using multipart upload, the ETag is not the MD5 of the entire object but the concatenation of MD5s of each part. This leads to false positives in deduplication. Use Content-MD5 or compute the full MD5 to verify exact duplicates.

Automate Cost Reporting and Alerting

Generate cost reports by querying storage usage and pricing per tier. Use a script to compute total cost per storage class.

#!/bin/bash
# storage-cost-report.sh
set -euo pipefail

BUCKET="my-data-lake"
COST_PER_GB = "{
  \"STANDARD\": 0.023,
  \"STANDARD_IA\": 0.0125,
  \"GLACIER\": 0.004,
  \"GLACIER_DEEP_ARCHIVE\": 0.00099
}"

# Get total size per storage class
aws s3api list-objects-v2 \
  --bucket "$BUCKET" \
  --query 'Contents[].{Size: Size, StorageClass: StorageClass}' \
  --output json \
  | jq -c 'group_by(.StorageClass) | map({class: .[0].StorageClass, total_size: map(.Size) | add})' \
  > $TEMP_DIR/sizes.json

# Compute total cost
jq -n \
  --argjson cost_map "$COST_PER_GB" \
  --argjson sizes "$(< $TEMP_DIR/sizes.json)" \
  '{
    total_cost: (
      $sizes | map(
        .class as $class |
        .total_size as $size |
        ($cost_map | .[$class]) * $size / 1024 / 1024 / 1024
      ) | add
    ),
    breakdown: $sizes | map(
      .class as $class |
      .total_size as $size |
      {
        storage_class: $class,
        size_gb: $size / 1024 / 1024 / 1024,
        cost_per_gb: ($cost_map | .[$class]),
        total_cost: ($cost_map | .[$class]) * $size / 1024 / 1024 / 1024
      }
    )
  }'

Send the report via email or to a monitoring system.

# Send report to Slack
curl -X POST -H "Content-Type: application/json" \
  -d @- \
  https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX \
  <<EOF
{
  "text": "Storage cost report for my-data-lake",
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Total cost: $1,243.21*"
      }
    },
    {
      "type": "section",
      "fields": [
        {
          "type": "mrkdwn",
          "text": "*STANDARD*\nSize: 2.1 TB\nCost: $48.30"
        },
        {
          "type": "mrkdwn",
          "text": "*STANDARD_IA*\nSize: 4.5 TB\nCost: $56.25"
        }
      ]
    }
  ]
}
EOF

Failure Mode: Cost Report Based on Incorrect Pricing

The cost per GB is often based on monthly pricing, but the report uses daily average usage. Without a BillingPeriod field or a StartTime in the GetBucketMetrics call, the report underestimates cost by up to 30% due to monthly billing cycles and tiered pricing.

Monitor and Iterate on Optimization

Set up a monitoring loop that evaluates storage cost optimization monthly. Use a set of metrics:

Use cron jobs or a lightweight scheduler to run the evaluation pipeline.

# crontab entry
0 2 * * * /opt/scripts/storage-cost-optimization.sh

Failure Mode: Over-Optimization in High-Access Periods

During peak load (e.g., end-of-month reporting), data is accessed frequently but not yet aged. The tiering policy moves data to warm or cold too early, increasing cost. Use a cold-window parameter in the policy to delay transitions during high-activity periods.

Common Pitfalls and Sharp Edges

Optimization is not a one-time task. It is an ongoing process that requires continuous evaluation, iteration, and alignment with application and business needs.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.