Cryptographic Key Management

Production engineering guide for cryptographic key management covering patterns, implementation strategies, and operational best practices.

Cryptographic Key Management is the practice of generating, storing, distributing, rotating, protecting, and revoking cryptographic keys used to secure data at rest, in transit, and in use. It matters when systems must maintain confidentiality, integrity, and authenticity across multiple services, environments, and lifecycles—especially in distributed systems, regulated industries, and high-availability platforms.

Key Lifecycle Management

Creating and managing keys across their entire lifecycle is foundational. Keys are not static assets; they evolve from creation to retirement. The most common pattern begins with key generation, followed by storage, usage, rotation, and eventual revocation.

To generate a new AES-256 key, use openssl with a standard command:

openssl rand -out key.aes256 -hex 32

This produces a 32-byte binary key, encoded in hex. For a key pair used in RSA-4096 signing, generate both private and public components:

openssl genrsa -out private.key 4096
openssl rsa -pubout -in private.key -out public.key

The private key is typically encrypted with a passphrase using AES-256-CBC:

openssl rsa -aes256 -in private.key -out private.encrypted.key

When a service needs to load a key, it must read the key from storage and make it available in memory. A typical configuration file for a web application using a JSON-based config format might include:

{
  "crypto": {
    "keys": [
      {
        "id": "db-encryption",
        "algorithm": "aes-256-gcm",
        "mode": "file",
        "path": "/etc/secrets/db.key",
        "rotation": {
          "interval": "7d",
          "threshold": 80,
          "strategy": "rolling"
        }
      },
      {
        "id": "jwt-signing",
        "algorithm": "rsa-4096",
        "mode": "vault",
        "endpoint": "https://vault.example.com/v1/secret/api/jwt",
        "auth": {
          "method": "approle",
          "role_id": "a1b2c3d4",
          "secret_id": "s1t2u3v4"
        }
      }
    ]
  }
}

The mode field determines how the key is retrieved: file, vault, aws-kms, gcp-kms, azure-keyvault, or database. The rotation section defines how and when keys are rotated. The strategy field can be rolling, switch, or active-passive, each with distinct behaviors during key transitions.

Key Storage and Protection

Keys must be protected from unauthorized access, especially at rest. The weakest link is often the key storage mechanism itself. Keys stored in plain text files are vulnerable to compromise, while keys in hardware security modules (HSMs) or cloud key management services offer stronger protection.

A key stored in a file is typically encrypted with a master key. The master key itself is stored separately—often in a key store or a configuration management system. The following kustomize overlay config demonstrates how a Kubernetes cluster uses a secret to hold a master key:


resources:
  - base
patchesStrategicMerge:
  - patches/keys.yaml
configMapGenerator:
  - name: master-keys
    literals:
      - MASTER_KEY=68a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2

This master key is used to encrypt all application keys. For a rotating key, the system must track which key is currently active. A common schema in a relational database stores key metadata:

CREATE TABLE crypto_keys (
  id UUID PRIMARY KEY,
  name TEXT NOT NULL,
  algorithm TEXT NOT NULL,
  version INTEGER NOT NULL,
  active BOOLEAN DEFAULT TRUE,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  expires_at TIMESTAMPTZ,
  rotation_policy JSONB,
  encrypted_key BYTEA,
  key_metadata JSONB
);

The rotation_policy column holds a JSON object describing how and when the key should be rotated. When a key is rotated, the system generates a new version and marks it as active. The old key remains available for backward compatibility.

Key Rotation

Key rotation ensures that long-lived keys are periodically replaced, reducing exposure to attacks and enabling key lifecycle policies. A rotation can be time-based, event-driven, or both.

For time-based rotation, a cron job runs daily to check if any key has exceeded its rotation interval:

# /etc/cron.daily/rotate-keys
#!/bin/bash
KEY_DIR="/etc/secrets"
LOG_FILE="/var/log/key-rotation.log"

for key in $(find "$KEY_DIR" -name "*.key" -type f); do
  key_name=$(basename "$key" .key)
  key_path="$KEY_DIR/$key_name"

  # Read rotation policy from config
  rotation_policy=$(jq -r ".crypto.keys[] | select(.id == \"$key_name\") | .rotation" /etc/app/config.json)

  interval=$(echo "$rotation_policy" | jq -r '.interval')
  threshold=$(echo "$rotation_policy" | jq -r '.threshold')

  now=$(date -Iseconds)
  created=$(stat -c %Y "$key_path" | date -Iseconds -d @)

  # Calculate days since creation
  days_since=$(date -d "$now" +%s -u | awk -v c=$(date -d "$created" +%s -u) '{print int(($c - $1) / 86400)}')

  if [ "$days_since" -ge "$interval" ]; then
    # Generate new key
    openssl rand -out "$key_path.new" -hex 32

    # Encrypt with current master key
    openssl enc -aes-256-cbc -in "$key_path.new" -out "$key_path.new.encrypted" -pass pass:$(cat /etc/secrets/master.key)

    # Update database
    psql -c "
      INSERT INTO crypto_keys (id, name, algorithm, version, active, created_at, encrypted_key)
      VALUES (
        gen_random_uuid(),
        '$key_name',
        'aes-256-gcm',
        (SELECT COALESCE(MAX(version), 0) + 1 FROM crypto_keys WHERE name = '$key_name'),
        true,
        '$now',
        pg_read_binary_file('$key_path.new.encrypted')
      )
    "

    # Mark old key as inactive
    psql -c "
      UPDATE crypto_keys
      SET active = false, expires_at = '$now'
      WHERE name = '$key_name' AND active = true
    "

    # Move new key into place
    mv "$key_path.new.encrypted" "$key_path"
    mv "$key_path.new" "$key_path.new.unencrypted"

    echo "$(date): Rotated $key_name (interval=$interval, threshold=$threshold)" >> "$LOG_FILE"
  fi
done

When a key reaches the threshold percentage of its lifetime, the system signals a warning. A threshold of 80 means that a key is rotated when it is 80% through its interval. The strategy field determines how the transition occurs:

Key Distribution and Access

Distributed systems require a mechanism to distribute keys efficiently and securely. A common pattern uses a key server that exposes a REST API for key retrieval.

A service retrieves a key via HTTP:

GET /v1/keys/{key-id} HTTP/1.1
Host: keys.example.com
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c
Accept: application/json

The response includes the key data and metadata:

{
  "id": "jwt-signing",
  "version": 3,
  "algorithm": "rsa-4096",
  "active": true,
  "created_at": "2023-08-15T08:00:00Z",
  "expires_at": "2024-08-15T08:00:00Z",
  "key_data": {
    "public": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA...",
    "private": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC...",
    "metadata": {
      "format": "pem",
      "encryption": "aes-256-gcm",
      "key_id": "rsa-4096-jwt-3"
    }
  }
}

The key_data field may contain the key in multiple formats. The metadata field describes how the key was generated and stored, which is useful for debugging and auditing.

When the key server is unavailable, the service must fall back to a local cache. A common failure mode occurs when the cache is stale and the key server is unreachable. The service returns a 503 error:

HTTP/1.1 503 Service Unavailable
Content-Type: application/json

{
  "error": "key_server_unavailable",
  "message": "Failed to retrieve key from key server",
  "retry_after": 30,
  "cached_key_version": 2,
  "last_updated": "2023-08-15T07:55:00Z"
}

The client can use the Retry-After header to schedule a retry and fall back to the cached key.

Key Revocation and Expiry

Revocation is the process of deactivating a key before its natural expiration. It is critical during security incidents, key compromises, or policy changes.

A revocation request is sent via a PATCH request:

PATCH /v1/keys/{key-id}/revocation HTTP/1.1
Host: keys.example.com
Authorization: Bearer <token>
Content-Type: application/json

{
  "reason": "compromised",
  "effective_at": "2023-08-15T08:15:00Z"
}

The server updates the crypto_keys table:

UPDATE crypto_keys
SET active = false, revoked_at = '2023-08-15T08:15:00Z', revocation_reason = 'compromised'
WHERE id = 'jwt-signing';

The system can now reject any signature or encryption using a revoked key. A common failure is the mismatch between the effective_at time and the actual time the revocation was applied. A key might be marked as revoked but still used for data written during the gap between the request and the database update.

Common Failure Modes

Several patterns emerge when key management fails in production:

Summary

Cryptographic Key Management is not a one-off task. It is a continuous process involving generation, storage, distribution, rotation, and revocation. The success of a system’s security hinges on the consistency and reliability of key management. Every failure—be it a missing file, a stale cache, a misconfigured policy, or a misaligned clock—is a failure in key management. The tools are standard: openssl, curl, psql, systemctl, jq, and kustomize. The patterns are universal, not tied to a product, and the details—config keys, error messages, HTTP headers, and database schemas—are the concrete reality of engineering practice.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.