Cloud Secrets Management

Production engineering guide for cloud secrets management covering patterns, implementation strategies, and operational best practices.

Managing credentials, tokens, API keys, and other sensitive data across cloud environments requires a structured approach to prevent exposure in logs, configuration files, or source code repositories. This applies when deploying applications that access databases, external APIs, or cloud services where secrets must be available at runtime without being stored in plaintext.

Storing Secrets

Store secrets using key-value pairs with explicit versioning and metadata. Each secret is identified by a unique key within a namespace, typically a path like /prod/app/database-password or /staging/services/external-api-key.

To store a secret:

# Example using a standard HTTP API for secret storage
curl -X POST https://secret-store.example.com/v1/secrets \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "key": "/prod/app/database-password",
    "value": "s3cr3t_p@ssw0rd",
    "metadata": {
      "description": "Primary database password for production app",
      "owner": "backend-team"
    }
  }'

Store secrets in encrypted form, using a key management service or built-in encryption. Secrets are typically stored in an encrypted format, with the plaintext only available during runtime.

Retrieving Secrets

Retrieve secrets by their key, with optional versioning and access control. A secret must be fetched through an authenticated interface, using either a token or certificate-based authentication.

# Example retrieval
curl -X GET https://secret-store.example.com/v1/secrets/prod/app/database-password \
  -H "Authorization: Bearer $TOKEN"

Response format:

{
  "key": "/prod/app/database-password",
  "value": "encrypted_value_here",
  "version": 3,
  "created_at": "2024-01-15T10:00:00Z",
  "updated_at": "2024-01-16T14:30:00Z"
}

The value field contains the encrypted secret. The application must decrypt it using a key management interface or pre-shared encryption keys.

Rotating Secrets

Rotate secrets by creating a new version of a secret, then updating all applications to use the new version. Rotation is performed in two steps:

  1. Update the secret with a new value.
  2. Re-deploy or restart services to pick up the updated secret.
# Rotate a secret
curl -X PUT https://secret-store.example.com/v1/secrets/prod/app/database-password \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "value": "n3w_s3cr3t_p@ssw0rd",
    "metadata": {
      "rotation_reason": "monthly_rotation"
    }
  }'

Applications can fetch the latest version by specifying version=latest or omitting the version, which defaults to the most recent.

Access Control

Access to secrets is controlled through identity-based policies. Each secret has an access control list (ACL) that defines which identities can read or write it. ACLs are typically defined at the namespace level or per secret.

# Example ACL update
curl -X PUT https://secret-store.example.com/v1/secrets/prod/app/database-password/acl \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "readers": ["service-account-frontend", "service-account-backend"],
    "writers": ["team-backend"]
  }'

Access is denied with a 403 Forbidden response when a principal lacks permission to read or write a secret.

Secret Lifecycle Management

Secrets must be rotated regularly and deleted after use. The lifecycle includes:

# Delete a secret
curl -X DELETE https://secret-store.example.com/v1/secrets/prod/app/database-password \
  -H "Authorization: Bearer $TOKEN"

Deletion is immediate, but the system may keep a backup for a period (e.g., 30 days) before permanent removal.

Application Integration

Applications retrieve secrets at runtime using client libraries or direct HTTP calls. A typical integration fetches all required secrets during startup and caches them for the duration of the process.

Example in Python:

import requests

def get_secret(key):
    response = requests.get(
        f'https://secret-store.example.com/v1/secrets/{key}',
        headers={'Authorization': f'Bearer {TOKEN}'}
    )
    if response.status_code != 200:
        raise Exception(f"Failed to fetch secret: {response.text}")
    return response.json()['value']

Applications must not log or store secrets in plaintext. Secrets should be replaced with placeholders or masked before logging.

Secret Exposure Prevention

Prevent secrets from appearing in logs, configuration files, or environment variables by using a secret injection pattern. Applications read secrets from a secure interface and never store them directly.

Secrets are exposed through:

Example of masking in logs:

# Instead of logging the full secret
print(f"DB_PASSWORD={secret_value}")  # Logs plaintext

# Log with placeholder or hash
print(f"DB_PASSWORD=***")  # Logs masked

Secret Rotation Automation

Automate rotation using a scheduled job that updates secrets and notifies services. The automation process must:

  1. Fetch current secret
  2. Generate new value
  3. Update secret in store
  4. Notify applications to reload

Example cron job:

# /etc/cron.d/secret-rotation
0 0 1 * * root /usr/local/bin/rotate-secret.sh /prod/app/database-password

Handling Access Failures

Access failures are typically returned with HTTP status codes:

Example error response:

{
  "error": "Forbidden",
  "message": "Access denied to secret /prod/app/database-password"
}

Applications must handle these failures gracefully, either by failing fast or implementing retry logic.

Versioning and Rollback

Each secret update creates a new version. Applications can specify which version to use:

# Retrieve specific version
curl -X GET https://secret-store.example.com/v1/secrets/prod/app/database-password?v=2 \
  -H "Authorization: Bearer $TOKEN"

Rollbacks are performed by updating the secret to a previous value, not by deleting versions. A rollback requires a new write operation.

Audit and Monitoring

Secret access is logged with metadata including:

Example audit log entry:

{
  "timestamp": "2024-01-16T14:30:00Z",
  "principal": "service-account-backend",
  "action": "read",
  "secret_key": "/prod/app/database-password",
  "ip_address": "192.168.1.100"
}

Failure Modes

Several silent failures can occur:

Common Pitfalls

Confusing version=latest with the current value leads to stale secrets if not updated. Using PUT instead of POST for secret creation results in an error because creation requires a new key.

Misconfiguring access control is a frequent issue: setting read permissions but not write permissions causes rotation failures.

Secret Backup and Recovery

Backups are maintained per secret, with retention policies. The system may maintain a history of up to 100 versions or 90 days of changes, whichever is shorter.

Recovery is performed by restoring from a known good version:

# Restore a secret to a previous version
curl -X PUT https://secret-store.example.com/v1/secrets/prod/app/database-password \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "value": "backup_value_from_2024-01-01",
    "version": 5
  }'

Configuration and Deployment

Secrets are deployed using a standard API, not through configuration files. The system enforces that secrets are never stored in repository files.

Deployment scripts must:

  1. Fetch secrets before starting the application
  2. Validate secret access
  3. Fail if secrets cannot be retrieved
#!/bin/bash
# Deployment script example
SECRET_VALUE=$(curl -s -X GET https://secret-store.example.com/v1/secrets/prod/app/database-password \
  -H "Authorization: Bearer $TOKEN")
if [ $? -ne 0 ]; then
    echo "Failed to retrieve secret"
    exit 1
fi
# Start application with secret in environment
export DB_PASSWORD="$SECRET_VALUE"
exec ./my-app

Troubleshooting

Common issues include:

Use the audit logs to trace access and identify exposure points. Monitor for unauthorized access attempts.

Integration Points

Secrets integrate with:

These integrations are typically handled through API endpoints or webhook mechanisms, not by embedding secrets directly in configuration files.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.