Secret Management Devops | The Garnet Wiki

Secrets are sensitive data—API keys, database credentials, TLS certificates, encryption keys, OAuth tokens, and private configuration values—that must be securely managed, versioned, distributed, and rotated across environments. Secret Management DevOps is the discipline of ensuring secrets are available, consistent, and secure from development to production, with minimal manual effort and zero configuration drift. It matters when a service fails because a database password is wrong, a third-party API fails due to expired tokens, or a deployment breaks because a new environment was missing a key.

Managing Secrets Across Environments

Secrets must be present and correct in every environment: development, staging, and production. The most common failure is a mismatch between environments, where a service connects to a test database in staging but fails in production due to a missing or incorrect DATABASE_URL secret.

The foundation is a secrets store: a central repository where secrets are stored, versioned, and accessed. A typical setup uses a secrets file per environment, stored in version control:


DATABASE_URL: postgres://staging-user:staging-pass@staging-db:5432/app
REDIS_URL: redis://staging-redis:6379/0
AWS_ACCESS_KEY_ID: AKIAEXAMPLE123456
AWS_SECRET_ACCESS_KEY: s3cr3tK3yF0rSt4ging
JWT_SECRET: a56b23b4c1d9e8f7a6b5c4d3e2f1a0b9

This file is checked into Git, and a CI pipeline applies it to the target environment using kubectl:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: staging
type: Opaque
data:
  DATABASE_URL: cG9zdGdyZXM6Ly9zdGFnaW5nLXVzZXI6c3RhbGlnZy1wYXNzQHN0YWdpbmdkYjoxNDMyL2FwcA==
  REDIS_URL: cmVkaXM6Ly9zdGFnaW5ncmVkaXM6NjM3OS8w
  AWS_ACCESS_KEY_ID: QUtJQUVYQU1QTEUxMjM0NTY=
  AWS_SECRET_ACCESS_KEY: c3NjcmUzV0tleUZvclN0YWdpbmc=
  JWT_SECRET: YDU2YjIzYjRjMWQ5ZThmN2E2YjVjNGQzZTJmMWExYjky
EOF

The data field uses base64 encoding. A common mistake is forgetting to encode values, resulting in binary data in the secret:

$ kubectl get secret app-secrets -o yaml
data:
  DATABASE_URL: cG9zdGdyZXM6Ly9zdGFnaW5nLXVzZXI6c3RhbGlnZy1wYXNzQHN0YWdpbmdkYjoxNDMyL2FwcA==
  REDIS_URL: cmVkaXM6Ly9zdGFnaW5ncmVkaXM6NjM3OS8w
  AWS_ACCESS_KEY_ID: QUtJQUVYQU1QTEUxMjM0NTY=
  AWS_SECRET_ACCESS_KEY: c3NjcmUzV0tleUZvclN0YWdpbmc=
  JWT_SECRET: YDU2YjIzYjRjMWQ5ZThmN2E2YjVjNGQzZTJmMWExYjky

A deployment that expects DATABASE_URL as a plain text key fails silently when the value is binary. The fix is to ensure the base64 encoding is correct:

$ echo -n "postgres://staging-user:staging-pass@staging-db:5432/app" | base64
cG9zdGdyZXM6Ly9zdGFnaW5nLXVzZXI6c3RhbGlnZy1wYXNzQHN0YWdpbmdkYjoxNDMyL2FwcA==

Secret Rotation and Lifecycle

Secrets must be rotated regularly. A common pattern uses a cron job that generates new secrets and updates the store.

A rotation script might generate a new JWT secret and update the staging secrets file:

#!/bin/bash
# rotate-secrets.sh

export JWT_SECRET=$(openssl rand -hex 32)
export AWS_SECRET_ACCESS_KEY=$(openssl rand -base64 64)

cat <<EOF > secrets/staging.yaml
DATABASE_URL: postgres://staging-user:staging-pass@staging-db:5432/app
REDIS_URL: redis://staging-redis:6379/0
AWS_ACCESS_KEY_ID: AKIAEXAMPLE123456
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
JWT_SECRET: ${JWT_SECRET}
EOF

git add secrets/staging.yaml
git commit -m "rotate secrets: JWT and AWS"
git push origin main

The script uses openssl to generate cryptographically strong secrets. The rand -hex 32 command produces a 64-character hexadecimal string, ideal for JWT secrets.

After rotation, the updated secrets must be deployed. A kubectl apply or helm upgrade is used to roll out changes. The most common failure is a deployment that updates secrets but does not restart the application. The service continues to use the old secrets until the next restart.

To ensure freshness, the deployment must include a readiness probe that checks the current secret values:

readinessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - |
        test "$(cat /etc/secrets/JWT_SECRET)" = "$(cat /app/expected/jwt-secret.txt)"
  initialDelaySeconds: 10
  periodSeconds: 5

This probe fails if the JWT_SECRET in the pod’s filesystem does not match the expected value, indicating a missing or outdated secret.

Distributing Secrets to Services

Services must access secrets at runtime. The most common method is mounting secrets as files or environment variables.

For a Kubernetes pod, secrets are mounted via a volume:

spec:
  containers:
    - name: app
      image: myapp:latest
      envFrom:
        - secretRef:
            name: app-secrets
      volumeMounts:
        - name: secrets
          mountPath: /etc/secrets
          readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: app-secrets

This configuration mounts the app-secrets secret to /etc/secrets, with each key as a file:

/etc/secrets/DATABASE_URL
/etc/secrets/REDIS_URL
/etc/secrets/AWS_ACCESS_KEY_ID
/etc/secrets/AWS_SECRET_ACCESS_KEY
/etc/secrets/JWT_SECRET

The service reads these files at startup. A common mistake is assuming the environment variable JWT_SECRET is automatically populated from the secret. It is not: envFrom populates environment variables only from the data fields, not from mounted files.

To use mounted files, the application must explicitly read them:

// Go service example
func main() {
    jwtSecret, err := os.ReadFile("/etc/secrets/JWT_SECRET")
    if err != nil {
        log.Fatal("failed to read JWT secret:", err)
    }
    os.Setenv("JWT_SECRET", string(jwtSecret))
    // ... rest of startup
}

Another pattern is using a secret loader service that runs alongside the application. This loader polls a secrets store and writes secrets to a shared directory:

# secrets-loader.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secrets-loader
spec:
  replicas: 1
  selector:
    matchLabels:
      app: secrets-loader
  template:
    metadata:
      labels:
        app: secrets-loader
    spec:
      containers:
        - name: secrets-loader
          image: gcr.io/myorg/secrets-loader:v1
          env:
            - name: SECRETS_STORE_URL
              value: http://secrets-store:8080
            - name: TARGET_DIR
              value: /secrets
          volumeMounts:
            - name: secrets
              mountPath: /secrets
              readOnly: false
      volumes:
        - name: secrets
          emptyDir: {}

The secrets-loader service fetches secrets via HTTP and writes them to /secrets:

$ curl http://secrets-store:8080/secrets/staging
{
  "DATABASE_URL": "postgres://staging-user:staging-pass@staging-db:5432/app",
  "REDIS_URL": "redis://staging-redis:6379/0",
  "AWS_ACCESS_KEY_ID": "AKIAEXAMPLE123456",
  "AWS_SECRET_ACCESS_KEY": "s3cr3tK3yF0rSt4ging",
  "JWT_SECRET": "a56b23b4c1d9e8f7a6b5c4d3e2f1a0b9"
}

The service writes each key as a file in the shared volume, and the app uses this directory as a source of secrets.

Centralized Secrets Store

A centralized secrets store is the heart of the system. It provides a single source of truth for all secrets. The store must support:

Versioning (e.g., each secret has a revision)
Access control (e.g., roles and permissions)
Rotation (e.g., automatic key rotation with metadata)
API for reading and updating secrets

A typical API endpoint returns a secret by name and environment:

GET /secrets/app/staging HTTP/1.1
Host: secrets-store.example.com
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.xxxxx

{
  "name": "app",
  "environment": "staging",
  "version": "v3",
  "data": {
    "DATABASE_URL": "postgres://staging-user:staging-pass@staging-db:5432/app",
    "REDIS_URL": "redis://staging-redis:6379/0",
    "AWS_ACCESS_KEY_ID": "AKIAEXAMPLE123456",
    "AWS_SECRET_ACCESS_KEY": "s3cr3tK3yF0rSt4ging",
    "JWT_SECRET": "a56b23b4c1d9e8f7a6b5c4d3e2f1a0b9"
  },
  "updated_at": "2023-10-05T12:34:56Z"
}

The Authorization header uses a JWT token, and the response includes metadata such as version and update time. The client uses the version field to cache secrets and detect changes.

A common failure is caching secrets without checking for updates. The client reads secrets/app/staging once, but subsequent requests do not check for changes. The fix is to use the ETag or Last-Modified headers:

GET /secrets/app/staging HTTP/1.1
Host: secrets-store.example.com
If-None-Match: "v3-abc123"

If the secret has not changed, the server returns 304 Not Modified. The client uses the cached version. When the version changes, the client fetches the new data.

Error Patterns and Failure Modes

Several failure modes are silent but critical:

Missing secrets: The service starts but cannot connect to the database because DATABASE_URL is not in the environment or volume. The error is a Connection refused or Invalid connection string, but the root cause is a missing or incorrect secret.

Wrong environment: A service in production uses staging secrets. The AWS_ACCESS_KEY_ID is AKIAEXAMPLE123456, but the production bucket is in a different region. The fix is to verify the environment name in the secrets store.

Delayed deployment: A new secret is added to the store, but the deployment script does not include it. The service uses an old secrets file, and the new secret is ignored.

Incomplete rotation: A rotation script updates the secrets file but does not deploy it. The secrets are in Git but not in Kubernetes. The service continues to use old secrets until the next deployment.

Secrets not reloaded: A service reads secrets at startup but does not reload them when they change. A new JWT secret is generated but not picked up until the service is restarted.

Invalid base64: The data field in a Kubernetes secret is not properly base64-encoded. The service receives binary data instead of text, leading to corrupted values.

Missing secrets in volume mounts: The volume mount is specified, but the secretName is misspelled. The secret is created but not mounted. The error is a No such file or directory when reading /etc/secrets/JWT_SECRET.

Incorrect file paths: The application expects secrets in /etc/secrets, but the volume mount uses /secrets. The service fails to find any secrets.

Cache invalidation: The secrets store returns a versioned response, but the client does not validate the cache. The client uses stale secrets for days, even after a rotation.

Authentication failures: The secrets store requires a token, but the token is expired or malformed. The error is 401 Unauthorized, but the client does not refresh the token.

Summary

Secret management in DevOps is not about storing secrets—it is about ensuring they are correct, available, and up to date across the entire stack. It requires a consistent process for creating, rotating, distributing, and validating secrets. The key patterns are: versioned secrets in Git, deployed via CI/CD, mounted as files or environment variables, and accessed via a centralized store. The most common failure is a mismatch between the expected and actual secrets, often due to missing steps in the pipeline. The solution is a clear, repeatable process that includes validation, caching, and error detection.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.