Secrets Management Done Right: Vault, SOPS, and the Practices That Actually Work

This document describes practical techniques for managing secrets across environments using tools that actually work in production. It focuses on how to store, rotate, access, and audit secrets with minimal operational overhead and maximum security.

Storing Secrets

Store plaintext values in version-controlled repositories using SOPS

Use SOPS to encrypt secrets at rest in Git repositories. The encryption is applied per-key, not full-file:

# example-secrets.yaml
database_password: ENC[AES256_GCM,data:...,iv:...,tag:...]
api_key: ENC[AES256_GCM,data:...,iv:...,tag:...]

Encrypt with:

sops --encrypt --encrypted-regex '^(database_password|api_key)$' example-secrets.yaml

SOPS uses age keys or AWS KMS for encryption. The encrypted file remains readable as YAML, but the values are base64-encoded and wrapped in a structured format.

Store sensitive data in HashiCorp Vault

Vault provides a consistent secret storage interface with TTLs and access control:

vault kv put secret/myapp/config \
  database_password="supersecret" \
  api_key="anothersecret"

Access via HTTP API:

curl -H "X-Vault-Token: $VAULT_TOKEN" \
  https://vault.example.com/v1/secret/data/myapp/config

Vault returns JSON with data and metadata keys:

{
  "data": {
    "data": {
      "database_password": "supersecret",
      "api_key": "anothersecret"
    },
    "lease_id": "secret/myapp/config/...",
    "lease_duration": 2592000,
    "renewable": true
  }
}

Vault also supports dynamic secrets like database credentials or SSH keys.

Rotate secret values

For static secrets, use a two-step process:

1. Write new values to Vault:

   vault kv put secret/myapp/config \
     database_password="newpassword123" \
     api_key="newkey456"

1. Update SOPS-encrypted files with new values:

   sops --decrypt example-secrets.yaml > decrypted.yaml
   # Manually update the plaintext fields
   sops --encrypt decrypted.yaml > example-secrets.yaml

Vault allows reading without rotation:

vault kv get secret/myapp/config

Accessing Secrets

Read Vault secrets from a container or service

Use Vault's AppRole authentication:

1. Create an AppRole:

   vault auth enable approle
   vault write auth/approle/role/myapp \
     policies="myapp-policy" \
     token_policies="myapp-policy"

1. Generate a role ID and secret ID:

   vault read auth/approle/role/myapp/role-id
   vault write -f auth/approle/role/myapp/secret-id

1. In application code, use Vault client or HTTP API:

   curl -H "X-Vault-Token: $VAULT_TOKEN" \
     https://vault.example.com/v1/secret/data/myapp/config

Read SOPS-encrypted secrets in Kubernetes

Use a sidecar container that decrypts files on mount:

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: secrets
      mountPath: /etc/secrets
  volumes:
  - name: secrets
    emptyDir: {}
  initContainers:
  - name: decrypt-secrets
    image: mozilla/sops:3.7.1
    command:
    - /bin/sh
    - -c
    - |
      sops --decrypt /etc/sops/secrets.yaml > /tmp/secrets.yaml
      cp /tmp/secrets.yaml /etc/secrets/
    volumeMounts:
    - name: sops-secrets
      mountPath: /etc/sops
    - name: secrets
      mountPath: /etc/secrets

Managing Access Control

Use Vault policies to control access

Policies are written in HCL:

# myapp-policy.hcl
path "secret/data/myapp/*" {
  capabilities = ["read", "list"]
}

path "secret/metadata/myapp/*" {
  capabilities = ["read", "list"]
}

Apply to a token:

vault policy write myapp-policy myapp-policy.hcl
vault token create -policy="myapp-policy"

Vault returns errors like:

Error: permission denied on secret/myapp/config

Control access with SOPS key management

SOPS supports multiple encryption keys. Each team member has their own age key:

# List current encryptors
sops --decrypt example-secrets.yaml | grep -A 10 "encryptors"

Adding a new key:

sops --encrypt \
  --add-encryption-key age1xyz \
  example-secrets.yaml > new-secrets.yaml

Auditing and Monitoring

Audit Vault access

Enable audit logging:

vault audit enable file file_path=/var/log/vault-audit.log

Vault logs entries like:

{"time":"2023-01-01T00:00:00Z","type":"request","path":"secret/data/myapp/config","operation":"read"}

Track SOPS changes in Git

SOPS files are binary, so git diff shows raw base64:

git diff example-secrets.yaml

Output:

-  database_password: ENC[AES256_GCM,data:...,iv:...,tag:...]
+  database_password: ENC[AES256_GCM,data:...,iv:...,tag:...]

Use sops --decrypt to see plaintext content:

sops --decrypt example-secrets.yaml

Failure Modes

Vault token expiration

Vault returns error on expired tokens:

Error: invalid token

Applications must handle this by acquiring new tokens via AppRole or Vault agent.

SOPS key loss

If a team member loses their age key, they cannot decrypt files:

sops --decrypt example-secrets.yaml

This produces:

Error: failed to decrypt data: unable to decrypt with any key

Recovery requires:

1. Re-encrypting with another key
2. Adding the new key to SOPS config

Mixed encryption in SOPS files

SOPS allows mixing encrypted and plaintext fields. If a field is not encrypted, it can leak:

# This is safe
database_password: ENC[AES256_GCM,...]
api_key: ENC[AES256_GCM,...]

# This leaks plaintext
plaintext_field: "should be encrypted but isn't"

Use --encrypted-regex to enforce encryption:

sops --encrypt --encrypted-regex '^(database_password|api_key)$' example.yaml

Common Pitfalls

Vault token reuse

Reusing tokens across services creates a single point of failure:

# Bad: Reuse token across multiple services
vault token create -token-policies="common-policy"

Better:

# Good: Per-service tokens
vault token create -token-policies="myapp-policy"
vault token create -token-policies="database-policy"

SOPS file versioning

SOPS files are not versioned by default. A new encryption key creates a new file:

sops --encrypt example-secrets.yaml > example-secrets.yaml

To preserve history, commit both old and new versions:

git add example-secrets.yaml
git commit -m "Update secrets"

Vault path naming

Vault paths should be consistent. Avoid ambiguous names:

# Good
vault kv put secret/myapp/config/database_password="..."

# Bad
vault kv put secret/myapp/database_password="..."

Best Practices Summary

1. Never store plaintext secrets in Git
2. Use Vault for dynamic and rotating secrets
3. Use SOPS for static secrets in version control
4. Encrypt only sensitive fields, not entire files
5. Set up access control policies at the path level
6. Audit all access to secrets
7. Plan for key rotation and recovery

The combination of Vault and SOPS provides a robust foundation for secrets management where Vault handles dynamic secrets and SOPS manages static ones, with clear separation of concerns and minimal operational overhead.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.