Securely store, distribute, and rotate secrets across applications and infrastructure. Covers vault architectures, secret injection patterns, rotation strategies, and the common mistakes that lead to secret leaks.
Secrets management is the practice of securely storing, distributing, and rotating sensitive data—such as database passwords, API keys, TLS certificates, and encryption keys—across a distributed system. It matters when services need to access secrets at runtime, especially in cloud-native environments with multiple environments, frequent deployments, and dynamic scaling. A misconfigured secret can cause outages, data breaches, or security policy violations.
To register a new secret, use a standardized interface: a POST request to /secrets with JSON payload. The secret is identified by a unique name and environment, and may include metadata such as description, owner, and rotation_interval.
POST /secrets
{
"name": "database.auth",
"environment": "prod",
"value": "s3cr3t_p@ssw0rd",
"description": "Credentials for primary PostgreSQL instance",
"owner": "[email protected]",
"rotation_interval": "7d"
}On success, the server responds with a 201 Created status and the full secret object, including a created_at timestamp and a secret_id.
{
"secret_id": "sec-7a2d1f8b",
"name": "database.auth",
"environment": "prod",
"value": "s3cr3t_p@ssw0rd",
"description": "Credentials for primary PostgreSQL instance",
"owner": "[email protected]",
"rotation_interval": "7d",
"created_at": "2024-03-15T10:30:00Z",
"updated_at": "2024-03-15T10:30:00Z"
}If a secret with the same name and environment already exists, the server returns 409 Conflict and includes the existing secret in the response body.
Services retrieve secrets via a well-known endpoint, typically GET /secrets/{name}/{environment}. The response is a JSON object with the value and metadata fields.
GET /secrets/database.auth/prod HTTP/1.1
Host: secrets.acme.com
Authorization: Bearer <service-token>
Accept: application/json{
"value": "s3cr3t_p@ssw0rd",
"metadata": {
"description": "Credentials for primary PostgreSQL instance",
"owner": "[email protected]",
"rotation_interval": "7d",
"created_at": "2024-03-15T10:30:00Z"
}
}The service can cache the response for up to max-age seconds, derived from a Cache-Control header, and revalidate using ETag or If-None-Match.
Secret rotation is triggered either manually or automatically. A manual rotation is a PUT request to the secret’s endpoint with a new value.
PUT /secrets/database.auth/prod HTTP/1.1
Host: secrets.acme.com
Authorization: Bearer <admin-token>
Content-Type: application/json
{
"value": "n3w_s3cr3t_p@ssw0rd",
"description": "Rotated for quarterly security audit"
}The server updates the value, sets updated_at, and logs the rotation event. The old value remains accessible for a grace period, typically 24 hours, during which both old and new values are valid.
Automatic rotation occurs via a periodic job that fetches secrets with rotation_interval set. For 7d, the job runs daily and checks if the current value is older than 7 days. If so, it generates a new value (e.g., using openssl rand -base64 32) and updates the secret.
Deploying services with secrets requires a consistent contract between platform and application. The platform provides secrets as environment variables, mounted as files, and accessible via a configuration file.
The platform injects secrets into the application process via environment variables. The convention is SECRET_{NAME}_{ENVIRONMENT}.
For database.auth/prod, the platform sets:
export DATABASE_AUTH_PROD="s3cr3t_p@ssw0rd"Applications can read this via standard os.getenv("DATABASE_AUTH_PROD") or process.env.DATABASE_AUTH_PROD. The platform ensures this variable is set before the application starts.
The platform mounts a directory, typically /etc/secrets, with one file per secret. The filename is name.environment, and the contents are the value.
For database.auth/prod, the file /etc/secrets/database.auth.prod contains:
s3cr3t_p@ssw0rdApplications can read this file using standard file I/O: File.read("/etc/secrets/database.auth.prod") or with open("/etc/secrets/database.auth.prod") as f: value = f.read().
The platform generates a JSON file at /etc/secrets/config.json that includes all secrets for the current environment.
{
"database": {
"auth": "s3cr3t_p@ssw0rd",
"host": "db.prod.acme.com"
},
"cache": {
"redis": "r3d1s_p@ssw0rd"
}
}The application reads this file during startup and merges it with its own configuration.
In Kubernetes, secrets are managed through Secret resources. The platform creates a Secret object per secret, with data and metadata.
apiVersion: v1
kind: Secret
metadata:
name: database-auth-prod
namespace: prod
labels:
app: web
environment: prod
type: Opaque
data:
value: c3NjcmV0X3Bhc3N3b3Jk
description: Q2VydGlmaWNhdGlvbiBmb3IgcHJvZGljIEBjZW50ZXIgaW50ZXJhY3Rpb24gc3RhdGlvbg==The value is base64-encoded. The application can mount this secret as a volume or inject it as environment variables.
To mount the secret, use a volumeMount in the pod spec:
volumeMounts:
- name: secrets
mountPath: /etc/secrets
readOnly: true
volumes:
- name: secrets
secret:
secretName: database-auth-prodTo inject as environment variables, use envFrom:
envFrom:
- secretRef:
name: database-auth-prodThis populates DATABASE_AUTH_PROD in the container.
When a service expects a secret and it is missing, it fails silently unless the platform provides a default fallback mechanism. The most common failure is a 500 Internal Server Error with a log entry:
ERROR [main] - Failed to load secret: database.auth/prod
Caused by: HTTP 404 Not Found
Response: {"error": "Secret not found", "details": {"name": "database.auth", "environment": "prod"}}This occurs when the secret was never created, or the name or environment is misspelled. A common typo is prod vs production.
During rotation, the old and new secrets coexist, but not all services update immediately. A service that reads the secret once at startup and caches it may use the old value for hours or days. This leads to drift: some services use the new secret, others the old.
To detect drift, applications log the secret value at startup and periodically. A log entry like:
INFO [startup] - Connected to database using secret: s3cr3t_p@ssw0rdreveals that a service is still on the old secret even after rotation.
When deploying a new version of a service, the platform may fail to inject all required secrets. The service starts, but fails to connect to a database because DATABASE_AUTH_PROD is missing. The error is not in the application log, but in the platform’s deployment logs.
A common cause is a missing Secret object in Kubernetes. The Secret exists, but the Secret resource is not mounted to the pod. The platform logs a 400 Bad Request when creating the deployment:
{
"error": "Deployment failed",
"details": {
"step": "mount-secrets",
"error": "Secret not found in namespace",
"secret_name": "database-auth-prod",
"namespace": "prod"
}
}The platform must validate that every required secret is present and correctly referenced.
In CI/CD pipelines, secrets are used to authenticate with external services: deploying to AWS, pushing to a container registry, or running integration tests.
The pipeline reads secrets from a configuration file, typically secrets.yaml, with a flat structure.
aws:
access_key_id: AKIA234567890
secret_access_key: s3cr3t_k3y
region: us-west-2
github:
token: ghp_abc123xyz
owner: acme
repo: platform-engineering
postgres:
connection_string: postgres://user:[email protected]:5432/devThe pipeline uses curl to fetch secrets from the secrets manager and sets them as environment variables.
#!/bin/bash
set -e
SECRETS=$(curl -s -H "Authorization: Bearer $CI_TOKEN" \
"https://secrets.acme.com/secrets/$(git branch --show-current)/ci")
# Convert to environment variables
echo "$SECRETS" | while IFS= read -r line; do
export "$line"
doneThe CI_TOKEN is a long-lived secret used to authenticate the CI system with the secrets manager.
The platform supports versioning of secrets. Each update to a secret creates a new version, identified by a version_id.
{
"secret_id": "sec-7a2d1f8b",
"name": "database.auth",
"environment": "prod",
"value": "n3w_s3cr3t_p@ssw0rd",
"version_id": "ver-1a3b2c4d",
"created_at": "2024-03-16T08:15:00Z",
"updated_at": "2024-03-16T08:15:00Z"
}To roll back, the platform applies a PUT request to the secret’s version_id endpoint:
PUT /secrets/database.auth/prod/versions/ver-1a3b2c4d HTTP/1.1
Host: secrets.acme.com
Authorization: Bearer <admin-token>
Content-Type: application/json
{
"value": "s3cr3t_p@ssw0rd",
"description": "Rolled back to original value"
}The platform updates the active value, and logs the rollback event. All services using the secret now see the old value.
For services that use many secrets, the platform supports lazy loading. The service makes an initial GET /secrets request to discover available secrets, then fetches only those it needs.
GET /secrets?include=database.auth,cache.redis,api.keys HTTP/1.1
Host: secrets.acme.com
Authorization: Bearer <service-token>The response is a list of secrets, each with name, environment, value, and metadata.
[
{
"name": "database.auth",
"environment": "prod",
"value": "s3cr3t_p@ssw0rd",
"metadata": { ... }
},
{
"name": "cache.redis",
"environment": "prod",
"value": "r3d1s_p@ssw0rd",
"metadata": { ... }
}
]The service uses only the returned secrets, reducing startup time and memory usage.
The platform exposes a REST API for managing secrets, with endpoints for listing, creating, updating, and deleting secrets. The API is versioned, with v1 as the default.
GET /v1/secrets?environment=prod&name=database.authThe server returns a paginated list of secrets, with next_page and prev_page links in the Link header.
Link: <https://secrets.acme.com/v1/secrets?environment=prod&page=2>; rel="next"The API supports filtering, sorting, and searching across secrets. The platform uses OAuth 2.0 for authentication, with Bearer tokens.
For multi-region deployments, the platform supports region-specific secrets. The environment field includes a region component.
{
"name": "database.auth",
"environment": "prod-us-west-1",
"value": "usw1_s3cr3t"
}Services in us-west-1 use prod-us-west-1, while services in us-east-1 use prod-us-east-1. The platform can route requests to the nearest region and fallback to a global secret if a regional one is missing.
The platform also supports cross-region replication: a new secret in us-west-1 is automatically replicated to us-east-1 with a configurable delay.
Secrets management is not just storing credentials—it is the operational backbone of secure, reliable, and scalable systems. The core operations of creating, reading, rotating, and deploying secrets are grounded in standard HTTP, JSON, and file I/O patterns. The failure modes—missing secrets, drift during rotation, incomplete deployments—are common and subtle. Advanced patterns like lazy loading, versioning, and multi-region support enable robust, production-grade systems. The platform must ensure that secrets are discoverable, consistent, and resilient across the entire lifecycle of an application.
This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy