Automate the rotation of secrets, API keys, and certificates to eliminate manual credential management. Covers rotation strategies, vault integration, zero-downtime rotation, certificate lifecycle, and the patterns that prevent expired credentials from causing outages.
Secrets rotation automation is the practice of regularly replacing stored credentials, keys, and tokens with new values without manual intervention or service disruption. It matters when secrets have finite lifespans, when compliance or policy mandates periodic change, or when a breach risk demands rapid key turnover. This page covers the operational pattern: how to detect secret locations, fetch current values, apply a rotation algorithm, persist the new secret, and ensure all consumers pick up the replacement. The focus is on concrete, repeatable steps using standard tooling and generic mechanisms that map to any environment.
Before a secret can be rotated, the system must know where it lives. Secrets are rarely confined to a single file; they surface in version‑controlled configs, container orchestration stores, database connections, and runtime environment bundles.
A first step is scanning repositories for strings that look like credentials. The git grep family of commands can surface matches across branches and history:
git grep -rni 'export DB_PASSWORD' .or, for a broader pattern across all tracked files:
grep -Rni 'password\s*=' --include='*.yaml' --include='*.json' --include='*.env' . | grep -v '^#' | cut -d= -f2 | sort -uThese commands produce a list of file paths and line numbers, which can be fed into a rotation script. Note that grep will also match comments and documentation; a production‑grade inventory typically adds YAML/JSON parsing to distinguish actual assignments from examples.
Cluster‑deployed secrets are stored as native Kubernetes objects. The kubectl CLI can list every secret across namespaces:
kubectl get secrets --all-namespaces -o jsonpath='{.items[*].metadata.name}'To inspect a specific secret’s current contents (useful for verifying what’s being rotated):
kubectl get secret db-credentials -o yamlThe output reveals the data and stringData fields, which are base64‑encoded. Rotation will replace these values while preserving the secret’s metadata (type, annotations, labels).
Services often load secrets from files mounted by a container runtime, from systemd unit files, or from configuration management tools. A generic check might look like:
systemctl cat my‑service | grep -i 'ENV\|SECRET\|KEY'or, for Docker‑compose deployments:
docker compose config | grep -i 'password\|token'These commands do not rotate anything; they simply expose where a runtime expects to find a secret value. Any rotation workflow must ensure the new value ends up in each of these locations before consumers restart.
With an inventory in hand, the rotation cycle proceeds through four deterministic steps: generate, persist, propagate, and verify. The following sub‑sections show concrete, real‑world commands that embody each step.
Cryptographically random values are preferred for high‑entropy secrets. The POSIX openssl tool can produce a 32‑byte base64 string suitable for most use cases:
NEW_SECRET=$(openssl rand -base64 32)If the secret must follow a specific format (e.g., a 16‑character alphanumeric API key), the generation step may invoke a platform‑specific SDK or a deterministic algorithm keyed by a master key. The pattern remains the same: output a single string variable that will be handed off to the persistence step.
The method of persistence depends on where the secret lives. For Kubernetes, the atomic replace‑or‑create pattern is common:
kubectl create secret generic db-credentials \
--from-literal=password="$NEW_SECRET" \
--dry-run=client -o yaml | kubectl apply -f -The --dry-run=client flag renders the object to YAML without sending it to the API server; piping through kubectl apply writes the change. If the secret already exists, apply performs a strategic merge; if not, it creates a new object. For single‑file stores (e.g., .env checked into Git), a sed replacement followed by a commit works:
sed -i "s/^DB_PASSWORD=.*/DB_PASSWORD=$NEW_SECRET/" .env
git add .env && git commit -m "rotate db password" && git pushFor cloud‑hosted secret managers, the CLI surface varies. The pattern is always “fetch current → replace value → write back,” but the exact flags are product‑specific and therefore left generic here.
A rotated secret is only effective once every consumer has loaded the new value. In Kubernetes, a restart of the workload forces a re‑read of mounted secrets:
kubectl rollout restart deployment/db-proxyThe kubectl rollout status command can then be used to watch the transition:
kubectl rollout status deployment/db-proxy --timeout=120sIn non‑Kubernetes environments, the refresh mechanism might be a service reload, a process signal, or a cache invalidation call. For systemd‑managed services:
systemctl restart my‑serviceFor CI pipelines that inject secrets at job start, a new pipeline trigger (often a webhook or a git push to a protected branch) ensures the fresh value is fetched from the store.
After propagation, a brief validation window confirms that the new secret works and the old one no longer causes errors. A lightweight health check against the affected service is typical:
curl -sf http://localhost:8080/healthzIf the endpoint returns 200 OK and the service logs show no authentication failures, the rotation is considered successful. Conversely, examining recent pod events (kubectl get events -n prod --field-selector reason=Failed) can reveal whether a consumer missed the update.
Even with a deterministic script, production rotations can encounter silent failure modes and confusing option interactions.
Kubernetes secrets are not instantly propagated to all nodes. After a kubectl apply, the control plane updates the etcd store, but kubelets cache the secret for a configurable sync interval (default 1 minute, adjustable via --secret‑sync‑period). A pod that started before the sync may continue using the old base64 value until it restarts. Operators often underestimate this lag and mark a rotation “failed” when only patience is needed.
--dry-run vs --dry-run=clientThe kubectl create and kubectl apply families expose two dry‑run modes. --dry-run=client evaluates the request locally and prints the resulting object without contacting the API server. --dry-run (without =client) sends the request to the server but returns the dry‑run response. A common mistake is using --dry-run in a script that expects no network traffic, causing the script to hang waiting for API server response, or conversely using --dry-run=client when a server‑side validation is required, producing a YAML that fails apply due to missing immutable fields. Verifying the exact flag semantics for the deployed kubectl version is a pre‑rotation checklist item.
When multiple automation instances target the same secret—e.g., a cron‑based script and a CI pipeline both triggered by a git push—concurrent writes can race. The first writer’s kubectl apply may be overwritten by the second writer’s output, or the API server may reject the second write with a conflict error. Some environments mitigate this with optimistic concurrency control (resourceVersion checks), but not all secret stores implement it. A rotation script that does not lock or idempotently check the current value before writing can silently corrupt the secret store, leaving a hybrid of old and new values that neither consumers nor audit tools can reconcile.
When a rotation workflow is shared across dev, staging, and prod namespaces, subtle syntax differences often surface. A kubectl version newer than the cluster’s minimum may expose flags (--confirm or --save-config) that older client binaries reject. Conversely, an older kubectl may not support jsonpath output used in inventory scripts, causing the rotation pipeline to abort with a parsing error. Maintaining a pinned kubectl version per environment, or wrapping commands in version‑detect logic, prevents these silent failures.
Platforms that provide secret rotation capabilities include HashiCorp Vault, AWS Secrets Manager, and Kubernetes native secrets controllers. These services expose HTTP or CLI interfaces that implement the rotate‑fetch‑persist pattern described above, often with built‑in lease management, versioning, and audit logging. Readers can map the generic commands and failure modes shown here to the specific SDK or UI of their chosen platform, replacing kubectl with vault kv put, aws secretsmanager rotate-secret, or the equivalent API calls their organization standardizes on.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy