GitOps Secrets Management

Manage secrets in GitOps workflows without storing them in git. Covers sealed secrets, external secrets operator, vault integration, secret rotation, and the patterns that keep sensitive data secure while maintaining the GitOps single-source-of-truth principle.

GitOps workflows require managing secrets in a way that maintains the declarative, version-controlled nature of infrastructure while protecting sensitive data from exposure in source repositories. This approach becomes critical when secrets must be synchronized between multiple environments, automated deployments depend on secret availability, or compliance requirements mandate audit trails for credential changes.

Synchronizing Secrets Between Git and Kubernetes

Reading Secrets from Git Repository

The basic workflow requires that secrets stored in Git be accessible to Kubernetes clusters through controllers like Argo CD or Flux. A typical approach stores secrets as Kubernetes Secret resources in the repository, with a pattern similar to:

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: production
type: Opaque
data:
  database_password: cGFzc3dvcmQxMjM=
  api_key: YWJjZGVmZ2hpams=

The secret data field uses base64 encoding. When applying this through GitOps, the controller retrieves the file from the repository and creates or updates the corresponding Kubernetes Secret.

Applying Secrets to Multiple Environments

When managing multiple environments (staging, production), the same pattern applies but requires careful consideration of how environment-specific values are handled. A common approach uses Kustomize overlays or Helm values files that inject environment-specific secrets:

# kustomization.yaml
resources:
  - ../base
patches:
  - path: patches/secret-patch.yaml

The patch file can modify the secret's data field to contain environment-specific values without duplicating the entire resource definition.

Handling Secret Rotation

Rotating secrets in a GitOps workflow requires updating the source of truth in Git and ensuring controllers detect and propagate changes. A typical rotation involves:

  1. Updating the secret value in the repository
  2. Committing the change with descriptive message including "secrets rotation"
  3. Verifying controller detects and applies changes

The command kubectl get secret app-secrets -n production -o yaml displays current values, showing the base64-encoded data that must be updated consistently.

Error Conditions for Secret Synchronization

Failed secret synchronization produces errors like:

Error: unable to sync secrets: secret 'app-secrets' not found in namespace 'production'

Or when decoding fails:

Error: failed to decode secret data: illegal base64 data at input byte 0

These errors typically appear in controller logs or through status reporting mechanisms.

Managing Encrypted Secrets in Git

Using External Secret Managers

Secrets can be stored encrypted in Git using tools like SOPS, Sealed Secrets, or Vault. A typical workflow with SOPS involves:

# Encrypting a secret file
sops --encrypt --encrypted-regex '^(data|stringData)$' secret.yaml > secret.sops.yaml

The resulting file contains both encrypted and plaintext sections:

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: production
data:
  database_password: ENC[AES256_GCM,data:...,iv:...]
  api_key: ENC[AES256_GCM,data:...,iv:...]

Decrypting Secrets During Deployment

When using encrypted secrets, controllers must handle decryption before applying to Kubernetes. This requires proper configuration of the controller's access to encryption keys or vault credentials.

The error message when decryption fails:

Error: failed to decrypt secret data: unable to decrypt with any key

Handling Key Rotation

Rotating encryption keys requires updating all secrets encrypted with old keys. The command:

sops --decrypt secret.sops.yaml | sops --encrypt --output secret.yaml -

Updates the entire file content to use new encryption parameters.

Version Control Considerations

Committing Secrets to Repository

Secrets should be committed to Git with clear naming conventions and commit messages that indicate purpose and ownership. The pattern:

git add secret.yaml
git commit -m "feat(secrets): update database password for production"

Managing Secret Lifecycles

When secrets are no longer needed, they must be removed from the repository and Kubernetes clusters. This involves:

  1. Deleting the secret resource from Git
  2. Committing the deletion
  3. Verifying controller removes the secret from cluster

The command kubectl delete secret app-secrets -n production removes the resource from the cluster.

Avoiding Secret Exposure

When secrets are accidentally committed to Git, the recovery process requires:

  1. Removing the file from Git history using git filter-branch
  2. Rotating the exposed secrets
  3. Committing new values to repository

Handling Large Secrets

Kubernetes has limits on secret size (1MB). When exceeding this limit, use external storage like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: large-config
data:
  config.properties: |
    key1=value1
    key2=value2

Configuration and Controller Setup

Required Permissions for Secret Access

Controllers must have appropriate RBAC permissions to manage secrets:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch"]

Monitoring Secret Status

Controllers provide status information about secret synchronization. The command kubectl get secrets shows current state, while controller-specific commands may show:

kubectl get application my-app -o jsonpath='{.status.sync.status}'

Operational Patterns and Best Practices

Using Git Submodules for Secrets

Some teams use Git submodules to manage secrets separately from main code repositories:

git submodule add https://github.com/myorg/secrets.git secrets/

This pattern separates secrets from application code, enabling different access controls.

Implementing Secret Validation

Validation can be performed before committing secrets by using:

kubectl apply --dry-run=client -f secret.yaml

This verifies that the secret structure is valid before applying to cluster.

Handling Secrets in CI/CD Pipelines

In automated pipelines, secrets must be available during deployment steps:

steps:
  - name: Deploy Secrets
    run: |
      kubectl apply -f secret.yaml
      kubectl rollout status deployment/my-app

Common Failure Scenarios

Secret Decryption Failures

When controllers cannot decrypt secrets, they typically log errors about missing keys or incorrect encryption format. The most common failure mode is:

Error: failed to create secret from file: unable to read encrypted data

This often results in deployment stalls where the controller waits for valid secret data.

Permission Issues

Controllers without proper permissions cannot access secrets:

Error: secrets is forbidden: User "system:serviceaccount:argocd:argocd-application-controller" cannot get secrets in the namespace "production"

Base64 Encoding Errors

Incorrect base64 encoding produces errors during secret creation or updates. The error:

Error: failed to create secret: invalid data format

Typically occurs when values are not properly encoded.

Secret Synchronization Delays

Controllers may take time to synchronize secrets due to polling intervals. This delay can cause:

Warning: secret not yet applied to cluster

Multi-Environment Consistency Issues

When environments have different secret requirements, inconsistency manifests as:

Error: secret data mismatch between Git and cluster

The most critical failure mode occurs when a controller applies an outdated secret while newer versions are already committed to Git, causing drift between source of truth and actual deployment state.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.