Implement advanced GitOps workflows for multi-cluster, multi-environment deployments. Covers ArgoCD ApplicationSets, Flux Kustomizations, progressive delivery, drift detection, secret management, and the operational patterns that make GitOps reliable at scale.
GitOps is a workflow where the desired state of a system is declared in a version-controlled repository, and a continuous reconciliation loop ensures that the actual system state matches the declared state. It matters when teams need consistent, auditable, and repeatable deployments across multiple environments, especially in cloud-native, microservices-based systems where infrastructure and applications are tightly coupled.
The foundation of GitOps is expressing system state—infrastructure, services, and configuration—as declarative YAML manifests. These manifests define what the system should look like, not just how to build it.
Each deployment artifact is a set of Kubernetes manifests: Deployment, Service, ConfigMap, Secret, Ingress, and CustomResource definitions. These are stored in a Git repository, typically under manifests/ or environments/<env>/.
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: production
spec:
replicas: 3
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:7.2-alpine
ports:
- containerPort: 6379
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "200m"Changes to this file are committed with a descriptive message:
git add manifests/redis.yaml
git commit -m "feat(redis): increase replicas to 3 for production"
git push origin mainThe Git repository becomes the single source of truth for the system’s configuration.
A GitOps controller watches the Git repository and applies changes to the target cluster. It continuously reconciles the desired state from Git against the actual state in the cluster.
The controller performs the following loop:
kubectl apply -f manifests/.This loop runs every 5 minutes by default, but can be tuned via a reconcileInterval configuration key.
When a new commit is pushed, the controller detects the change, applies the manifests, and records the reconciliation in a status file or event log.
{
"commit": "a1b2c3d4e5f67890",
"timestamp": "2024-05-10T12:34:56Z",
"status": "reconciled",
"changes": [
{
"resource": "Deployment/redis",
"kind": "Deployment",
"namespace": "production",
"action": "updated",
"from": "replicas: 2",
"to": "replicas: 3"
}
],
"errors": []
}A reconciliation failure is logged as an event in the cluster:
kubectl get events -n gitops-controllerOutput:
LAST SEEN TYPE REASON OBJECT MESSAGE
2m Warning ReconcileFailed manifest/redis.yaml Failed to apply manifest: Error: failed to parse YAML at manifests/redis.yaml: line 15: expected '}' but got '}' in mapThe error indicates a malformed YAML file, but the controller does not stop after the first failure. It continues applying other manifests, and later logs the full set of issues.
The workflow becomes iterative when deployments are driven by pull requests (PRs) on the Git repository.
A developer creates a feature branch, modifies a manifest, and opens a PR:
git checkout -b feature/add-user-service
# edit manifests/userservice.yaml
git add manifests/userservice.yaml
git commit -m "feat(userservice): add user service with PostgreSQL"
git push origin feature/add-user-serviceThe PR includes a description, and CI runs tests on the changes. A GitOps controller automatically applies the PR’s manifest changes to a staging environment.
Upon PR merge, the controller deploys the changes to production. This is known as merge-triggered deployment.
To enable this, the controller monitors the main branch and watches for merge events. It applies the main branch’s manifests to the production cluster.
# gitops-controller.yaml
controller:
source:
type: git
url: https://git.example.com/infra.git
branch: main
path: manifests/
reconciliation:
interval: 5m
strategy: merge-triggered
environments:
- name: staging
namespace: staging
branch: feature/*
- name: production
namespace: production
branch: mainThe controller applies the staging manifests when a new commit appears in the feature/* branches, and applies production manifests when main is updated.
Canary deployments are implemented by managing multiple, overlapping manifests in Git. Each canary release is represented by a separate set of manifests in a canaries/ directory.
# manifests/canaries/v2.1.0/userservice.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: userservice
namespace: production
annotations:
gitops.canary: "true"
canary.version: "2.1.0"
spec:
replicas: 1
selector:
matchLabels:
app: userservice
version: v2.1.0
template:
metadata:
labels:
app: userservice
version: v2.1.0
spec:
containers:
- name: userservice
image: registry.example.com/userservice:v2.1.0
ports:
- containerPort: 8080The GitOps controller detects that a canaries/ directory exists. It applies all canary manifests in parallel with the main deployment.
After a week, the team promotes the canary release to full production by merging the canaries/v2.1.0/ directory into manifests/:
git checkout main
git merge canaries/v2.1.0
git push origin mainThe controller detects the merge, applies the full set of manifests/ including the new canary version, and scales it from 1 to 10 replicas.
A critical failure mode is canary drift: the canary manifests are not updated in sync with the main deployment. A service may be running v2.1.0 in canary, but the v2.1.0 image is not yet available in the image registry.
The controller logs a warning:
kubectl describe deployment userservice -n productionOutput:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ImagePullFailed 3m kubelet Failed to pull image "registry.example.com/userservice:v2.1.0": rpc error: code = Unknown desc = Error response from daemon: manifest unknownThis indicates that the image tag v2.1.0 was not pushed, or the Dockerfile was not rebuilt after the manifest change.
Rollbacks are trivial in GitOps: simply revert a commit.
git revert HEAD
git push origin mainThe GitOps controller detects the new commit, applies the previous version of the manifest, and reconciles the cluster.
The controller compares the desired state at the reverted commit with the current state in the cluster and applies only the necessary updates.
If a rollback fails, the controller logs a RollbackFailed event:
{
"commit": "f4e3d2c1b0a98765",
"timestamp": "2024-05-10T13:45:22Z",
"status": "rollback_failed",
"details": [
{
"resource": "Deployment/userservice",
"action": "rollback",
"from": "v2.0.0",
"to": "v1.9.0",
"error": "failed to apply manifest: Error: Deployment.apps/userservice not found"
}
]
}This indicates that the userservice Deployment was missing during the rollback, likely due to a configuration error in the v1.9.0 manifest.
Configuration changes are managed as separate manifests. A config/ directory holds configuration for services, with environment-specific overrides.
# config/staging/database.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: database-config
namespace: staging
data:
DB_HOST: "postgres-staging.example.com"
DB_PORT: "5432"
LOG_LEVEL: "debug"When a team wants to change the log level in staging, they modify the file:
data:
DB_HOST: "postgres-staging.example.com"
DB_PORT: "5432"
LOG_LEVEL: "info"They commit and push:
git add config/staging/database.yaml
git commit -m "config: set log level to info in staging"
git push origin mainThe controller applies the updated ConfigMap and triggers a rolling update of all services that consume it.
A subtle failure occurs when a service consumes a ConfigMap but does not restart after a change.
# manifests/userservice.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: userservice
namespace: production
spec:
template:
spec:
containers:
- name: userservice
image: registry.example.com/userservice:v2.0.0
envFrom:
- configMapRef:
name: database-config
resources:
requests:
memory: "256Mi"
cpu: "200m"The controller applies the ConfigMap but does not restart the pods. The service continues running with the old LOG_LEVEL until it is manually rolled or the controller adds a rollingUpdate strategy.
To fix this, the controller must detect when a ConfigMap is updated and trigger a rolling update of all referencing deployments.
The controller queries the cluster:
kubectl get configmap database-config -n staging -o yamlOutput:
apiVersion: v1
kind: ConfigMap
metadata:
name: database-config
namespace: staging
annotations:
gitops.reconcile: "true"
last-applied: "2024-05-10T13:45:22Z"
data:
DB_HOST: "postgres-staging.example.com"
DB_PORT: "5432"
LOG_LEVEL: "info"The last-applied annotation is set during reconciliation. If the controller sees that a ConfigMap has been updated since the last reconciliation, it applies the changes and schedules a rollout.
GitOps scales to multiple environments by organizing the repository into environment-specific directories.
manifests/
├── base/
│ ├── crds/
│ └── kustomization.yaml
├── staging/
│ ├── kustomization.yaml
│ ├── config/
│ └── overlays/
│ └── production/
│ ├── kustomization.yaml
│ └── patches/
└── production/
├── kustomization.yaml
├── config/
└── overlays/
└── canary/
├── kustomization.yaml
└── patches/The controller applies base/ first, then applies staging/ overlays, and then production/ overlays.
The kustomization.yaml files define the composition:
# manifests/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
patchesStrategicMerge:
- config/database.yaml
- config/logging.yaml
configMapGenerator:
- name: app-config
files:
- config/app.yamlWhen a change is merged to main, the controller applies the production manifests. A new PR to staging updates the staging cluster with the latest changes.
A common error is missing patchesStrategicMerge in the kustomization.yaml file, leading to a configuration drift where the staging environment diverges from the production baseline.
The controller logs this as a KustomizationError:
{
"error": "KustomizationError",
"message": "missing patchesStrategicMerge in kustomization.yaml",
"details": [
"expected: patchesStrategicMerge: [config/database.yaml]",
"actual: patches: [config/database.yaml]"
]
}This error is silent until a service fails to start, because the database.yaml patch was not applied.
image field in a manifest points to a tag that is not present in the registry.canaries/ directory is not updated in sync.patchesStrategicMerge is misspelled as patches, leading to missing patches.The GitOps workflow is not just a sequence of commands—it is a system of continuous reconciliation, driven by Git and enforced by declarative configuration. It is the practice of letting the repository define the world.
This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy