Gitops Automation Patterns | The Garnet Wiki

GitOps automation patterns are the recurring workflows and structural decisions that turn a declarative configuration repository into a self-healing, continuously deployed system. They matter when infrastructure and application configuration must be versioned, tested, and applied automatically in response to changes in a Git repository, particularly in cloud-native environments where services are distributed, stateful, and frequently updated.

Declarative Configuration as a Single Source of Truth

In GitOps, the desired state of a system is captured in a Git repository, typically in a manifests/ directory or a config/ tree. Every change to configuration—whether it’s a new service, a version bump, or a scaling policy—manifests as a Git commit. The system continuously reconciles the actual state with the declarative state, applying drift corrections and propagating changes.

Apply Configuration on Push

When a developer pushes a change to the main branch, the system must detect the change and apply the updated configuration. This is the core loop.


git commit -m "Increase frontend replica count to 6"
git push origin main

The pipeline must:

Fetch the latest Git revision.
Render manifests using a templating engine (e.g., Helm, Kustomize, or plain YAML).
Apply the manifests to the target cluster.

# Example: Render Helm chart and apply with kubectl
helm template app --set replicaCount=6 --output-dir ./rendered
kubectl apply -k ./rendered

A failure here is silent: if helm template fails but the pipeline continues, the rendered manifests may be incomplete or outdated. Common issues include:

Missing --set values in Helm (e.g., --set image.tag=latest missing).
kubectl apply failing due to a timeout or a resource conflict, but the pipeline logs show only kubectl apply without verbosity.
kubectl not using --wait or --timeout, so the pipeline proceeds before the rollout is complete.

Reconcile the System with a Git-Driven Loop

Reconciliation is the process of continuously checking that the actual state matches the desired state. It runs in a loop, polling the Git repository and applying updates as needed.

The loop typically executes every 30 seconds and performs:

Fetch latest Git state.
Compute diff between current and desired state.
Apply changes using a declarative tool.

# Example reconciliation loop
while true; do
  git pull origin main
  kubectl diff -k ./manifests
  kubectl apply -k ./manifests
  sleep 30
done

This loop can be run as a Kubernetes Job, a CronJob, or a long-running Pod. A common failure mode is that kubectl diff returns a diff, but kubectl apply applies only a subset of changes, especially when the diff output is piped into apply without --filename -.

To reduce noise, use --prune and --force in apply:

kubectl apply -k ./manifests --prune --force

--prune removes resources not present in the current manifest, while --force triggers a rolling update even if no changes are detected. The difference between --prune and --prune=true is subtle but critical: --prune enables pruning without a value, while --prune=true explicitly sets it. Misunderstanding this leads to orphaned resources that accumulate over time.

Automate Rollouts with Rollout Triggers

Rollouts in GitOps are not just one-time deployments. They are orchestrated sequences of events triggered by Git changes.

Versioned Rollouts via Git Tags

When a new version of an application is released, a Git tag is created, and the system triggers a rollout.

# Create a new release tag
git tag v1.2.3
git push origin v1.2.3

The automation must:

Fetch the tagged revision.
Render manifests for that version.
Apply the manifests to a staging or production environment.

A common oversight is not setting the image.tag in Helm or Kustomize to match the Git tag:

# helm/values.yaml
image:
  repository: myapp
  tag: {{ .Chart.Version }}

If the Chart.yaml version is 1.2.3, but the Helm chart is not built and pushed before tagging, the tag field is 1.2.3 but the image is myapp:latest. This leads to a mismatch between the Git version and the deployed image.

To fix this, the pipeline must explicitly build and push the image:

docker build -t myapp:1.2.3 .
docker push myapp:1.2.3
helm template app --set image.tag=1.2.3 --output-dir ./rendered

A silent failure occurs when docker build is not using --pull and --cache-from, so the image is built from a stale local cache, missing updates from upstream layers.

Rolling Updates with Canary and Blue-Green Patterns

For zero-downtime rollouts, GitOps systems use canary or blue-green strategies.

A canary rollout begins with a small fraction of traffic routed to the new version, verified by metrics, and then gradually increased.

# manifests/rollout-canary.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: frontend
spec:
  replicas: 6
  selector:
    matchLabels:
      app: frontend
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: {}
        - setWeight: 50
        - pause: { duration: 300s }
        - setWeight: 100
      canaryAnalysis:
        templates:
          - name: latency
            templateRef:
              name: latency-analysis
              apiVersion: argoproj.io/v1alpha1
        args:
          - name: duration
            value: 600s

The system monitors the rollout and pauses when a metric (e.g., latency) exceeds a threshold. If the analysis fails, it rolls back to the previous version.

A failure mode: canaryAnalysis runs but the templateRef is not properly defined in the cluster. The AnalysisTemplate must exist and be labeled correctly. A missing argoproj.io/analysis-template: true label leads to a silent failure—analysis runs, but no results are recorded.

Automate Environment Promotions

Promoting configuration from one environment to another—e.g., from dev to staging to prod—is a common pattern.

Environment-Specific Configuration via Branching

Each environment has its own branch: dev, staging, main.

# Promote from dev to staging
git checkout staging
git merge dev
git push origin staging

The promotion triggers a pipeline that:

Builds and pushes images.
Renders manifests for the target environment.
Applies the configuration to the target cluster.

A common mistake is not merging the staging branch back into main after a successful promotion, leading to divergent configurations.

To automate this, use a promote script:

#!/bin/bash
set -euo pipefail

BRANCH=$1
TARGET=$2

git checkout $TARGET
git merge $BRANCH
git push origin $TARGET

# Apply to cluster
kubectl apply -k ./manifests/$TARGET

This script must handle merge conflicts and run helm lint and kubectl apply --dry-run.

A subtle failure: kubectl apply runs in a non-interactive context (e.g., CI), but --dry-run produces output that is not captured, so the pipeline logs show no changes, but the actual apply is not executed.

Cross-Environment Drift Detection

Drift between environments is detected by comparing the state of dev, staging, and prod clusters.

# Detect drift between dev and staging
kubectl diff -k ./manifests/dev -k ./manifests/staging

The output is a list of differences—missing resources, mismatched fields, different image tags.

To automate drift detection, create a nightly job:

# drift-detection.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: drift-detection
spec:
  template:
    spec:
      containers:
        - name: drift-check
          image: alpine:latest
          command:
            - sh
            - -c
            - |
              kubectl diff -k ./manifests/dev -k ./manifests/staging
              kubectl diff -k ./manifests/staging -k ./manifests/main
              kubectl diff -k ./manifests/dev -k ./manifests/main
          volumeMounts:
            - name: manifests
              mountPath: /manifests
      volumes:
        - name: manifests
          configMap:
            name: git-manifests
      restartPolicy: OnFailure

A common issue: the diff output is not written to a file, so it appears only in the job logs, but not in a shared location. The system must explicitly write the diff output to a file or upload it to a storage system.

Automated Promotion with Pull Request Review

Promotions are often gated by a pull request.

# Example PR: promote from dev to staging
git checkout -b feature/new-ui
# make changes
git push origin feature/new-ui

A pull request is opened. The CI runs tests and applies the configuration to a temporary cluster.

A reviewer approves the PR. The automation must:

Merge the PR.
Apply the configuration to staging.
Wait for a health check.
Promote to production if successful.

The automation uses kubectl get pods -l app=frontend --watch to wait for a service to be ready.

A failure occurs when the --watch flag is not used, so the command exits immediately after the first get, missing a new pod rollout.

The automation must also check for readiness probes and liveness probes.

# Wait for readiness
kubectl wait --for=condition=ready pod -l app=frontend --timeout=300s

If the --timeout is not set, the wait can hang indefinitely, especially in high-latency clusters.

Handle Configuration Drift and Rollback

Configuration drift occurs when the actual state diverges from the desired state. GitOps systems detect and correct this drift automatically.

Detect and Apply Drift

The system runs a drift detection loop that:

Compares the current cluster state with the Git-manifests state.
Applies changes if the state differs.

# drift-detection.sh
#!/bin/bash
set -euo pipefail

cd /app

git pull origin main

diff_output=$(kubectl diff -k ./manifests --output yaml)
if [[ -n "$diff_output" ]]; then
  echo "$diff_output" | tee /var/log/drift.log
  kubectl apply -k ./manifests
  git add .
  git commit -m "Apply drift: $(date)"
  git push origin main
fi

A silent failure: diff_output is empty, but kubectl diff returns no output. This happens when --output yaml is used but the diff is not properly formatted.

Automated Rollback on Failure

When a rollout fails, the system rolls back to the previous configuration.

A common pattern: on failure, the system:

Records the failure in a log.
Restores the previous configuration.
Notifies stakeholders.

# rollback.sh
#!/bin/bash
set -euo pipefail

BRANCH=$(git rev-parse --abbrev-ref HEAD)
PREV_COMMIT=$(git rev-parse HEAD~1)

git checkout $PREV_COMMIT
kubectl apply -k ./manifests
git add .
git commit -m "Rollback to $PREV_COMMIT due to failure"
git push origin $BRANCH

The system must also record the reason for rollback.

A failure mode: the kubectl apply command is not using --prune, so resources are left behind but not updated. The rollback is incomplete.

Configuration Validation and Pre-Apply Checks

Before applying a configuration, the system validates it.

# validate-manifests.sh
#!/bin/bash
set -euo pipefail

# Validate Helm charts
helm lint ./charts

# Validate Kubernetes manifests
for file in manifests/*.yaml; do
  kubectl apply -f "$file" --dry-run=server
done

# Check for common issues
kubectl get crd --no-headers | awk 'NF == 0 { print "No CRDs defined" }'

A common error: kubectl apply --dry-run=server returns Error: cannot apply:, but the error is not captured. The system must use --output json and parse the output.

A silent failure: kubectl get crd returns no output, but the awk command does not print anything. The NF == 0 condition is not triggered because get crd outputs a header row but no data.

Summary

GitOps automation patterns are not a set of tools but a set of workflows that transform a Git repository into a self-managing system. The patterns described—declarative configuration, reconciliation loops, versioned rollouts, environment promotions, drift detection, and automated rollbacks—are the core mechanisms that enable continuous delivery and operational resilience. Each pattern comes with subtle failure modes, precise commands, and critical flags. Mastering these patterns means building a system that not only deploys but also observes, corrects, and evolves itself.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.