GitOps Workflow Patterns | The Garnet Wiki

GitOps Workflow Patterns

GitOps treats a Git repository as the single source of truth for declarative infrastructure and application state. When desired configuration lives in git and a controller continuously reconciles the actual cluster to match that state, the team is operating a GitOps workflow. This matters when you want to enforce immutability, audit every change, and automate delivery without manual kubectl scripts—or when you’re evaluating whether your current “push-to-deploy” process can scale past a handful of services. The patterns below are organized by what you’re trying to accomplish, with concrete commands, flags, and the sharp edges you’ll only learn after a production incident.

Establishing Git as the Source of Truth and Initial Sync

If you’re starting from an existing cluster and need to capture its state into git, or bootstrapping a new cluster from a git manifest, the goal is a one-time export followed by automated reconciliation.

Export existing cluster state

  kubectl get all,cm,svc,ing -A -o yaml > manifests/all.yaml

Review the output carefully; kubectl get includes auto-generated fields (e.g., creationTimestamp, uid) that are not idempotent if committed verbatim.

Bootstrap a cluster from git

  helm install my-app ./charts/my-app --namespace prod --create-namespace

After the initial install, the GitOps controller (whatever brand) watches the repo and runs kubectl apply -f on every commit merge. The pattern is: edit manifests → commit → PR merge → webhook fires → controller reconciles.

Idempotent re-application

  kubectl apply -f k8s/ -R

apply tracks the last-applied annotation; if the manifest on disk hasn’t changed, no action is taken. This is the baseline unit of reconciliation in most GitOps systems.

Common first‑failure mode

  error: the server could not find the requested resource

This usually means a resource kind (e.g., Ingress, Crd) isn’t available in the target cluster, or the namespace was omitted from the manifest header.

Branch‑Driven Development and Pull‑Request Workflows

Teams that want every infrastructure or configuration change to go through code review adopt a branch‑PR pattern. This mirrors application development but applies to Helm charts, Kustomize bases, or raw YAML.

Feature branch workflow

  git checkout -b feat/update-api-version
  # edit values.yaml or a kustomize patch
  git add manifests/
  git commit -m "api: bump image tag to v2.4.1"
  git push origin feat/update-api-version

Open a PR. Required CI checks typically include:

  helm lint ./charts/my-app
  kustomize build .  # validates structural correctness
  kubectl apply --dry-run=client -f rendered-manifest.yaml  # dry-run check

Merge‑triggered sync

Once the PR is approved and merged, the GitOps controller’s webhook fires. In Argo CD or Flux, this is a git sync that runs kubectl apply against the merged commit SHA. The controller does not re‑run from HEAD; it syncs the exact commit that was merged, ensuring reproducibility.

Helm upgrade on merge

If using Helm within GitOps, the merge can trigger an upgrade with atomic rollback:

  helm upgrade --install my-app ./chart \
    --namespace prod \
    --set image.tag=v2.4.1 \
    --atomic \
    --wait \
    --timeout 5m

The --atomic flag ensures the release is rolled back entirely if the post‑upgrade hooks fail; without it, a partially deployed release may remain in PENDING_UPGRADE state.

Gotcha: binary merge of YAML/Helm files

Git’s default text merge will corrupt multi‑line Helm template blocks. Use a .gitattributes entry or a merge driver that invokes helm template to resolve conflicts intelligently:

  # .gitattributes
  *.yaml merge=helm

Without this, a developer resolving a conflict may accidentally keep both replicaCount: 3 and replicaCount: 5, leading to silently mismatched state.

Automated Reconciliation and Drift Detection

Over time, manual kubectl edit commands, API server changes, or external configuration management can push the cluster away from the git‑desired state. Detecting and remediating drift is a core GitOps workflow pattern.

Dry‑run diff before commit

  kustomize build . | kubectl apply --dry-run=client -f -

This prints a summary of resources that would be created, updated, or deleted, without touching the cluster.

Explicit diff against rendered manifest

  kubectl diff -f prod-kustomize/

(Provided by kustomize 5+ or third‑party plugins.) Shows which fields would change if apply were run now.

Reconciling after drift is detected

  kubectl apply -f prod-kustomize/

If the controller’s auto‑sync is disabled or throttled, you may need to trigger it manually:

  flux sync  # Flux CLI
  argo app sync my-app  # Argo CD CLI

Drift‑only notification

Many teams install a Prometheus alert or Flux notification hook that fires when flux get kustomization reports STATUS: ProgressiveSyncDegraded or SYNCED after a failed reconciliation. The error text typically includes:

  failed to reconcile: admission webhook denied the change: policy violation

This usually means a validating webhook (e.g., OPA Gatekeeper, Kyverno) rejected the manifest change. The workflow pattern here is: fix the manifest to satisfy the policy, commit, and let the controller retry.

Sharp edge: aggressive auto‑sync masking manual changes

If a controller reconciles every 20 seconds and you’ve manually edited a Deployment’s replica count via kubectl edit, the change will be overwritten on the next sync. There is no implicit “merge conflict” warning—the cluster simply snaps back to git state. The pattern to avoid this: use a dedicated branch for hot‑fixes, or gate manual edits behind a feature‑flag annotation that the controller respects.

Progressive Delivery (Canary, Blue‑Green) via Git Declarative Manifests

GitOps is frequently used to orchestrate gradual traffic shifts. The pattern is: change the manifest in git, and the controller (or an adjacent service mesh driver) enacts the rollout strategy.

Canary via Helm value override

Edit values.yaml:

  # values.yaml
  canary:
    enabled: true
    replicaCount: 2
    image:
      tag: v2.3.0

Commit and merge. The Helm controller releases a new Release with strategy.type: RollingUpdate and the configured number of pods. Traffic routing (e.g., via Ingress, Service, or service mesh) is what actually splits production traffic; GitOps only ensures the correct pod versions are running.

Kustomize strategic merge patch for canary

Create a patch file overlays/canary/patch.yaml:

  # overlays/canary/patch.yaml
  resources:
    - ../base
  patchesStrategicMerge:
    - deployment.yaml
  configMapGenerator:
    - name: app-config
      literals:
        - CANARY=true

Deploy with:

  kubectl apply -k overlays/canary/

The controller reconciles the overlay, and if you have an ingress controller with canary routing (e.g., NGINX canary annotation), you can shift percentages without further git changes.

Blue‑green via separate Helm releases

Maintain two Helm releases, my-app-blue and my-app-green, each with its own values file and image tag. A DNS or ingress switch flips traffic from one release to the other. The GitOps pattern is: merge the new release’s manifest, verify health, then update the ingress to point at the new release name.

Common failure: stuck canary pods

If the canary replica count is set but pods remain ImagePullBackOff, the error text will be:

  Warning  FailedMount  12s  kubelet  Failed to mount...

This usually means the image tag in the canary overlay doesn’t match a digest in the registry, or the image pull secret expired. The GitOps workflow offers no automatic retry for credential failures; the controller will keep reporting SYNC_FAILED until the secret is rotated and the commit is pushed again.

Sharp Edges: What Silently Fails and What You Learn Too Late

Pattern	Silent Failure	What You Learn After It Breaks
`kubectl apply -f` after manual `edit`	Manual change overwritten on next reconcile; no error logged beyond controller `SyncCompleted`	You notice missing replicas or wrong image tag after a deployment window
Git merge of binary Helm/YAML files	Corrupted manifest committed; `apply` reports `error: resource mapping mismatch`	Production rollout fails; you discover the merge kept both old and new `replicaCount` values
Helm `--atomic` upgrade	If pre‑upgrade hooks fail, release stays in `PENDING_UPGRADE`; `helm status` shows no error, just “pending”	Team spends hours debugging “why isn’t my new version running?” only to find the rollback flag saved them
Controller webhook throttling	Sync appears successful but skips resources; drift accumulates unnoticed	Audit reveals cluster is running v1.2 manifests while git has v1.5; no alert fired because webhook returned `200` on a partial match
Image pull secret rotation without re‑sync	Pods restart in a loop; `kubectl get pods` shows `CrashLoopBackOff`	You check logs and see `token expired`; the git commit that updated the secret never triggered a controller sync because the webhook endpoint was misconfigured

Key takeaway: GitOps patterns are only as reliable as the reconciliation loop that watches the repo. Every pattern above assumes a controller (Flux, Argo CD, Weaveworks, or a custom operator) is watching the default branch and running kubectl apply (or equivalent) on every merge. If that loop is paused, rate‑limited, or mis‑configured, the “single source of truth” claim breaks silently. Always verify the controller’s sync status with flux get kustomization or argo app get <app> before and after merges, and treat the webhook/credential configuration as critical infrastructure, not a “set‑and‑forget” component.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.