Gitops Implementation Guide

GitOps treats a git repository as the single source of truth for declarative infrastructure and application configuration. When your cluster runs workloads that you want version‑controlled, auditable, and automatically reconciled, GitOps provides a repeatable pattern: every change lives as a commit, review happens via pull request, and a sync controller reconciles the cluster state to match git. This matters whenever you need audit trails, zero‑downtime rollouts, or the ability to restore a known‑good state without manual scripts, and it becomes especially valuable when managing multiple clusters or fleets where manual kubectl drift is a constant risk.

Initialize a GitOps repository

Start with a bare git repo that will hold your desired‑state manifests. Standard tools suffice—no vendor‑specific scaffolding is required.

git init gitops-repo
cd gitops-repo
git config user.name "Ops Team"
git config user.email "[email protected]"

Create a minimal kustomize layout. kustomize is an open‑source tool that many GitOps controllers support out of the box.

kustomize init --resourcePrefix app

You’ll now have a kustomization.yaml that references any base resources you place alongside it. Add a sample k8s manifest to prove the loop works:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: nginx:1.25
          ports:
            - containerPort: 80

Commit and push:

git add deployment.yaml kustomization.yaml
git commit -m "initial kustomize base with my-app deployment"
git branch -M main
git remote add origin [email protected]:your-org/gitops-repo.git
git push -u origin main

Declare desired state in git

Your repo should contain k8s manifests that describe every resource the sync agent needs to manage. Use standard apiVersion/kind blocks; the syntax is identical to what you’d pass to kubectl apply -f.

If you prefer Helm charts, you can store them alongside kustomize bases. A real helm template invocation looks like this:

helm template my-release ./charts/my-app \
  --set image.tag=1.25 \
  --namespace demo

Both approaches produce valid declarative yaml that a GitOps sync loop can ingest. The key invariant is: the file on disk is the source of truth. Anything not tracked in git is, for the purposes of GitOps, out of scope.

Connect the cluster and activate sync

With manifests in git, the next step is to point a reconciliation loop at the repository. Most GitOps implementations share a common pattern: the agent is configured with a git host, a branch name, and a path prefix. When a commit appears under that path, the agent runs kubectl apply (or an equivalent API write) for every manifest that changed, then watches the cluster until the observed state matches the declared state.

Apply the base from the repo you just pushed:

kubectl apply -k ./ -n demo

Verify the resources came up:

kubectl get all -n demo

You should see the my-app Deployment with 2 ready replicas. The sync agent’s reconciliation loop (whether built into your chosen tool or a custom watcher) will now keep this state in sync with future commits.

Everyday workflow: PR‑driven changes

The core GitOps loop is a pull‑request based workflow. To change something—say, bump the image tag or scale the replica count—you never kubectl edit directly. Instead:

Create a topic branch from main:

   git checkout -b increase-replicas

Edit the manifest:

   # deployment.yaml (modified)
   spec:
     replicas: 4

Commit and push:

   git add deployment.yaml
   git commit -m "increase replicas to 4 for load‑test"
   git push origin increase-replicas

Open a pull request. Reviewers check the diff—they’re reviewing infrastructure changes the same way they’d review code changes.

After merge, the sync agent detects the new commit on main under the tracked path and triggers kubectl apply -k ./ -n demo (or its equivalent). The Deployment is patched to 4 replicas.

Confirm the rollout:

   kubectl rollout status deployment/my-app -n demo

Output will show deployment "my-app" successfully rolled out to 4 available replicas.

If your environment uses a sealed‑secret or external‑secrets controller, the same PR workflow applies: secret manifests are version‑controlled, and the secret‑sync component reconciles them to the cluster when the PR merges.

Verify and validate after sync

After any merge, it’s good practice to confirm that the cluster actually reflects the git state. A few standard commands cover most cases:

# List running pods and their readiness
kubectl get pods -l app=my-app -n demo -o wide

# Show the current replica count declared in the API server
kubectl get deployment my-app -n demo -o jsonpath='{.status.replicas}'

# Inspect the rollout history if you need to audit changes
kubectl rollout history deployment/my-app -n demo

If any resource reports 0/2 READY or the rollout history shows unexpected pauses, kubectl describe deployment my-app -n demo will surface events such as Failed to pull image or BackOff pulling image. Those events are emitted by the kubelet and are not specific to any GitOps product—they’re the same messages you’d see after a manual kubectl apply.

Sharp edges

Missed sync loops

The most common silent failure is a sync agent that never receives the “new commit” signal. This happens when:

The agent is pointed at a stale branch reference or a disabled webhook.
Branch‑protection rules reject the merge commit (e.g., required reviewers are missing), so the default branch never advances.
The agent’s clock or watch interval is misconfigured, causing it to poll less frequently than the team’s merge cadence.

Recovery: Verify the agent’s configuration (branch name, path prefix, auth token). Force a manual sync if the product supports a sync now endpoint, or simply git fetch && git checkout main on the agent host and trigger kubectl apply -k . locally to close the gap.

Drift from direct `kubectl` edits

When a team member runs kubectl set replicas=5 deployment/my-app outside a PR, the cluster state diverges from git. The next sync attempt will either:

Reconcile (if the agent is configured with prune: true or equivalent), which will force the deployment back to the git‑declared replica count, potentially cutting pods.
Report a conflict (if the agent is conservative), leaving the resource in a “out‑of‑sync” state and emitting an event like resource versions diverge.

Recovery: Treat drift as a configuration error. The standard fix is to git commit the intended change, merge, and let the sync loop bring the cluster into alignment. If you need a temporary override, document it in a separate “ops” branch and merge back when ready.

Image‑tag automation pitfalls

Using kustomize edit set image app=my-app:latest can inadvertently promote an untagged or dev build into production if the latest tag is being force‑pushed by CI. Because the sync agent watches git and applies the change instantly, a bad latest tag propagates to all replicas before anyone notices.

Mitigation: Pin image versions in manifests (e.g., image: my-registry/my-app@sha256:abc…) and let CI push new‑version manifests with explicit tags. Avoid latest in GitOps‑managed Deployments.

Secret handling gotchas

Manifests that reference secretKeyRef without a sealed‑secret or external‑secrets wrapper will either:

Fail to sync if the agent lacks permissions to read the secret namespace.
Leak cleartext into git history if the secret YAML is committed verbatim.

Pattern: Store only the secret name in git; use a dedicated secrets‑sync tool that reads from a vault or KMS and creates the k8s Secret object. Never commit stringData or binaryData blocks directly unless your policy explicitly allows it and you understand the audit trail.

RBAC under‑provisioning

A sync agent needs get, list, watch, create, update, and delete (or patch) on the resource kinds it manages. If the service account attached to the agent has only get and list, you’ll see errors like the server could not find the requested resource or forbidden: rolebinding.rbac.authorization.k8s.io "gitops-sync" not found when the agent tries to patch a Deployment.

Fix: Deploy the agent with a ClusterRole that scopes the necessary verbs to the target namespace (or namespaced Role for multi‑tenant clusters). Test with kubectl auth can-i create deployments -n demo --as=system:serviceaccount:default:gitops-sync before enabling the full loop.

Production breakage from accidental `replicas: 0`

Committing a manifest where spec.replicas: 0 causes the sync agent to immediately scale the Deployment to zero. Because the reconciliation loop runs on every commit, the outage is instantaneous and can affect user‑facing services.

Recovery: The only safe rollback is a new commit that restores replicas: <desired> and lets the sync agent apply it. Guardrails include pre‑commit hooks that reject replicas: 0 in Deployment manifests, or CI pipelines that validate manifests against a schema before they ever reach main.

Following the patterns above gives you a functional GitOps pipeline using only git, kubectl, and standard declarative manifests. The sharp edges listed are learned from real clusters that have been running GitOps in production for years—none of them are product‑specific internals, and all of them are avoidable with the right guardrails and a disciplined PR‑first workflow.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.