GitOps: Infrastructure Delivery

Implement GitOps for infrastructure. Covers Git as single source of truth, ArgoCD, Flux, deployment strategies, drift detection, and GitOps workflow design.

GitOps: Infrastructure Delivery is a disciplined approach to managing infrastructure as code, where the desired state of the system is declared in a Git repository and continuously reconciled to the live environment. It matters when teams need predictable, auditable, and reproducible infrastructure across multiple environments—production, staging, and development—where changes to infrastructure are as frequent and versioned as application code.

Defining Desired State in Git

The foundation of GitOps is a Git repository that acts as the single source of truth for infrastructure. Each environment—dev, staging, prod—has its own branch or directory, and infrastructure manifests are versioned alongside application code. A typical repository layout:

infrastructure/
├── environments/
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   └── overlays/
│   │       └── aws/
│   │           └── config.yaml
│   ├── staging/
│   └── prod/
├── modules/
│   ├── networking/
│   │   ├── cluster.yaml
│   │   └── ingress.yaml
│   └── database/
│       └── postgres.yaml
├── config/
│   └── terraform.tfvars
└── README.md

The kustomization.yaml file in each environment declares dependencies and overlays:


apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../modules/networking
  - ../../modules/database
  - config.yaml
patchesStrategicMerge:
  - overlays/aws/config.yaml
commonLabels:
  env: dev
  app: myapp

This structure enables teams to define infrastructure declaratively and compose it across layers. Changes to infrastructure are made by committing to Git, and the system automatically applies the new state.

Reconciling to Desired State

A GitOps controller watches the infrastructure repository and ensures the cluster matches the state declared in Git. The controller performs a reconciliation loop: it reads the latest commit, applies changes to the cluster, and reports the current state back to Git.

The most common mechanism is a pull-based loop: a controller periodically polls the Git repository for changes, then applies them using a tool like kubectl, terraform, or a custom orchestrator. The loop is idempotent: applying the same manifest multiple times produces the same result.

A typical reconciliation process:

  1. Controller fetches the latest commit from the main branch.
  2. It applies kubectl apply -f environments/prod to the cluster.
  3. It updates a status field in the repository to reflect the current state.
# Example: apply infrastructure changes
$ git checkout prod
$ kubectl apply -f environments/prod/

The controller reports the result in a Git commit status:

{
  "state": "success",
  "description": "Applied infrastructure to production cluster",
  "target_url": "https://dashboard.example.com/reconcile/123",
  "context": "gitops/prod"
}

This status is visible in the Git UI and can be used to gate pull requests.

Managing Change with Pull Requests

Every infrastructure change is a pull request. A developer creates a feature branch, modifies a manifest, and opens a PR. The CI system runs tests, and the GitOps controller evaluates the PR.

The PR includes a preview of the intended state: the controller renders the final manifest for the PR and applies it to a test cluster. This is often called a "preview environment" or "staging rollout".

# environments/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../modules/database
  - ../../modules/networking
  - config.yaml
patchesStrategicMerge:
  - overlays/gcp/config.yaml
  - overlays/preview-123.yaml

When merged, the PR triggers the next reconciliation cycle. The controller ensures the live system matches the prod branch, and updates the status accordingly.

Handling Drift and Failure

Despite continuous reconciliation, infrastructure can drift. A pod is manually scaled, a config map is edited in the cluster, or a node fails. The GitOps controller detects drift by comparing the current cluster state with the desired state in Git.

The controller runs a diff between the live cluster and the manifest:

$ kubectl get all -o yaml > current-state.yaml
$ kubectl diff -f environments/prod/

The output shows differences:

+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: web
+spec:
+  replicas: 5
+  selector:
+    matchLabels:
+      app: web
+  template:
+    metadata:
+      labels:
+        app: web
+    spec:
+      containers:
+      - image: nginx:1.21
+        name: web
+      - image: registry.example.com/logging-agent:latest
+        name: logging
+        ports:
+        - containerPort: 80
+        - containerPort: 8080
+      - image: registry.example.com/metrics-exporter:1.2
+        name: metrics
+        ports:
+        - containerPort: 9090
+        env:
+        - name: METRICS_ENDPOINT
+          value: http://metrics.example.com
+        - name: LOG_LEVEL
+          value: debug
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: app-config
+data:
+  log_level: debug
+  feature_flags: |
+    analytics: true
+    dashboard: false
+    notifications: true
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: web
+spec:
+  ports:
+  - port: 80
+    targetPort: 80
+  selector:
+    app: web
+  type: LoadBalancer

If a difference is detected, the controller logs it and updates the PR status:

{
  "state": "failure",
  "description": "Drift detected: 3 resources out of sync",
  "target_url": "https://dashboard.example.com/drift/456",
  "context": "gitops/drift"
}

The controller can automatically fix drift by applying the missing changes, or it can wait for human review.

Handling Rollbacks and Rollouts

When a change fails, teams roll back by creating a new commit that reverts the infrastructure state. The GitOps controller detects the rollback and applies the new state.

A common pattern is to use Git’s merge commit message to trigger a rollout:

$ git commit -m "Rollback to v1.2.0 after web service crash"
$ git push origin main

The controller reads the commit message and triggers a rollout. It can also use the commit message to determine the rollout strategy:

Rollback to v1.2.0 after web service crash

- Rollback to deployment v1.2.0
- Scale down logging-agent to 2 replicas
- Update app-config to use legacy database
- Apply to all environments

The controller parses the message and applies the changes in sequence. It can also generate a rollout report:

{
  "rolloutId": "2023-10-15-01",
  "commit": "a1b2c3d4e5f6",
  "status": "in-progress",
  "steps": [
    {
      "phase": "update-deployment",
      "target": "web",
      "version": "v1.2.0",
      "startedAt": "2023-10-15T10:00:00Z",
      "completedAt": "2023-10-15T10:05:00Z"
    },
    {
      "phase": "update-configmap",
      "target": "app-config",
      "changes": [
        { "key": "log_level", "from": "info", "to": "debug" },
        { "key": "feature_flags", "from": "analytics:true", "to": "analytics:false" }
      ],
      "startedAt": "2023-10-15T10:05:00Z"
    }
  ]
}

This report is stored in the Git repository and linked from the PR.

Automating Infrastructure Lifecycle

GitOps enables automation of the entire infrastructure lifecycle. When a new environment is needed, a team creates a new branch and populates it with base manifests.

$ git checkout -b feature/new-environment
$ mkdir environments/new-env
$ cp -r modules/* environments/new-env/
$ echo 'env: new-env' >> environments/new-env/kustomization.yaml
$ git add environments/new-env
$ git commit -m "Add new environment: staging"
$ git push origin feature/new-environment

The controller detects the new environment and applies it to the cluster. It can also trigger a CI job to validate the environment.

Similarly, when a new version of a module is released, the team updates the modules/networking directory and creates a release commit:

$ git commit -m "Release v2.1.0 of networking module"
$ git tag v2.1.0
$ git push origin v2.1.0

The controller watches for new tags and applies the updated module to all environments that use it.

Common Failure Modes and Pitfalls

The most common failure is slow reconciliation: the controller polls the Git repository too infrequently. A change is committed, but the cluster doesn’t update for 10 minutes. The team assumes the system is live, but it isn’t.

To diagnose, they check the controller logs:

2023-10-15T10:00:00Z INFO gitops: polling git repo at https://git.example.com/infra.git
2023-10-15T10:10:00Z INFO gitops: fetched commit a1b2c3d4e5f6
2023-10-15T10:10:05Z INFO gitops: applying to cluster
2023-10-15T10:10:30Z INFO gitops: applied successfully
2023-10-15T10:10:30Z INFO gitops: updating status

The logs reveal a 25-second gap between commit and application.

Another failure is incomplete state tracking: the controller applies changes but forgets to update the Git status. The PR shows "pending" even though the cluster is up to date.

{
  "state": "pending",
  "description": "Waiting for infrastructure to be applied",
  "target_url": "https://dashboard.example.com/apply/123",
  "context": "gitops/prod"
}

The fix is to ensure the controller writes the status after each reconciliation.

A subtle but critical issue is confusion between apply and replace. The controller uses kubectl apply, which is incremental, but the team expects full replacement. A kubectl replace is needed when updating a resource that should be entirely replaced.

$ kubectl replace -f environments/prod/web-deployment.yaml

The difference shows in the logs:

# kubectl apply
- deployment.apps/web configured
- configmap/app-config unchanged

# kubectl replace
- deployment.apps/web replaced
- configmap/app-config created

Another pitfall is missing drift detection. The controller applies changes, but never checks if the cluster matches the desired state. The team discovers drift only after a failure.

The solution is to schedule a drift check as a separate job:

$ kubectl get all -o yaml | diff -u current-state.yaml -

This job runs nightly and updates a drift-report.json file in the repository.

Configuration and Monitoring

The GitOps controller is configured via a standard config.yaml file in the repository:

# config.yaml
controller:
  git:
    url: https://git.example.com/infra.git
    branch: main
    pollingInterval: 30s
    credentials:
      type: token
      token: abc123xyz
  reconciliation:
    strategy: incremental
    dryRun: true
    maxRetries: 5
    timeout: 5m
  status:
    updateInterval: 10s
    context: gitops/prod
    targetUrlTemplate: https://dashboard.example.com/reconcile/{id}

The controller uses this config to manage its behavior. When the config changes, it reloads the settings and applies the new logic.

Monitoring the GitOps system is done via logs, metrics, and status updates. The controller exposes metrics via Prometheus:

gitops_reconciliation_duration_seconds{job="prod",status="success"} 1.23
gitops_drift_detected_total{environment="prod"} 42
gitops_polling_interval_seconds 30.0

These metrics are visualized in Grafana, showing the health of the infrastructure delivery pipeline.

Conclusion

GitOps: Infrastructure Delivery is a robust, scalable, and observable way to manage infrastructure. It turns infrastructure into a first-class citizen in the development lifecycle. Every change is versioned, tested, and audited. Every failure is traceable. The system is self-healing, self-documenting, and highly predictable. It is not a tool or a product, but a pattern—deeply rooted in Git, driven by reconciliation, and built for continuous delivery of infrastructure.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.