ESC
Type to search guides, tutorials, and reference documentation.

GitOps: Git as the Source of Truth for Deployment

Declarative desired state in Git plus a reconciler that continuously converges the running system toward it. Covers pull versus push delivery, repository topology, drift, promotion, secrets, and where the model stops working.

GitOps is a deployment methodology where infrastructure and application configuration are declared in Git repositories and automatically synchronized to target environments. This approach makes Git the single source of truth for deployments, enabling declarative infrastructure management through version-controlled configuration files.

Understanding GitOps Workflow

In GitOps, changes flow from Git to your deployment system through automated pipelines or operators. The process begins with developers committing configuration changes to a repository, which then triggers synchronization to target environments. This workflow ensures that all deployed artifacts are traceable back to specific commits and can be rolled back to any previous state.

Setting up GitOps with Kubernetes

Deploying Configuration from Git

To deploy a Kubernetes application using GitOps principles, you typically store your manifests in a Git repository and use a controller to synchronize them to the cluster:

# Example deployment manifest stored in Git
kubectl apply -f ./manifests/deployment.yaml

The Git repository structure should separate environment-specific configurations from common ones. For example, a typical repository might contain:

/
├── base/
│   ├── deployment.yaml
│   └── service.yaml
├── overlays/
│   ├── production/
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   └── staging/
│       ├── deployment.yaml
│       └── service.yaml

Managing Secrets in Git

Secrets should be stored in Git using tools like Sealed Secrets or external secret management solutions. Never store plain-text secrets in Git repositories.

# Example of a sealed secret
kubectl create secret generic my-secret \
  --from-literal=username=secretuser \
  --from-literal=password=secretpassword \
  -o yaml --dry-run=client | \
  kubeseal --format yaml > sealed-secret.yaml

Handling Deployment Failures

Rollback to Previous Git State

When deployments fail, you can rollback by checking out a previous commit:

# Check current status
git status

# View recent commits
git log --oneline -10

# Reset to previous commit
git reset --hard HEAD~1

# Force update the deployment
kubectl apply -f ./manifests/

Dealing with Configuration Drift

Configuration drift occurs when changes are made directly to the cluster instead of through Git:

# Detect drift between Git and cluster state
kubectl diff -f ./manifests/

# Apply only what's in Git
kubectl apply -f ./manifests/ --prune -l app=myapp

Error Handling During Sync

When synchronization fails, GitOps systems typically log errors to indicate what went wrong. Common error messages include:

  • Error: unable to decode "": no kind "Deployment" is registered for version "apps/v1"
  • Error: failed to apply configuration: resource already exists
  • Error: timeout waiting for deployment to be ready

Managing Multiple Environments

Environment-Specific Configuration

Use Git subtrees, branches, or overlays to manage environment-specific settings:

# Create separate branch for production
git checkout -b production

# Merge changes from staging
git checkout production
git merge staging

Staging and Production Separation

Separate deployment pipelines for staging and production environments ensure controlled rollouts:

# Deploy to staging
kubectl apply -f ./overlays/staging/

# Deploy to production after approval
kubectl apply -f ./overlays/production/

Git Hooks for Deployment Automation

Pre-Commit Validation

Use pre-commit hooks to validate configuration before allowing commits:

#!/bin/bash
# .git/hooks/pre-commit

# Validate manifests
kubectl apply --dry-run=client -f ./manifests/

# Check for common errors
if grep -r "image:.*latest" ./manifests/; then
  echo "Warning: Found 'latest' tag in images"
fi

exit 0

Post-Receive Triggering

Configure Git hooks to trigger deployments on specific branch updates:

#!/bin/bash
# .git/hooks/post-receive

while read oldrev newrev refname; do
  if [[ $refname = "refs/heads/main" ]]; then
    echo "Deploying to production"
    kubectl apply -f ./manifests/
  fi
done

Version Control Best Practices

Commit Message Standards

Adhere to conventional commit messages for better traceability:

git commit -m "feat(k8s): add resource limits to deployment"
git commit -m "fix(config): correct service port in staging"
git commit -m "chore(deploy): update image tag to v1.2.3"

Branch Protection Rules

Enforce branch protection to prevent direct cluster changes:

# Example GitHub branch protection rules
# Require pull requests
# Require status checks to pass
# Require reviews from code owners

Handling Rollbacks and Rollouts

Automated Rollback Process

When a deployment fails, you can automate rollback through Git history:

# Find the last successful commit
LAST_SUCCESSFUL=$(git log --grep="deploy" --oneline -10 | grep "success" | head -1 | cut -d' ' -f1)

# Reset to that commit
git reset --hard $LAST_SUCCESSFUL

# Redeploy
kubectl apply -f ./manifests/

Canary Deployments via Git

Implement canary deployments by creating specific Git branches:

# Create canary branch
git checkout -b feature/canary-deployment

# Deploy to canary environment
kubectl apply -f ./overlays/canary/

# Monitor for success before promoting
kubectl rollout status deployment/myapp-canary

Configuration Management Patterns

Using Kustomize for Overlays

Manage environment differences with Kustomize:

# overlays/staging/kustomization.yaml
bases:
- ../base
patches:
- deployment-patch.yaml

Managing Dependencies

Track external dependencies through Git submodules or vendoring:

# Add a dependency
git submodule add https://github.com/example/infrastructure.git dependencies/infrastructure

# Update dependencies
git submodule update --remote --recursive

Error Recovery and Maintenance

Reconciling Configuration Drift

When drift occurs, use kubectl diff to identify changes:

# Compare current state with Git
kubectl diff -f ./manifests/

# Apply the difference
kubectl apply -f ./manifests/

Handling Failed Deployments

When deployments fail, examine both Git status and cluster state:

# Check deployment status
kubectl get deployments

# View events for failing resources
kubectl get events --sort-by=.metadata.creationTimestamp

# Describe problematic resources
kubectl describe deployment myapp

GitOps Integration with CI/CD

Pipeline Configuration

Set up pipeline triggers based on Git events:

# Example pipeline configuration
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

Testing Before Deployment

Validate changes before applying them:

# Run tests
make test

# Validate manifests
kubectl apply --dry-run=client -f ./manifests/

# Apply configuration
kubectl apply -f ./manifests/

Operational Considerations

Monitoring GitOps State

Monitor for synchronization issues:

# Check for failed reconciliations
kubectl get deployments -o jsonpath='{.items[*].status.conditions[?(@.type=="Available")].status}'

# Watch for events
kubectl get events --watch

Handling Secrets Management

Manage secrets through external systems rather than Git:

# Example of secret handling
kubectl create secret generic database-config \
  --from-literal=host=prod-db.example.com \
  --from-literal=port=5432

Common Pitfalls and Gotchas

Silent Failures

The most common silent failure occurs when configuration validation is bypassed:

# This command succeeds even with malformed manifests
kubectl apply -f ./manifests/

# Always validate first
kubectl apply --dry-run=client -f ./manifests/

Misunderstanding Pruning Behavior

Pruning removes resources not defined in current manifests:

# Without pruning, old resources remain
kubectl apply -f ./manifests/

# With pruning, unused resources are removed
kubectl apply -f ./manifests/ --prune -l app=myapp

Git History Pollution

Frequent commits can pollute history with unnecessary changes:

# Avoid committing generated files
echo "*.yaml" >> .gitignore

# Use specific paths for updates
kubectl apply -f ./manifests/app.yaml

Advanced GitOps Techniques

Rollout Strategies

Implement progressive rollouts by managing deployment strategies in Git:

# Deployment with rolling update strategy
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

Health Checks and Readiness

Define proper health checks to ensure deployments are ready:

# Container readiness probe
readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Resource Management

Manage resource requests and limits through Git:

# Resource specification
resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "500m"

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.