GitOps in Practice: Managing Infrastructure Through Pull Requests

Implement GitOps for infrastructure and application deployment. Covers ArgoCD vs Flux, repository structures, secret management in Git, drift detection, and the operational model that makes GitOps work beyond the demo.

This page describes how to implement infrastructure management workflows using Git pull requests as the primary mechanism for changes. It covers the practical patterns and failure modes that emerge when treating infrastructure configuration as code, using Git as the source of truth, and requiring review and approval before applying changes to running systems.

Creating a new environment through PR workflow

To create a new environment using this approach, you need a Git repository containing infrastructure definitions and a CI/CD system that can process pull requests.

Repository structure for environment creation

The repository must contain the following structure:

environments/
  staging/
    k8s/
      deployment.yaml
      service.yaml
      configmap.yaml
  production/
    k8s/
      deployment.yaml
      service.yaml
      configmap.yaml

Each directory contains Kubernetes manifests for that environment.

Creating a new branch and commit

Start by creating a new branch with a descriptive name:

git checkout -b feature/new-environment-staging

Then add the necessary manifest files to create a new environment:

# environments/staging/k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: app
        image: myapp:1.0
# environments/staging/k8s/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: myapp-service
spec:
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 8080

Submitting a pull request

When submitting the PR, ensure it includes a description of what will change in the environment and links to any relevant documentation or issue tracking.

Reviewing changes through Git diff

The pull request should show all changes using standard Git diff output. The review process involves checking for:

Merging infrastructure changes

When merging a pull request, the system must apply the changes to the target environment.

Using kubectl to apply changes

After a PR is merged, run the following command to update the environment:

kubectl apply -f environments/staging/k8s/

This command applies all manifests in the directory to the Kubernetes cluster. If any manifest has errors, kubectl will output specific error messages indicating which resource failed and why.

Handling merge conflicts

If multiple PRs target the same infrastructure resources, you may encounter merge conflicts:

git pull origin main
# Auto-merging environments/staging/k8s/deployment.yaml
# CONFLICT (content): Merge conflict in environments/staging/k8s/deployment.yaml
# Automatic merge failed; fix conflicts and then commit the result.

Applying changes with specific flags

For production environments, you may want to use more verbose output:

kubectl apply -f environments/staging/k8s/ --dry-run=client -o yaml

This shows what would be applied without making any changes.

Rolling back problematic deployments

When a change causes issues, you need a way to revert to a previous state.

Using git checkout for rollback

You can revert to a previous commit by checking out the specific commit:

git checkout <commit-hash>
kubectl apply -f environments/staging/k8s/

Handling partial failures

If kubectl apply fails partway through applying multiple manifests, it will show which resources were successfully updated and which failed:

deployment.apps/myapp configured
service.myapp-service unchanged
error: error when applying patch:
{"op":"replace","path":"/spec/replicas","value":3}
from resource: 
Error from server (BadRequest): admission webhook "validation.gatekeeper.sh" denied the request: validation error: [myapp] replicas must be 2

In this case, only the deployment was updated, while the service remained unchanged. The error indicates that the Gatekeeper admission controller rejected a change to the replica count.

Managing secrets through GitOps

Secrets in GitOps workflows should never contain actual secret values in plain text repositories.

Creating Kubernetes secrets from files

Use kubectl create secret to generate secrets:

kubectl create secret generic myapp-secrets \
  --from-literal=username=admin \
  --from-literal=password=secret123 \
  -n staging

Storing secret references in Git

Instead of storing actual secrets, store references in Git:

# environments/staging/k8s/secret-reference.yaml
apiVersion: v1
kind: Secret
metadata:
  name: myapp-secrets
type: Opaque

Using external secret management tools

Some workflows use tools like Sealed Secrets or external secret managers, which store encrypted secrets in Git and decrypt them during deployment:

kubectl apply -f sealed-secret.yaml

Handling configuration drift

Over time, your actual infrastructure may drift from what's defined in Git.

Detecting drift with kubectl diff

Use kubectl diff to compare current state with desired state:

kubectl diff -f environments/staging/k8s/

This command will show exactly which resources differ between the definition in Git and the actual running configuration.

Addressing drift through PR workflow

When drift is detected, create a new pull request that corrects the drift. The review process ensures that changes are intentional and consistent with the desired state.

Managing cross-environment dependencies

When multiple environments depend on each other, you need to be careful about change propagation.

Versioning infrastructure definitions

Use semantic versioning for your infrastructure definitions:

# Tagging a release
git tag v1.2.0
git push origin v1.2.0

Using environment-specific overlays

Some workflows use kustomize or similar tools to manage environment-specific variations:

# environments/staging/kustomization.yaml
resources:
- k8s/deployment.yaml
- k8s/service.yaml
patches:
- patch: |
    - op: replace
      path: /spec/replicas
      value: 3
  target:
    kind: Deployment
    name: myapp

Managing pull request lifecycle

The PR lifecycle includes several stages that must be handled correctly.

Creating feature branches with descriptive names

Branches should clearly indicate what change they implement:

git checkout -b feat/add-monitoring-stack

Writing effective pull request descriptions

PR descriptions should include:

Managing review status

PRs should be reviewed by at least one other engineer before merging. The system should prevent merging until all required reviews are complete.

Handling failed deployments

When a deployment fails, you need clear signals and recovery paths.

Monitoring deployment status

Use kubectl get to check deployment status:

kubectl get deployments -n staging
kubectl describe deployment myapp -n staging

Troubleshooting common failures

Common failures include:

Using kubectl logs for diagnostics

For troubleshooting, use:

kubectl logs -l app=myapp -n staging
kubectl logs deployment/myapp -n staging --previous

Managing multiple environments with shared components

When you have multiple environments sharing common infrastructure, you need to be careful about how changes propagate.

Using shared configuration directories

Structure your repository to allow shared components:

shared/
  base/
    common.yaml
environments/
  staging/
    k8s/
      deployment.yaml
  production/
    k8s/
      deployment.yaml

Managing component dependencies

Ensure that when you change a shared component, you understand the impact on all dependent environments.

Handling permissions and access control

Access control is critical in GitOps workflows.

Restricting branch protection rules

Use Git branch protection rules to ensure:

Managing repository permissions

Ensure that only authorized users can push to protected branches:

git push origin feature/new-feature
# This will fail if user doesn't have write access

Common failure modes and mitigation strategies

Silent failures in manifest application

Some Kubernetes manifests may appear to succeed but silently fail to apply changes:

kubectl apply -f environments/staging/k8s/
# Output shows success, but deployment replicas remain unchanged

To catch these, always verify with kubectl get after applying.

Race conditions during concurrent changes

When multiple PRs are merged simultaneously, race conditions can cause unexpected behavior. Use kubectl apply with appropriate timeouts and check for conflicts before merging.

Misconfigured admission controllers

Admission controllers can silently reject changes:

Error from server (Forbidden): error when applying patch:
{"op":"replace","path":"/spec/replicas","value":3}
from resource: 
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request

This error indicates that an admission controller rejected the change. You should always test changes in a staging environment before applying to production.

Incorrect resource ordering

Kubernetes manifests may be applied in incorrect order:

kubectl apply -f environments/staging/k8s/ --validate=false

Using --validate=false can help avoid early validation errors but may mask deeper issues.

Handling large-scale changes

For large deployments, consider using kubectl patch or kubectl edit to make incremental changes rather than applying entire manifests at once.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.