Environment Management

Production engineering guide for environment management covering patterns, implementation strategies, and operational best practices.

Environment management governs how software systems are configured, deployed, and operated across distinct runtime contexts such as development, staging, and production. This discipline becomes critical when configuration drift creates operational inconsistencies, or when deployment processes require environment-specific settings that must be reliably applied and versioned.

Provisioning New Environments

To create a new environment from scratch, define its target infrastructure using declarative configuration:

# example-environment.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: myapp-staging
  labels:
    environment: staging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: myapp-staging
data:
  DATABASE_URL: "postgresql://staging-db:5432/myapp"
  LOG_LEVEL: "info"

Apply this configuration using kubectl:

kubectl apply -f example-environment.yaml

The key failure mode occurs when a namespace is created without proper resource quotas, leading to unexpected consumption of cluster resources. The system silently accepts creation but eventually fails to schedule new pods due to quota exhaustion.

Managing Configuration Variants

Configuration management requires distinguishing between static and dynamic values per environment:

# config-template.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: myapp-{{ENV}}
data:
  DATABASE_URL: "{{DATABASE_URL}}"
  LOG_LEVEL: "{{LOG_LEVEL}}"
  FEATURE_FLAG_A: "{{FEATURE_FLAG_A}}"

Use environment variables to substitute template values:

export DATABASE_URL="postgresql://prod-db:5432/myapp"
export LOG_LEVEL="warn"
export FEATURE_FLAG_A="true"

Deploy with a script that substitutes placeholders:

envsubst < config-template.yaml | kubectl apply -f -

This approach fails silently if an environment variable is unset, defaulting to empty strings in the resulting ConfigMap. The most common failure pattern involves missing variables that cause runtime errors only after deployment.

Sharing Resources Across Environments

Shared resources like databases or caches must be provisioned with clear ownership and naming conventions:

# shared-resources.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-db-pvc
  namespace: shared-infrastructure
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

Ensure shared resources have unique names to avoid conflicts. For example, a database named myapp-db in both staging and production namespaces should be distinguished by namespace-level scoping.

Environment-Specific Secrets Management

Secrets must be isolated per environment while maintaining consistent structure:

# secret-template.yaml
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: myapp-{{ENV}}
type: Opaque
data:
  DATABASE_PASSWORD: "{{DATABASE_PASSWORD}}"
  API_KEY: "{{API_KEY}}"

The command to create a secret from literal values:

kubectl create secret generic app-secrets \
  --namespace=myapp-staging \
  --from-literal=DATABASE_PASSWORD=secret123 \
  --from-literal=API_KEY=apikey456

A common failure occurs when secrets are stored with overly broad permissions, causing unintended access to staging secrets in production contexts.

Lifecycle Management of Environments

Environments require clear deletion and archival policies:

kubectl delete namespace myapp-staging

Before deletion, ensure all resources are cleaned up:

kubectl get pods -n myapp-staging --no-headers | wc -l
# Should return 0 before proceeding with deletion

The most common failure occurs when deleting a namespace that still contains active workloads, resulting in orphaned resources and potential data loss.

Environment Isolation and Resource Limits

Environments must have distinct resource allocations to prevent interference:

# resource-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: env-quota
  namespace: myapp-staging
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi

Apply resource quotas to ensure environments don't consume more than allocated:

kubectl apply -f resource-quota.yaml

Failure to set quotas leads to resource contention and unpredictable performance across environments.

Cross-Environment Dependency Management

Dependencies between environments must be explicitly declared:

# dependency-manifest.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: environment-dependencies
  namespace: myapp-staging
data:
  DEPENDENCY_SERVICE: "myapp-prod"
  DEPENDENCY_VERSION: "v2.1.0"

This ensures that staging environments know exactly which production services they depend on and can alert when dependencies change.

Environment State Verification

Validate environment state after changes:

kubectl get pods -n myapp-staging -o jsonpath='{.items[*].status.phase}'
# Expected output: Running

Check that all expected resources exist:

kubectl get configmap -n myapp-staging --no-headers | wc -l
# Should match expected count of configmaps

The most frequent issue arises when verification commands pass but runtime behavior fails due to subtle configuration differences.

Environment Cleanup Automation

Automated cleanup prevents resource sprawl:

#!/bin/bash
# cleanup-old-environments.sh
kubectl get namespaces --no-headers | grep -E 'myapp-(staging|dev)' | \
  awk '$1 ~ /2023-/ {print $1}' | xargs -r kubectl delete namespace

This script removes environments older than a specific date, preventing accumulation of unused resources.

Handling Environment-Specific Overrides

Some configuration must be overridden per environment:

# override-example.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: myapp-staging
data:
  LOG_LEVEL: "debug"

Apply overrides after base configuration:

kubectl apply -f base-config.yaml
kubectl apply -f override-example.yaml

The override mechanism silently fails if a key in the override ConfigMap does not exist in the base configuration, resulting in missing configuration values.

Monitoring Environment Health

Establish health checks for environments:

kubectl get pods -n myapp-staging --no-headers | grep -v Running
# Should return no output for healthy environment

Monitor resource usage:

kubectl top pods -n myapp-staging

A failure occurs when monitoring tools are not configured to check environment-specific namespaces, leading to missed health issues.

Managing Environment Access Control

Access control must be specific per environment:

# rbac-example.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: myapp-staging
  name: env-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

Apply role bindings to grant access:

kubectl apply -f rbac-example.yaml
kubectl create rolebinding staging-access \
  --role=env-role \
  --user=dev-user \
  --namespace=myapp-staging

The most common mistake is granting excessive permissions, allowing users to access resources beyond their intended environment scope.

Environment Migration Between Clusters

Moving environments between clusters requires careful handling of persistent data:

kubectl get all -n myapp-staging -o yaml | \
  kubectl apply -n myapp-prod -f -

Ensure stateful resources are properly migrated:

kubectl get pvc -n myapp-staging --no-headers | \
  awk '{print $1}' | xargs -r -I {} \
  kubectl get pvc {} -n myapp-staging -o yaml | \
  sed 's/myapp-staging/myapp-prod/' | \
  kubectl apply -n myapp-prod -f -

Migration fails silently if persistent volume claims are not properly re-attached to the new cluster, causing data loss.

Environment Versioning and Tagging

Tag environments consistently:

kubectl annotate namespace myapp-staging \
  environment-version="v1.2.3" \
  environment-type="staging"

Use these annotations for automated management:

kubectl get namespaces -l environment-type=staging --no-headers | \
  awk '{print $1}' | xargs -r kubectl delete namespace

The failure mode occurs when version tags are inconsistent, making automation unpredictable.

Environment Rollback Procedures

Define rollback procedures for environment changes:

kubectl rollout undo deployment/myapp-deployment -n myapp-staging --to-revision=1

Maintain a record of successful deployments:

kubectl get deployment myapp-deployment -n myapp-staging -o jsonpath='{.metadata.annotations}'

The most common rollback failure occurs when the original configuration is not properly preserved, requiring manual reconstruction of environment state.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.