Configuration Management

Production engineering guide for configuration management covering patterns, implementation strategies, and operational best practices.

Configuration Management is the disciplined practice of tracking, versioning, and deploying system settings across environments—development, staging, production—ensuring that every machine, service, and deployment behaves consistently and predictably. It matters when a service fails due to a missing environment variable, a misconfigured Nginx server, or a drifted database schema, and when a team must deploy dozens of components with interdependencies across multiple clusters.

Defining Configuration

Configuration is any data that controls how a system behaves without changing its code. It includes environment variables, service-specific settings, runtime parameters, and infrastructure metadata. Configuration Management ensures that every component—from a Python web app to a Kubernetes cluster—uses the correct values across all stages.

The core unit is a configuration *set*, which may be a single file, a directory, or a bundle of resources. Each set is versioned, audited, and applied through a defined process. The configuration is not stored in a single place—it is *managed*: created, validated, deployed, and monitored.

Configuration Sources

Configuration can originate from multiple sources, each with increasing precedence:

All sources are combined at startup, with later sources overriding earlier ones. A service may load defaults from a defaults.yaml, merge in config.yaml, then override with environment variables, and finally accept runtime flags.

Versioning and Tracking

Configuration must be versioned in source control. The preferred approach is to treat configuration as code: a config/ directory in the repository, with per-environment subdirectories:

config/
├── common/
│   ├── database.yaml
│   └── logging.yaml
├── development/
│   ├── app.yaml
│   └── database.yaml
├── staging/
│   ├── app.yaml
│   └── database.yaml
└── production/
    ├── app.yaml
    └── database.yaml

Each configuration file uses a standard format—YAML or JSON—with clearly named keys. Keys are namespaced, and values are typed:


app:
  port: 8080
  debug: false
  cache:
    backend: "redis"
    ttl: 3600
    servers:
      - "redis.prod.internal:6379"
      - "redis-backup.prod.internal:6379"
  logging:
    level: "info"
    format: "json"
    output: "stdout"

A change to config/production/app.yaml is a versioned artifact, committed with a Git commit message like:

feat: increase cache TTL and enable Redis backup nodes

- Set cache.ttl to 3600
- Add backup Redis server at redis-backup.prod.internal
- Update logging format to JSON

Deploying Configuration

Configuration deployment is the process of applying a configuration set to one or more environments. It is not a one-off task—it is a repeatable, idempotent operation.

Deploying to a Single Service

A service running via systemctl expects its configuration to be available at /etc/myapp/. The deployment process involves:

  1. Building a configuration bundle.
  2. Copying it to the target host.
  3. Applying it via a deploy script or systemctl command.
# Deploy script: deploy.sh
#!/bin/bash

set -euo pipefail

ENV=$1
CONFIG_DIR="config/$ENV"
TARGET_DIR="/etc/myapp"

echo "Deploying configuration for $ENV"

# Copy all config files
cp -r "$CONFIG_DIR"/* "$TARGET_DIR/"

# Reload systemd service
systemctl reload myapp.service

# Validate configuration
if ! systemctl is-active --quiet myapp.service; then
  echo "Error: myapp.service failed to reload"
  exit 1
fi

echo "Deployment completed"

The service uses systemd's EnvironmentFile to load environment variables from a file:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target

[Service]
ExecStart=/usr/bin/myapp --config /etc/myapp/app.yaml
EnvironmentFile=/etc/myapp/env.conf
Restart=always

[Install]
WantedBy=multi-user.target

The env.conf file is sourced at startup:

# /etc/myapp/env.conf
APP_PORT=8080
APP_DEBUG=false
REDIS_URL=redis.prod.internal:6379

Deploying to Multiple Services

When multiple services depend on shared configuration, the deployment becomes more complex. A common pattern uses a deployment pipeline that applies configuration across a fleet:

# .github/workflows/deploy.yml
name: Deploy Configuration

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Deploy to staging
        run: |
          ./deploy.sh staging
          # Validate service health
          curl -f http://staging.myapp.com/health || exit 1
          systemctl status myapp.service --no-pager

      - name: Deploy to production
        run: |
          ./deploy.sh production
          # Apply database migrations
          psql -h db.prod.internal -U app -d myapp -f migrations/2024-03-15-add-user-roles.sql
          # Verify all services
          for service in myapp auth gateway; do
            if ! systemctl is-active --quiet "$service.service"; then
              echo "Service $service failed to start"
              exit 1
            fi
          done

Configuration Validation

A common failure mode is a configuration file that parses correctly but contains invalid values. A service might start without error, but silently misbehave due to a typo or wrong type.

Validation must occur at multiple levels:

A service using jsonschema to validate its configuration:

# app.py
import json
import jsonschema
from jsonschema import ValidationError

SCHEMA = {
    "type": "object",
    "properties": {
        "app": {
            "type": "object",
            "properties": {
                "port": {"type": "integer", "minimum": 1, "maximum": 65535},
                "debug": {"type": "boolean"},
                "cache": {
                    "type": "object",
                    "properties": {
                        "backend": {"enum": ["redis", "memcached", "inmemory"]},
                        "ttl": {"type": "integer", "minimum": 300},
                        "servers": {
                            "type": "array",
                            "items": {"type": "string", "format": "hostname"}
                        }
                    },
                    "required": ["backend", "ttl"]
                }
            },
            "required": ["port", "debug"]
        }
    },
    "required": ["app"]
}

def load_and_validate_config(path: str) -> dict:
    with open(path, 'r') as f:
        raw = yaml.safe_load(f)

    try:
        jsonschema.validate(raw, SCHEMA)
    except ValidationError as e:
        print(f"Configuration validation failed: {e.message}")
        sys.exit(1)

    return raw

A misconfigured app.yaml that fails validation:

app:
  port: 8080
  debug: true
  cache:
    backend: "redis"
    ttl: 450
    servers:
      - "redis.prod.internal"
      - "192.168.1.10"

Error on startup:

Configuration validation failed: '192.168.1.10' is not a valid hostname

The service exits with code 1, and the orchestrator (e.g., systemd, Kubernetes) restarts it.

Managing Configuration Drift

Configuration drift occurs when a system’s actual configuration diverges from its expected configuration. It is silent and cumulative, leading to inconsistent behavior across environments.

Detecting Drift

Drift is detected by comparing the *current state* of a system against the *desired state* defined in configuration files.

The simplest tool is a diff script:

# check-drift.sh
#!/bin/bash

ENV=$1
CONFIG_DIR="config/$ENV"
TARGET_DIR="/etc/myapp"

echo "Checking drift for $ENV"

# Compare all config files
diff -r "$CONFIG_DIR" "$TARGET_DIR" || echo "Drift detected"

# Check systemd service file
if ! diff "$CONFIG_DIR/myapp.service" "/etc/systemd/system/myapp.service"; then
  echo "Service file drift"
fi

# Check environment variables
if ! diff <(env | sort) <(cat "$CONFIG_DIR/env.conf" | sort); then
  echo "Environment variable drift"
fi

A nightly CI job runs check-drift.sh staging and reports any mismatches.

Fixing Drift

Fixing drift involves applying the desired configuration and recording the change.

The apply command synchronizes the target system with the config set:

# apply.sh
#!/bin/bash

ENV=$1
CONFIG_DIR="config/$ENV"
TARGET_DIR="/etc/myapp"

echo "Applying configuration for $ENV"

# Copy all files
rsync -av --delete "$CONFIG_DIR/" "$TARGET_DIR/"

# Apply systemd service
cp "$CONFIG_DIR/myapp.service" "/etc/systemd/system/myapp.service"

# Reload and enable
systemctl daemon-reload
systemctl enable myapp.service
systemctl reload myapp.service

# Record the change in a log
echo "$(date): Applied configuration from $CONFIG_DIR" >> /var/log/config-applied.log

Handling Configuration Rollbacks

When a new configuration breaks the system, a rollback must be fast and reliable.

A rollback is not simply copying an old config file—it is a full configuration restore.

The rollback.sh script:

#!/bin/bash

ENV=$1
VERSION=$2
CONFIG_DIR="config/$ENV/$VERSION"
TARGET_DIR="/etc/myapp"

echo "Rolling back to $VERSION in $ENV"

# Restore from backup
rsync -av "$CONFIG_DIR/" "$TARGET_DIR/"

# Apply service file
cp "$CONFIG_DIR/myapp.service" "/etc/systemd/system/myapp.service"

# Reload and restart
systemctl daemon-reload
systemctl restart myapp.service

# Record rollback
echo "$(date): Rolled back to $VERSION" >> /var/log/config-rollback.log

Rollback is triggered by a health check failure:

# health-check.sh
#!/bin/bash

if ! curl -f http://localhost:8080/health; then
  echo "Health check failed. Initiating rollback."
  ./rollback.sh staging 2024-03-10
  exit 1
fi

Failure Modes and Sharp Edges

Configuration Management is not just about files—it is about processes, validation, and recovery. It is the glue that ensures systems behave as expected, even as complexity grows.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.