Manage application and infrastructure configuration consistently across environments. Covers configuration hierarchy, environment-specific overrides, feature flags, secrets separation, configuration drift detection, and the patterns that prevent configuration-related outages.
Configuration management at scale is the practice of maintaining consistent, reliable, and version-controlled infrastructure and application state across hundreds or thousands of machines, services, and environments. It matters when a single manual change risks introducing drift, when deployment frequency exceeds weekly cadence, and when a failure in one environment cascades across others. This is not just about applying configurations—it is about orchestrating a continuous, auditable, and self-healing state across the entire system.
Configuration at scale begins with a clear definition of what constitutes configuration: the desired state of a system. This state is expressed as declarative, versioned artifacts—typically YAML or JSON—that describe how every component should be set up, from kernel parameters and service manifests to application secrets, user roles, and network policies.
The core artifact is a configuration manifest, stored in source control. Each manifest includes a name, version, targets, and apply directives.
name: production
version: 2025.04.08
targets:
- role: web
hosts: ["web-01.example.com", "web-02.example.com"]
environment: production
apply:
- service: nginx
config:
worker_processes: 4
error_log: "/var/log/nginx/error.log"
access_log: "/var/log/nginx/access.log"
files:
"/etc/nginx/sites-available/app":
source: "templates/app.conf.j2"
mode: "0644"
owner: "root"
group: "root"
- package: postgresql
version: "15.3"
options:
max_connections: 200
shared_buffers: "2GB"Changes to this manifest are tracked via Git. A commit triggers a configuration deployment pipeline. The config/ directory is the source of truth, and every change is a versioned delta.
Once defined, configuration must be applied consistently across a fleet of servers, containers, and services. The most common method is pull-based configuration, where each node periodically checks for updates and applies them.
Each node runs a configuration agent—a daemon that polls a central configuration store for updates. The agent applies the configuration in sequence, resolving dependencies and reporting status.
# Agent invocation on every node
systemctl start config-agent
# Config agent runs every 10 minutes
# Polls the configuration server at config.example.com/api/configThe agent fetches the latest manifest and applies changes using a sequence of configuration steps—each step is an atomic unit of change.
Each step is defined as a task with a type, name, and parameters:
{
"type": "apply-file",
"name": "configure-nginx-site",
"parameters": {
"path": "/etc/nginx/sites-available/app",
"content": "server {\n listen 80;\n location / {\n root /var/www/html;\n }\n}",
"mode": "0644",
"owner": "root",
"group": "root"
}
}The agent applies tasks in order, using idempotent operations—operations that can be repeated without side effects. A create-file task, for example, creates a file if it does not exist, and updates it if it does, but does not fail on an existing file.
A failure in a task halts the pipeline and logs the error:
{
"task": "apply-file",
"step": "configure-nginx-site",
"status": "failed",
"error": {
"code": "file-write-error",
"message": "Failed to write /etc/nginx/sites-available/app: Permission denied",
"details": {
"path": "/etc/nginx/sites-available/app",
"uid": 1000,
"gid": 1000,
"expected_owner": "root",
"expected_group": "root"
}
},
"timestamp": "2025-04-08T14:22:13Z"
}The agent reports status via a heartbeat endpoint:
GET /api/agent/status
Host: config.example.com
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.xxxxxx
{
"agent_id": "web-01.example.com",
"last_updated": "2025-04-08T14:22:13Z",
"config_version": "2025.04.08",
"tasks": [
{
"name": "apply-file",
"status": "success",
"duration": 124,
"started": "2025-04-08T14:21:49Z"
},
{
"name": "restart-service",
"status": "failed",
"error": "service-unavailable",
"details": {
"service": "nginx",
"command": "systemctl restart nginx",
"exit_code": 1,
"output": "Failed to restart nginx.service: Unit nginx.service not found."
}
}
]
}Over time, configuration drift occurs when a node diverges from the desired state. This can be due to manual changes, failed deployments, or transient failures.
To detect drift, the agent runs a drift check before applying updates. It compares the current state of the system (as reported by systemctl, ls, cat, ps, curl, and stat) against the expected state defined in the configuration.
A drift check produces a diff report:
{
"agent_id": "web-01.example.com",
"config_version": "2025.04.08",
"drift_detected": true,
"changes": [
{
"type": "file",
"path": "/etc/nginx/nginx.conf",
"expected": {
"mode": "0644",
"owner": "root",
"group": "root",
"content": "user www-data;\nworker_processes auto;"
},
"actual": {
"mode": "0644",
"owner": "root",
"group": "www-data",
"content": "user www-data;\nworker_processes 4;"
},
"diff": [
{
"field": "group",
"expected": "root",
"actual": "www-data",
"reason": "Manual change during maintenance"
},
{
"field": "worker_processes",
"expected": "auto",
"actual": "4",
"reason": "Updated after load test"
}
]
},
{
"type": "service",
"name": "nginx",
"expected": { "enabled": true, "active": true },
"actual": { "enabled": true, "active": false },
"diff": [
{
"field": "active",
"expected": true,
"actual": false,
"reason": "Failed to start after deployment"
}
]
}
]
}The agent can act on drift: either auto-correcting (reapplying configuration) or notifying administrators via email, Slack, or a central dashboard.
At scale, configurations are not flat. They depend on each other. A web server depends on a database, which depends on a shared file system.
Configuration units must be ordered and versioned. Each unit declares its dependencies, and the configuration engine resolves the dependency graph.
# config/databases/postgres.yaml
name: postgres
version: 15.3
dependencies:
- "filesystem:shared"
- "network:internal-vpc"
- "service:postgresql"
requires:
- "user:postgres"
- "secret:db-password"The configuration engine resolves dependencies by querying a dependency registry, a simple key-value store accessible via HTTP.
GET /api/dependencies?name=postgresql&version=15.3
Host: config.example.com
{
"name": "postgresql",
"version": "15.3",
"dependencies": [
{ "type": "service", "name": "postgresql", "version": "15.3" },
{ "type": "package", "name": "libpq", "version": "15.3" },
{ "type": "file", "path": "/etc/postgresql/15/main/postgresql.conf" }
],
"required_secrets": [
"db-password",
"ssl-cert"
],
"provides": [
"database:main",
"network:db-internal"
]
}When a configuration unit is applied, the engine ensures all its dependencies are present and up-to-date. If a dependency is missing or outdated, it schedules a dependency resolution job.
Configuration changes are not applied globally at once. They are rolled out incrementally to avoid downtime and to allow rollback.
A deployment plan defines the rollout strategy. It includes:
{
"plan": "production-rollout",
"phases": [
{
"name": "staging",
"targets": ["staging-01", "staging-02"],
"delay": "PT30M",
"conditions": [
{ "metric": "http-5xx-rate", "threshold": 0.05, "window": "PT5M" }
]
},
{
"name": "canary",
"targets": [
"web-01.example.com",
"web-02.example.com",
"web-03.example.com"
],
"scale": 0.3,
"delay": "PT1H",
"conditions": [
{ "metric": "request-latency", "threshold": 200, "window": "PT10M" }
]
},
{
"name": "production",
"targets": "all",
"scale": 1.0,
"conditions": [
{ "metric": "uptime", "threshold": 0.99, "window": "PT1H" }
]
}
],
"rollback": {
"on": [
{ "metric": "error-rate", "threshold": 0.1, "window": "PT10M" },
{ "metric": "memory-usage", "threshold": 0.9, "window": "PT30M" }
],
"strategy": "blue-green"
}
}Each phase is executed by a deployment orchestrator that coordinates configuration agents across nodes. The orchestrator records the rollout state in a deployment registry.
Configuration management at scale fails silently. The most common silent failure is missing or stale configuration data.
An agent may fail to fetch the latest manifest due to a network outage, resulting in stale configuration for hours. The agent logs:
{
"agent_id": "web-01.example.com",
"error": {
"code": "fetch-failed",
"message": "Failed to fetch configuration manifest from config.example.com",
"details": {
"url": "https://config.example.com/api/config",
"timeout": 10,
"error": "Connection timed out",
"retry_count": 3,
"last_successful": "2025-04-08T12:00:00Z"
}
}
}The orchestrator detects this and triggers a fallback mode: the agent uses a local copy of the last known configuration, and notifies the team.
Another common issue is configuration version mismatch. A node may be configured with version 2025.04.08, but the deployment plan expects 2025.04.09. The orchestrator checks the config_version in the agent’s heartbeat and flags the mismatch.
{
"deployment": "production-rollout",
"phase": "canary",
"status": "warning",
"message": "Nodes are running outdated config version",
"details": {
"expected_version": "2025.04.09",
"actual_versions": {
"web-01.example.com": "2025.04.08",
"web-02.example.com": "2025.04.08",
"web-03.example.com": "2025.04.08"
},
"nodes": [
"web-01.example.com",
"web-02.example.com",
"web-03.example.com"
],
"resolution": "Re-deploying config manifest to all nodes"
}
}The system supports manual override of configuration: an operator can force a node to use a specific configuration version, bypassing the automatic rollout.
A single configuration management system across thousands of nodes generates vast amounts of data. To monitor it, a configuration dashboard displays:
The dashboard queries a configuration analytics database, where all agent heartbeats, task results, and drift reports are stored.
-- Query: Nodes with high drift in the last 24 hours
SELECT
agent_id,
COUNT(*) AS drift_count,
AVG(diff_count) AS avg_diffs,
MAX(last_drift_time) AS last_drift
FROM drift_reports
WHERE created_at >= NOW() - INTERVAL '24 hours'
GROUP BY agent_id
HAVING AVG(diff_count) > 5
ORDER BY drift_count DESC;The system logs configuration change history as a sequence of events, each describing a configuration update.
{
"event": "config-applied",
"timestamp": "2025-04-08T14:22:13Z",
"version": "2025.04.08",
"author": "[email protected]",
"changes": [
{
"type": "file",
"path": "/etc/nginx/sites-available/app",
"from": "0.1.2",
"to": "0.1.3",
"description": "Updated server block for SSL termination"
},
{
"type": "service",
"name": "nginx",
"action": "restart",
"reason": "Configuration changed"
}
],
"nodes": [
"web-01.example.com",
"web-02.example.com"
]
}This history enables auditing, troubleshooting, and compliance. A change in a configuration file is not just a file—it is an event, a decision, a responsibility.
This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy