Environment Management: Reproducible Dev, Staging, and Production

Build environment management that guarantees consistency between development, staging, and production. Covers environment parity, infrastructure-as-code patterns, data management, feature toggles across environments, and preventing the 'works on staging' category of production outages.

Environment management ensures that development, staging, and production environments behave identically across teams, machines, and deployment pipelines. It matters when a bug appears in staging but not locally, when a configuration change breaks production without warning, or when onboarding a new developer takes more than a day. The goal is to eliminate "it works on my machine" through a repeatable, versioned process that defines every aspect of an environment: dependencies, services, configurations, and data.

Defining the Environment

An environment is a complete, self-contained system that runs a service or application. It includes the operating system, runtime, libraries, binaries, configuration files, and data. The core artifact is a *manifest*—a file that defines the environment’s state. This manifest must be version-controlled and checked into the same repository as the application source.

The manifest is typically a docker-compose.yml or terraform.tfvars file, but the underlying principle is consistent across tools: a declarative description of the environment state. For example, a docker-compose.yml file might define three services:

version: '3.8'
services:
  web:
    image: myapp:v1.2.0
    ports:
      - "8080:80"
    environment:
      - DATABASE_URL=postgresql://db:5432/app
      - LOG_LEVEL=info
    depends_on:
      - db
  db:
    image: postgres:15
    volumes:
      - pgdata:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=app
      - POSTGRES_USER=app
      - POSTGRES_PASSWORD=secret
volumes:
  pgdata:

The image field specifies the exact container image, which includes the app code, dependencies, and runtime. The environment block defines key-value pairs passed to the container at startup. The volumes section maps a named volume (pgdata) to a directory on the host, preserving database state across restarts.

When this file is committed to Git, it becomes a *reference environment*. Any machine that runs docker-compose up -d from this directory will reproduce the exact same environment.

Provisioning Environments from Manifests

To create a new environment, a developer checks out the project and runs a single command:

docker-compose up -d

This command pulls the specified images, starts the services, and wires them together. If the image field in docker-compose.yml references a tagged image (e.g., myapp:v1.2.0), the engine checks for the image locally. If missing, it pulls it from a registry (e.g., Docker Hub, AWS ECR).

If the registry is private, the environment must include credentials. These are typically provided via a ~/.docker/config.json file or via environment variables passed to docker-compose:

export DOCKER_REGISTRY_USER=ci
export DOCKER_REGISTRY_PASS=supersecret
docker-compose up -d

The provisioning step is idempotent: running docker-compose up -d multiple times produces the same state. If a service crashes, docker-compose restarts it. If a service’s config changes, the next up command applies the update.

Versioning Environments

Each environment—dev, staging, production—must be tied to a specific version of the manifest and its dependencies. The most common method is to use Git branches or tags.

For example, a dev environment uses the main branch, a staging environment uses the staging branch, and production uses the v1.0.0 tag. The docker-compose.yml file in each branch includes a version field that reflects the environment:


version: '3.8'
services:
  web:
    image: myapp:dev-${CI_COMMIT_SHA}
    # ...

The CI_COMMIT_SHA variable is injected by a CI system (e.g., GitHub Actions, GitLab CI). When a new commit is pushed to main, the CI pipeline builds and pushes the myapp:dev-abc123 image and updates the docker-compose.yml file with the new image tag.

For staging, the pipeline runs on the staging branch and produces an image like myapp:staging-${CI_COMMIT_SHA}. Production is triggered by a git tag:

git tag -a v1.0.0 -m "Release 1.0.0"
git push origin v1.0.0

The v1.0.0 tag becomes the anchor for the production environment. The CI system builds the image myapp:v1.0.0 and deploys it via docker-compose up -d using a docker-compose.prod.yml file that overrides the base docker-compose.yml.

Environment-Specific Configuration

Different environments require different settings. A dev environment might use a local database, while production uses a managed PostgreSQL instance. These differences are managed via configuration overlays.

The base docker-compose.yml defines the core services. An environment-specific file, such as docker-compose.staging.yml, extends it:

version: '3.8'
services:
  web:
    environment:
      - LOG_LEVEL=warn
      - DATABASE_URL=postgresql://staging-db:5432/app
      - FEATURE_FLAGS=enable-new-ui,enable-analytics
    volumes:
      - ./logs:/app/logs
    deploy:
      replicas: 3
      update_config:
        parallelism: 1
        delay: 10s
  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_DB=app_staging
      - POSTGRES_USER=staging
      - POSTGRES_PASSWORD=staging-secret
    volumes:
      - staging-data:/var/lib/postgresql/data

This file is merged with the base configuration. The deploy section, not present in the base file, is applied only to staging.

To deploy, the user combines both files:

docker-compose -f docker-compose.yml -f docker-compose.staging.yml up -d

The -f flag allows multiple files to be merged in order. Later files override earlier ones. This pattern allows fine-grained control: environment-specific variables, service overrides, and resource constraints.

Ensuring Reproducibility

Reproducibility is achieved by freezing all moving parts. The image field in the manifest locks the app and all its dependencies. The docker-compose.yml file locks the service topology, network, and volume layout.

But a common failure is mismatched versions. A service might start with myapp:v1.2.0, but the v1.2.0 image is built from a commit that is not the same as the current docker-compose.yml. This happens when the image is built before the manifest is committed.

To fix this, the CI pipeline must build the image *after* committing the manifest. A typical CI job looks like:

- name: Build and deploy
  run: |
    docker build -t myapp:${{ github.sha }} .
    docker push myapp:${{ github.sha }}
    git add docker-compose.yml
    git commit -m "Update image tag to ${{ github.sha }}"
    git push

This ensures that the docker-compose.yml file and the image tag refer to the same source state.

Another failure mode is missing or stale volumes. When a docker-compose.yml file uses named volumes, they are created on-demand. But if the volume already exists from a previous deployment, it may contain old data.

To prevent this, the deployment script should remove and recreate volumes:

docker-compose down --volumes
docker-compose up -d

The --volumes flag removes the named volumes after stopping the services. When up runs, it creates fresh volumes. This is critical for testing database migrations: a new migration script runs against a clean database.

Data Management Across Environments

Data is a first-class citizen in environment management. Each environment needs a consistent dataset. This is achieved through data snapshots and seeding.

A common pattern is to export the database state as a SQL dump:

pg_dump -h db -U app app > dump-2024-03-15.sql

This file is stored in the repository under data/dumps/. A docker-compose.yml file includes a command override to load the dump on startup:

services:
  db:
    image: postgres:15
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./data/dumps/dump-2024-03-15.sql:/docker-entrypoint-initdb.d/01-init.sql
    command: >
      sh -c "
        echo 'Waiting for postgres...'
        while ! pg_isready -h db -U app -d app; do
          sleep 1
        done
        echo 'Loading initial data...'
        psql -h db -U app -d app -f /docker-entrypoint-initdb.d/01-init.sql
        echo 'Starting PostgreSQL...'
        exec postgres -D /var/lib/postgresql/data
      "

The command field runs a shell script that waits for the database to be ready, applies the SQL dump, and starts the PostgreSQL server. The volumes section maps the SQL file into the container’s docker-entrypoint-initdb.d directory, where PostgreSQL automatically runs all .sql files on startup.

When a developer runs docker-compose up -d, the database is initialized with the exact data from the dump. This guarantees that the dev environment matches staging and production.

Debugging Environment Drift

Even with strong practices, environments drift. Services run on different images, configurations diverge, and data evolves.

A common failure is a service starting but not receiving requests. The error is silent: logs show the service starting, but the API endpoint returns 503. The cause is often a missing depends_on or incorrect networks configuration.

For example, the web service might depend on db, but the db service does not expose its port:

services:
  web:
    depends_on:
      - db
    networks:
      - app
  db:
    networks:
      - app
    ports:
      - "5432:5432"

The depends_on field ensures that db starts before web, but it does not guarantee that web can reach db via the network. The ports field exposes the PostgreSQL port to the host, but the networks section connects both services to a shared network named app.

Without the networks section, web and db are on separate networks, and web cannot resolve the db hostname.

To diagnose this, a developer runs:

docker-compose exec web ping db

If the ping fails, the network configuration is incorrect. If it succeeds, the next step is checking the DATABASE_URL environment variable.

A common mistake is using DATABASE_URL=postgresql://db:5432/app in docker-compose.yml, but the db service does not run the postgres process. The command field is missing or incorrect.

The docker-compose logs web command shows the logs. If the service starts but never receives requests, the developer checks the docker-compose ps output:

docker-compose ps

This shows the status of each service. If web is running but db is stopped or restarting, the issue is in the db service.

When debugging, the developer often runs docker-compose logs -f to follow the logs in real time. The --follow (-f) flag keeps the logs streaming until interrupted.

Best Practices and Pitfalls

A subtle failure occurs when a service depends on another but does not wait for it to be ready. The depends_on field only waits for the container to start, not for the service to be healthy.

services:
  web:
    depends_on:
      db:
        condition: service_healthy

This requires that the db service exposes a HEALTHCHECK in its Dockerfile:

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --start-timeout=10s \
  CMD pg_isready -h localhost -U app -d app

Without this, the web service starts before the database is ready, leading to connection timeouts.

Finally, always document the environment setup process. A DEVELOPMENT.md file should include:

This ensures that every developer, regardless of their machine, can reproduce the exact environment.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.