Feature Flags at Scale: Decoupling Deployment from Release

Feature flags at scale decouple the act of deploying code from releasing it to users. When your team deploys a new service, a database migration, or a UI overhaul, the code is live—but users see it only when the release is *cut*. Feature flags allow you to deploy code anytime, and release features incrementally, gradually, or even instantly, without any downtime or coordination. This is essential when multiple teams release independently, when release cadences differ, or when rollbacks must be fast and low-risk.

Planning a Feature Flag Strategy

A feature flag strategy begins with a clear mapping of *what* is controlled by flags and *how* they are managed across environments and teams.

Each feature flag is defined by a unique identifier (e.g., new-checkout-flow), a default value (e.g., false), and a set of targeting rules. The flag’s state is evaluated at runtime, and the evaluation is consistent across all services.

In production, a feature flag may be configured via a centralized configuration service. The configuration is a JSON object stored in a database or a key-value store. The schema for a flag’s configuration is:

{
  "id": "new-checkout-flow",
  "name": "New Checkout Flow",
  "description": "Replace the legacy checkout with a modern, multi-step interface.",
  "default": false,
  "targets": [
    {
      "userId": "12345",
      "variant": "beta"
    },
    {
      "emailDomain": "acme.com",
      "variant": "stable"
    }
  ],
  "variants": {
    "beta": {
      "enabled": true,
      "config": {
        "showShippingEstimates": true,
        "enableGiftWrapping": true
      }
    },
    "stable": {
      "enabled": true,
      "config": {
        "showShippingEstimates": false,
        "enableGiftWrapping": false
      }
    }
  }
}

The flag is evaluated by checking if any target matches the current context (e.g., user ID, email domain, region, device type). Evaluation order is: specific user > domain > region > default. The system returns the variant object as the effective configuration for the flag.

This configuration is updated via a REST API:

PATCH /api/feature-flags/new-checkout-flow HTTP/1.1
Host: config.platform.example.com
Authorization: Bearer <token>
Content-Type: application/json

{
  "default": true,
  "targets": [
    {
      "userId": "12345",
      "variant": "beta"
    },
    {
      "emailDomain": "acme.com",
      "variant": "stable"
    }
  ],
  "variants": {
    "beta": {
      "enabled": true,
      "config": {
        "showShippingEstimates": true
      }
    }
  }
}

Deploying a Feature Flagged Service

When a service uses feature flags, it must evaluate them at runtime. The evaluation process is idempotent and must be fast. A common pattern is to load flag configurations once at startup and refresh them periodically.

In a Go service using a feature flag client, the initialization code is:

func main() {
    cfg := &config.Config{
        FeatureFlags: config.FeatureFlagConfig{
            Endpoint: "https://config.platform.example.com/api/feature-flags",
            RefreshInterval: 30 * time.Second,
            CacheTTL: 60 * time.Second,
        },
    }

    client, err := feature.NewClient(cfg.FeatureFlags)
    if err != nil {
        log.Fatal("failed to create feature client:", err)
    }

    // Start background refresh
    go client.Start()

    // Use flag in request handler
    http.HandleFunc("/checkout", func(w http.ResponseWriter, r *http.Request) {
        if client.BoolValue("new-checkout-flow", false) {
            // Use new checkout flow
            w.Write([]byte("Welcome to the new checkout!"))
        } else {
            // Use legacy flow
            w.Write([]byte("Welcome to the old checkout."))
        }
    })

    log.Fatal(http.ListenAndServe(":8080", nil))
}

The client fetches flag definitions from the config service using GET /api/feature-flags?env=prod&service=checkout. It caches the results in memory and uses Cache-Control: max-age=60 to avoid stale flags.

If the config service is unreachable, the client falls back to the last known configuration. The RefreshInterval is used to pull new configs in the background. If the service restarts, it uses the cached config until the first refresh completes.

Managing Feature Flag Rollouts

Rollouts are the primary mechanism for releasing features. A rollout defines a phased release strategy—canary, gradual, staged, or phased rollout.

A rollout is represented as a sequence of steps, each with a duration and a flag state. Each step can be a simple update or a complex split.

Example rollout configuration:

{
  "id": "rollout-new-checkout",
  "name": "Roll out new checkout to 100% of users",
  "steps": [
    {
      "start": "2024-03-15T08:00:00Z",
      "duration": "PT1H",
      "flag": "new-checkout-flow",
      "value": true,
      "conditions": {
        "region": "us-west",
        "device": "mobile"
      }
    },
    {
      "start": "2024-03-15T10:00:00Z",
      "duration": "PT2H",
      "flag": "new-checkout-flow",
      "value": true,
      "conditions": {
        "region": "eu-central",
        "device": "desktop"
      }
    },
    {
      "start": "2024-03-15T12:00:00Z",
      "duration": "PT24H",
      "flag": "new-checkout-flow",
      "value": true,
      "conditions": {}
    }
  ]
}

Each step is evaluated in chronological order. The system checks for active rollouts every 15 seconds. When a step starts, it updates the flag’s default value and optionally adds new targets.

A rollout can be triggered via an API:

POST /api/rollouts HTTP/1.1
Host: rollout.platform.example.com
Authorization: Bearer <token>
Content-Type: application/json

{
  "name": "Roll out new checkout",
  "steps": [
    {
      "start": "2024-03-15T08:00:00Z",
      "duration": "PT1H",
      "flag": "new-checkout-flow",
      "value": true
    }
  ]
}

The rollout system maintains a log of execution, including:

• rollout.id and step.id
• started_at, completed_at
• status: pending, running, completed, failed
• error: message, code, details

If a step fails—e.g., the config service is down during a flag update—the rollout enters failed state and sends alerts via webhook or email.

Debugging Feature Flag Failures

Feature flags fail silently in production. The most common failure is *flag drift*: the flag value in the database does not match the effective value used by services.

A service may report that new-checkout-flow is true, but users see the old checkout. The cause is often a misconfigured default value.

Debugging starts with querying the flag configuration:

GET /api/feature-flags/new-checkout-flow?env=prod&service=checkout HTTP/1.1
Host: config.platform.example.com
Authorization: Bearer <token>

The response includes:

• last_updated: ISO 8601 timestamp
• refreshed_at: when the client last fetched the config
• effective_value: the value computed from evaluation rules
• evaluation_context: the input used to compute the value (e.g., user_id=12345, region=us-west)

A mismatch between effective_value and default indicates a targeting rule is not being applied.

Another common issue is *flag staleness*. The client fetches the config every 30 seconds, but a rollout step updates the flag at 12:00, and the next refresh happens at 12:31. During that time, users may see inconsistent behavior.

To detect staleness, the service logs the refreshed_at timestamp with every flag evaluation:

[INFO] feature_flag: new-checkout-flow, effective=true, evaluated_at=2024-03-15T12:05:23Z, refreshed_at=2024-03-15T11:55:00Z

If evaluated_at is older than refreshed_at, the flag is stale.

A silent failure occurs when a variant is defined but not used. The variants map may contain a beta variant, but the default value is true, so the beta variant is never applied. This happens when a rollout step sets value: true but forgets to set variant: "beta".

To catch this, the config service logs every flag evaluation:

{
  "flag": "new-checkout-flow",
  "user_id": 12345,
  "region": "us-west",
  "device": "mobile",
  "effective_value": true,
  "variant": "beta",
  "evaluated_at": "2024-03-15T12:05:23Z",
  "config_version": "v3"
}

The evaluation logs are used to detect regression: a user with user_id=12345 sees new-checkout-flow=false, but the evaluation log shows variant: "beta" and effective_value: true. The cause is likely a missing or misconfigured targets rule.

Rolling Back a Feature

Rollbacks are fast and low-risk when using feature flags. A rollback is not a deployment, but a configuration change.

To roll back new-checkout-flow, the team sends a request:

PATCH /api/feature-flags/new-checkout-flow HTTP/1.1
Host: config.platform.example.com
Authorization: Bearer <token>
Content-Type: application/json

{
  "default": false,
  "targets": [
    {
      "emailDomain": "acme.com",
      "variant": "stable"
    }
  ],
  "variants": {
    "stable": {
      "enabled": true,
      "config": {
        "showShippingEstimates": false
      }
    }
  }
}

The change is applied immediately across all services. The default value is set to false, so all users fall back to the old checkout. Services that use the flag evaluate it on the next request, and within minutes, users see the rollback.

For a gradual rollback, the team can set up a new rollout with a series of steps:

{
  "id": "rollback-new-checkout",
  "steps": [
    {
      "start": "2024-03-15T13:00:00Z",
      "duration": "PT30M",
      "flag": "new-checkout-flow",
      "value": false,
      "conditions": {
        "region": "us-west"
      }
    },
    {
      "start": "2024-03-15T14:00:00Z",
      "duration": "PT1H",
      "flag": "new-checkout-flow",
      "value": false,
      "conditions": {
        "region": "eu-central"
      }
    }
  ]
}

This allows the team to roll back in phases, monitoring metrics before fully disabling the feature.

Monitoring and Observability

Feature flags are only effective if they are observable. The system must provide a dashboard that shows:

• Flag state across all environments
• Active rollouts and their progress
• Evaluation logs and errors
• User-facing metrics: percentage of users on new flow, time-to-activation, error rate

A critical metric is *flag activation rate*: the ratio of users who see the flag’s effect to those who should.

The system tracks activation via a flag-activation log:

{
  "user_id": 12345,
  "flag": "new-checkout-flow",
  "activated_at": "2024-03-15T12:05:23Z",
  "effective_value": true,
  "variant": "beta",
  "context": {
    "region": "us-west",
    "device": "mobile"
  }
}

A user is considered *activated* when the flag’s value is true for their session. The system computes the activation rate per rollout and per variant.

An alert triggers when the activation rate drops below 80% for a flag. The alert includes:

• flag_name
• current_activation_rate
• target_rate
• last_30_minutes
• failed_evaluations

Failed evaluations are logged with the raw evaluation context and the result of each rule.

Best Practices and Pitfalls

• Always use default + targets: avoid relying solely on default. A flag with only a default value is static and cannot evolve.
• Evaluate flags early: in middleware or request handlers. Do not evaluate flags only in controllers.
• Use variants for configuration: config fields in variants allow feature-specific settings without code changes.
• Version flag configurations: include config_version in the evaluation logs. This allows tracing changes through time.
• Disable flags on failure: when a service fails to evaluate a flag, fall back to a default configuration, and log the error.
• Use refreshed_at as a health signal: if refreshed_at is stale, the config service may be slow or down.

A common mistake is *flag drift*: the default value is set in production, but a service is started with a cached config that predates the change. The service uses the old default until the next refresh.

Another pitfall is *overlapping rollouts*. A rollout updates new-checkout-flow to true, but another rollout updates it to false in the same window. The system must handle concurrent updates—preferably using ETag or if-match in HTTP.

Finally, never assume that a flag is *on* just because its value is true. A flag is *active* only when its effective value matches the expected behavior. A true flag with no variants is often a misconfiguration.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.