Mobile A B Testing

Production engineering guide for mobile a b testing covering patterns, implementation strategies, and operational best practices.

Mobile A/B testing is the practice of releasing two or more versions of an app to different user segments and measuring which version performs better on defined metrics. It matters whenever you need to validate a change in behavior, UI, or data flow before committing to a full release, or when you need to optimize an existing experience without forcing every user onto a new version. In mobile, testing is harder than web because deployments require app store cycles, so the mechanism must work through feature flags, remote configuration, or runtime branching without requiring a new binary for every experiment.

Defining variants and routing

The core unit of mobile A/B testing is the feature flag or toggle that switches behavior at runtime. A typical implementation stores a JSON payload remote-config style, fetched on app launch and cached locally. The payload maps experiment keys to variant assignments, often hashed from user ID to ensure consistent grouping.

{
  "experiment_new_onboarding": {
    "variant_a": 0.5,
    "variant_b": 0.5
  },
  "experiment_button_color": {
    "primary_blue": "0xff0066ff",
    "secondary_green": "0x00ff6600"
  }
}

The SDK initializes this payload on first launch, hashes the current user identifier against the defined percentages, and stores the resulting variant. Subsequent runs read the cached value, so the assignment is fixed for the session but can change across app updates when the remote payload is refreshed.

Concrete check pattern (generic SDK-agnostic):

// Pseudocode: feature flag check at point of divergence
val isVariantB = features.getBoolean("experiment_new_onboarding") == false
if (isVariantB) {
    showNewOnboardingFlow()
} else {
    showClassicOnboardingFlow()
}

In practice, the flag key, the variant logic, and the percentage split are defined in a remote config system or a feature-flag service. The mobile app never computes the split locally; it defers to the fetched payload. If the payload is stale or missing, the app must have a sensible default—usually the control variant—to avoid crashing the launch sequence.

Deploying variants without an app store release

The primary advantage of mobile A/B testing is deploying variants without submitting a new binary. This is achieved through remote configuration services that push JSON payloads over HTTPS, or through in-app update mechanisms that fetch flag definitions at runtime.

A typical flow:

  1. Config change is made in the remote-control console (or via API if you manage your own).
  2. Payload is published and propagated to edge endpoints.
  3. App fetches the payload on next launch, usually within the first few seconds of onCreate/applicationDidFinishLaunching.
  4. Variant assignment is read from the local cache; the UI branches accordingly.

Typical curl-based inspection if you have access to the remote-config endpoint:

curl -H "Authorization: Bearer $TOKEN" \
  https://config.example.com/experiments \
  | jq '.["experiment_button_color"]'

If you are using a managed service, the API surface will differ, but the pattern is always the same: a GET request returning feature-flag state, cached locally, and read at the point of divergence.

Common confusion: percentage splits are often expressed as integers that sum to 100, or as floating-point values that sum to 1.0. Mixing these representations causes silent misrouting—half the users see the treatment while the other half sees control, but the assignment is no longer balanced. Always verify the schema of the payload your platform emits.

Measuring results and event attribution

Once variants are running, you need to measure whether the observed difference is statistically meaningful. Mobile analytics pipelines typically instrument two categories of events:

The critical linkage is the experiment ID. Every conversion event must carry the experiment identifier as a tag so that analytics can aggregate by variant. If that tag is missing or mismatched, you will see aggregate numbers but cannot attribute them to a specific arm.

Typical event payload shape (generic):

{
  "event_name": "purchase_completed",
  "experiment_id": "exp_01234",
  "experiment_variant": "variant_b",
  "price": 19.99,
  "timestamp": "2024-04-01T14:32:10Z"
}

If your analytics backend does not automatically append the experiment tag, you must inject it in the SDK layer before dispatch. A frequent failure mode is forgetting to tag events after a variant switch mid-experiment, which bifurcates the data set and renders the test inconclusive.

Sharp edges: what silently fails

Stale cache after remote-config update

If the remote-config service pushes a new payload but the app process does not restart or re-fetch, the old variant assignment persists until the next app launch. This is especially problematic in long-running sessions or backgrounded apps. The cure is to force a config refresh on a known trigger—such as a pull-to-refresh gesture, a deep-link open, or a daily background sync—rather than relying on the OS to invalidate the cache.

Percentage drift

When multiple experiments share the same user-ID hash bucket, the effective split can drift from the intended percentages. A user assigned to 70% for experiment A might effectively be in the 80% bucket for experiment B because the hash functions overlap. This contaminates the A/B results because the groups are no longer independent. The standard mitigation is to salt the user ID with a per-experiment secret before hashing, ensuring disjoint buckets.

Feature-flag race condition

If two experiments toggle the same underlying UI component, the order of flag evaluation matters. Experiment A might enable a new button, while Experiment B hides it based on a different condition. The final rendered state depends on which flag SDK method runs last, which is non-deterministic across app launches and OS restarts. Always gate experiments on independent flags, or compose them explicitly in a single decision point rather than scattering checks throughout the view hierarchy.

Attribution window mismatch

Experiments that span more than one analytics reporting period can produce fragmented data. If an experiment runs from the last two days of one month into the first three days of the next, and your reporting resets monthly, you may see incomplete funnels. Define a fixed experiment window up front, and ensure your analytics pipeline can slice by custom date ranges rather than calendar boundaries.

Debugging a broken experiment

When a test produces no difference between variants, or worse, crashes the app on launch, follow this checklist:

  1. Verify the payload—fetch the remote-config endpoint and confirm the experiment key exists and the variant percentages are what you expect.
  2. Check the local cache—clear the app’s remote-config cache (often under SharedPreferences on Android or UserDefaults on iOS) and relaunch. If the variant changes, the issue was a stale cache.
  3. Inspect the flag evaluation order—add temporary logging at the point of divergence to print which flags are active and in what order. This reveals hidden dependencies between experiments.
  4. Confirm event tagging—verify that every conversion event includes the experiment ID. Use a debug build with verbose logging to see the outbound payload.
  5. Roll back one variable at a time—if multiple experiments are running, disable all but one and re‑measure. This isolates which flag is causing the observed behavior.

Platforms in this space

This pattern is implemented across hosted remote-config services, open-source feature-flag libraries, and in-house config servers. The mechanics—HTTP fetch, JSON payload, hash-based assignment, local caching—are consistent; the surface area (API keys, console UI, SDK methods) varies by vendor.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.