How do I debug an A/B test that changed traffic split unexpectedly?

Here’s a practical way to debug an A/B test when the traffic split changed unexpectedly.

1) Confirm the symptom

First determine whether the split really changed or whether it’s a measurement issue.

• Check raw assignment counts by variant, not just downstream metrics.
• Compare:
  • assigned users/sessions
  • eligible users
  • exposed users
  • logged events
• Make sure you’re looking at the same time window and same population.

2) Identify where the split is applied

Traffic split can change at different layers:

• Experiment platform (feature flag / bucketing service)
• Application code (custom assignment logic)
• CDN / edge / routing layer
• Analytics pipeline (assignment recorded correctly but reported incorrectly)

Find the exact place where users are assigned to variants.

3) Check for recent config changes

Look for:

• Experiment allocation weights changed
• Ramp-up settings modified
• Targeting rules changed
• Exclusion rules added/removed
• Holdout or mutual exclusion group changes
• Kill switch or fallback behavior triggered

Also check whether the experiment was re-published or partially rolled out.

4) Verify bucketing logic

Common causes:

• Different hash inputs: user ID, session ID, device ID, anonymous ID
• ID instability: users changing IDs between requests
• Salt changes: experiment key or salt changed
• Modulo / rounding errors in allocation percentages
• Non-deterministic assignment due to randomness instead of stable hashing

Questions to ask:

• Is bucketing done on a stable identifier?
• Did the identifier source change?
• Is the same user being assigned consistently across devices/sessions?

5) Look for population drift

The split may appear changed because the population changed.

• Traffic source mix changed
• Mobile vs desktop mix changed
• Logged-in vs logged-out mix changed
• Geography changed
• Bot filtering changed
• Eligibility criteria filtered out one variant more than the other

If assignment is correct but exposure differs, the issue may be in eligibility or page rendering.

6) Check sample ratio mismatch (SRM)

Run an SRM test to see if observed counts deviate significantly from expected allocation.

If there’s SRM:

• It often indicates an instrumentation, targeting, or assignment bug
• Or traffic loss in one arm

If there’s no SRM but metrics look odd:

• The split may be fine, but exposure or logging is biased

7) Inspect logs end-to-end

Trace a small set of users through the system:

• Incoming request
• Assignment decision
• Variant served
• Event logged
• Analytics ingestion

Compare expected vs actual for each step. This helps locate where the split diverges.

Useful debug fields:

• user/session ID
• experiment key
• allocated variant
• allocation timestamp
• eligibility reason
• serving variant
• request ID / trace ID

8) Check for caching or sticky sessions

Unexpected split changes can happen when:

• CDN caches variant-specific pages
• Sticky sessions are enabled/disabled
• Load balancer affinity changes
• A cookie is missing, expired, or blocked
• Server restarts clear in-memory assignment state

If variant is stored in cookie/local storage, verify persistence and scope.

9) Review deployment and rollback history

Correlate the split change with:

• code deploys
• config updates
• experiment platform releases
• infra changes
• database migrations
• feature flag service outages

A timing match often reveals the culprit.

10) Reproduce locally or in staging

Try:

• Reassigning a fixed set of test IDs
• Using the same bucketing inputs
• Comparing expected allocation with actual
• Simulating cookie loss, new sessions, or ID changes

This is especially useful if assignment depends on multiple identifiers.

11) Check analysis/reporting logic

Sometimes the experiment is fine, but reporting is wrong:

• Wrong denominator
• Deduping issue
• Late-arriving events
• ETL filters dropping one variant
• Join keys missing
• Time zone or windowing bug

Validate raw assignment data before trusting dashboard aggregates.

12) Common root causes

Most frequent causes of unexpected split changes:

• Allocation config changed
• Bucketing ID changed or became unstable
• SRM due to logging loss
• Eligibility filter bug
• Cookie/local storage issue
• Caching/sticky-session issue
• Dashboard/reporting bug

A simple debugging checklist

1. Compare expected vs observed allocation.
2. Run SRM.
3. Verify assignment logs.
4. Trace a few users end-to-end.
5. Check recent config/deploy changes.
6. Validate bucketing ID stability.
7. Inspect eligibility and exposure logic.
8. Confirm reporting pipeline isn’t distorting the split.

If you want, I can also give you:

• a step-by-step SRM debugging workflow
• a SQL query template to check assignment imbalance
• a checklist for feature flag systems
• or a root-cause tree for A/B traffic split issues