Prompt
Why is my A/B test not splitting traffic evenly?
Latest observation
Uneven A/B traffic splits usually come from one of a few places:
-
The randomization isn’t truly random
If assignment is based on a weak hash, time, IP, browser fingerprint, or “first come, first served,” the split can drift. -
You’re splitting at the wrong level
If users can see the experiment on multiple devices, sessions, or pages, and the unit of randomization is inconsistent, traffic won’t look even. -
Filtering after assignment
If you assign 50/50 but then exclude bots, internal traffic, errors, or users who fail tracking, the observed split can become uneven. -
Sample ratio mismatch from implementation bugs
Common causes: one variant failing to load, redirect loops, JS errors, one bucket dropping more requests, or server-side routing issues. -
Caching or load balancing effects
A CDN, edge cache, or sticky session can send more traffic to one variant than intended. -
Consent or ad blockers
If analytics only records users who accept consent or don’t block scripts, measurement can be biased. -
The experiment is segmented unevenly
If you’re looking at a subset by geography, device, or campaign, the split may be even overall but not within that segment. -
Small sample size / normal variance
Early in a test, a 50/50 split can still look off just by chance.
What to check:
- Compare assigned traffic vs tracked traffic.
- Verify the split at the assignment point, not just in analytics.
- Look for variant-specific errors, missing events, or redirects.
- Confirm the randomization unit is user/session/device as intended.
- Check for filters, consent loss, and bot exclusion.
If you want, I can help you debug it systematically—tell me whether your test is client-side or server-side, and what platform/tool you’re using.
Brands