flaky ui tests in ci

Flaky UI tests in CI are usually caused by timing, environment differences, shared state, or test/data instability. The key is to make failures reproducible, then attack the root cause—not just retry forever.

Common causes

• Async timing issues: element not ready yet, animations still running, network calls unresolved
• Race conditions: test actions happen faster/slower in CI than locally
• Selector instability: brittle CSS/XPath selectors, dynamic IDs
• Shared state: tests depend on prior tests, leftover cookies/localStorage/db state
• Environment differences: headless browser behavior, viewport, CPU throttling, timezone/locale
• External dependencies: real APIs, third-party services, unstable test data
• Test pollution: parallel tests writing to same user/account/data

How to stabilize them

• Prefer stable selectors like data-testid
• Wait for specific conditions, not arbitrary sleeps
• Mock or stub network calls where possible
• Reset state before each test:
  • cookies/localStorage/sessionStorage
  • DB/test fixtures
  • user/account data
• Make tests independent and idempotent
• Use consistent CI config:
  • viewport
  • timezone
  • locale
  • browser version
• Avoid relying on exact animation timing; disable animations in tests
• Run tests in isolation to find order dependencies
• Capture artifacts on failure:
  • screenshots
  • video
  • console logs
  • network logs
  • DOM snapshot

Debugging strategy

1. Rerun the failed test alone
2. Run it repeatedly (e.g., 20–50 times) to confirm flakiness
3. Compare local vs CI environment
4. Look at the first point of divergence in logs/artifacts
5. Remove arbitrary waits and replace with explicit waits
6. Reduce external dependencies with mocks/fixtures
7. Check for hidden shared state

CI-specific improvements

• Increase browser startup stability and resource limits
• Use dedicated test containers/VMs
• Serialize tests that touch shared resources
• Quarantine known flaky tests temporarily, but track and fix them
• Add automatic retry only as a short-term mitigation

Good rule of thumb

If a UI test fails intermittently, assume:

• it’s waiting on something it doesn’t explicitly know about, or
• it’s depending on state it doesn’t fully control.

If you want, I can help you with:

• a flaky test debugging checklist
• Playwright/Cypress/Selenium-specific fixes
• a strategy to isolate and reproduce a CI-only UI flake