How can I catch performance regressions in CI before deploy

To catch performance regressions in CI before deploy, the goal is to make performance repeatable, measurable, and gated. A practical approach is:

1. Define a few critical performance budgets

Pick the metrics that matter most for your app, for example:

• API latency: p50 / p95 / p99
• Page load metrics: LCP, INP, TTFB
• Throughput: requests/sec
• Resource usage: CPU, memory
• Query time / DB load

Set explicit thresholds, e.g.:

• “p95 checkout API must stay under 250 ms”
• “LCP must not exceed 2.5 s on a reference device/network”
• “Regression > 10% fails CI”

2. Run performance tests in a controlled environment

Keep test conditions as stable as possible:

• Dedicated CI runner or isolated environment
• Fixed CPU/memory limits
• Same test data and seed
• Same software versions
• Disable noisy background jobs
• Run multiple iterations and use median/percentiles

This reduces flakiness and false alarms.

3. Compare against a baseline

Performance checks are most useful when compared to a known-good baseline:

• Store historical metrics from main branch
• Compare the PR build to the latest successful baseline
• Fail or warn if the delta exceeds your threshold

Useful rule:

• small changes: warn
• large regressions: fail

4. Automate benchmarks in CI

Add a CI job that runs:

• Unit-level microbenchmarks for hot functions
• Integration benchmarks for critical flows
• End-to-end synthetic tests for user journeys

Examples:

• Backend: benchmark login, search, checkout endpoints
• Frontend: run Lighthouse/WebPageTest or browser automation
• Database: measure key query plans and timings

5. Use performance testing tools

Depending on stack, common options include:

• Frontend: Lighthouse CI, WebPageTest, Playwright + performance marks
• Backend/API: k6, JMeter, Locust, wrk, hey
• Node: benchmark.js, autocannon
• JVM: JMH
• Python: pytest-benchmark
• Go: go test -bench

6. Make regressions visible in PRs

Don’t just fail the build silently. Show:

• metric name
• baseline value
• current value
• percent change
• pass/fail reason

This helps developers understand and fix the issue quickly.

7. Track trends over time

Store results in a time-series system or dashboard:

• Grafana / Prometheus
• Datadog
• BigQuery / Snowflake
• GitHub checks + artifacts

Trend visibility helps spot gradual degradation that thresholds may miss.

8. Reduce flakiness

Performance tests can be noisy, so:

• Warm up the system before measuring
• Run several samples
• Use medians or trimmed means
• Retry only if you have strong evidence of test noise
• Avoid shared CI runners for hard gates

9. Separate fast checks from deep tests

Use two layers:

• Per-PR quick checks: lightweight benchmarks, smoke perf tests
• Nightly/weekly deep tests: load tests, longer soak tests, broader scenarios

This keeps CI fast while still catching meaningful regressions.

10. Alert only on meaningful changes

Set thresholds that reflect real user impact, not tiny noise:

• 5–10% for stable benchmarks
• Larger thresholds for noisy environments
• Absolute thresholds for latency SLOs

---

Example CI flow

1. Build PR
2. Run tests
3. Run performance benchmark job
4. Compare metrics against baseline
5. Fail if regression > threshold
6. Upload report and trend data

---

Practical recommendation

If you want the highest value with the least effort, start with:

• 1–3 critical user journeys
• one or two key metrics each
• a baseline from main
• a CI gate with a 10% regression threshold
• trend dashboards for everything else

If you tell me your stack (frontend/backend language, CI system, and what you care about most), I can suggest a concrete setup and tools.