Prompt
How do I set up end-to-end tests that run on pull requests and nightly builds?
Latest observation
A good pattern is to split fast PR checks from full nightly validation:
- On pull requests: run a smaller, targeted end-to-end suite to catch regressions quickly.
- Nightly: run the full E2E suite, including slower, flaky, or environment-heavy tests.
Recommended setup
1) Organize tests by scope
Tag or split tests into groups, for example:
smoke→ critical flows onlypr→ a small subset suitable for pull requestsnightly→ full suite
If you use Playwright, Cypress, or similar, you can filter by tags, file names, or test markers.
Example:
tests/e2e/smoke/tests/e2e/nightly/
Or with tags:
@smoke@regression@slow
2) Run a quick suite on PRs
Configure your CI to trigger on pull requests and execute only the fast subset.
Typical goals:
- finish in 5–15 minutes
- run in parallel if possible
- fail fast on real regressions
Example CI behavior:
- install dependencies
- start app/services
- run
e2e --grep @smokeor equivalent
3) Run the full suite nightly
Schedule a workflow to run once a day during off-hours.
Typical goals:
- cover all E2E scenarios
- include cross-browser/device matrices if relevant
- capture artifacts like traces, screenshots, and videos
Use the scheduler in your CI platform:
- GitHub Actions:
schedule - GitLab CI: scheduled pipelines
- Jenkins: cron trigger
- CircleCI: scheduled workflows
4) Make the environment reproducible
E2E tests are much easier to maintain if the environment is consistent.
Best practices:
- use containerized test environments
- provision test data/reset DB before each run
- mock external services where appropriate
- use seeded accounts and fixtures
- isolate tests from each other
5) Keep PR tests stable
To avoid noisy failures:
- use retries sparingly
- quarantine flaky tests from PR runs
- keep selectors stable
- wait on app state, not arbitrary sleeps
- use timeouts appropriate to CI
6) Collect artifacts
For failed tests, save:
- screenshots
- videos
- logs
- traces / DOM snapshots
This is especially helpful for nightly runs, where debugging failures may happen later.
Example: GitHub Actions
Here’s a simple pattern using two workflows.
PR workflow
name: e2e-pr
on:
pull_request:
jobs:
e2e:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
- run: npm ci
- run: npm run build
- run: npm run test:e2e:smoke
Nightly workflow
name: e2e-nightly
on:
schedule:
- cron: "0 2 * * *" # every day at 2 AM UTC
workflow_dispatch:
jobs:
e2e:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
- run: npm ci
- run: npm run build
- run: npm run test:e2e
Example npm scripts
{
"scripts": {
"test:e2e:smoke": "playwright test --grep @smoke",
"test:e2e": "playwright test"
}
}
Practical recommendations
- Start with a very small PR suite and expand gradually.
- Keep nightly runs authoritative for full coverage.
- If nightly failures happen, automatically notify the team via Slack/email.
- Treat E2E tests as product code: review, refactor, and maintain them.
If you want, I can also give you:
- a GitHub Actions example for Playwright/Cypress,
- a GitLab CI/Jenkins setup, or
- a recommended test folder/tagging strategy.