Prompt
How do I automate browser tasks in a way that is reliable enough for production?
Latest observation
To automate browser tasks reliably in production, treat browser automation like a distributed system with a flaky UI, not like a simple script. The main goal is to make it deterministic, observable, and recoverable.
1) Prefer APIs over browser automation when possible
If the website has:
- a public API,
- an internal endpoint you can use legally,
- or an export/import mechanism,
use that instead. Browser automation should be the last resort because UIs change, load unpredictably, and break often.
2) Use a solid browser automation stack
Common choices:
- Playwright: usually the best choice for production reliability
- Puppeteer: good, but less comprehensive than Playwright
- Selenium: broad support, but generally more brittle and slower
For most modern production use cases, Playwright is the default recommendation.
3) Make every step explicit and wait for the right condition
Avoid arbitrary sleeps like:
await page.waitForTimeout(5000)
Instead wait for:
- element visibility
- element enabled state
- network idle
- specific URL change
- a DOM marker that indicates the app is ready
Example:
await page.waitForSelector('button[type="submit"]', { state: 'visible' })
await page.click('button[type="submit"]')
await page.waitForURL('**/dashboard')
4) Use stable selectors
Don’t rely on fragile selectors like:
- CSS generated by frameworks
- text that changes often
- nth-child chains
Prefer:
data-testiddata-*attributes you control- accessible roles and labels
Example:
await page.getByTestId('login-button').click()
await page.getByRole('button', { name: 'Log in' }).click()
If you control the app, add automation hooks:
<button data-testid="checkout-submit">Submit</button>
5) Design for idempotency
Production automation should be safe to retry without causing duplicates or bad state.
Examples:
- If submitting a form creates a record, ensure duplicate submissions are detected.
- Store task state externally so a retry knows what happened.
- Use unique request IDs or correlation IDs if the backend supports it.
6) Add retries, but only around safe operations
Retry:
- navigation timeouts
- transient network failures
- temporary 5xx responses
- stale element issues
Do not blindly retry:
- form submissions that may create side effects
- payment actions
- destructive operations
Use a retry policy with:
- max attempts
- exponential backoff
- jitter
- clear failure classification
7) Build strong observability
You need to know what happened when things fail.
Log:
- job/task ID
- URL
- step name
- timing for each step
- browser version
- screenshots on failure
- HTML snapshot or DOM excerpt
- console errors
- network failures
Very useful artifacts:
- screenshot on every failure
- video for critical flows
- trace files if using Playwright
Playwright is especially good here:
- tracing
- screenshots
- video recording
- network inspection
8) Run in a controlled environment
Production browser automation is much more reliable when the environment is stable.
Recommendations:
- use Docker or another containerized runtime
- pin browser and dependency versions
- use the same locale/timezone/font stack consistently
- avoid headful mode unless necessary
- allocate enough CPU/RAM
- isolate jobs from each other
Avoid “works on my laptop” setups.
9) Handle authentication carefully
Authentication is a major source of flakiness.
Best practices:
- use service accounts where possible
- reuse authenticated session state carefully
- avoid logging in through the UI every run if you can persist a session
- handle MFA explicitly if unavoidable
- store secrets securely in a secret manager
If using cookies/session storage, refresh them before expiry.
10) Expect UI changes and plan for them
Production automation breaks because pages change.
Mitigations:
- contract tests for selectors
- a small “canary” job that runs frequently
- alerts when key selectors disappear
- version your automation flow
- keep the site owner in the loop if it’s your app
11) Add defensive checks at each step
Don’t assume a click worked. Verify the result.
Example:
- click “Save”
- wait for success toast
- confirm record exists in table
- confirm URL or server state changed
This prevents silent failures.
12) Use a queue and worker model
For production, don’t run browser tasks as ad hoc scripts.
Use:
- a queue for tasks
- workers with concurrency limits
- job timeout and retry policy
- dead-letter queue for failures
- progress tracking
This makes the system manageable and scalable.
13) Keep tasks small and composable
Break large flows into steps:
- login
- navigate
- fill form
- submit
- verify outcome
That way you can:
- retry specific steps
- isolate failures
- test parts independently
14) Add human fallback where needed
Some tasks will never be fully reliable due to:
- CAPTCHA
- account verification
- ambiguous UI
- changing anti-bot defenses
In those cases, use:
- human-in-the-loop review
- manual approval queue
- partial automation
15) Test like production
Have:
- unit tests for parsing and decision logic
- integration tests against a staging environment
- smoke tests on every deploy
- nightly end-to-end tests
Also test:
- slow network
- element not found
- login expiration
- popups/modals
- browser crashes
A practical production pattern
A good architecture looks like this:
- Task enters queue
- Worker picks up task
- Worker launches isolated browser context
- Executes steps with explicit waits and stable selectors
- Verifies each step
- Captures logs/screenshots/traces
- Marks task success or failure
- Retries only safe failures
- Sends alert if repeated failures occur
A short rule of thumb
If you want browser automation to be production-grade, make sure it is:
- deterministic
- observable
- idempotent
- retryable
- isolated
- tested
- easy to debug
If you want, I can also give you:
- a Playwright production template,
- a retry/error-handling pattern, or
- a production architecture diagram for browser automation.