Prompt

How do I automate browser tasks in a way that is reliable enough for production?

Technology / Testing & QA1 observationLast seen Oct 26, 2026

Latest observation

Oct 26, 2026OpenAI APIWeb search: off

To automate browser tasks reliably in production, treat browser automation like a distributed system with a flaky UI, not like a simple script. The main goal is to make it deterministic, observable, and recoverable.

1) Prefer APIs over browser automation when possible

If the website has:

  • a public API,
  • an internal endpoint you can use legally,
  • or an export/import mechanism,

use that instead. Browser automation should be the last resort because UIs change, load unpredictably, and break often.

2) Use a solid browser automation stack

Common choices:

  • Playwright: usually the best choice for production reliability
  • Puppeteer: good, but less comprehensive than Playwright
  • Selenium: broad support, but generally more brittle and slower

For most modern production use cases, Playwright is the default recommendation.

3) Make every step explicit and wait for the right condition

Avoid arbitrary sleeps like:

await page.waitForTimeout(5000)

Instead wait for:

  • element visibility
  • element enabled state
  • network idle
  • specific URL change
  • a DOM marker that indicates the app is ready

Example:

await page.waitForSelector('button[type="submit"]', { state: 'visible' })
await page.click('button[type="submit"]')
await page.waitForURL('**/dashboard')

4) Use stable selectors

Don’t rely on fragile selectors like:

  • CSS generated by frameworks
  • text that changes often
  • nth-child chains

Prefer:

  • data-testid
  • data-* attributes you control
  • accessible roles and labels

Example:

await page.getByTestId('login-button').click()
await page.getByRole('button', { name: 'Log in' }).click()

If you control the app, add automation hooks:

<button data-testid="checkout-submit">Submit</button>

5) Design for idempotency

Production automation should be safe to retry without causing duplicates or bad state.

Examples:

  • If submitting a form creates a record, ensure duplicate submissions are detected.
  • Store task state externally so a retry knows what happened.
  • Use unique request IDs or correlation IDs if the backend supports it.

6) Add retries, but only around safe operations

Retry:

  • navigation timeouts
  • transient network failures
  • temporary 5xx responses
  • stale element issues

Do not blindly retry:

  • form submissions that may create side effects
  • payment actions
  • destructive operations

Use a retry policy with:

  • max attempts
  • exponential backoff
  • jitter
  • clear failure classification

7) Build strong observability

You need to know what happened when things fail.

Log:

  • job/task ID
  • URL
  • step name
  • timing for each step
  • browser version
  • screenshots on failure
  • HTML snapshot or DOM excerpt
  • console errors
  • network failures

Very useful artifacts:

  • screenshot on every failure
  • video for critical flows
  • trace files if using Playwright

Playwright is especially good here:

  • tracing
  • screenshots
  • video recording
  • network inspection

8) Run in a controlled environment

Production browser automation is much more reliable when the environment is stable.

Recommendations:

  • use Docker or another containerized runtime
  • pin browser and dependency versions
  • use the same locale/timezone/font stack consistently
  • avoid headful mode unless necessary
  • allocate enough CPU/RAM
  • isolate jobs from each other

Avoid “works on my laptop” setups.

9) Handle authentication carefully

Authentication is a major source of flakiness.

Best practices:

  • use service accounts where possible
  • reuse authenticated session state carefully
  • avoid logging in through the UI every run if you can persist a session
  • handle MFA explicitly if unavoidable
  • store secrets securely in a secret manager

If using cookies/session storage, refresh them before expiry.

10) Expect UI changes and plan for them

Production automation breaks because pages change.

Mitigations:

  • contract tests for selectors
  • a small “canary” job that runs frequently
  • alerts when key selectors disappear
  • version your automation flow
  • keep the site owner in the loop if it’s your app

11) Add defensive checks at each step

Don’t assume a click worked. Verify the result.

Example:

  • click “Save”
  • wait for success toast
  • confirm record exists in table
  • confirm URL or server state changed

This prevents silent failures.

12) Use a queue and worker model

For production, don’t run browser tasks as ad hoc scripts.

Use:

  • a queue for tasks
  • workers with concurrency limits
  • job timeout and retry policy
  • dead-letter queue for failures
  • progress tracking

This makes the system manageable and scalable.

13) Keep tasks small and composable

Break large flows into steps:

  • login
  • navigate
  • fill form
  • submit
  • verify outcome

That way you can:

  • retry specific steps
  • isolate failures
  • test parts independently

14) Add human fallback where needed

Some tasks will never be fully reliable due to:

  • CAPTCHA
  • account verification
  • ambiguous UI
  • changing anti-bot defenses

In those cases, use:

  • human-in-the-loop review
  • manual approval queue
  • partial automation

15) Test like production

Have:

  • unit tests for parsing and decision logic
  • integration tests against a staging environment
  • smoke tests on every deploy
  • nightly end-to-end tests

Also test:

  • slow network
  • element not found
  • login expiration
  • popups/modals
  • browser crashes

A practical production pattern

A good architecture looks like this:

  1. Task enters queue
  2. Worker picks up task
  3. Worker launches isolated browser context
  4. Executes steps with explicit waits and stable selectors
  5. Verifies each step
  6. Captures logs/screenshots/traces
  7. Marks task success or failure
  8. Retries only safe failures
  9. Sends alert if repeated failures occur

A short rule of thumb

If you want browser automation to be production-grade, make sure it is:

  • deterministic
  • observable
  • idempotent
  • retryable
  • isolated
  • tested
  • easy to debug

If you want, I can also give you:

  1. a Playwright production template,
  2. a retry/error-handling pattern, or
  3. a production architecture diagram for browser automation.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (1 observation for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.