Prompt

How do I set up browser automation with proxy rotation and session management?

Technology / Testing & QA1 observationLast seen Oct 26, 2026

Latest observation

Oct 26, 2026OpenAI APIWeb search: off

Here’s a practical way to think about it:

1) Pick your automation stack

Common choices:

  • Playwright: best modern option for reliable automation
  • Selenium: older, very common
  • Puppeteer: great if you’re mainly using Chromium

If you want good session handling and easy proxy per context, Playwright is usually the easiest.


2) Separate three concerns

You want to manage:

A. Browser automation

The code that drives the browser:

  • open page
  • click
  • fill forms
  • scrape data

B. Proxy rotation

A pool of proxies, e.g.

  • datacenter proxies
  • residential proxies
  • mobile proxies

Rotation strategy:

  • rotate every session
  • rotate every N requests
  • rotate on block / captcha / error
  • sticky sessions for a fixed time window

C. Session management

Keep track of:

  • cookies
  • localStorage / sessionStorage
  • login state
  • user-agent
  • fingerprints if needed

The safest pattern is:

  • one proxy = one browser context/session
  • persist cookies/storage per identity
  • reuse the same proxy for that session until it expires

3) Recommended architecture

A clean design is:

  • Proxy Manager

    • hands out proxies
    • tracks health / failures
    • supports sticky assignment
  • Session Store

    • saves cookies, localStorage, auth tokens
    • keyed by session ID / account / workflow
  • Browser Runner

    • starts browser instances or contexts
    • loads session data
    • binds to a proxy
    • performs tasks
    • saves updated session state

4) Playwright example pattern

In Playwright, you can set a proxy when creating a browser context or launching the browser.

Basic idea

  • create a context with proxy
  • use storageState to persist session

Example in Node.js

const { chromium } = require('playwright');

async function run(proxy, storageFile) {
  const browser = await chromium.launch({
    headless: true,
  });

  const context = await browser.newContext({
    proxy: {
      server: proxy.server, // e.g. http://host:port
      username: proxy.username,
      password: proxy.password,
    },
    storageState: storageFile || undefined,
  });

  const page = await context.newPage();
  await page.goto('https://example.com');

  // do stuff...

  await context.storageState({ path: storageFile });
  await browser.close();
}

5) Proxy rotation strategy

Option 1: Rotate per session

Best for:

  • logged-in workflows
  • stable identities
  • fewer detection issues

Flow:

  1. pick a proxy
  2. create browser context
  3. run tasks
  4. save session state
  5. retire or reuse the same proxy for that session

Option 2: Rotate per task/request

Best for:

  • public scraping
  • high-volume, stateless access

Downside:

  • breaks login/session continuity
  • more likely to trigger suspicious behavior

Option 3: Sticky sessions

Use one proxy for a time window, e.g. 10–30 minutes, then rotate.

This is useful if:

  • site tracks IP consistency
  • you need a stable identity during a workflow

6) Session persistence

Playwright

Use storageState to save/load:

  • cookies
  • localStorage

Example:

await context.storageState({ path: 'session.json' });

Later:

const context = await browser.newContext({
  proxy: {...},
  storageState: 'session.json',
});

Selenium

You typically:

  • save cookies manually
  • restore cookies after login
  • optionally reuse a browser profile directory

Chrome profile approach

For persistent sessions, you can use a dedicated user profile directory per identity:

  • --user-data-dir=/path/to/profile1

This preserves more than cookies, but can be heavier and harder to manage at scale.


7) Good proxy hygiene

A proxy pool should track:

  • success rate
  • recent failures
  • latency
  • country/geo
  • ISP/type (datacenter vs residential)

Suggested rules:

  • mark proxy unhealthy after repeated timeouts / bans
  • cooldown proxies after failures
  • prefer proven proxies for login flows
  • don’t switch proxies mid-login unless needed

8) Detect and handle blocks

Your automation should watch for:

  • HTTP 403/429
  • unexpected redirects
  • captcha pages
  • empty/blocked responses
  • navigation timeouts

When detected:

  • retry with a fresh proxy/session
  • back off
  • reduce request rate
  • avoid hammering the site

9) Concurrency model

If running many sessions:

  • one worker thread/process per session is simpler
  • map each session to a proxy
  • store session metadata in Redis / database / file system

Example state per session:

{
  "sessionId": "acct_123",
  "proxy": "proxy-7",
  "storageFile": "sessions/acct_123.json",
  "status": "active",
  "lastUsed": "2026-10-26T12:00:00Z"
}

10) Practical best practices

  • Keep identity consistency: same proxy, cookie state, and UA for a session
  • Don’t rotate proxies too aggressively within an authenticated session
  • Add random delays and human-like pacing
  • Use retries with exponential backoff
  • Log proxy/session errors separately
  • Encrypt session files if they contain auth tokens
  • Respect site terms and legal requirements

11) If you want a simple production-ready pattern

Use:

  • Playwright
  • one session per browser context
  • proxy assigned at session creation
  • persist storage state to disk
  • proxy health tracking in Redis/Postgres
  • sticky sessions
  • automatic fallback on failure

If you want, I can give you:

  1. a Playwright + Node.js starter project,
  2. a Python Selenium version, or
  3. a proxy/session manager design with code.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (1 observation for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.