Prompt
How do I set up browser automation with proxy rotation and session management?
Latest observation
Here’s a practical way to think about it:
1) Pick your automation stack
Common choices:
- Playwright: best modern option for reliable automation
- Selenium: older, very common
- Puppeteer: great if you’re mainly using Chromium
If you want good session handling and easy proxy per context, Playwright is usually the easiest.
2) Separate three concerns
You want to manage:
A. Browser automation
The code that drives the browser:
- open page
- click
- fill forms
- scrape data
B. Proxy rotation
A pool of proxies, e.g.
- datacenter proxies
- residential proxies
- mobile proxies
Rotation strategy:
- rotate every session
- rotate every N requests
- rotate on block / captcha / error
- sticky sessions for a fixed time window
C. Session management
Keep track of:
- cookies
- localStorage / sessionStorage
- login state
- user-agent
- fingerprints if needed
The safest pattern is:
- one proxy = one browser context/session
- persist cookies/storage per identity
- reuse the same proxy for that session until it expires
3) Recommended architecture
A clean design is:
-
Proxy Manager
- hands out proxies
- tracks health / failures
- supports sticky assignment
-
Session Store
- saves cookies, localStorage, auth tokens
- keyed by session ID / account / workflow
-
Browser Runner
- starts browser instances or contexts
- loads session data
- binds to a proxy
- performs tasks
- saves updated session state
4) Playwright example pattern
In Playwright, you can set a proxy when creating a browser context or launching the browser.
Basic idea
- create a context with
proxy - use
storageStateto persist session
Example in Node.js
const { chromium } = require('playwright');
async function run(proxy, storageFile) {
const browser = await chromium.launch({
headless: true,
});
const context = await browser.newContext({
proxy: {
server: proxy.server, // e.g. http://host:port
username: proxy.username,
password: proxy.password,
},
storageState: storageFile || undefined,
});
const page = await context.newPage();
await page.goto('https://example.com');
// do stuff...
await context.storageState({ path: storageFile });
await browser.close();
}
5) Proxy rotation strategy
Option 1: Rotate per session
Best for:
- logged-in workflows
- stable identities
- fewer detection issues
Flow:
- pick a proxy
- create browser context
- run tasks
- save session state
- retire or reuse the same proxy for that session
Option 2: Rotate per task/request
Best for:
- public scraping
- high-volume, stateless access
Downside:
- breaks login/session continuity
- more likely to trigger suspicious behavior
Option 3: Sticky sessions
Use one proxy for a time window, e.g. 10–30 minutes, then rotate.
This is useful if:
- site tracks IP consistency
- you need a stable identity during a workflow
6) Session persistence
Playwright
Use storageState to save/load:
- cookies
- localStorage
Example:
await context.storageState({ path: 'session.json' });
Later:
const context = await browser.newContext({
proxy: {...},
storageState: 'session.json',
});
Selenium
You typically:
- save cookies manually
- restore cookies after login
- optionally reuse a browser profile directory
Chrome profile approach
For persistent sessions, you can use a dedicated user profile directory per identity:
--user-data-dir=/path/to/profile1
This preserves more than cookies, but can be heavier and harder to manage at scale.
7) Good proxy hygiene
A proxy pool should track:
- success rate
- recent failures
- latency
- country/geo
- ISP/type (datacenter vs residential)
Suggested rules:
- mark proxy unhealthy after repeated timeouts / bans
- cooldown proxies after failures
- prefer proven proxies for login flows
- don’t switch proxies mid-login unless needed
8) Detect and handle blocks
Your automation should watch for:
- HTTP 403/429
- unexpected redirects
- captcha pages
- empty/blocked responses
- navigation timeouts
When detected:
- retry with a fresh proxy/session
- back off
- reduce request rate
- avoid hammering the site
9) Concurrency model
If running many sessions:
- one worker thread/process per session is simpler
- map each session to a proxy
- store session metadata in Redis / database / file system
Example state per session:
{
"sessionId": "acct_123",
"proxy": "proxy-7",
"storageFile": "sessions/acct_123.json",
"status": "active",
"lastUsed": "2026-10-26T12:00:00Z"
}
10) Practical best practices
- Keep identity consistency: same proxy, cookie state, and UA for a session
- Don’t rotate proxies too aggressively within an authenticated session
- Add random delays and human-like pacing
- Use retries with exponential backoff
- Log proxy/session errors separately
- Encrypt session files if they contain auth tokens
- Respect site terms and legal requirements
11) If you want a simple production-ready pattern
Use:
- Playwright
- one session per browser context
- proxy assigned at session creation
- persist storage state to disk
- proxy health tracking in Redis/Postgres
- sticky sessions
- automatic fallback on failure
If you want, I can give you:
- a Playwright + Node.js starter project,
- a Python Selenium version, or
- a proxy/session manager design with code.