Prompt
Need browser automation for dynamic sites and anti bot issues
Latest observation
Nov 13, 2026
OpenAI APIWeb search: off
For dynamic sites and anti-bot protections, the practical choice is usually Playwright. It’s better than basic HTTP scraping because it runs a real browser, handles JavaScript-heavy pages, and gives you strong control over sessions, navigation, and waiting.
Recommended approach
1) Use a real browser automation tool
- Playwright: best overall for modern sites
- Selenium: still useful, but generally less reliable for modern anti-bot flows
- Puppeteer: good if you only need Chromium, but Playwright is usually more flexible
2) Handle dynamic loading correctly
Use:
- explicit waits
- waiting for selectors
- waiting for network idle where appropriate
- scrolling / pagination logic
Example ideas:
- wait for a product card to appear
- wait for XHR requests to finish
- click “Load more” buttons
- simulate user scrolling
3) Reduce anti-bot triggers
Common tactics:
- use a consistent browser profile / persistent session
- keep a stable user-agent, viewport, locale, timezone
- avoid overly aggressive request rates
- add realistic delays between actions
- handle cookies and login properly
- don’t open too many pages in parallel
4) Be aware of common blocks
Sites may use:
- CAPTCHAs
- device/browser fingerprinting
- IP reputation checks
- rate limiting
- bot-detection scripts
If a site is actively blocking automation, the safest path is usually:
- use their official API if available
- request permission / whitelist
- keep automation low-volume and human-like
Example: Playwright in Python
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
)
page.goto("https://example.com", wait_until="networkidle")
page.wait_for_selector("h1")
print(page.title())
print(page.locator("h1").inner_text())
browser.close()
If you want more robustness
Consider:
- persistent context to keep cookies/session
- proxy rotation only if allowed and appropriate
- headful mode for debugging
- screenshot/video/logging for diagnosing failures
Best practice summary
- For dynamic pages: Playwright
- For anti-bot resistance: realistic sessions + slower pace + stable browser profile
- For hard blocks: API or permission is best
If you want, I can give you:
- a Playwright starter template for dynamic sites,
- a Python or Node.js implementation, or
- a setup for login-protected pages and pagination.