Prompt
How do I fix headless browser scripts that time out on dynamic content?
Latest observation
Headless browser timeouts on dynamic content usually come from one of a few issues: the page is still loading after your script thinks it’s done, the content is loaded by XHR/fetch after initial render, or the site is detecting/limiting automation. Here’s a practical checklist to fix it.
1) Wait for the right thing, not just load
Don’t rely only on page.goto(..., { waitUntil: "load" }) or a fixed sleep.
Use one of these instead:
- Wait for a specific selector
- Example: wait until the element that contains your dynamic data exists.
- Wait for network to settle
- Useful if the page loads content via API calls.
- Wait for a JS condition
- Example:
window.appReady === true
- Example:
Examples
Playwright
await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('.results-loaded', { timeout: 30000 });
Puppeteer
await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('.results-loaded', { timeout: 30000 });
2) Increase timeouts, but only after fixing the wait logic
If the page is just slower than your default timeout, increase it:
Playwright
page.setDefaultTimeout(60000);
page.setDefaultNavigationTimeout(60000);
Puppeteer
page.setDefaultTimeout(60000);
page.setDefaultNavigationTimeout(60000);
But if you only increase the timeout without waiting for the right condition, the script may still be flaky.
3) Use explicit polling for data that appears late
If content is inserted after an API response, poll for the value instead of sleeping.
await page.waitForFunction(() => {
return document.querySelectorAll('.item').length > 0;
}, { timeout: 30000 });
Or wait for a variable:
await page.waitForFunction(() => window.__DATA_READY__ === true);
4) Check whether the content is inside an iframe
Dynamic content often loads in an iframe, and your selectors won’t find it unless you target the frame.
Playwright
const frame = page.frame({ url: /example/ });
await frame.waitForSelector('.item');
Puppeteer
const frame = page.frames().find(f => /example/.test(f.url()));
await frame.waitForSelector('.item');
5) Inspect network requests
If the page times out, it helps to see what is still loading or failing.
Playwright
page.on('requestfailed', req => console.log('FAILED', req.url(), req.failure()));
page.on('response', res => {
if (!res.ok()) console.log('BAD RESPONSE', res.status(), res.url());
});
Puppeteer
page.on('requestfailed', req => console.log('FAILED', req.url(), req.failure()));
page.on('response', res => {
if (!res.ok()) console.log('BAD RESPONSE', res.status(), res.url());
});
Look for:
- API calls that never return
- 403/429 responses
- long-polling or websocket connections
- blocked assets
6) Avoid waiting for “network idle” on modern apps unless you know it fits
networkidle can hang forever if the page keeps background requests open.
Use it only when appropriate:
- good for simple pages
- risky for SPAs, ads, analytics-heavy pages, live apps
Prefer:
domcontentloaded- selector-based waits
- app-specific readiness checks
7) Make sure the browser is not being detected
Some sites behave differently for headless browsers:
- show CAPTCHA
- delay content
- block scripts
- return empty placeholders
Things that may help:
- run in non-headless mode for debugging
- set a realistic user agent
- set viewport and locale
- enable normal browser features
- avoid obvious automation fingerprints
Example:
await browser.newPage({
userAgent: 'Mozilla/5.0 ...',
});
In some cases, you may need stealth/evasion measures, but use them responsibly and within site terms.
8) Scroll or interact if content is lazy-loaded
Infinite scroll or “load more” pages need user-like actions.
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
await page.waitForTimeout(2000);
Or loop until no new content appears.
9) Capture screenshots and HTML when it fails
This helps you see what the script saw at timeout.
await page.screenshot({ path: 'timeout.png', fullPage: true });
console.log(await page.content());
If the page looks incomplete, you know it’s a wait issue.
If it shows a login/CAPTCHA/error page, it’s a site behavior issue.
10) Use the underlying API directly if possible
If the dynamic content comes from a JSON endpoint, it’s often more reliable to call that API directly instead of scraping the rendered page.
This avoids:
- rendering delays
- timeouts
- UI changes
- headless detection
A good debugging sequence
- Run in visible mode, not headless.
- Add logging for requests/responses.
- Screenshot at the failure point.
- Replace fixed sleeps with selector waits.
- Increase timeout only if needed.
- Check for iframe/lazy-load/login/CAPTCHA.
- If possible, use the API directly.
If you want, I can help you debug a specific Puppeteer or Playwright script if you paste the code and the timeout error.
Brands