Prompt

How do I fix headless browser scripts that time out on dynamic content?

Technology / Testing & QA2 observationsLast seen Nov 16, 2026

Latest observation

Nov 16, 2026OpenAI APIWeb search: off

Headless browser timeouts on dynamic content usually come from one of a few issues: the page is still loading after your script thinks it’s done, the content is loaded by XHR/fetch after initial render, or the site is detecting/limiting automation. Here’s a practical checklist to fix it.

1) Wait for the right thing, not just load

Don’t rely only on page.goto(..., { waitUntil: "load" }) or a fixed sleep.

Use one of these instead:

  • Wait for a specific selector
    • Example: wait until the element that contains your dynamic data exists.
  • Wait for network to settle
    • Useful if the page loads content via API calls.
  • Wait for a JS condition
    • Example: window.appReady === true

Examples

Playwright

await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('.results-loaded', { timeout: 30000 });

Puppeteer

await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('.results-loaded', { timeout: 30000 });

2) Increase timeouts, but only after fixing the wait logic

If the page is just slower than your default timeout, increase it:

Playwright

page.setDefaultTimeout(60000);
page.setDefaultNavigationTimeout(60000);

Puppeteer

page.setDefaultTimeout(60000);
page.setDefaultNavigationTimeout(60000);

But if you only increase the timeout without waiting for the right condition, the script may still be flaky.

3) Use explicit polling for data that appears late

If content is inserted after an API response, poll for the value instead of sleeping.

await page.waitForFunction(() => {
  return document.querySelectorAll('.item').length > 0;
}, { timeout: 30000 });

Or wait for a variable:

await page.waitForFunction(() => window.__DATA_READY__ === true);

4) Check whether the content is inside an iframe

Dynamic content often loads in an iframe, and your selectors won’t find it unless you target the frame.

Playwright

const frame = page.frame({ url: /example/ });
await frame.waitForSelector('.item');

Puppeteer

const frame = page.frames().find(f => /example/.test(f.url()));
await frame.waitForSelector('.item');

5) Inspect network requests

If the page times out, it helps to see what is still loading or failing.

Playwright

page.on('requestfailed', req => console.log('FAILED', req.url(), req.failure()));
page.on('response', res => {
  if (!res.ok()) console.log('BAD RESPONSE', res.status(), res.url());
});

Puppeteer

page.on('requestfailed', req => console.log('FAILED', req.url(), req.failure()));
page.on('response', res => {
  if (!res.ok()) console.log('BAD RESPONSE', res.status(), res.url());
});

Look for:

  • API calls that never return
  • 403/429 responses
  • long-polling or websocket connections
  • blocked assets

6) Avoid waiting for “network idle” on modern apps unless you know it fits

networkidle can hang forever if the page keeps background requests open.

Use it only when appropriate:

  • good for simple pages
  • risky for SPAs, ads, analytics-heavy pages, live apps

Prefer:

  • domcontentloaded
  • selector-based waits
  • app-specific readiness checks

7) Make sure the browser is not being detected

Some sites behave differently for headless browsers:

  • show CAPTCHA
  • delay content
  • block scripts
  • return empty placeholders

Things that may help:

  • run in non-headless mode for debugging
  • set a realistic user agent
  • set viewport and locale
  • enable normal browser features
  • avoid obvious automation fingerprints

Example:

await browser.newPage({
  userAgent: 'Mozilla/5.0 ...',
});

In some cases, you may need stealth/evasion measures, but use them responsibly and within site terms.

8) Scroll or interact if content is lazy-loaded

Infinite scroll or “load more” pages need user-like actions.

await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
await page.waitForTimeout(2000);

Or loop until no new content appears.

9) Capture screenshots and HTML when it fails

This helps you see what the script saw at timeout.

await page.screenshot({ path: 'timeout.png', fullPage: true });
console.log(await page.content());

If the page looks incomplete, you know it’s a wait issue.
If it shows a login/CAPTCHA/error page, it’s a site behavior issue.

10) Use the underlying API directly if possible

If the dynamic content comes from a JSON endpoint, it’s often more reliable to call that API directly instead of scraping the rendered page.

This avoids:

  • rendering delays
  • timeouts
  • UI changes
  • headless detection

A good debugging sequence

  1. Run in visible mode, not headless.
  2. Add logging for requests/responses.
  3. Screenshot at the failure point.
  4. Replace fixed sleeps with selector waits.
  5. Increase timeout only if needed.
  6. Check for iframe/lazy-load/login/CAPTCHA.
  7. If possible, use the API directly.

If you want, I can help you debug a specific Puppeteer or Playwright script if you paste the code and the timeout error.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (2 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.