Prompt

How do I automate a website with dynamic content, infinite scroll, and changing selectors?

Technology / Testing & QA2 observationsLast seen Nov 11, 2026

Latest observation

Nov 11, 2026OpenAI APIWeb search: off

Automating a site like that is usually a mix of browser automation + waits + resilient locators + network/data extraction. The key is to avoid treating it like a static page.

1) Use the right tool

For dynamic sites, prefer:

  • Playwright (often best choice)
  • Selenium (works, but usually more brittle)
  • Puppeteer (good for Chrome-only workflows)

Playwright is especially good for:

  • auto-waiting
  • handling SPA frameworks
  • robust selectors
  • network interception
  • scrolling and pagination

2) Don’t rely on brittle selectors

If selectors keep changing, avoid:

  • long CSS chains
  • auto-generated class names
  • exact text that changes frequently

Prefer stable locators:

  • data-testid
  • aria-label
  • role-based selectors
  • nearby stable text/structure
  • attributes like href, title, alt

Example strategies:

  • page.getByRole('button', { name: 'Load more' })
  • page.locator('[data-testid="product-card"]')
  • page.locator('a[href*="/item/"]')

If you control the site, add stable attributes like:

<div data-testid="user-card">

3) Wait for the right thing, not for arbitrary time

Avoid sleep(5000) unless you have no choice.

Instead wait for:

  • element to appear
  • network to go idle
  • API response
  • loading spinner to disappear
  • content count to increase

Examples:

await page.waitForSelector('[data-testid="result-item"]');
await page.waitForLoadState('networkidle');
await page.waitForResponse(resp => resp.url().includes('/api/items') && resp.ok());

4) Handle infinite scroll by scrolling incrementally

Most infinite scroll pages load more content when the user approaches the bottom.

Typical approach:

  1. read current item count
  2. scroll to bottom
  3. wait for new items
  4. repeat until no more items load

Playwright example (Node.js)

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://example.com');

let previousCount = 0;
let sameCountRounds = 0;

while (sameCountRounds < 3) {
  const items = page.locator('[data-testid="item"]');
  const count = await items.count();

  if (count === previousCount) {
    sameCountRounds++;
  } else {
    sameCountRounds = 0;
    previousCount = count;
  }

  await page.mouse.wheel(0, 2000);
  await page.waitForTimeout(1000); // small pause, only if necessary
}

console.log('Loaded items:', await page.locator('[data-testid="item"]').count());
await browser.close();

More reliable version

Wait for item count to increase:

const items = page.locator('[data-testid="item"]');
const before = await items.count();

await page.mouse.wheel(0, 3000);
await page.waitForFunction(
  prev => document.querySelectorAll('[data-testid="item"]').length > prev,
  before
);

5) Watch for virtualized lists

Some websites don’t keep all items in the DOM. They recycle rows as you scroll (common in React virtual lists).

Symptoms:

  • item count stays low
  • DOM changes as you scroll
  • only visible items exist

In that case:

  • extract data as you scroll
  • don’t depend on all items being present at once
  • collect content into a set/map to avoid duplicates

Example idea:

const seen = new Set();

while (true) {
  const items = await page.locator('[data-testid="item"]').allTextContents();
  for (const text of items) seen.add(text);

  const prevSize = seen.size;
  await page.mouse.wheel(0, 2000);
  await page.waitForTimeout(800);

  if (seen.size === prevSize) break;
}

6) Inspect network calls

Often the best approach is not scraping the UI at all, but reading the API behind it.

If the page loads data via XHR/fetch:

  • open DevTools → Network
  • reload and look for JSON endpoints
  • reproduce those requests directly in code

Benefits:

  • faster
  • more reliable than UI scraping
  • no selector issues
  • easier to paginate

You can use:

  • browser devtools manually
  • Playwright request interception
  • direct fetch/requests scripts

Example:

const response = await page.waitForResponse(resp =>
  resp.url().includes('/api/products') && resp.ok()
);
const data = await response.json();

If an API is accessible, scraping it directly is usually the best solution.


7) Use a robust retry strategy

Dynamic pages fail sometimes:

  • content not loaded yet
  • scroll triggered late
  • transient 429/503
  • selectors changed

Add:

  • retries
  • backoff
  • logging
  • screenshot on failure

Example concept:

  • try action
  • if it fails, wait a bit and retry
  • if it still fails, save page state

8) Handle pagination + infinite scroll together

Some sites load more on scroll, then eventually switch to “Next page,” or append until a threshold.

Your scraper should support both:

  • scroll until no new items
  • if a “Next” button appears, click it
  • continue

9) Keep state and deduplicate

When collecting data from dynamic content:

  • dedupe by ID, URL, or a stable field
  • store progress so you can resume
  • log what page/scroll step you’re on

10) Be careful with anti-bot measures

Dynamic sites often also have:

  • rate limits
  • CAPTCHAs
  • bot detection
  • login/session requirements

Best practices:

  • slow down requests
  • use realistic browser settings
  • keep sessions/cookies
  • avoid excessive parallelism
  • respect terms of service

Practical Playwright pattern

A good generic template:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });

const collected = new Set();
let unchangedRounds = 0;
let lastCount = 0;

while (unchangedRounds < 3) {
  // collect visible items
  const texts = await page.locator('[data-testid="item"]').allTextContents();
  texts.forEach(t => collected.add(t));

  const currentCount = collected.size;
  if (currentCount === lastCount) {
    unchangedRounds++;
  } else {
    unchangedRounds = 0;
    lastCount = currentCount;
  }

  // scroll down
  await page.evaluate(() => window.scrollBy(0, window.innerHeight * 2));
  await page.waitForTimeout(1000);
}

console.log([...collected]);
await browser.close();

Rule of thumb

If selectors change often, think in this order:

  1. Can I use the API instead of the UI?
  2. Can I use stable attributes/roles?
  3. Can I wait for content changes rather than fixed delays?
  4. Can I scroll incrementally and dedupe data?
  5. Do I need retries and anti-bot handling?

If you want, I can give you:

  • a Playwright template for infinite scroll scraping,
  • a Selenium version, or
  • help analyzing a specific site structure if you share an example.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (2 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.