Prompt
How do I automate a website with dynamic content, infinite scroll, and changing selectors?
Latest observation
Automating a site like that is usually a mix of browser automation + waits + resilient locators + network/data extraction. The key is to avoid treating it like a static page.
1) Use the right tool
For dynamic sites, prefer:
- Playwright (often best choice)
- Selenium (works, but usually more brittle)
- Puppeteer (good for Chrome-only workflows)
Playwright is especially good for:
- auto-waiting
- handling SPA frameworks
- robust selectors
- network interception
- scrolling and pagination
2) Don’t rely on brittle selectors
If selectors keep changing, avoid:
- long CSS chains
- auto-generated class names
- exact text that changes frequently
Prefer stable locators:
data-testidaria-label- role-based selectors
- nearby stable text/structure
- attributes like
href,title,alt
Example strategies:
page.getByRole('button', { name: 'Load more' })page.locator('[data-testid="product-card"]')page.locator('a[href*="/item/"]')
If you control the site, add stable attributes like:
<div data-testid="user-card">
3) Wait for the right thing, not for arbitrary time
Avoid sleep(5000) unless you have no choice.
Instead wait for:
- element to appear
- network to go idle
- API response
- loading spinner to disappear
- content count to increase
Examples:
await page.waitForSelector('[data-testid="result-item"]');
await page.waitForLoadState('networkidle');
await page.waitForResponse(resp => resp.url().includes('/api/items') && resp.ok());
4) Handle infinite scroll by scrolling incrementally
Most infinite scroll pages load more content when the user approaches the bottom.
Typical approach:
- read current item count
- scroll to bottom
- wait for new items
- repeat until no more items load
Playwright example (Node.js)
import { chromium } from 'playwright';
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://example.com');
let previousCount = 0;
let sameCountRounds = 0;
while (sameCountRounds < 3) {
const items = page.locator('[data-testid="item"]');
const count = await items.count();
if (count === previousCount) {
sameCountRounds++;
} else {
sameCountRounds = 0;
previousCount = count;
}
await page.mouse.wheel(0, 2000);
await page.waitForTimeout(1000); // small pause, only if necessary
}
console.log('Loaded items:', await page.locator('[data-testid="item"]').count());
await browser.close();
More reliable version
Wait for item count to increase:
const items = page.locator('[data-testid="item"]');
const before = await items.count();
await page.mouse.wheel(0, 3000);
await page.waitForFunction(
prev => document.querySelectorAll('[data-testid="item"]').length > prev,
before
);
5) Watch for virtualized lists
Some websites don’t keep all items in the DOM. They recycle rows as you scroll (common in React virtual lists).
Symptoms:
- item count stays low
- DOM changes as you scroll
- only visible items exist
In that case:
- extract data as you scroll
- don’t depend on all items being present at once
- collect content into a set/map to avoid duplicates
Example idea:
const seen = new Set();
while (true) {
const items = await page.locator('[data-testid="item"]').allTextContents();
for (const text of items) seen.add(text);
const prevSize = seen.size;
await page.mouse.wheel(0, 2000);
await page.waitForTimeout(800);
if (seen.size === prevSize) break;
}
6) Inspect network calls
Often the best approach is not scraping the UI at all, but reading the API behind it.
If the page loads data via XHR/fetch:
- open DevTools → Network
- reload and look for JSON endpoints
- reproduce those requests directly in code
Benefits:
- faster
- more reliable than UI scraping
- no selector issues
- easier to paginate
You can use:
- browser devtools manually
- Playwright request interception
- direct
fetch/requestsscripts
Example:
const response = await page.waitForResponse(resp =>
resp.url().includes('/api/products') && resp.ok()
);
const data = await response.json();
If an API is accessible, scraping it directly is usually the best solution.
7) Use a robust retry strategy
Dynamic pages fail sometimes:
- content not loaded yet
- scroll triggered late
- transient 429/503
- selectors changed
Add:
- retries
- backoff
- logging
- screenshot on failure
Example concept:
- try action
- if it fails, wait a bit and retry
- if it still fails, save page state
8) Handle pagination + infinite scroll together
Some sites load more on scroll, then eventually switch to “Next page,” or append until a threshold.
Your scraper should support both:
- scroll until no new items
- if a “Next” button appears, click it
- continue
9) Keep state and deduplicate
When collecting data from dynamic content:
- dedupe by ID, URL, or a stable field
- store progress so you can resume
- log what page/scroll step you’re on
10) Be careful with anti-bot measures
Dynamic sites often also have:
- rate limits
- CAPTCHAs
- bot detection
- login/session requirements
Best practices:
- slow down requests
- use realistic browser settings
- keep sessions/cookies
- avoid excessive parallelism
- respect terms of service
Practical Playwright pattern
A good generic template:
import { chromium } from 'playwright';
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });
const collected = new Set();
let unchangedRounds = 0;
let lastCount = 0;
while (unchangedRounds < 3) {
// collect visible items
const texts = await page.locator('[data-testid="item"]').allTextContents();
texts.forEach(t => collected.add(t));
const currentCount = collected.size;
if (currentCount === lastCount) {
unchangedRounds++;
} else {
unchangedRounds = 0;
lastCount = currentCount;
}
// scroll down
await page.evaluate(() => window.scrollBy(0, window.innerHeight * 2));
await page.waitForTimeout(1000);
}
console.log([...collected]);
await browser.close();
Rule of thumb
If selectors change often, think in this order:
- Can I use the API instead of the UI?
- Can I use stable attributes/roles?
- Can I wait for content changes rather than fixed delays?
- Can I scroll incrementally and dedupe data?
- Do I need retries and anti-bot handling?
If you want, I can give you:
- a Playwright template for infinite scroll scraping,
- a Selenium version, or
- help analyzing a specific site structure if you share an example.