How can I scrape websites built with React or Angular?

Websites built with React or Angular are usually client-rendered, which means the initial HTML may be sparse and the real content is loaded by JavaScript. To scrape them, you generally need one of these approaches:

1) Prefer the underlying API if possible

Often React/Angular apps fetch data from JSON APIs.

• Open DevTools → Network → XHR/Fetch
• Reload the page
• Look for requests returning JSON
• Scrape those endpoints directly with requests / httpx

This is usually the fastest, most stable, and cleanest method.

Example

import requests

url = "https://example.com/api/products"
data = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}).json()
print(data)

---

2) Use a browser automation tool for rendered pages

If the data is only available after JavaScript runs, use:

• Playwright (recommended)
• Selenium
• Puppeteer (Node.js)

These tools load the page like a real browser, execute JavaScript, and let you extract the rendered DOM.

Playwright example

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com", wait_until="networkidle")
    print(page.content())
    browser.close()

Then parse page.content() with BeautifulSoup if needed.

---

3) Wait for dynamic content

React/Angular pages may render content after a delay. Wait for selectors instead of relying on page.goto() alone.

Playwright

page.goto("https://example.com")
page.wait_for_selector(".product-card")
items = page.locator(".product-card").all_text_contents()
print(items)

Selenium

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example.com")
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".product-card"))
)
print(driver.page_source)
driver.quit()

---

4) Check for server-side rendering or hydration data

Some React apps ship pre-rendered HTML plus a JSON blob with page data.

Look for:

• <script id="__NEXT_DATA__"> in Next.js
• window.__INITIAL_STATE__
• Angular hydration/state scripts

You can extract those values directly from HTML.

Example with BeautifulSoup

import requests
from bs4 import BeautifulSoup
import json

html = requests.get("https://example.com").text
soup = BeautifulSoup(html, "html.parser")

script = soup.find("script", id="__NEXT_DATA__")
if script:
    data = json.loads(script.string)
    print(data)

---

5) Handle anti-bot protections carefully

Some sites use:

• Cloudflare
• CAPTCHA
• login/session checks
• rate limiting

In those cases:

• respect the site’s Terms of Service and robots.txt
• use sensible rate limits
• keep headers/cookies consistent
• avoid aggressive scraping

If access is blocked, the best path is often the official API or permission from the site owner.

---

Practical recommendation

Start in this order:

1. Inspect network requests for JSON APIs
2. If no usable API, use Playwright
3. Wait for specific selectors
4. Extract data from rendered HTML or embedded state

If you want, I can show you:

• a Playwright scraper template
• how to find hidden API endpoints
• or how to scrape a specific React/Angular site you have in mind