How do I extract data from a website that requires clicking buttons?

If the site needs button clicks to reveal data, the usual approach is to automate a browser rather than just download the HTML.

Common options

1) Use browser automation

Tools:

• Playwright (recommended)
• Selenium
• Puppeteer (Node.js)

These can:

• open the page
• click buttons
• wait for content to load
• read the resulting HTML or text

2) Inspect the network requests

Sometimes the button click just triggers an API call behind the scenes. If so, you can often skip browser automation and:

• open DevTools → Network
• click the button
• find the request that returns JSON/data
• call that API directly in Python/JavaScript

This is usually faster and more reliable.

---

Example with Playwright (Python)

Install:

pip install playwright
playwright install

Basic example:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com")

    # click a button
    page.click("text=Load More")

    # wait for data to appear
    page.wait_for_timeout(2000)

    # get page content
    html = page.content()
    print(html)

    browser.close()

If you need to extract text from elements:

items = page.locator(".item").all_inner_texts()
print(items)

---

Example with Selenium (Python)

Install:

pip install selenium

Basic example:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example.com")

button = driver.find_element(By.XPATH, "//button[text()='Load More']")
button.click()

WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".item"))
)

print(driver.page_source)
driver.quit()

---

Tips for button-based sites

• Use explicit waits instead of fixed sleeps when possible.
• Handle:
  • popups
  • cookie banners
  • pagination
  • “Load more” buttons
  • infinite scroll
• If the site uses React/Vue/Angular, the data may be loaded dynamically, so the raw page HTML may not contain it initially.
• Check whether the data is already available in:
  • embedded JSON
  • script tags
  • API responses

---

Important note

Make sure you have permission to scrape the site and comply with its terms of service and robots.txt where applicable.

If you want, I can show you:

1. a Playwright script for your specific site, or
2. how to find the API request behind the button click.