How do I extract structured data from HTML pages?

To extract structured data from HTML pages, you generally use web scraping or HTML parsing. The best approach depends on how the page is built.

Common approaches

1. Parse the HTML directly

If the data is present in the page source, use an HTML parser and select elements by tag, class, id, or CSS selectors.

Python example with BeautifulSoup

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
html = requests.get(url).text
soup = BeautifulSoup(html, "html.parser")

title = soup.select_one("h1").get_text(strip=True)
print(title)

2. Use structured metadata already on the page

Many sites include machine-readable data such as:

• JSON-LD (<script type="application/ld+json">)
• Microdata
• Open Graph tags (meta property="og:title")

Example: extract JSON-LD

import requests, json
from bs4 import BeautifulSoup

html = requests.get("https://example.com").text
soup = BeautifulSoup(html, "html.parser")

for script in soup.find_all("script", type="application/ld+json"):
    data = json.loads(script.string)
    print(data)

3. Use browser automation for dynamic pages

If the page loads data with JavaScript, the raw HTML may not contain it. Use tools like:

• Selenium
• Playwright
• Puppeteer

These render the page like a browser, then you can extract the final DOM.

4. Extract data from embedded APIs

Sometimes the HTML page calls an API behind the scenes. If you can find that API, it’s often easier to pull structured JSON directly instead of scraping HTML.

Use browser dev tools:

• Open Network tab
• Reload page
• Look for XHR/fetch requests returning JSON

Practical workflow

1. Inspect the page source.
2. Check whether the data is already in HTML or JSON-LD.
3. If not, see whether it’s loaded from an API.
4. If JavaScript-rendered, use Playwright/Selenium.
5. Parse results into a structured format like CSV, JSON, or a database.

Useful tools

• BeautifulSoup: simple HTML parsing
• lxml: fast parsing
• Scrapy: full scraping framework
• Playwright: modern browser automation
• Selenium: browser automation
• pandas.read_html(): easy table extraction

Example with pandas

import pandas as pd

tables = pd.read_html("https://example.com")
print(tables[0])

Tips

• Prefer APIs or JSON-LD when available.
• Handle pagination and rate limits.
• Respect robots.txt and site terms.
• Add retries and error handling.

If you want, I can show you:

1. a Python scraper example,
2. how to extract JSON-LD, or
3. how to scrape a JavaScript-rendered site.