How do I turn scraped web pages into clean structured datasets?

Turning scraped web pages into clean, structured datasets is usually a 4-step pipeline:

1. Extract
2. Normalize
3. Validate
4. Store

Here’s a practical way to do it.

---

1) Define the target schema first

Before cleaning anything, decide what your final dataset should look like.

Example schema for product pages:

• url
• title
• price
• currency
• brand
• availability
• rating
• review_count
• scraped_at

This helps you avoid keeping messy “just-in-case” data.

---

2) Convert HTML into usable text/fields

If you scraped raw HTML, parse it with tools like:

• BeautifulSoup for Python
• lxml
• Scrapy selectors
• pandas.read_html for tables

Typical extraction tasks:

• remove scripts/styles
• get text from tags
• pull attributes like href, src, content
• extract structured data like JSON-LD or microdata when available

Example:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
title = soup.select_one("h1").get_text(strip=True)
price = soup.select_one(".price").get_text(strip=True)

---

3) Clean the extracted values

This is where most of the work happens.

Common cleaning steps

• Trim whitespace
• Standardize case
• Remove symbols
• Parse dates
• Convert numbers
• Handle missing values
• Deduplicate records

Examples

Clean text

clean_title = "  New iPhone 15  ".strip()

Parse prices

import re

raw = "$1,299.00"
price = float(re.sub(r"[^\d.]", "", raw))

Normalize dates

from datetime import datetime

raw_date = "Jan 5, 2025"
date_obj = datetime.strptime(raw_date, "%b %d, %Y")

Remove extra whitespace

import re

text = re.sub(r"\s+", " ", text).strip()

---

4) Handle messy real-world issues

Scraped pages are rarely clean.

Common problems

• inconsistent label names
• missing fields
• multiple formats for the same value
• HTML entities
• duplicated rows
• pagination and repeated content
• hidden navigation text mixed into content

Fixes

• use fallback selectors
• write field-specific parsing rules
• map variants to standard values
• drop boilerplate text
• deduplicate by URL, ID, or a composite key

Example of fallback extraction:

title = (
    soup.select_one("h1.product-title")
    or soup.select_one("h1")
    or soup.select_one(".title")
)
title = title.get_text(strip=True) if title else None

---

5) Structure the output

Put the cleaned values into rows and columns.

Good output formats

• CSV for simple tabular data
• JSON for nested data
• Parquet for large analytics datasets
• SQL tables for querying and pipelines

Example with pandas:

import pandas as pd

data = [
    {"url": url, "title": title, "price": price}
]

df = pd.DataFrame(data)
df.to_csv("clean_data.csv", index=False)

---

6) Validate the dataset

Check that the data meets expectations.

Validation checks

• required fields present
• numeric values in range
• valid URLs
• dates parse correctly
• no duplicate primary keys
• consistent categories/currency codes

Tools:

• pydantic
• pandera
• great expectations

Example:

assert df["price"].notna().all()
assert (df["price"] >= 0).all()

---

7) Use a transformation pipeline

For anything non-trivial, make your cleaning modular:

• extract_html()
• parse_fields()
• clean_fields()
• validate_rows()
• write_output()

This makes the process repeatable and easier to debug.

---

8) If pages are semi-structured, use rules + heuristics

For many websites, a mix of approaches works best:

• CSS selectors for predictable fields
• regex for embedded values
• JSON-LD parsing for metadata
• NLP/spaCy for entity extraction from free text

If the page layout varies a lot, build a per-site parser.

---

9) Example end-to-end flow in Python

import re
import pandas as pd
from bs4 import BeautifulSoup

def clean_price(text):
    if not text:
        return None
    m = re.search(r"[\d,.]+", text.replace(",", ""))
    return float(m.group()) if m else None

def parse_page(html, url):
    soup = BeautifulSoup(html, "html.parser")
    
    title_el = soup.select_one("h1") or soup.select_one(".title")
    price_el = soup.select_one(".price") or soup.select_one('[data-testid="price"]')
    
    title = title_el.get_text(" ", strip=True) if title_el else None
    price = clean_price(price_el.get_text(" ", strip=True) if price_el else None)
    
    return {
        "url": url,
        "title": title,
        "price": price
    }

rows = [parse_page(html, url) for html, url in scraped_pages]
df = pd.DataFrame(rows).drop_duplicates(subset=["url"])

---

10) Best practices

• Save the raw HTML too, at least initially
• Keep cleaning logic versioned
• Log parse failures
• Build tests for known page examples
• Prefer structured data embedded in the page when available
• Don’t over-clean too early; preserve raw values if useful

---

If you want, I can also show you:

1. a Python cleaning pipeline template,
2. how to extract data from HTML tables, or
3. how to use pandas + BeautifulSoup on a real example.