How do I set up a web scraping tool for tracking pricing changes across merchants?

To set up a web scraping tool for tracking pricing changes across merchants, it helps to design it as a small pipeline:

1. Define what you want to track

   • Product name / SKU / merchant
   • Current price
   • Sale price vs regular price
   • Availability
   • Shipping cost
   • Timestamp of the scrape

2. Choose your approach

   • Static pages: use requests + BeautifulSoup or httpx + lxml
   • Dynamic JavaScript pages: use Playwright or Selenium
   • Large-scale / production: consider Scrapy with scheduled jobs

3. Build the scraper

   • Create a list of merchant product URLs
   • For each page:
     • fetch the HTML
     • parse the price element(s)
     • normalize the price into a consistent numeric format
     • store the result with a timestamp

4. Store the data

   • Use a database like:
     • PostgreSQL for structured tracking
     • SQLite for a small setup
     • BigQuery / Snowflake for analytics at scale
   • Suggested tables:
     • merchants
     • products
     • price_snapshots

5. Detect changes

   • Compare current price with the last stored price
   • Trigger alerts when:
     • price drops
     • price increases
     • product goes out of stock
     • new merchant appears

6. Schedule the scraping

   • Use cron, Airflow, or a cloud scheduler
   • Common schedule:
     • every hour for fast-changing merchants
     • daily for slower-moving catalogs

7. Add reliability

   • Respect robots.txt and site terms
   • Add rate limiting and retries
   • Rotate user agents carefully
   • Handle anti-bot protections
   • Log failures and parse errors

8. Send alerts or reports

   • Email, Slack, webhook, dashboard
   • Only notify on meaningful changes to avoid noise

Simple Python example

import requests
from bs4 import BeautifulSoup
from datetime import datetime

url = "https://example.com/product/123"
headers = {"User-Agent": "Mozilla/5.0"}

resp = requests.get(url, headers=headers, timeout=20)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "html.parser")
price_text = soup.select_one(".price").get_text(strip=True)

print({
    "url": url,
    "price": price_text,
    "scraped_at": datetime.utcnow().isoformat()
})

If pages are JavaScript-rendered

Use Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/product/123", wait_until="networkidle")
    price = page.locator(".price").inner_text()
    print(price)
    browser.close()

Recommended architecture

• Crawler: collects pages
• Parser: extracts price data
• Storage: database for snapshots
• Diff engine: compares old/new prices
• Notifier: sends alerts
• Scheduler: runs jobs automatically

If you want, I can also provide:

• a full Python starter project structure
• a database schema for price tracking
• or a merchant comparison scraper using Scrapy/Playwright