How can I scrape business listings from online directories?

Scraping business listings from online directories can be straightforward technically, but the important first step is to make sure you’re allowed to do it.

1) Check legality and site rules first

Before scraping:

• Read the site’s Terms of Service
• Check robots.txt
• Avoid scraping pages that require login if it violates terms
• Don’t bypass CAPTCHAs, rate limits, or anti-bot protections
• If possible, use an official API instead of scraping

2) Identify the data source

Business directories often show:

• business name
• category
• address
• phone number
• website
• hours
• rating/reviews

Look at:

• static HTML pages
• paginated listing pages
• search/filter result pages
• page requests made by JavaScript

3) Choose the right approach

A. Static HTML scraping

Use requests + BeautifulSoup if the content is in the page HTML.

Example:

import requests
from bs4 import BeautifulSoup

url = "https://example.com/directory"
headers = {
    "User-Agent": "Mozilla/5.0"
}

resp = requests.get(url, headers=headers)
soup = BeautifulSoup(resp.text, "html.parser")

for card in soup.select(".business-card"):
    name = card.select_one(".name").get_text(strip=True)
    address = card.select_one(".address").get_text(strip=True)
    phone = card.select_one(".phone").get_text(strip=True)
    print(name, address, phone)

B. JavaScript-rendered pages

If data loads dynamically, use:

• Playwright
• Selenium
• or inspect XHR/fetch requests and scrape the underlying JSON endpoint

Example with Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/directory", wait_until="networkidle")

    cards = page.locator(".business-card")
    for i in range(cards.count()):
        card = cards.nth(i)
        print(
            card.locator(".name").inner_text(),
            card.locator(".address").inner_text()
        )

    browser.close()

4) Handle pagination

Directories usually paginate or use infinite scroll.

Pagination example

import requests
from bs4 import BeautifulSoup

base_url = "https://example.com/directory?page={}"

for page_num in range(1, 6):
    resp = requests.get(base_url.format(page_num), headers={"User-Agent": "Mozilla/5.0"})
    soup = BeautifulSoup(resp.text, "html.parser")
    for card in soup.select(".business-card"):
        print(card.select_one(".name").get_text(strip=True))

Infinite scroll

You may need to:

• scroll with Playwright/Selenium
• or capture the API calls that load more results

5) Be polite and stable

Good scraping practice:

• use a custom User-Agent
• add delays between requests
• retry on temporary failures
• don’t hammer the server
• cache responses if possible

Example delay:

import time
time.sleep(1)

6) Store the data

Common formats:

• CSV
• JSON
• SQLite/PostgreSQL

Example CSV:

import csv

with open("businesses.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["name", "address", "phone"])
    writer.writerow(["Acme Co", "123 Main St", "555-1234"])

7) Watch for common obstacles

• CAPTCHA
• login walls
• obfuscated HTML
• duplicate listings
• missing values
• rate limiting / blocked IPs

8) Best practice: inspect network requests

Often the page is just a shell and the listings come from an API. Open browser dev tools:

• Network tab
• filter by XHR/fetch
• reload page
• look for JSON responses with listing data

Scraping the API endpoint directly is usually cleaner than parsing HTML.

If you want, I can help you with:

1. a generic scraper template,
2. scraping a specific directory site, or
3. a version using Playwright/Selenium.