How do I extract job postings from multiple job boards automatically?

To extract job postings from multiple job boards automatically, the safest and most maintainable approach is usually:

1. Use official APIs or RSS feeds when available
2. Build a small ingestion pipeline
3. Normalize all postings into one schema
4. Schedule the extraction regularly
5. Handle deduplication, retries, and site-specific changes

1) Prefer APIs over scraping

Many job boards offer:

• public APIs
• partner APIs
• XML/JSON feeds
• RSS feeds
• sitemap/job feeds

This is ideal because it is:

• more stable than HTML scraping
• less likely to break when the site changes
• usually more compliant with site terms

Examples:

• Greenhouse job board API
• Lever postings API
• Workable API
• SmartRecruiters API
• Indeed/LinkedIn often have restrictions and limited official access

2) If no API exists, scrape carefully

If you must scrape:

• check the site’s Terms of Service and robots.txt
• throttle requests
• use identifiable user-agent headers
• avoid bypassing anti-bot measures
• only collect data you’re allowed to access

Typical tools:

• Python requests + BeautifulSoup for simple pages
• Scrapy for larger crawls
• Playwright/Selenium for JavaScript-rendered pages

3) Create a unified data model

Different boards structure data differently, so map them into one schema, for example:

{
  "source": "greenhouse",
  "job_id": "12345",
  "title": "Data Analyst",
  "company": "Acme Inc",
  "location": "Remote",
  "employment_type": "Full-time",
  "description": "....",
  "apply_url": "https://...",
  "posted_at": "2026-10-04T00:00:00Z",
  "scraped_at": "2026-10-04T12:00:00Z"
}

4) Build a connector per source

Treat each job board as a separate connector:

• greenhouse_connector
• lever_connector
• workable_connector
• custom_scraper_connector

Each connector:

• fetches raw data
• parses source-specific fields
• returns standardized records

This keeps the system maintainable.

5) Schedule the extraction

Use a scheduler:

• cron for simple setups
• Airflow
• Prefect
• Celery beat
• serverless scheduled jobs

Run every few hours or daily depending on your needs.

6) Deduplicate and track changes

Store a unique key based on:

• source + job_id
• or apply_url
• or a hash of title/company/location

Track:

• new jobs
• updated jobs
• removed jobs

A database like PostgreSQL works well.

7) Example Python approach

Very simple pattern:

import requests
from bs4 import BeautifulSoup

def extract_jobs(url):
    r = requests.get(url, timeout=30)
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "html.parser")
    jobs = []

    for card in soup.select(".job-card"):
        jobs.append({
            "title": card.select_one(".title").get_text(strip=True),
            "location": card.select_one(".location").get_text(strip=True),
            "apply_url": card.select_one("a")["href"]
        })

    return jobs

For JSON APIs, it’s even simpler:

import requests

def extract_jobs_from_api(api_url):
    data = requests.get(api_url, timeout=30).json()
    return [
        {
            "title": job["title"],
            "location": job.get("location"),
            "apply_url": job["url"]
        }
        for job in data["jobs"]
    ]

8) Best architecture

A practical setup:

• Collector layer: fetch from each board
• Parser/transform layer: normalize fields
• Storage layer: save to DB
• Scheduler: runs jobs automatically
• Monitoring/logging: alert when a connector fails

9) Things to watch out for

• anti-bot protections
• pagination and infinite scroll
• rate limits
• duplicate postings across boards
• stale or expired jobs
• legal/compliance issues

10) Recommended stack

If you’re starting from scratch:

• Python
• Requests / BeautifulSoup
• Scrapy for scale
• Playwright for JS-heavy sites
• PostgreSQL
• cron or Airflow
• Docker for deployment

If you want, I can also give you:

1. a ready-to-run Python scraper template,
2. a Scrapy project structure for multiple job boards, or
3. a connector design for APIs like Greenhouse/Lever/Workable.