How do I set up a site crawler for scheduled audits with exportable issue tracking on a large site?

To set up a site crawler for scheduled audits with exportable issue tracking on a large site, you want a workflow that is:

1. Reliable at scale
2. Repeatable on a schedule
3. Able to track issues over time
4. Easy to export into spreadsheets, tickets, or BI tools

Here’s a practical setup.

---

1) Define the audit scope and crawl rules

Before choosing tools or configuring a crawler, define:

• Domains/subdomains to include
• URL patterns to include/exclude
• Authenticated areas if needed
• Maximum crawl depth
• Page types to focus on:
  • HTML pages
  • PDFs
  • JavaScript-rendered pages
  • Internationalized versions
• Respect robots.txt or not, depending on your policy
• Canonical handling
• Parameter handling to avoid duplicate crawling

For a large site, it helps to create:

• Allowed URL regex patterns
• Blocked patterns for faceted navigation, search result pages, calendars, infinite spaces, etc.

---

2) Choose a crawler with scheduling and exports

You’ll want a crawler that can do at least:

• Scheduled crawls
• Custom issue rules
• Export to CSV/XLSX/API
• Historical comparisons
• Authenticated crawling if needed
• Large-site crawl management

Common options:

• Screaming Frog SEO Spider + scheduling + exports
• Sitebulb with scheduled audits and issue reports
• Lumar / Oncrawl / Deepcrawl for enterprise-scale crawling
• Botify for very large sites and log/crawl analysis
• Custom crawler using Scrapy, Playwright, or Selenium if you need custom workflows

If you need exportable issue tracking over time, enterprise tools are often best because they support:

• recurring audits
• issue history
• dashboards
• integrations to Jira/Slack/Looker/BigQuery, etc.

---

3) Configure the crawler for large-site performance

For large sites, tuning matters.

Recommended settings:

• Increase memory and storage
• Use headless/browser rendering only where required
• Crawl with multiple threads/concurrency, but stay polite
• Set request throttling to avoid overloading the site
• Use URL deduplication
• Exclude low-value URL spaces
• Use sitemaps and internal links together
• Capture HTTP status codes, redirects, canonicals, hreflang, metadata, structured data, and response times

For JavaScript-heavy sites:

• Use a crawler that supports rendering
• Or a two-pass approach:
  • first crawl raw HTML
  • second crawl rendered pages only for critical sections

---

4) Create issue rules that can be tracked over time

Your issue tracking system should classify findings into stable rules, like:

• Broken links
• Redirect chains
• 4xx / 5xx pages
• Missing title tags
• Duplicate titles
• Missing meta descriptions
• Duplicate content
• Non-indexable important pages
• Canonical mismatches
• Missing H1s
• Orphan pages
• Pagination issues
• hreflang errors
• Slow pages
• Mixed content / insecure resources

For each issue, store:

• Issue name
• Severity
• Affected URL
• Source URL
• First detected
• Last detected
• Crawl date
• Status: new / recurring / fixed / ignored
• Owner/team
• Ticket ID

This is what makes exports useful for tracking, not just reporting.

---

5) Build a recurring crawl schedule

A good schedule depends on site size and change frequency.

Example schedules:

• Daily for high-traffic or high-change sites
• Weekly for medium sites
• Monthly for stable sites
• After-deploy audits for release validation

Best practice:

• Schedule:
  • a full crawl monthly
  • targeted crawls weekly for critical templates/sections
  • post-release checks for key areas

If the site is very large, you may need:

• segmented crawls by folder, template, or subdomain
• rotating crawl windows
• incremental audits rather than one massive crawl every time

---

6) Export issues in a format suitable for tracking

You’ll want both raw crawl data and issue summaries.

Export formats:

• CSV for spreadsheets
• XLSX for stakeholders
• JSON/API for automation
• Database table for long-term storage and trend analysis

Minimum export fields:

• Crawl date
• URL
• Issue type
• Severity
• HTTP status
• Source page
• Template/type
• Canonical target
• Indexability
• Title/meta/H1 fields
• Issue status
• Notes
• Assigned owner
• Ticket link

Helpful practice:

Maintain a master issue table where each crawl appends data. Then compare current vs previous crawl to identify:

• new issues
• resolved issues
• unchanged issues
• recurring issues

---

7) Set up issue triage and ownership

Exports become valuable when they feed a process.

Suggested workflow:

1. Crawl runs on schedule
2. Issues are exported automatically
3. Issues are grouped by:
   • severity
   • section
   • template
   • owner
4. New issues are pushed into:
   • Jira
   • Asana
   • Trello
   • Monday
   • GitHub Issues
5. Recurring issues are flagged
6. Fixed issues are validated in the next crawl

If your crawler supports it, use:

• webhooks
• API integration
• CSV-to-ticket automation via Zapier/Make or a script

---

8) Create a dashboard for trends

A crawler alone gives snapshots; you want trend visibility.

Track:

• Total errors over time
• Errors by severity
• Errors by section/template
• Fix rate
• Recurrence rate
• Pages crawled vs pages expected
• Indexable pages count
• Broken link count
• Redirect chain count
• Core web vitals or response time trends if available

Tools:

• Looker Studio
• Power BI
• Tableau
• Grafana
• Built-in crawler dashboards

---

9) Validate crawl completeness

Large sites often have crawl gaps.

Check:

• Was the sitemap fully used?
• Were important sections reachable?
• Did the crawler stop due to limits?
• Were JS pages rendered?
• Were blocked URLs excluded too aggressively?
• Were parameter variants causing duplication?

Useful validation sources:

• XML sitemaps
• Analytics landing pages
• Server logs
• CMS content inventory
• Search Console index coverage

Compare crawl coverage against expected URL inventory.

---

10) Recommended practical architecture

Simple stack:

• Crawler: Screaming Frog or Sitebulb
• Schedule: cron / Task Scheduler / built-in scheduler
• Storage: CSV/XLSX in shared drive or S3
• Tracking: spreadsheet + Jira
• Dashboard: Looker Studio / Power BI

Enterprise stack:

• Crawler platform: Botify / Lumar / Oncrawl
• Orchestration: scheduled jobs + API
• Storage: data warehouse
• Issue management: Jira/ServiceNow
• Dashboards: BI tool
• Validation: logs + Search Console

Custom stack:

• Crawler: Scrapy + Playwright
• Scheduler: Airflow / cron
• Storage: PostgreSQL / BigQuery
• Issue logic: custom rules engine
• Exports: CSV + API + ticket creation script

---

11) A good implementation pattern

If you want something maintainable:

1. Run weekly crawl
2. Export findings to CSV
3. Normalize into a database
4. Compare with previous crawl
5. Mark new/resolved/recurring issues
6. Auto-create tickets for new critical issues
7. Generate a summary report
8. Send report to stakeholders

That gives you both auditability and operational tracking.

---

12) What I’d recommend for a large site

If the site is very large and you need scheduled audits with exportable issue tracking, the strongest setup is usually:

• Enterprise crawler if budget allows
• Scheduled crawls
• Issue exports to CSV/API
• A database or warehouse for history
• Jira integration for issue tracking
• Dashboard for trend monitoring
• Coverage validation using sitemap + logs

---

If you want, I can also give you:

1. a tool-by-tool setup guide for Screaming Frog, Sitebulb, or an enterprise platform, or
2. a custom architecture using Scrapy/Playwright + cron + CSV/Jira for a very large site.