Prompt
what should I use to find duplicate URLs from filters and parameters
Latest observation
Oct 13, 2026
OpenAI APIWeb search: off
To find duplicate URLs caused by filters and URL parameters, use a URL normalization + deduplication approach.
Best tools / methods
- Screaming Frog SEO Spider
- Crawl the site and export URLs.
- Use filters to spot parameterized URLs and near-duplicates.
- Good for medium-sized sites.
- Sitebulb
- Great for finding duplicate pages, parameter issues, and crawl patterns.
- Google Search Console
- Check Pages and URL parameters issues indirectly through indexed URLs and duplicates.
- Log file analysis
- Useful if you want to see which parameter URLs are actually being crawled.
- Custom script / spreadsheet
- Export URLs and normalize them by removing tracking parameters and sorting/removing query strings.
What to look for
- URLs with different parameters but same content, like:
/category?sort=price/category?sort=new/category?page=2/category?filter=red
How to identify duplicates
- Crawl all URLs
- Normalize URLs
- Remove tracking parameters like
utm_*,gclid,fbclid - Decide which parameters matter and which don’t
- Remove tracking parameters like
- Group by canonical path
- Compare pages with the same main path and similar titles/content
- Check canonicals
- Make sure duplicates point to the preferred version
- Review indexing
- Use GSC to see what Google chose as canonical
Quick recommendation
If you want a practical SEO workflow, start with: Screaming Frog + a spreadsheet/pivot table + Google Search Console
If you want, I can give you:
- a step-by-step Screaming Frog workflow, or
- a formula/script to dedupe URLs in Excel/Google Sheets.