How do I find crawl issues on a website that's not getting indexed properly?

To find crawl issues on a site that isn’t being indexed properly, use a mix of Google Search Console, server/log data, and a quick technical audit.

1) Check Google Search Console first

Look at these reports:

• Pages / Indexing report

  • Shows which URLs are indexed and which aren’t
  • Pay attention to reasons like:
    • Crawled - currently not indexed
    • Discovered - currently not indexed
    • Blocked by robots.txt
    • Excluded by noindex tag
    • Duplicate without user-selected canonical
    • Soft 404
    • Server error (5xx)

• Crawl stats

  • Reveals whether Googlebot is crawling enough pages
  • Look for spikes in:
    • response time
    • 4xx / 5xx errors
    • crawl requests dropping unexpectedly

• URL Inspection tool

  • Test important URLs individually
  • Check:
    • whether Google can crawl the page
    • whether indexing is allowed
    • canonical chosen by Google
    • last crawl date
    • rendered page screenshot / HTML

2) Review robots and directives

Make sure you’re not accidentally blocking crawlers:

• robots.txt
  • Check for blocked folders or pages
• noindex meta tags
  • Ensure important pages aren’t marked noindex
• X-Robots-Tag HTTP headers
  • Sometimes pages are blocked here even if HTML looks fine
• Canonical tags
  • Make sure they point to the correct preferred URL

3) Look for server and response problems

Crawl problems often come from technical errors:

• 5xx errors
• slow server response
• timeouts
• redirect chains or loops
• broken links
• 404/410 pages being linked internally
• inconsistent HTTP/HTTPS or www/non-www versions

4) Crawl the site with SEO tools

Use tools like:

• Screaming Frog
• Sitebulb
• Ahrefs site audit
• Semrush site audit

These can uncover:

• blocked pages
• redirect chains
• broken internal links
• duplicate content
• missing canonicals
• thin pages
• pages buried too deep in the site structure

5) Check internal linking and site structure

If pages are hard to reach, Google may not crawl them well.

• Important pages should be reachable within a few clicks
• Ensure strong internal links from important pages
• Verify your XML sitemap includes only indexable, canonical URLs

6) Compare sitemap vs indexed pages

• Submit a clean XML sitemap in Search Console
• Remove:
  • redirects
  • 404s
  • noindex pages
  • non-canonical URLs
• If sitemap URLs aren’t indexing, that’s a sign of deeper crawl or quality issues

7) Inspect logs if possible

Server logs show exactly what bots are doing:

• Which URLs Googlebot requests
• How often it crawls
• Whether it hits errors or gets stuck on low-value pages

This is especially useful for large sites.

8) Check for JavaScript rendering issues

If the site relies heavily on JS:

• Google may not see content or links immediately
• Test with URL Inspection and “View crawled page”
• Make sure important content and links are available in the rendered HTML

---

Quick priority checklist

Start with:

1. Search Console Indexing report
2. URL Inspection on key pages
3. robots.txt / noindex / canonicals
4. crawl audit with Screaming Frog or similar
5. server logs and crawl stats

If you want, I can also give you a step-by-step crawl issue diagnosis workflow or a checklist you can use on any site.