Prompt

How do I troubleshoot pages that are crawled but not indexed?

Technology / SEO & AEO Tools1 observationLast seen Oct 5, 2026

Latest observation

Oct 5, 2026OpenAI APIWeb search: off

Pages that are crawled but not indexed usually mean Google found the URL, fetched it, but decided not to keep it in the index. The fix depends on why.

1) Confirm the exact reason in Google Search Console

In URL Inspection, check:

  • Crawled - currently not indexed
  • Discovered - currently not indexed
  • Duplicate, Google chose different canonical
  • Alternate page with proper canonical
  • Excluded by ‘noindex’
  • Blocked by robots.txt

This reason tells you where to look next.

2) Make sure the page is indexable

Check the basics:

  • The page returns 200 OK
  • It does not have a noindex meta tag or header
  • It is not blocked by robots.txt
  • The canonical tag points to the correct URL
  • It is not redirecting unexpectedly
  • It is accessible without login/paywall issues

3) Check for duplication and canonical issues

Google may skip pages that look like duplicates. Look for:

  • Similar or near-identical content on multiple URLs
  • Parameters, filters, faceted navigation creating many versions
  • Canonicals pointing to another page
  • Canonical chains or conflicts

Fix by:

  • Consolidating duplicate URLs
  • Using the correct canonical
  • Strengthening the unique value of the page

4) Improve page quality and uniqueness

A common reason is that the page is technically crawlable but not valuable enough to index. Improve:

  • Unique, substantial main content
  • Clear purpose and search intent match
  • Strong title and heading
  • Unique text, images, data, or tools
  • Internal links from important pages

Thin, template-heavy, or near-empty pages are often excluded.

5) Strengthen internal linking

Google indexes pages it sees as important. Do this:

  • Link to the page from relevant high-authority pages
  • Add it to category pages, hubs, or breadcrumbs
  • Avoid orphan pages
  • Use descriptive anchor text

6) Check crawl budget and site quality at scale

If many pages are affected:

  • Remove low-value pages from indexation strategy
  • Noindex or consolidate thin pages
  • Improve site architecture
  • Reduce parameter-generated duplicates
  • Fix soft 404s and broken internal links

7) Re-test after fixes

After changes:

  • Use URL Inspection → Test Live URL
  • Request indexing
  • Monitor coverage over the next few days/weeks

8) Common patterns by reason

Crawled - currently not indexed

Often:

  • Low-quality or thin content
  • Duplicate-like content
  • Weak internal linking

Discovered - currently not indexed

Often:

  • Crawl budget constraints
  • Too many low-value URLs
  • Poor site architecture

Duplicate, Google chose different canonical

Often:

  • Canonical mismatch
  • Duplicate content
  • URL variations

Blocked by robots.txt / noindex

Often:

  • Accidental technical settings

9) What not to do

  • Don’t keep asking for indexing without fixing the underlying issue
  • Don’t mass-publish thin pages
  • Don’t canonical everything to the homepage
  • Don’t use noindex on pages you actually want indexed

If you want, I can give you a step-by-step troubleshooting checklist or help diagnose a specific URL/status from Search Console.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (1 observation for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.