Indexed Though Blocked by robots.txt: What It Means + How to Fix It

The contradiction in this status dissolves once you accept one fact: robots.txt controls crawling, not indexing. Blocking a URL in robots.txt tells Google don't fetch this — it does not say don't list this. So Google can, and sometimes does, put the URL in the index on outside signals alone (most often inbound links from other sites), having never read a word of the page.

The result is the worst of both worlds. The URL shows up in search, but with no title, a generic or stale description, or the giveaway snippet “No information is available for this page.” Google is ranking a page it can't see.

So the real question isn't how do I make this go away — it's did I want this page indexed or not? The two answers need opposite fixes, and picking the wrong one is how most people get stuck here.

First, confirm what Google can and can't do

Open Pages → Why pages aren't indexed, click “Indexed, though blocked by robots.txt,” and pull one URL into URL Inspection. Two fields tell you everything:

• Crawl allowed: No — confirms a robots.txt rule is blocking the fetch.
• Page fetch: Failed (blocked) alongside Indexing: URL is on Google — confirms the page is indexed despite that block.

If Crawl allowed says Yes, you're not looking at this status — recheck the report. While you're in URL Inspection, note the robots.txt rule GSC names as the blocker; you'll need to find that exact Disallow: line.

Decide which page this is

Everything downstream depends on this one call. Run the affected URL against both — only one is yours.

This page should be indexed

You blocked it by accident — a leftover Disallow: from a migration, an overly broad rule like Disallow: /products/ that swept up pages you want, or a staging rule that shipped to production.

Tell: it's a real page you'd be happy to see rank — a product, an article, a category — and you can't remember deliberately wanting it out of Google.

This page should not be indexed

It's a thank-you page, a faceted-filter URL, an internal search result, a PDF you'd rather hide — and you reached for robots.txt to keep it out of Google. That's the wrong tool: robots.txt stops the crawl but not the listing, which is precisely why it's stuck here.

Tell: you intended to keep this URL out of search, and robots.txt was your method.

Fix it for the page you have

Match the path to the call you just made — you only need one.

1. Should be indexed: remove the Disallow, then re-crawl

   Open https://yourdomain.com/robots.txt and find the rule URL Inspection named — for example Disallow: /products/. Delete it, or narrow it so it no longer matches the URL you want crawled. Order and specificity matter: a broad Disallow: can override a later Allow:, so test the exact path in GSC's robots.txt report after editing.

   Then re-run URL Inspection, confirm Crawl allowed: Yes, and click Request indexing. Google re-crawls, reads the real content, and rebuilds the index entry with a proper title and snippet.

2. Should NOT be indexed: allow crawling, then noindex

   This is the counterintuitive part. To get a page out of the index, you have to let Google in. The deindex signal is a noindex tag — and Google can only see it by crawling the page.

   First, remove the Disallow: rule from robots.txt so the page is crawlable. Then add a noindex directive to the page itself:

   <meta name="robots" content="noindex" />
   

   Now Google can fetch the page, read the noindex, and drop it from the index — typically within days to a few weeks. Once it's gone, you can re-block in robots.txt, but you rarely need to; the noindex does the job on its own. See Excluded by noindex tag for what the clean end-state looks like.

3. Then pull it out of your sitemap

   For the deindex path: if the URL is in your XML sitemap, remove it. A sitemap entry tells Google this URL matters — the opposite of the noindex signal you're sending. Leaving it in keeps re-suggesting the page Google is trying to forget.

Confirm the fix landed

Give it a few days to a few weeks, then verify against what you were going for:

• If you wanted it indexed — re-inspect in URL Inspection for Crawl allowed: Yes, then watch the Performance report for that page; impressions and a real snippet appearing is the proof it's indexed and readable.
• If you wanted it deindexed — re-inspect and confirm Google processed the noindex; the URL should move to Excluded by ‘noindex’ tag in the Pages report and fall out of site:yourdomain.com/your-page-url results.

If a page you wanted gone is still listed after a couple of weeks, the usual cause is that the block went back on too soon — Google never got a clean crawl with the noindex visible. Lift the block, leave it lifted until the page clears, and re-check.

Don't confuse it with these neighbors

The line that sets this apart: it's an indexed page, and the block is the cause of the problem — not the solution to it.