Index bloat: when the fix is fewer pages

What's index bloat?

More URLs eligible for indexing than the site has distinct things to say. It shows up as pagination, filter parameters, tag archives, per-answer or per-comment permalinks, and templated pages generated for combinations nobody searches for. Each one returns 200 and looks healthy in a crawl report. Nothing is broken. That's the tricky bit: a crawler grades pages one at a time, and index bloat is a property of the set.

The remedy is unpopular because it means deleting work someone was paid to produce. It's also, at scale, one of the very few interventions with a documented before and after.

The version where deleting 5 million pages was correct

Grant Simmons was hired into Homes.com in 2014, after the site had lost roughly 60% of its organic traffic to what turned out to be Panda. The diagnosis wasn't subtle:

We should clean up a lot of crap, a lot of thin pages that existed purely for just existing

The scale of the cut is the part people struggle to believe:

it was a matter of getting rid of millions of pages from a 110 million page site, taking some really big risks in chopping.

Five million pages came out of the index. Grant then did the thing every SEO fantasises about, which is get a straight answer from Google in person. At SMX London, with John Mueller in the room, he described the site and the fix:

He got on his laptop, he typed in and 20 seconds later he goes, yeah, you're under a Panda penalty. And I go, all right, well, they know that stuff.

And the outcome:

we made those changes and when Panda was pushed out at that time, the rolling updates, we got back all our traffic and then some.

His summary of why the previous agency hadn't fixed it is the transferable lesson:

They were not thinking to be aggressive. I've seen in those days recovery had to be aggressive.

Grant's full episode covers entity SEO, measurement, and what he thinks reporting still gets wrong: listen here.

Three things made deletion the right call at Homes.com, and they're worth saying, because they're the conditions most sites don't meet. The pages were near-duplicates coming out of a template. There were millions of them, enough to change how the whole domain was assessed. And there was a measured traffic collapse to recover from, not a hunch.

The version where deleting is a waste of a weekend

Now the small-site version, from a crawl of seoarcade.com.

The site runs a Q&A section. Every question has a page, and every individual answer to that question also gets its own permalink. Filter the crawl to a single question about keyword forecasts and you get 20 of the 867 crawled URLs back. Eight of those came back 200, all carrying the same title and between 824 and 1,127 words of overlapping text.

Each of those eight sits in the sitemap and reports isIndexable: true. By any crawl-based definition this is index bloat, and the obvious move is to noindex the answer permalinks and ship it.

Then you check what Google actually did with them, which is the step almost nobody takes.

Search Console URL inspection on six URLs from that site returned "URL is unknown to Google" for five of them. The keyword-forecasts question page, two of its answer permalinks, a third answer permalink from a different question, and the parameter archive at /?post_type=answer. The only one that came back "Submitted and indexed" was an ordinary question page, which is the page that should be indexed. It reported a Google-selected canonical matching its declared canonical, three referring URLs, and a Q&A rich result.

There was nothing to remove. The crawler found a mess, and the index didn't have one.

Check before you cut

Here's the difference between the two cases, and it costs nothing to establish. Google's URL Inspection API is free and uses no Role SEO credits, and it answers the only question that matters: does Google know this URL exists, and if it does, which URL did it pick as canonical?

Four verdicts come back, and each one implies a different move.

• If the URL is unknown to Google, there's nothing to remove. Fix the sitemap so you stop submitting it and move on.
• "Duplicate, Google chose a different canonical" also leaves you nothing to remove, because Google's already consolidated the set. Confirm the URL it picked is the one you want.
• The real bloat signal is "Crawled, currently not indexed" or "Discovered, currently not indexed". Google's seen the page and turned it down, so either make the page worth indexing or take it out of the sitemap and the internal link graph.
• A page that comes back "Submitted and indexed" when you never wanted it indexed is the one case for noindex or removal.

Patrick Stox, who built Site Audit and Page Inspect at Ahrefs, points out that the discovery side has more inputs than most people account for, which is why bloat keeps reappearing after you delete it:

But what are the data sources for these systems? Most people will tell you sitemaps, or crawling. Well, there's more. RSS feeds are a thing.

Pulling a URL out of a sitemap doesn't pull it out of an RSS feed, an internal link, or somebody else's page. If a URL keeps coming back after you deleted it, one of the other sources is still publishing it. Patrick's episode on building for the GEO era is on the Unscripted SEO podcast.

The workflow

1. Crawl the site and read the pages report, not the issues report. Bloat isn't an issue type. It's a pattern in URLs, titles and word counts.
2. Sort by title. Repeated titles across a lot of URLs are where templated duplication lives. Then sort by word count, because the low end is where thin templated output collects.
3. Take a sample of 10 suspect URLs and inspect them in Search Console.
4. Act on the verdicts, not the crawl. Unknown and duplicate URLs need a sitemap fix. Indexed-but-unwanted URLs need noindex or a 410.
5. Re-inspect the same sample in 30 days. Index changes are slow, and re-checking is the only way to know the change landed.

Run this with the Role SEO MCP

get_audit_pages gives you the crawl side and inspect_urls gives you Google's side, so an agent can join them in one pass. Both are free calls. They read stored audit state and Search Console rather than a paid data provider.

Using the Role SEO MCP on project [name]:

1. From the most recent site audit, pull all crawled pages. Group
   them by exact title and show me every title used by more than
   two URLs, with the URL count and the word-count range.

2. Do the same for URL patterns: show me any path segment or query
   parameter that appears on more than ten URLs (things like
   /answer/, /page/, /tag/, ?filter=).

3. Pick the ten worst-looking URLs from those groups, weighted
   toward the largest group, and run inspect_urls on them.

4. Report each URL with its coverage state and its Google-selected
   canonical. Then sort them into: unknown to Google, canonicalised
   away, crawled but not indexed, and indexed.

5. Tell me the count in each bucket and what fraction of the
   suspect set is in the index. Recommend an action per
   bucket, and say plainly if the answer is that there is nothing
   to remove.

Point five is the instruction that keeps this workflow honest. Ask an agent to find index bloat and it'll find index bloat. Ask it what fraction of the suspects are in the index and you get a number that can come back as zero.

Deletion is a scale decision

Both stories in this guide are true, and they point in opposite directions, which is the useful bit.

At 110 million pages with a measured 60% traffic loss, removing 5 million near-duplicates was the intervention that worked, and the agency that wouldn't be aggressive enough had already failed at it. At 867 pages with no traffic collapse, the same instinct would have burned a weekend removing URLs Google had never fetched.

What separates the two isn't the size of the mess in the crawl report. It's whether the mess made it into the index and whether anything measurable got worse. Establish both before you delete a page, and the audit becomes evidence instead of an argument.

If part of your crawl came back blocked, hold off on this workflow until that's fixed, because a blocked page reports no title and zero words and will look exactly like thin content.

Index bloat FAQ

How do I know if my site has index bloat?

Compare the number of URLs you want indexed against the indexed count in Search Console's page indexing report. A big gap is the signal. Then inspect a sample of the surplus URLs, because a crawler can hand you hundreds of near-duplicate pages that Google never fetched, which isn't bloat.

Does index bloat hurt rankings?

It can, at scale, when a big share of a domain's pages are thin or duplicated and the domain gets assessed as a whole. On a site of a few hundred pages the more common effects are wasted crawling and internal link equity spread over URLs that were never going to rank. Neither is urgent on its own.

Should I noindex or delete duplicate pages?

Use noindex when the page has a purpose for users, like a filtered listing or a paginated archive. Use a 410 or a 301 when the page has no purpose at all. Pulling a URL out of the sitemap alone doesn't deindex it, it only stops you asking for indexing.

How many pages should a website have indexed?

As many as there are distinct things worth ranking, which for most small business sites is dozens rather than thousands. The count matters less than the ratio. If most of your indexed URLs get no impressions in a year, the set is bigger than the site can support.

Can a site audit tool find index bloat?

Not directly, because every bloated page returns 200 and passes its per-page checks. What a crawler gives you is the raw material: repeated titles, thin word counts, recurring URL patterns. Pair it with Search Console URL inspection to work out which of those URLs Google holds. Role SEO does both: crawls up to 50 pages on the free plan and 10,000 on the £15/month plan, and runs URL inspection against your connected property at no credit cost.