XML sitemaps and indexing

A page that isn’t indexed can’t rank, however good it is. This service finds out which of your pages Google has, which it has looked at and decided not to keep, and which it has never found, then fixes the reasons why.

Crawled but not indexed is almost never a technical fault. It is Google telling you the page was not worth keeping.

What sitemaps and indexing work means

Indexing is the step between a search engine finding a page and being willing to show it. A page can be crawled and rejected, found and ignored, or never discovered at all, and those are three different problems with three different fixes.

An XML sitemap is the list of URLs you’re asking to have indexed. It helps with discovery, particularly on large sites and new sections. It doesn’t oblige a search engine to keep anything, and a sitemap full of contradictions makes things worse rather than better.

Who it’s for

This suits sites where pages aren’t showing up:

  • new sections or products that never appear in search
  • sites showing large numbers of “crawled, currently not indexed” in Search Console
  • anyone who has just migrated, replatformed or changed URLs
  • large catalogues where only part of the range is indexed
  • headless or custom builds, which often have no usable sitemap at all

The problems it usually solves

Redirected URLs left in the sitemap. After a migration the file often still lists the old addresses. Every one is a URL you’re asking to have indexed and then redirecting away from, which is a contradiction and a waste of crawling.

Whole sections missing. Usually a template a content system doesn’t include by default. Those pages are then relying on internal links alone to be discovered, and if the internal linking is also thin, they simply aren’t found.

Crawled and not indexed. The most common question I get on this. It usually means the page was reachable but wasn’t judged worth keeping, which is a content and duplication problem rather than a technical one. Resubmitting it changes nothing.

Lastmod that means nothing. A date that updates on every deployment, whether or not the page changed, tells a search engine nothing useful. Either make it real or leave it out.

Headless builds with no sitemap. A headless setup doesn’t hand you one by default. On a project where the redirects had been planned carefully, the sitemap was the thing that nearly caught us out, because on a normal build the platform just produces one.

What’s included

  1. A sitemap audit against a live crawl: what’s listed that shouldn’t be, and what’s missing
  2. An indexing diagnosis using Search Console’s own reports and live URL inspection
  3. A separation of the three problems: not discovered, discovered and rejected, or actively excluded
  4. Corrected sitemaps, generated where your platform can’t produce a usable one
  5. Robots and canonical fixes where they’re contradicting each other
  6. Internal linking recommendations, since discovery usually fails there first
  7. A resubmission and monitoring plan

What you get

  • A clear answer on which pages Google has and which it doesn’t
  • A corrected sitemap listing only URLs that should be indexed
  • The reason each missing page is missing, rather than a general recommendation
  • A monitoring routine so new indexing problems get caught early

The honest answer about “crawled, currently not indexed”

There’s an industry habit of treating this as a technical fault with a technical fix. Usually it isn’t. It means a search engine looked at the page and didn’t think it was worth keeping, which points at duplication, thin content, or a page that’s one of several covering the same thing.

The fix is almost always to merge, improve or remove, not to resubmit. Telling clients that is less satisfying than a checklist, and it’s the truth.

Limitations

Indexing is a search engine’s decision and there’s no way to compel it. Submitting a sitemap, fixing the canonicals and improving the pages all improve the odds, and none of them is a guarantee.

Search Console’s reports also cap their rows, so on a large site the totals you can export are a floor rather than a complete picture.

What this won’t solve

Getting a page indexed doesn’t make it rank. If the page is indexed and invisible, that’s a content and competition problem, covered by content quality and E-E-A-T.

Talk to the person doing the work

There is no account manager here and no team to be handed to. If you get in touch, it is me who reads it, me who looks at your site, and me who does the work if we go ahead.

Get in touch, or start with a free assessment.

Engagement and price

There are no list prices here, and that is deliberate. What a piece of work costs depends on the state of the site, how much of it there is and what you actually need, none of which I know before looking at it.

So it starts with a free analysis. I look at your site, your Search Console and who you are really competing with, then come back with what I would do first and what it would cost to do. No obligation attached, and if the honest answer is that you do not need me yet, I will say so.

How the work gets done

Sitemaps run through the sitemap toolkit, with crawl and status data from the technical SEO checks and live inspection through the crawl and data integrations. This service sits under SEO, and for replatforming work see site migrations.

FAQ

Why is my page crawled but not indexed?

Usually because the search engine looked at it and didn’t judge it worth keeping, rather than because of a technical fault. That normally points to duplication, thin content or several pages covering the same ground. Merging or improving the page works. Resubmitting it doesn’t.

Do I need an XML sitemap?

Most sites benefit, and large or frequently updated sites benefit most. On a small, well-linked site it makes little practical difference. It matters most when pages are hard to reach through internal links, or when a whole section has recently been added.

How long does indexing take?

There’s no reliable figure, and it varies by site, page and how often a search engine crawls you. New pages on an established, well-linked site are usually picked up quickly. Pages on a new domain with few links can take considerably longer, and no submission speeds that up much.

Can you guarantee my pages get indexed?

No, and neither can anyone else. Indexing is the search engine’s decision. What can be done is remove every reason not to index a page, make sure it’s discoverable, and make sure it’s worth keeping, which is what this service covers.

Find out what’s actually indexed

Get in touch, or start with a free assessment, which includes an indexing check.

Get started

Proof, not promises.

Tell me what's going on and I'll come back with what I'd do first. No obligation, and no report you need a translator for.

Over 925,000 impressions and 13,795 clicks in a year, from people asking one awkward question: what size do I need?condoms.uk, past 12 months

Brands and sites I have worked on

Brands worked on across nearly twenty years, in agency roles and directly.