Tools help·3 min read

Inside LOCRA: what a sitemap live check actually tests

How LOCRA checks every loc in a sitemap for status, robots, canonical, redirects and host, the three expectations it can judge against, and why a dirty sitemap is a trust problem.

Nikko Nanji

Nikko Nanji - technical SEO consultant

Published - Last reviewed

Short answer: a sitemap is a list of pages you're asking Google to treat as canonical, indexable and live. Every redirect, noindex, 404 or conflicting canonical on that list weakens it. LOCRA fetches each loc the way a crawler would and marks it PASS, FAIL or ERROR against the state you expect: live and indexable, noindex, or gone.

The quick start is in LOCRA: how it works. This is what each check does.

What you paste in

A sitemap URL for a urlset. A sitemap index is refused, with a count of its child sitemaps. Use ATLAS to map the index, then run LOCRA on each child. Duplicate locs are removed. LOCRA checks the first 2,000 by default, and you can raise that to 5,000 or 10,000. Tick random sample to check a random slice instead of the first N, which is better on a big sitemap where the first rows are all one template.

Then choose the expectation:

  • 200 OK (live and indexable): the default, for a normal sitemap.
  • noindex (deindex list): for a sitemap you're using to get pages dropped.
  • Gone (410 or 404): for removal lists.

The checks, under the hood

For the default expectation, LOCRA follows redirects and reads the final response. A row fails if any of these is true:

  • The status isn't 200: 404 or 410 ("page gone"), or anything else.
  • The page carries noindex in the meta robots tag or the X-Robots-Tag header.
  • The final URL differs from the loc: "Stale / redirected URL in sitemap". Host case, trailing slash and fragment are normalised first.
  • The loc's host isn't the sitemap's host, ignoring www: "Wrong host vs sitemap host". If only www versus apex changed on the way, it's "www / apex mismatch".
  • The canonical points elsewhere: "Conflicting canonical -> URL".

All the reasons are listed, so one row can carry three problems.

noindex expectation: a 200 with no noindex fails as "still indexable". A noindex, a 404 or a 410 all pass.

Gone expectation: LOCRA doesn't follow redirects here. It reads the first response. 410 passes. 404 passes, with a note that 410 is preferred. A redirect fails and shows its Location. A 200 fails as "still live", even with a noindex. Server errors and rate limits are ERROR, not FAIL.

Being polite to the host

LOCRA runs two requests at a time, with a pause after each one, and backs off when a host answers 429. If a host keeps rate-limiting, it slows to one at a time and lengthens the pause between batches. A run with 400 or more locs takes a while on purpose. Hammering a client's site to finish quicker is not a feature.

Edge cases

  • Relative canonicals are resolved against the final URL before they're compared.
  • Sitemap XML quirks: CDATA and escaped ampersands in locs are decoded, so & in a URL doesn't create a false mismatch.
  • Non-HTML locs (PDFs, images) are judged on status and headers only. There's no canonical to read.

Worked example

CASE FILE: product sitemap, samplebrand.test
<url><loc>https://samplebrand.test/kits/blue-kit</loc></url>
<url><loc>https://www.samplebrand.test/kits/red-kit</loc></url>
<url><loc>https://samplebrand.test/kits/green-kit?ref=nav</loc></url>

Live: row 1 200, self-canonical
Live: row 2 301 -> https://samplebrand.test/kits/red-kit (200)
Live: row 3 200, canonical -> https://samplebrand.test/kits/green-kit

VERDICT: PASS, FAIL (stale + www / apex), FAIL (conflicting canonical)

Row 2 is listed on the wrong host and redirects. Write the apex URL in the sitemap. Row 3 lists a tracking-parameter URL whose canonical points at the clean one. List the canonical URL, not the variant.

Two template-level fixes to the sitemap generator and the whole file is clean.

Run it: open LOCRA with a sitemap prefilled

That link prefills this site's own sitemap, so you can watch a full run before you point it at yours.

The money angle

A clean sitemap doesn't earn money by itself. A dirty one wastes crawl and blurs which URL you want to rank, and on a commercial template that means the wrong URL earning, or none. Sitemap generator fixes are also some of the cheapest in technical SEO: one change to a template, and thousands of rows are fixed.

That's why LOCRA is one of the first runs in a technical SEO audit, and the last check after a migration.

Sources