Inside ATLAS: mapping a sitemap estate from robots.txt down

Short answer: before you judge a site's pages, look at what it asks Google to crawl. ATLAS starts at robots.txt, follows every Sitemap line into indexes and child sitemaps, counts what each one declares, and groups the URLs into sections by path. It maps what's declared, never what's indexed. It's the floor plan of the site, and it's often the fastest way to see where the weight really sits.

The basics are in ATLAS: how it works. This is the walk and the grouping.

What you paste in

A site address or a sitemap URL:

• A site address: ATLAS reads /robots.txt and collects every Sitemap: line. If there aren't any, it tries /sitemap.xml and /sitemap_index.xml.
• A sitemap URL (the path contains "sitemap", or ends in .xml or .xml.gz): ATLAS starts there.

Every step is logged in plain words: "robots.txt declares 3 sitemaps", "this is an index with 12 child sitemaps", and so on. You can see exactly how it got where it got.

The walk, under the hood

• Indexes add their children to the queue. Urlsets are counted and grouped.
• Caps keep it honest: 4 levels of nesting, 80 sitemaps and 50,000 declared URLs. Hitting any of them marks the map as capped and names the cap in the log, so a partial map never pretends to be whole.
• Statuses per sitemap: ok, fetch failed (with the HTTP code), empty, empty index, or not sitemap XML.
• No duplicates: each sitemap is read once, even if two indexes list it.

How the sections are grouped

Within each child sitemap, ATLAS groups URLs by their first path segment: /kits/, /guides/, /tags/, or (root). One refinement does most of the work. If a single section holds more than 20 URLs and 70% or more of that sitemap, and splits further, ATLAS breaks it down one more level (/shop/kits/, /shop/tags/). Locale folders such as /en/ or /en-gb/ are never split this way. Past 40 groups, the smallest are folded into Other. Each group shows its count and up to 8 sample URLs.

Edge cases

• Declared isn't indexed. ATLAS never claims a URL is in Google. For live status, run the child sitemap through LOCRA.
• Gzipped sitemaps. ATLAS reads the response as text. A .xml.gz served as a raw compressed file, rather than with gzip encoding, comes back as "not sitemap XML". That's worth knowing, because Google can read those files.
• robots.txt you can't see. If robots.txt fails, the log says so and ATLAS falls back to the usual locations. Missing sitemaps can mean the site has none, or that they're simply not declared.

Worked example

robots.txt -> Sitemap: https://samplebrand.test/sitemap_index.xml
sitemap_index.xml -> 3 child sitemaps
  sitemap-pages.xml ....... 40 URLs   (root) 22, /guides/ 18
  sitemap-products.xml .... 900 URLs  /kits/ 900
  sitemap-tags.xml ........ 6,200 URLs /tags/ 6,200

The site sells kits. Yet nearly nine in ten URLs it asks Google to crawl are tag pages. That's the shape a quality reviewer sees first. The next moves: sample the tag template in THINN, decide improve, merge or remove for the tag section in writing, and check the result in LOCRA.

Run it: open ATLAS

The money angle

Site shape is a business decision disguised as a technical one. If most of what you declare is low-value templates, your crawl and your quality signals are weighted towards pages that don't sell. A map makes that visible in minutes, before anyone spends weeks rewriting pages in the wrong section.

ATLAS is the first look in a technical SEO audit. In Google penalty recovery, it's the evidence that the thin section has been contained, not just reworded.