Short answer: robots.txt controls crawling, and noindex controls indexing. Put a Disallow in front of a noindex and Google can never fetch the page to read the noindex. A URL that's linked from elsewhere can then stay indexed, with no snippet. VEXA checks each URL against the robots.txt rule that actually applies to Googlebot, fetches the page, and returns one of five verdicts with the exact rule to change.
The step-by-step is in VEXA: how it works. This is the rule engine.
What you paste in
URLs, up to 50 per run. VEXA fetches robots.txt once per host and reuses it for every URL on that host.
How the robots.txt rule is chosen
This is where hand checks go wrong, so the code follows Google's documented behaviour:
- Group choice. If there's a group for
googlebot, only that group's rules apply. Otherwise the*group applies. They're not merged. A site with a Googlebot group that forgets a rule from*has, in effect, allowed it for Google. - Longest match wins. Each Allow and Disallow path is turned into a pattern, with
*matching anything and a trailing$anchoring the end. It's tested against the path plus the query string. The rule with the longest path (wildcards not counted) wins. - Ties go to Allow. If an Allow and a Disallow match with the same length, Allow wins.
- An empty
Disallow:blocks nothing. - robots.txt missing or failing. A 404 or 410 means no rules: everything is allowed. Any other error means VEXA can't prove the robots side, and says so.
The five verdicts
- TRAP: blocked for Googlebot, and the page says noindex. Lift the Disallow, keep the noindex, request a recrawl. It's also a TRAP if the page is blocked and VEXA couldn't fetch it either: you have to decide which way to go.
- KEEP_PRIVATE: blocked, and the page looks private. The path contains login, sign-in, account, checkout, cart, admin, /api/, survey or password, or the title does. Keep the block, and put the page behind auth or a 404 so links can't get it indexed.
- ALLOW_PUBLIC: blocked, but it looks like a normal public page with no noindex. Remove the rule so Google can crawl it.
- NOINDEX_OK: crawlable and noindex. The correct way to remove a page from the index. Leave it.
- CLEAR: crawlable, indexable, no conflict.
NOINDEX_OK and CLEAR pass. Everything else, including fetch errors, fails.
Edge cases
- The query string counts.
Disallow: /*?sort=blocks/kits?sort=pricebut not/kits. VEXA tests the full path and query string, as Google does. - "Private" is a pattern guess. VEXA looks for login, checkout and similar words in the path or title. It's a strong hint, not proof. Read the page before you keep a block on something earning money.
- noindex is read from three places: the robots meta tag, the googlebot meta tag and the X-Robots-Tag header.
Worked example
robots.txt:
User-agent: *
Disallow: /tags/
Disallow: /account
Allow: /tags/featured
https://samplebrand.test/tags/teal (meta robots: noindex)
https://samplebrand.test/tags/featured
https://samplebrand.test/account/ordersVERDICT: TRAP, CLEAR, KEEP_PRIVATE
/tags/teal says noindex behind Disallow: /tags/. That's the trap: Google can't read the noindex. /tags/featured matches both /tags/ (5 characters) and /tags/featured (14), so the longer Allow wins and it's crawlable. /account/orders is blocked and has "account" in its path, so keep it private. The fix is one line: remove Disallow: /tags/, and let the noindex do its job.
The money angle
The trap costs twice. Pages you wanted gone stay in the index, often thin ones a reviewer will notice. And pages blocked by mistake can't be crawled or refreshed, which puts real revenue at risk.
In a thin-content clean-up during Google penalty recovery, getting this wrong can make a site look as if it ignored the fix. VEXA is how I prove the noindex can actually be seen.