Short answer: anyone can type "Googlebot" into a user agent. Only Google crawls from Google's IP ranges. BOTCHECK reads your access log, checks every Google-claiming request against the ranges Google publishes, and then asks the question that matters commercially: did your own stack block the real one?
The step-by-step log walkthrough is in check Googlebot in access logs. This is the engine room.
What you paste in
Raw access log lines. BOTCHECK understands three shapes:
- The combined log format that Apache and nginx write by default.
- JSON lines with common CDN field names, such as client IP, edge status, user agent and response bytes.
- Anything else, parsed with a looser fallback that hunts for an IP, a status and the last quoted string as the user agent.
Comment lines and blank lines are skipped.
How a request gets classified
Two facts per line: what the user agent claims, and where the IP sits.
- The claim. The user agent is matched against a list of Google crawler names: Googlebot and its Image, Video and News variants, Google-InspectionTool, GoogleOther, AdsBot-Google, Storebot-Google, Google-Extended and others.
- The IP. It's matched against Google's published range files for common crawlers, special crawlers and user-triggered fetchers. Both IPv4 and IPv6 are handled, with exact CIDR maths rather than prefix guessing.
From those two facts, each request gets one of five classes:
- Verified common crawler: claims Google, and the IP is in the common crawler ranges.
- Verified special crawler: claims Google, and the IP is in the special crawler ranges.
- User-triggered fetcher: the IP is in the user-triggered ranges.
- Spoofed: claims to be Google, but the IP isn't in any Google range.
- Unknown: the IP can't be parsed, or it sits in a Google range without a Google claim.
Range files are fetched live when possible. If a live file fails, BOTCHECK falls back to a bundled snapshot for that file and labels which one it used, so you always know what you're comparing against.
What counts as a block
A request counts as blocked if it got a 403, 429 or 503, or if it was challenged. A challenge is detected three ways: a request to the CDN challenge path, a JSON security or WAF action that mentions a challenge, or a marker pattern you supply. 5xx responses are counted separately, as server errors.
For verified common crawler traffic only, BOTCHECK builds the evidence you need for a ticket:
- Blocked hits per UTC day, with the status mix and the URLs affected.
- The full list of at-risk paths that were blocked or got a 5xx.
- Oversized responses: any response to verified Googlebot over 2 MB, with path and date.
Why only verified traffic? Blocking a spoofer is your firewall doing its job. Blocking the real Googlebot is your firewall costing you money.
Edge cases
- Spoofed isn't always malicious. Some monitoring tools fake a Googlebot user agent. Spoofed means "not Google", nothing more.
- Unknown rows deserve a look. An IP in a Google range with no Google claim is often a Google service that isn't a search crawler. Don't count it as search.
- Dates come out in UTC. Log timestamps with offsets are converted, so a block at 23:30 local time can land on the next UTC day. Match the date to your deploy log in UTC.
Worked example
66.249.66.1 - - [18/Oct/2026:09:12:03 +0000] "GET /pricing HTTP/1.1" 403 512 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.1 - - [18/Oct/2026:09:12:09 +0000] "GET /guides/setup HTTP/1.1" 200 48211 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.0.113.7 - - [18/Oct/2026:09:13:40 +0000] "GET /pricing HTTP/1.1" 200 51002 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"VERDICT: real Googlebot blocked on /pricing, fake one let through
The 66.249.x address falls in Google's common crawler ranges, so lines 1 and 2 are verified. Line 1 is a 403 on the pricing page, so it's a blocked verified hit, and /pricing goes on the at-risk list. Line 3 claims Googlebot from a documentation-range IP, so it's spoofed. The firewall had it backwards.
Before you trust the classes on your own log, check that sample IP against the live range files in the tool.
Run it: open BOTCHECK and press Sample input
The money angle
If Google can't fetch a page, it can't refresh it, and a commercial page that stays stale or drops out stops earning. The fix is usually one rule: allow verified Google ranges before the challenge fires.
That makes BOTCHECK one of the cheapest fixes to value. Send the blocked paths to PAYBACK, put in your own clicks and conversion value, and compare that with an hour of firewall work.
When I run a traffic drop audit, log evidence like this is one of the first things I ask for. It separates "Google stopped liking us" from "we stopped letting Google in".