How to check Googlebot in your access logs

Short answer: a request that says "Googlebot" is a claim, not proof. Anyone can type that into a request. The proof is the IP address. Check it with a reverse DNS lookup and then a forward lookup, or match it against Google's published IP lists. Once you know which hits are really Google, look at what your server gave them. Real Googlebot getting 403, 429, 5xx or a bot challenge is the problem I'd fix first.

An access log is the file your server writes every request into: the IP address, the time, the URL, the status code and the user agent. The user agent is the name a visitor gives itself, like "Googlebot" or "Chrome".

What is Googlebot?

Googlebot is the program Google uses to fetch pages for Search. It follows links and sitemaps, downloads pages, and passes them on so Google can index them. Indexing means storing the page so it can show in search results.

Google groups its crawlers into three types, and it matters when you read logs:

• Common crawlers, such as Googlebot. Google says they always follow robots.txt rules for automatic crawls. robots.txt is the file that tells crawlers where they can't go.
• Special-case crawlers, such as AdsBot. They may or may not follow robots.txt.
• User-triggered fetchers, which fetch a page because a person asked for it, for example Google Site Verifier. Google says these ignore robots.txt, because a person requested the fetch.

So a hit from a user-triggered fetcher isn't Googlebot crawling your site. Keep them apart in your numbers.

What does Googlebot see on your page?

Googlebot sees what your server sends it, and then what the page looks like once it's rendered. Rendered means after the JavaScript has run, like in a browser.

Two limits to know:

• Size. Google's Googlebot doc says that, for Search, Googlebot fetches the first 2MB of a supported file, or the first 64MB of a PDF. Anything after that isn't used.
• What you serve it. If a firewall or bot check sends Googlebot a challenge page, that's what Google sees.

To see it the way Google does, use the URL Inspection tool in Search Console. Run a live test, then click View tested page. You get the rendered HTML, a screenshot and the HTTP response headers. Your logs then tell you whether real Googlebot got that same page or something else.

How to verify Googlebot

Google's verify doc gives two methods.

1. Reverse DNS, then forward DNS. Good for a few IPs.

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

• Run host on the IP from your log.
• Check the name ends in googlebot.com, google.com or googleusercontent.com.
• Run host on that name and check it gives back the same IP.

Do both steps. The forward lookup isn't optional, whatever some guides say. Google lists it as part of the method. A reverse lookup on its own can be faked by whoever controls that IP's DNS records. Only the real owner of the name can make the forward lookup point back to the same IP.

A note on googleusercontent.com: Google's doc accepts it, but it belongs to user-triggered fetchers, which act for a person and ignore robots.txt. If you're checking Googlebot crawling, expect googlebot.com.

2. Match the IP against Google's published lists. Good at scale. See the next section.

Googlebot IP addresses

Google publishes its crawler IP ranges as JSON files. The ranges are in CIDR format, which is a short way of writing a block of addresses, like 66.249.64.0/27.

The files live at https://developers.google.com/static/crawling/ipranges/ (for example common-crawlers.json). The old address, developers.google.com/static/search/apis/ipranges/googlebot.json, now redirects to common-crawlers.json, so update any script or firewall rule that still uses the old one. Each file carries a creation time and Google updates them, so don't save a copy once and forget it.

Can I verify Googlebot by switching my user agent in Chrome?

No. Changing your user agent in Chrome only changes the label your browser sends. The requests still come from your own IP address, so they won't pass a DNS or IP list check. It's handy for seeing how a site responds to the label, and it's exactly what fake Googlebots do. To see what Google really gets, use the URL Inspection live test.

Fake Googlebot: how to spot it

A fake Googlebot is a request that uses a Google user agent from an IP that isn't Google's. They're usually scrapers, hoping you let in anything that says "Googlebot".

Signs in your logs:

• The user agent says Googlebot, but the IP isn't in any of Google's lists and reverse DNS doesn't end in a Google domain.
• It ignores your robots.txt rules. Real Googlebot follows them for automatic crawls.
• It's getting 200s on pages you meant to protect, because a rule lets in anything with "Googlebot" in the user agent.

It's safe to challenge or block fakes at your CDN or firewall. Check a few with reverse DNS first, so your rule doesn't catch real Google too.

How to check if Googlebot is blocked

Start in Search Console, then confirm in your logs.

1. robots.txt report. In Search Console settings, open the robots.txt report. It shows the robots.txt files Google found for your top hosts, when it last fetched them, and any errors. If Google can't fetch robots.txt, nothing else matters yet.
2. URL Inspection. Inspect a URL you care about. "Crawl allowed?" tells you whether robots.txt blocks it. "Page fetch" tells you whether Google could fetch it.
3. Crawl Stats. In Settings, open Crawl Stats. Look at host status and the split of responses by status code. A rise in 403, 429 or 5xx is the warning sign.
4. Your logs. Search Console shows Google's side. Your logs show which rule did it. Filter to verified Googlebot and look for 403, 429, 503 and challenge pages. Bot protection, rate limits and firewall rules are the usual cause, and nobody notices because the site works fine for people.

What Google says about the codes you might be sending it:

• 429 and 5xx make Google slow its crawling. Indexed URLs are kept for a while but eventually dropped.
• 503 or 429 to shed load is fine for a short time. Google's crawl rate doc says not for longer than 1 to 2 days. If Googlebot sees them on the same URL for several days, the URL may drop out of the index.
• 401 and 403 don't slow crawling. Google says don't use them to limit crawl rate.
• robots.txt itself returning 429 or 5xx is serious. Google's Crawl Stats help says that for the first 12 hours Google stops crawling the site. After that, for up to 30 days, it uses the last robots.txt it fetched successfully. A 404 for robots.txt is fine. Google reads it as "no rules".

On bot challenges: letting verified Googlebot through is fine. Showing it different content from what a person sees after passing the challenge is cloaking, which is against Google's spam policies.

How many pages does Googlebot crawl per day?

There's no set number. It depends on your site's size, speed and how much Google wants to crawl it. Crawl Stats in Search Console shows your total crawl requests over time, so you can see your own site's normal. Your logs show the same from your side, request by request.

How to read a .log file

An access log is plain text, one request per line. Most servers write the "combined" format. Here's one line, made up but in the real shape:

66.249.66.1 - - [21/Oct/2026:09:14:02 +0100] "GET /guides/kettles HTTP/1.1" 200 18432 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Left to right:

• 66.249.66.1 is the IP address. This is what you verify.
• [21/Oct/2026:09:14:02 +0100] is the date, time and time zone.
• "GET /guides/kettles HTTP/1.1" is the request: the method and the URL path.
• 200 is the status code your server sent.
• 18432 is the bytes sent.
• "-" is the referrer, the page the visitor came from. Crawlers usually leave it empty.
• The last part is the user agent. This is the claim.

To open a log, any text editor works for small files. For big ones use the command line: less access.log to scroll, grep to filter. Older logs are often zipped as .gz. Read them without unzipping with zcat access.log.1.gz | less or search them with zgrep -i googlebot access.log.*.gz.

If you need to strip or hash IP addresses for privacy, verify the bots first. Once the IPs are gone, you can't tell real Googlebot from a fake.

How to check Googlebot in your logs by hand

You don't need a log platform for a first look. Log file analysis just means reading these lines in bulk to see what crawlers did.

1. Pull a sample. A day or a week of access logs is enough to start.
2. Filter to Google claims. grep -i "googlebot" access.log > google.log. Widen the pattern if you care about AdsBot, Google-InspectionTool or other Google agents.
3. List the IPs. For the combined log format: awk '{print $1}' google.log | sort | uniq -c | sort -rn.
4. Verify the top IPs. Reverse then forward DNS with host, as above. For more than a handful, match them against the JSON lists.
5. Split real from fake. Keep two files: verified and spoofed.
6. Read what real Googlebot got. awk '{print $9}' verified.log | sort | uniq -c gives the status mix. Look for 403, 429, 5xx and challenge pages.
7. Note the paths. Which URLs did real Googlebot fail on? Those go in the ticket.

What good looks like: verified Googlebot gets mostly 200s and 304s, a few 301s and 404s, and fakes are challenged or blocked. What bad looks like: verified Googlebot getting 403, 429, 503 or a challenge, or fakes getting 200s.

Which tool is best for log analysis?

It depends on how much log you have. There's no single best tool, so here's how I'd pick:

Whatever you use, check it verifies Googlebot by IP, not by user agent. Some tools only read the label.

The manual check tells you whether the bigger tool is worth setting up.

How BOTCHECK makes it quicker

BOTCHECK does steps 2 to 7 for the log lines you paste. It's a free online check that runs in your browser, with no sign-in. Your log lines stay in the page and aren't uploaded. The only thing the page fetches is Google's published IP lists.

What you paste: Apache or Nginx combined format, or one JSON object per line (CDN fields like ClientIP, EdgeResponseStatus, ClientRequestUserAgent and SecurityAction work). If your CDN writes a challenge marker, add it as a pattern, because a plain log can't always tell a challenge from a 403.

What it checks against: Google's five published IP lists. It fetches them live, and falls back to a bundled copy if the live fetch fails, with the date of each list shown. A browser can't do reverse DNS, so BOTCHECK gives you the host commands to copy for anything you want to double-check.

How to read the result:

• Verified Googlebot requests blocked per day: the headline. Requests from verified common crawlers that got 403, 429, 503 or a challenge, by date (UTC).
• Lines read, Claim to be Google, Verified Google hits, Spoofed Googlebot hits.
• What verified Googlebot was served: the status mix.
• Problems to fix: blocked verified Googlebot, verified 5xx, spoofed IPs, spoofed requests served 2xx, and responses Over 2MB.
• By IP: each IP gets a class: Verified - common crawler, Verified - special crawler, User-triggered fetcher, Spoofed or Unknown. Download the CSV, or copy reverse DNS commands for the spoofed IPs.

Value, if you want it: a Revenue at risk box puts a worst, base and best monthly value on the paths where verified Googlebot was blocked, challenged or got a 5xx. Every input is a number you type in. The starting values are placeholders, not benchmarks. You can carry it into PAYBACK.

Example (made-up numbers)

A team pastes one day of CDN logs, 5,000 lines.

• 620 lines claim to be Google.
• 540 are Verified - common crawler. 80 come from 4 IPs in none of Google's lists: Spoofed.
• Of the 540 verified hits, 38 got a challenge and 6 got a 503.
• 12 of the spoofed hits were served 200, because a firewall rule lets in any "Googlebot" user agent.

That's two tickets: stop challenging verified Googlebot (check by IP, not user agent), and swap the user-agent rule for an IP-based one. Re-run the next day's logs to confirm.

What to do next

• Fix blocks on verified Googlebot before anything else. Check firewall, bot protection and rate-limit rules first, then server errors.
• Check robots.txt itself returns 200 or 404, never 429 or 5xx.
• For URLs blocked in robots.txt but still indexed, use VEXA.
• Before you ask Google to recrawl, check the URLs with GATE.
• Want a full review? See the technical SEO audit.