ten crawlers, one URL, one control

A 403 to Googlebot does not prove a block.

aeo-lab requests your page as each named AI crawler and as an ordinary browser, parses robots.txt the way crawlers actually implement it, and reports the ambiguity instead of resolving it for you.

Read the raw run Read the source

✓ zero dependencies ✓ no hosted demo, by choice ✓ Node 22+

www.notion.com — 2026-09-05 captured run

user-agent sentcodepayload

browser (control) 200

GPTBot200

OAI-SearchBot200

ChatGPT-User200

ClaudeBot200

Claude-User200

PerplexityBot200

Perplexity-User200

CCBot200

Googlebot403

Google-Extendedn/a

Applebot-Extendedn/a

payload bar = bytes returned. 241,703 for every 200. Googlebot got 4,547 bytes in 16 ms — a challenge page, not the site.

⚠ reported as ambiguous, not as blocked

robots.txt allows Googlebot, so the rejection is server-side. Vendors publish crawler IP ranges so sites can reject impostors — and this request came from an ordinary IP. Nothing outside Notion's own server logs can tell a real block from correct anti-spoofing.

The last two rows were never requested: Google-Extended and Applebot-Extended are robots.txt opt-out tokens. No crawler sends those user agents.

9agents servedbyte-identical to the control

1rejection403, cause undecidable

2never probedrobots.txt tokens, not crawlers

0blocks claimednot from outside the logs

the finding it refuses to resolve

Two explanations fit the same 403. Neither can be dismissed from outside.

Most tools in this category print "blocked" and move on. This one prints both branches and says which evidence would separate them.

explanation A

Notion really does reject Googlebot

A deliberate server-side rule, applied to the crawler that identified itself. If true, Google is not fetching this page and the fix is a Notion configuration change.

would be confirmed by: a matching deny rule in Notion's own edge config

explanation B

Notion is correctly rejecting an impostor

Google publishes Googlebot's IP ranges precisely so sites can reject anything claiming the name from elsewhere. This scan sends the name from an ordinary IP. Rejecting it is the correct behaviour.

would be confirmed by: real Googlebot IPs succeeding in Notion's server logs

aeo-lab reports: ambiguous. A rejection is only raised to high severity when robots.txt independently agrees with it. Here it does not — so the finding stays ambiguous and says exactly which log line would settle it.

verified 2026-09-05 · slack.com

A scan can manufacture its own finding.

250 ms delay GPTBot 429

2000 ms delay GPTBot 200

Requesting one URL as ten agents in quick succession is a burst, and some sites rate-limit it. The 429 was not evidence of a block — it was the scan hitting its own target too fast. Default delay is 1000 ms; the README recommends 2000 ms and treats every 429 as a scan artifact until a slower re-run reproduces it.

five signals per run

What it actually measures.

User-agent rejection

A browser gets 200 and a named crawler does not, from the same IP, seconds apart.

robots.txt, as implemented

Grouped agents, most-specific group wins, longest path wins, Allow beats Disallow on a tie, an empty Disallow means allow-all.

Payload divergence

Same status code, materially smaller body — the shape of a challenge page or a bot-specific stub.

Render gap

How much readable text exists in the served HTML before any JavaScript runs, since most AI crawlers do not execute it.

Rate limiting

Reported as rate limiting, always — never promoted to a block.

one command

Run it against your own URL.

bash

$ node bin/scan.mjs https://example.com/ --delay-ms 2000
  control (browser)   200   241,707 B
  GPTBot              200   241,703 B
  ClaudeBot           200   241,703 B
  Googlebot           403     4,547 B   ambiguous
  Google-Extended       -           -   never probed

  1 finding · 0 high severity · exit 0

exit codes

0 no high-severity findings 1 at least one high-severity finding 2 usage error, or the control could not fetch — so nothing is comparable

stated up front

What it will not tell you.

N01

Whether you get cited

Crawl access is necessary, not sufficient. Measuring citation honestly is a separate method, and this tool does not pretend to do it.

N02

Anything from a real browser's TLS fingerprint

This is an HTTP client sending browser headers, not a browser. Serious bot management can challenge it exactly as it would any automated client.

N03

Anything about a crawl or a sitemap

One URL per run, no traversal. Crawl-graph and orphan analysis is a sibling tool's job.