Crawl Census

Tools

Token cost is the o200k count of the tools list JSON, including schemas, before any tool is called.

• scan_site

  Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of

• site_report

  Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.

• census_stats

  Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.

• crawl_preflight

  Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each do

• agent_profile

  What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.

• census_facts

  Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line,

• submit_domains

  Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth subm