Book a demo
Product
AI visibilityCompetitorsCitationsGEO score and page fixesPrompts and demandAI crawlers and referralsContent StudioAgents and automationsAurora strategistCommerceEarned mediaLocal networkAPI and MCPThe whole platform
Solutions
Brand and marketingSEO and GEO teamsContent teamsPR and communicationsE-commerceMulti-location and localAgenciesExecutives
Free tools
Free AI visibility scanPrompt set generatorAI readiness checkTemplates
Resources
Learn GEOUse casesHoliday AI Index 2026BlogNewsletterMethodologyDevelopersChangelogIntegrationsSecurity and trust
PricingAboutSign in
Check your AI visibilityBook a demo
AI crawlers and referrals

See which AI bots read your site, and which answers send visits.

Lighthouse logs GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and the other AI crawlers on every page, and shows your GA4 and Search Console data beside your AI visibility.

Check your AI visibilityBook a demo
app.ahoylighthouse.com/bottracking
AI crawlers and referrals in the Lighthouse app, showing the sample brand Ridgeline Outfitters

Sample data for Ridgeline Outfitters, a fictional brand.

How do I know if AI crawlers can read my site?

Check your server logs or a crawler tracker for the user agents of each AI engine, and check robots.txt for rules that block them. Lighthouse does both continuously: it records every visit from AI crawlers by page, flags pages they cannot reach, and reports which blocked bots are costing you answers.

10AI crawler families identified by nameOpenAIAnthropicPerplexityGoogleBingMeta AIAmazonApple
1line of robots.txt hiding the sample brand from ChatGPT searchUser-agent: OAI-SearchBot
Disallow: /

What can you do with AI crawlers and referrals?

Every AI crawler, by page

A one-line snippet or a Cloudflare integration logs each AI bot visit with the page, status code and time.

Blocked and broken pages

Pages that return errors to bots, sit behind scripts they cannot render, or are disallowed in robots.txt.

AI referral traffic

Visits from AI engines in GA4 and the pixel, shown beside your AI visibility trend.

Shadow pages

Machine-readable pages built for AI agents, served alongside your site, with their own crawl stats.

Measured, not modeled

Every number opens to the answers behind it.

Every answer collected from the engine’s own logged-out page in a real browser, stored with its sources and a screenshot, and never estimated from a panel.

Read the methodology
Next in the platform

Content Studio

Write the page AI wants to cite.

Questions people ask

Should I block GPTBot?

GPTBot collects training data; OAI-SearchBot fetches pages for ChatGPT search answers. Many brands block the first and allow the second. Blocking OAI-SearchBot removes you from ChatGPT search citations, which is the most common fix we find.

Do I need to install anything?

Either a Cloudflare worker or a small tracking pixel. Neither sets cookies or tracks people. Server-log upload is not available yet.

See what AI says about you.

A real Lighthouse scan of your site: your AI visibility, who is named instead, and what to fix. Free to start.

Check your AI visibilityBook a demo