Crawling Implementation

We use fetch() with custom headers to request your pages, then parse HTML using Cheerio (a Node.js HTML parser). We extract:

• <title> tag — primary page identifier
• <h1> through <h3> — content structure
• <meta name="description"> — summary content
• Body text — full content stripped of HTML/scripts
• Schema.org JSON-LD — structured data

We limit to ~20 pages per analysis to balance depth with API costs. Internal links are followed to discover connected pages.

// Simplified crawl logic
const response = await fetch(url, { headers: { 'User-Agent': 'Lighthouse-GEO/1.0' }});
const html = await response.text();
const $ = cheerio.load(html);
const title = $('title').text();
const bodyText = $('body').text().substring(0, 30000);

How Embeddings Work

We use OpenAI's text-embedding-3-large model to convert text into 3,072-dimensional vectors. Each dimension captures a different aspect of meaning.

Example: The words "shaver" and "razor" will have vectors pointing in similar directions because they're semantically related. "Shaver" and "pizza" will point in very different directions.

// Embedding generation
const response = await openai.embeddings.create({
  model: 'text-embedding-3-large',
  input: pageContent.substring(0, 30000),
  dimensions: 3072
});
const embedding = response.data[0].embedding; // [0.023, -0.041, 0.089, ...]

Why this matters: This is the same technology ChatGPT uses to understand content. By analyzing your embeddings, we see your content the way AI sees it.

Cosine Similarity Explained

Cosine similarity measures the angle between two vectors. If two vectors point in exactly the same direction, similarity = 1.0 (100%). If perpendicular, similarity = 0. If opposite, similarity = -1.0.

// Cosine similarity calculation
function cosineSim(a, b) {
  let dot = 0, normA = 0, normB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}

Topic clusters are generated dynamically based on your industry using GPT-4o-mini. We then embed each topic cluster and compare it against each page's embedding.

Interpretation: A page with 70% similarity to "sensitive skin shaving" is semantically well-aligned with that topic. AI is more likely to cite it when users ask related questions.

Prompt Testing Methodology

We use the OpenAI Responses API with gpt-4o and force the web_search tool to ensure real-time, cited results.

Prompt types:

• Unbranded: "best electric shaver for sensitive skin" — tests if AI discovers you organically
• Branded: "Philips shaver reviews" — tests brand recognition

// AI visibility test
const response = await openai.responses.create({
  model: 'gpt-4o',
  tools: [{ type: 'web_search' }],
  tool_choice: { type: 'web_search' }, // Force live search
  input: "What's the best electric shaver for sensitive skin?"
});
// Parse response for brand mentions, competitors, citations

Citation extraction: We parse the response's annotations array to extract URLs the AI cited as sources.

Competitor & Citation Extraction

We use regex patterns and NLP heuristics to identify company names mentioned in AI responses. We filter out:

• Generic terms (e.g., "Premium Choice", "Best Value")
• Product attributes (e.g., "Sensitive Skin", "Long Battery")
• The brand being analyzed (to focus on competitors)

We maintain a whitelist of known brands (Braun, Panasonic, Remington, etc.) for accurate detection and aggregate mentions across all prompts.

Citations are extracted from the API's url_citation annotations, which tell us exactly which sources the AI used to formulate its answer.

GEO Score Components (0-100)

Each page gets four sub-scores (0-25 each):

• Answer Capsules (0-25): Detects 120-150 character answer summaries after headings. AI loves citing these concise answers.
• Section Structure (0-25): Optimal paragraph length is 120-180 words. Too short = lacks depth. Too long = loses focus.
• Original Data (0-25): Counts statistics (%), research citations, years, specific numbers. Data-rich content is more authoritative.
• Technical SEO (0-25): Schema markup (especially FAQPage/HowTo), lists, and tables. These make content machine-readable.

// GEO Score calculation
const geoScore = answerCapsuleScore + sectionStructureScore 
               + originalDataScore + technicalSEOScore;
// citabilityRating = geoScore >= 70 ? "High" : geoScore >= 45 ? "Medium" : "Low"

Citation Gap Analysis

We track every URL cited by AI and cross-reference with competitor mentions:

• Citation Gap: A source that appeared with competitor mentions but not your brand. These are PR opportunities.
• Authority Sources: High-trust domains (.edu, .gov, news sites) that carry more weight.
• Category Breakdown: News, reviews, educational, forums — each has different outreach strategies.

// Citation gap detection
const citationGaps = topSources.filter(source => 
  source.competitorsMentioned.length > 0 && 
  !source.includesYourBrand
);

Action: Target citation gap sources for guest posts, press releases, or product reviews. Getting featured there means AI will cite you too.

Why 120-150 Characters?

Research on ChatGPT citation patterns found:

• 72.4% of cited posts have concise answer summaries immediately after headings
• Optimal length: 120-150 characters — long enough for detail, short enough to cite verbatim
• Direct answers (not questions or lead-ins) perform best

We use GPT-4o-mini to analyze your page sections and generate capsules that match your content while following citation best practices.

Why Multi-LLM Matters

Each AI has different training data, biases, and search capabilities:

• 🟢 ChatGPT: OpenAI's GPT-4o with live web search — largest market share
• 🟣 Claude: Anthropic's AI — known for accuracy and safety focus
• 🔵 Gemini: Google's AI — integrated with Google Search data
• 🟤 DeepSeek: Chinese AI gaining global traction — different training corpus
• 🔶 Qwen: Alibaba's multilingual AI — strong in Chinese, English, and other languages
• 🟠 Perplexity: Search-first AI with real-time web access — heavily citation-focused
• 🔴 Google AI Overview: Appears in Google Search results — massive visibility impact

We aggregate visibility across all platforms to give you a true cross-platform AI visibility score.

// Parallel testing across all LLMs
const results = await Promise.allSettled([
  searchChatGPT(prompt),  // Real-time search
  searchClaude(prompt),    // Training data
  searchGemini(prompt),    // Google ecosystem
  searchDeepSeek(prompt),  // Asian market focus
  searchQwen(prompt),      // Alibaba multilingual
  searchPerplexity(prompt), // Search-first
  searchGoogleAIOverview(prompt) // via SERPapi
]);

Shopping Query Monitor

We generate shopping-intent prompts tailored to your industry:

• General: "best [category] for [use case]", "top rated [category] 2026"
• Comparison: "[brand] vs [competitor] [product]"
• Price-based: "[category] under $[price]"
• Head-to-head: "should I buy [product A] or [product B]?"

Key Metrics:

• AI Share of Shelf: % of shopping queries where your product appears vs. competitors
• Recommendation Position: Average rank in AI's product recommendations
• Sentiment Score: How positively AI describes your products
• Merchant Distribution: Which retailers AI recommends buying from

// Commerce query execution
const shoppingQueries = generateShoppingQueries(config);
for (const query of shoppingQueries) {
  const results = await searchAllLLMs(query);
  // Parse: product mentions, prices, merchants, sentiment
  // Track: brand position, competitor mentions
}

Competitive Battlecards: We run head-to-head comparison queries and analyze which product AI prefers, along with pros/cons mentioned for each.

How Bot Tracking Works

We provide a lightweight tracking pixel you add to your website. It works in two ways:

• Human Tracking: A 1x1 transparent pixel fires on page load, recording visits, referrers, and device info
• Bot Detection: Server-side user-agent analysis identifies AI crawlers by their signature patterns

// AI bots we detect
const AI_BOTS = [
  { pattern: /GPTBot/i, provider: 'openai', type: 'training' },
  { pattern: /ChatGPT-User/i, provider: 'openai', type: 'browsing' },
  { pattern: /ClaudeBot/i, provider: 'anthropic', type: 'training' },
  { pattern: /PerplexityBot/i, provider: 'perplexity', type: 'search' },
  { pattern: /Google-Extended/i, provider: 'google', type: 'training' },
  // ... and more
];

Key Metrics:

• Human Pageviews: Total human visits tracked via pixel
• Bot Visits: AI crawler visits by provider (OpenAI, Anthropic, etc.)
• Bot Traffic %: Ratio of AI bots to human visitors
• Top Crawled Pages: Which pages AI is most interested in
• Top Referrers: Where your human traffic comes from

Why it matters: If AI bots are crawling your site frequently, your content is being ingested into AI training data or real-time search indexes. More crawls = higher likelihood of being cited.