Bot configuration verification

Welcomed user-agents

Every agent below is listed independently in robots.txt with Allow: /. Vendor docs linked for verification.

• Anthropic . Claude vendor docs

  • ClaudeBot . training corpus
  • Claude-SearchBot . Claude search index
  • Claude-User . real-time fetch from Claude users

• OpenAI . ChatGPT vendor docs

  • GPTBot . training corpus
  • OAI-SearchBot . ChatGPT search index
  • ChatGPT-User . real-time fetch from ChatGPT, Custom GPTs, and GPT Actions
  • ChatGPT-User/2.0 . ChatGPT-User newer build
  • ChatGPT-User/3.0 . ChatGPT-User newest build

• Perplexity . Perplexity vendor docs

  • PerplexityBot . search index
  • Perplexity-User . real-time fetch

• Google . Gemini, AI Mode, AI Overviews vendor docs

  • Googlebot . search and AI Overviews and AI Mode use the same crawl
  • Google-Extended . Gemini training opt-in directive
  • GoogleOther . background variants

• Apple . Apple Intelligence vendor docs

  • Applebot . search and assistant
  • Applebot-Extended . training opt-in directive

• Microsoft . Bing, Copilot vendor docs

  • bingbot . search and Copilot

• Meta . Meta AI vendor docs

  • meta-externalagent . Meta AI
  • FacebookBot . Facebook crawler

• Mistral . Mistral vendor docs

  • MistralAI-User . Mistral assistant fetch

• DuckDuckGo . DuckAssist vendor docs

  • DuckAssistBot . DuckDuckGo assistant

• ByteDance . Doubao vendor docs

  • Bytespider . ByteDance / Doubao

• Amazon . Alexa+, Rufus vendor docs

  • Amazonbot . Alexa+ and Rufus

• Common Crawl . Common Crawl vendor docs

  • CCBot . used by hundreds of downstream RAG systems

• LinkedIn . LinkedIn vendor docs

  • LinkedInBot . preview rendering when authors share

Live robots.txt

Rendered from the same module that serves /robots.txt.

# Markwright is a publishing platform built for AI citation.
# Bots are welcomed. Sitemaps and llms.txt are below.

# === Anthropic (Claude) ===
User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# === OpenAI (ChatGPT) ===
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ChatGPT-User/2.0
Allow: /

User-agent: ChatGPT-User/3.0
Allow: /

# === Perplexity ===
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# === Google (Gemini, AI Mode, AI Overviews) ===
User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: GoogleOther
Allow: /

# === Apple ===
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

# === Microsoft (Bing, Copilot) ===
User-agent: bingbot
Allow: /

# === Meta ===
User-agent: meta-externalagent
Allow: /

User-agent: FacebookBot
Allow: /

# === Mistral ===
User-agent: MistralAI-User
Allow: /

# === DuckDuckGo ===
User-agent: DuckAssistBot
Allow: /

# === Common Crawl ===
User-agent: CCBot
Allow: /

# === Amazon ===
User-agent: Amazonbot
Allow: /

# === ByteDance ===
User-agent: Bytespider
Allow: /

# === LinkedIn (for previews when authors share) ===
User-agent: LinkedInBot
Allow: /

# === Default for everything else ===
User-agent: *
Allow: /
Disallow: /api/internal/
Disallow: /dashboard/
Disallow: /login
Disallow: /signup

Sitemap: https://markwright.app/sitemap.xml