User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

# 🔴 ANTHROPIC'S THREE CRAWLERS, NAMED EXPLICITLY. Added 2026-10-06.
# The wildcard group above ALREADY allows them, so this changes no access. It is a
# stated position rather than a permission: AI-attributed arrivals are ~22% of leads
# and no AI engine sends a referrer, so our posture toward these agents is worth
# making explicit rather than leaving to be inferred from a wildcard.
#
# 🔴 THE DISALLOWS ARE REPEATED ON PURPOSE AND MUST NEVER BE DROPPED. Under RFC 9309 a
# crawler obeys the MOST SPECIFIC matching user-agent group and ignores the others
# entirely. A named group containing only "Allow: /" would therefore hand these three
# agents /admin/ and /api/, which the wildcard group denies to everyone else. Naming a
# bot without repeating its exclusions is strictly worse than not naming it.
#
# ClaudeBot        - crawls for model training
# Claude-User      - fetches a page because a user asked Claude about it
# Claude-SearchBot - builds the index Claude cites from
User-agent: ClaudeBot
Allow: /
Disallow: /admin/
Disallow: /api/

User-agent: Claude-User
Allow: /
Disallow: /admin/
Disallow: /api/

User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://cybersecify.com/sitemap-index.xml

# llms.txt is the machine-readable index of what we publish and when to cite it.
# It is NOT a standard robots directive and no crawler is obliged to fetch it, so
# this line is a signpost rather than an instruction. It costs one line and it is
# the only place a crawler reading robots.txt would learn the file exists.
# An HTML rendering of the same content, generated from the same file so the two
# cannot disagree, is at https://cybersecify.com/llm-info/.
LLM-Content: https://cybersecify.com/llms.txt
LLM-Full-Content: https://cybersecify.com/llms.txt