User-agent: * Allow: / # Sitemap (index -> conflicts, pages, briefings) Sitemap: https://armedconflicts.org/sitemap.xml # RSS feed: https://armedconflicts.org/feed.xml # LLM-readable summaries: # https://armedconflicts.org/llms.txt # https://armedconflicts.org/llms-full.txt # AI usage policy: https://armedconflicts.org/ai.txt # LLM info (start here): https://armedconflicts.org/llm-info # No Crawl-delay: it is non-standard (ignored by Googlebot/Bingbot), this is # a static site with no server load to protect, and the whole point of this # workstream is to remove friction for the AI retrieval crawlers driving # citation growth (OAI-SearchBot, ChatGPT-User, Google-Extended, # Applebot-Extended, CCBot, Amazonbot, Meta-ExternalAgent, Bytespider, etc.) # — none of which have their own group below and would otherwise inherit a # delay from User-agent: * for no benefit. # AI training crawlers — allowed but please respect the source attribution User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / # Search crawlers: everything except the raw data downloads. # # /data/ holds 97 .json/.csv/.geojson files, linked from all 62 record and map # pages as "Download data". They are payloads, not pages -- a .geojson cannot # carry a -- so Bing's SEO scanner flags every one it reaches as # "Missing Title Tag" (9 and climbing, checked 2026-09-20) and neither engine # has anything there worth ranking. Blocking them clears that report and stops # 97 files competing for crawl budget on a site where Google has indexed 61 of # 99 pages. Disallow is listed first so the rule holds under both the # longest-match and the older first-match reading of robots.txt. # # ONLY these two groups are narrowed, deliberately. The AI retrieval crawlers # must keep /data/: llms.txt hands them a data URL for every theater and that # is the entire point of the file. The named ones (GPTBot, ClaudeBot, # anthropic-ai, PerplexityBot) match their own groups below; the unnamed ones # (OAI-SearchBot, ChatGPT-User, CCBot, Amazonbot, Meta-ExternalAgent, # Google-Extended, Applebot-Extended, Bytespider) fall back to User-agent: * # above, which still says Allow: /. # # Not /assets/, ever: Googlebot needs the CSS and JS to render these pages, # and blocking them would break rendering rather than save anything. User-agent: Googlebot Disallow: /data/ Allow: / User-agent: Bingbot Disallow: /data/ Allow: /