跳到正文
arXiv cs.CY· Benjamin Jensen, Ian Reynolds, Yasir Atalan, Martin Pollack, Austin Woo, Anna McConkie, Robert Sincero·· 9 小时前AI 评分58

NDM Bench:评估前沿 LLM 在核决策场景中的倾向

The Nuclear Decision-Making Benchmark: Evaluating Frontier LLM Tendencies in Escalation, Proliferation, Non-Proliferation, and Arms Control Scenarios

AI 导读

研究者提出 Nuclear Decision-Making Benchmark(NDM Bench),用 151 个由国际关系专家撰写的场景测试 LLM 在核政策问题上的回答倾向,覆盖核升级(76)、军控(25)、防扩散(25)与扩散(25)四个领域。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large language models (LLMs) are entering defense and national-security workflows. Seven of the nine nuclear-armed states have committed to keeping humans in control of decisions to use nuclear weapons, but LLMs may still shape the assessments and options that reach decision-makers. Scrutinizing that influence calls for evidence about how these models behave on nuclear-policy questions. Such evidence should cover more than crisis escalation and should separate a model's tendencies from artifacts of a single phrasing, ordering, or run; evaluations to date have not focused on these questions. We introduce the Nuclear Decision-Making Benchmark (NDM Bench), a dataset and reproducible framework for testing how the choice of model and the design of a question shape LLMs' answers on nuclear policy. Its 151 scenarios, written by international relations experts, span four domains: nuclear escalation (76), arms control (25), non-proliferation (25), and proliferation (25). Each scenario can be posed with different states in the decision-maker's role and with added framing sentences. Eight frontier LLMs answered every prompt five times, about 380{,}000 responses in all. Results highlight that models' choices differ by model and domain, and that in escalation, how often they choose a nuclear threat or nuclear use depends on the state being advised and, for some models, the framing. However, agreement across identical runs is often low, with all five runs agreeing on only 43\% of escalation prompts. We release the scenarios, the prompts with their answer orders, and the parsed responses, along with documentation of how they were built and recommended practices for evaluations where there is no correct answer to grade against.
Comments: 46 pages, 7 figures
Subjects: Computers and Society (cs.CY)
Cite as: arXiv:2608.05180 [cs.CY]
  (or arXiv:2608.05180v2 [cs.CY] for this version)
  https://doi.org/10.48550/arXiv.2608.05180

arXiv-issued DOI via DataCite

Submission history

From: Martin Pollack [view email]
[v1] Mon, 29 Jun 2026 01:45:20 UTC (1,275 KB)
[v2] Mon, 16 Nov 2026 22:15:18 UTC (272 KB)

来源:arXiv cs.CY · arxiv.org