arXiv cs.CL· Rodney Lafuente-Mercado, Robert Sincero, Phil Sage, Yuan Xue·· 3 天前AI 评分29
MA-Bench：衡量语言模型的任务规划能力
MA-Bench: Measuring Operational Planning in Language Models
AI 导读
研究者推出 MA-Bench，基于军事作战规划第一步"任务分析"构建，包含 10 个专家场景、10 至 12 份文档和 1,257 条专家编写的通过/失败要求。在二十个前沿与开放权重模型中，最佳模型仅答对 62% 的评分项，文档中明述事实正确率 58%，需推断的判断仅 33%，六成失败属于遗漏而非错误。
正文 · 原文
Abstract:How well can a language model turn a large, incomplete and partly contradictory body of documents into a plan when no single correct answer exists? We introduce Mission Analysis (MA)-Bench, built on mission analysis, the first step of military operational planning, in which a staff reads the full situation and states the problem before anyone proposes a solution. Its 10 expert-built scenarios pair synthetic packets of ten to twelve documents with 1,257 expert-written pass/fail requirements, each containing information on their source, question type, and importance level. A reasoning-model judge grades each brief against the requirements without seeing the documents; it grades like a fifth credentialed planner, leaving a four-planner panel's agreement unchanged (Fleiss $\kappa$ 0.42 to 0.41), and a judge from a second provider reproduces the model ranking (Spearman 0.94). Across twenty frontier and open-weights models the best gets 62% of the rubric items right. Models are strongest on facts stated in the documents (58%) and weakest on judgments that must be inferred (33%), and six failures in ten are omissions rather than errors. MA-Bench grades the determinations, not the prose. We release the packets, rubrics, prompts, scorer and every graded brief with item-level verdicts.
| Comments: | 32 pages. Dataset: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19138 [cs.LG] |
| (or arXiv:2610.19138v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19138 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rodney Lafuente-Mercado [view email]
[v1]
Tue, 13 Oct 2026 20:07:50 UTC (325 KB)
来源：arXiv cs.CL · arxiv.org