arXiv cs.CL· Mateusz Modrzejewski, Tomasz Radzikowski·· 13 小时前AI 评分34
工具增强模型比较中的提示词敏感性研究
Prompt Sensitivity in Tool-Augmented Model Comparisons
AI 导读
一项 arXiv 研究提出匹配协议，将工具调用拆分为答案读取、推理、调用决策、工具返回结果与无调用报告等环节，在五个领域、五种措辞下测试六个开源模型。结果显示，工具在程序能直接回答问题时提升准确率，却扩大了多数两两比较的措辞敏感性，且扩大程度与模型调用频率相关。作者建议在准确率之外同时报告调用频率和多措辞下的比较稳定性。
正文 · 原文
Abstract:Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher. We ask whether giving the models tools makes such comparisons more stable. An accuracy gain cannot answer this, because tools change several steps between the question and the scored answer at once. We propose a matched protocol that separates those steps: the answer readout, reasoning, the decision to call, the tool's answer, and the report supplied without a call. We apply it to six open models in five domains under five wordings. We find that tools raise accuracy where a program answers the question and widen the wording sensitivity of many pairwise comparisons. The widening follows how often a model calls, and call frequency follows the wording for some models and not others. It also depends on the interface and the instructions: a native tool-calling interface and a second tool description leave no detected widening in molecules, and sensitivity persists where tools return intermediate results. These findings show that deterministic reports do not guarantee stable comparisons. We recommend reporting call frequency and comparison stability under several wordings beside accuracy.
| Comments: | 9 pages plus 8 pages of supplementary material; 2 figures and 4 tables in the main text, 2 figures and 5 tables in the supplement |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.31148 [cs.CL] |
| (or arXiv:2610.31148v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31148 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tomasz Radzikowski [view email]
[v1]
Thu, 22 Oct 2026 12:07:01 UTC (115 KB)
来源：arXiv cs.CL · arxiv.org