arXiv cs.MA· Joss Armstrong·· 9 天前AI 评分30
MARGIN：面向多智能体基础模型协调的运行时置信度校准
MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination
AI 导读
MARGIN 是一种运行时校准方法，通过跟踪各模型近期准确率与自报置信度的比值来修正置信度，无需重训练模型或预留校准集。在覆盖代码生成、问答和数学的评测中，使用 18 个模型池，MARGIN 在两次代码生成迁移上的预期校准误差低于全部五个在线校准基线，问答迁移上低于其中四个。在代码生成协调实验中，校准使答案选择准确率相比未校准的置信度加权提升 4.3 和 14.0 个百分点。
正文 · 原文
Abstract:When a coordinator compares answers from heterogeneous foundation models, self-reported confidence may have different meanings across responders and changing workloads. This paper presents MARGIN (Multi-Agent Runtime Grading via Incremental Normalisation), a runtime calibration method that learns model-specific confidence corrections from observed answer outcomes without retraining the models or requiring a held-out calibration set. MARGIN tracks recent accuracy and stated confidence within confidence bands, uses their ratio to correct reported confidence, and blends sparse-band corrections toward a model-level estimate. The corrected scores weight candidate answers in a collective decision. Evaluation covers code generation, question answering, and mathematics, using an 18-model pool and a nine-model subset for distribution-shift experiments. On BigCodeBench, model-mean confidence is negatively related to accuracy; among correct/incorrect response pairs, choosing the more confident responder performs below chance. Against five online calibration baselines receiving identical feedback and retaining their learned state across each transition, MARGIN achieves lower post-shift expected calibration error than all five in two code-generation transitions and than four in a question-answering transition; the remaining question-answering comparison is inconclusive. In separate code-generation coordination experiments, calibration improves the ranking of correct responses and increases answer-selection accuracy by 4.3 and 14.0 percentage points on two of three benchmarks relative to uncalibrated confidence weighting. These results support model-specific runtime calibration for coordination under changing workloads when correctness feedback is available for the participating responders.
| Subjects: | Machine Learning (cs.LG); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2605.22949 [cs.LG] |
| (or arXiv:2605.22949v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.22949 arXiv-issued DOI via DataCite |
Submission history
From: Joss Armstrong [view email]
[v1]
Thu, 21 May 2026 18:25:05 UTC (207 KB)
[v2]
Wed, 27 May 2026 16:57:17 UTC (208 KB)
[v3]
Fri, 31 Jul 2026 14:06:03 UTC (237 KB)
[v4]
Thu, 8 Oct 2026 11:10:45 UTC (147 KB)
来源：arXiv cs.MA · arxiv.org