arXiv cs.AI· Chenglin Yang·· 10 天前AI 评分36
评估整个堆栈而非单层：智能体动作的确定性规则与 LLM 门控是否独立失效？
Evaluate the Stack, Not the Layer: Do Deterministic and LLM Gates for Agent Actions Fail Independently?
AI 导读
研究用 1,119 条已标注智能体动作测试运行时门控堆栈的失效是否独立，将堆栈折算为等效层数 n_mult。任意两个 LLM 评委组合约等效 1.2 至 1.4 层（6 对全部显著），规则层加一个评委约 1.86 至 2.09 层（4 对均不显著）。单独准确率无法预测某层带来的增量，且有一个评委层级在 112 个批次中有 50 个被非请求的模型版本服务，该事件推翻了预先声明的分析规则。
正文 · 原文
Abstract:Runtime gates for agent tool calls are stacked on the assumption that their errors multiply. We test it on 1,119 labelled agent actions from three corpora, without an adaptive adversary. The stack has one deterministic rule layer and four LLM judges, three of them re-collected with the served model recorded on every call. We read each stack as a number of multiplication-equivalent layers, n_mult, with its floor under perfect coupling. Under the STRICT miss definition (escalation to a human scored as not stopped), any two judges compose to about 1.2 to 1.4 layers ({\phi} median +0.430, 6 of 6 pairs significant, floors 1.02 to 1.17). The rule layer plus one judge composes to 1.86 to 2.09 layers ({\phi} median +0.014, 0 of 4 significant, floors 1.01 to 1.09). Under PRIMARY (escalation scored as caught) the bands are 1.21 to 1.57 and 1.80 to 2.13. Intervals separate on the pooled data, point estimates split on each corpus, and a third-vendor judge lands in the judge band. Solo accuracy does not predict what a layer adds: a cloud rule pack lowers the rule layer's solo miss rate by 20% and adds no new joint coverage. The difficulty share of judge coupling is not identifiable: 31.8% to 61.8% depending on the probe and the miss definition. One judge tier was served by an unrequested model version in 50 of 112 batches, concentrated on the external corpus. That event overturned a pre-declared analysis rule, and the scoring of review verdicts reversed five conclusions. We report both.
| Comments: | 15 pages, 1 figure, 10 tables. Artifact (data, scripts, provenance): this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07359 [cs.AI] |
| (or arXiv:2610.07359v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07359 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chenglin Yang [view email]
[v1]
Mon, 5 Oct 2026 20:26:58 UTC (61 KB)
来源：arXiv cs.AI · arxiv.org