arXiv cs.CY· Mehmet Murat Huyut·· 11 小时前AI 评分58
研究发现智能体脚手架会改变固定LLM对银行行为规则的合规表现，且各模型差异不均
The Harness Is the Model: Agent Scaffolding Changes a Fixed LLM's Compliance with Banking Conduct Rules, Unevenly Across Models
AI 导读
一项预注册研究让四个开源权重模型分别在五种智能体脚手架下运行156个模拟零售银行案例，共3120段两轮对话，按执行工具轨迹确定性评分。汇总看脚手架只把正确率从87.3%拉到92.1%，但单个模型内最好与最差脚手架相差11.5个百分点，足以让它在四个模型中从第一跌到末位，而另一个模型几乎不受影响。
正文 · 原文
Abstract:Banks validate language models, but they deploy systems: a model inside a harness that decides what it sees, which tools it can call and who checks its actions. We test whether that harness changes compliance with banking conduct rules when the model is held fixed. Four open-weight models each ran five harnesses (a pipeline that makes one structured decision, a ReAct-style tool loop, a loop in which the policy is reachable only through search, a loop with a long benign customer memory, and a loop with a same-model compliance reviewer) on 156 pre-registered cases in a simulated retail bank whose 11 rule families paraphrase EU consumer-credit, payments, investor-protection, data-protection and anti-money-laundering provisions. Every one of the 3120 two-turn episodes was scored deterministically from the executed tool trace, with violations and over-refusals counted separately. Pooled across models the harness moved the correct-case rate only between 87.3% and 92.1%, but within a model it mattered a great deal: for one model the spread between its best and worst harness was 11.5 percentage points, enough to move it from first to last place among the four models, while for another the harness made no detectable difference; the model x harness interaction itself was suggestive rather than conclusive. The same-model reviewer cut violations from 9.6% to 6.4% without raising over-refusals, contrary to our pre-registered prediction. Rules that require date arithmetic failed in every harness. In an exploratory analysis with two LLM judges and a blind human check, 26% of violations were accompanied by a customer message that denied the action the agent had just executed, which a transcript-only review would record as a correct refusal. For model-risk management the unit of validation should be the model-harness pair, and conduct testing should read the action log, not the conversation.
| Comments: | 16 pages, 4 tables. Pre-registered. Code, data and all 3120 episode traces: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2611.06116 [cs.CL] |
| (or arXiv:2611.06116v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06116 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mehmet Murat Huyut PhD [view email]
[v1]
Wed, 4 Nov 2026 07:59:59 UTC (21 KB)
来源：arXiv cs.CY · arxiv.org