arXiv cs.AI· Saisab Sadhu, Aadit Sengupta, Vinay kumar Sankarapu, Pratinav Seth·· 9 天前AI 评分29
LLM 合规系统对监管规则不敏感：五模型 20 领域审计研究
Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems
AI 导读
研究在五个模型、20 个监管与平台政策领域上测试：删除、替换或否定所给规则而保持案例不变，看结论是否改变（OCS）或模型内部合规表征是否偏移（ICS-delta），结果两者都几乎不动。其中 guard 模型规则敏感度与准确率最低，仅 51%，通用模型为 90-92%。在删除规则会改变原本正确预测的案例上，模型能紧密跟随规则；更好的提示词或直接干预内部表征都无法弥合这一差距。
正文 · 原文
Abstract:Large language model compliance systems are deployed on the assumption that a verdict depends on the regulatory rule it is given. We test this directly across five models and 20 regulatory and platform-policy domains: delete, swap, or negate the governing rule while holding the case fixed, and check whether the verdict changes (OCS) or the model's internal representation of compliance shifts at all (ICS-delta). Neither moves much: models' verdicts are often invariant to substantial perturbations of the supplied rule, and the guard model, evaluated here under a custom-rule adaptation of its native taxonomy, is the least rule-sensitive and least accurate of the five, barely above chance (51%, versus 90-92% for general-purpose models). This reflects easy cases more than blanket neglect: on cases where deleting the rule changes a previously correct model prediction, models do track it closely. Neither better prompting nor direct intervention on the model's internal representations closes this gap. Accuracy alone does not establish that a compliance verdict is grounded in the supplied rule.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2610.12313 [cs.AI] |
| (or arXiv:2610.12313v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12313 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Saisab Sadhu [view email]
[v1]
Thu, 8 Oct 2026 16:59:40 UTC (1,807 KB)
来源：arXiv cs.AI · arxiv.org