arXiv cs.AI· Tony Mason·· 9 天前AI 评分32
Arbiter：检测 LLM 智能体系统提示词中的干扰模式
Arbiter: Detecting Interference in LLM Agent System Prompts
AI 导读
Arbiter 结合形式化评估规则与多模型 LLM 扫描，检测 LLM 编程智能体系统提示词中的干扰模式。对 Claude Code、Codex CLI 和 Gemini CLI 三个系统提示词的分析共发现 152 项扫描结果，并在针对单一厂商的定向分析中人工标注出 21 种干扰模式。
正文 · 原文
Abstract:System prompts for LLM-based coding agents are software artifacts that govern agent behavior, yet lack the testing infrastructure applied to conventional software. We present Arbiter, a framework combining formal evaluation rules with multi-model LLM scouring to detect interference patterns in system prompts. Applied to three major coding agent system prompts: Claude Code (Anthropic), Codex CLI (OpenAI), and Gemini CLI (Google), we identify 152 findings across the undirected scouring phase and 21 hand-labeled interference patterns in directed analysis of one vendor. We show that prompt architecture (monolithic, flat, modular) strongly correlates with observed failure class but not with severity, and that multi-model evaluation discovers categorically different vulnerability classes than single-model analysis. One scourer finding was structural data loss in Gemini CLI's memory system was consistent with an issue filed and patched by Google, which addressed the symptom without addressing the schema-level root cause identified by the scourer. Total cost of cross-vendor analysis: \$0.27 USD.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Programming Languages (cs.PL) |
| Cite as: | arXiv:2603.08993 [cs.SE] |
| (or arXiv:2603.08993v2 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2603.08993 arXiv-issued DOI via DataCite |
Submission history
From: Tony Mason [view email]
[v1]
Mon, 9 Mar 2026 22:29:47 UTC (52 KB)
[v2]
Wed, 7 Oct 2026 23:03:53 UTC (51 KB)
来源：arXiv cs.AI · arxiv.org