arXiv cs.AI· Sizhe Chen, Sanjay Kariyappa, Liwei Jiang, Chong Xiang, Jihye Choi, Chaowei Xiao, G. Edward Suh·· 14 小时前AI 评分38
通过目标-目标冲突学习改进 LLM 指令层级
Improving Instruction Hierarchy by Learning from Goal-Goal Conflicts
AI 导读
研究者发现现有 OpenAI 和 NVIDIA 的指令层级（IH）训练集缺失一类"目标-目标冲突"，用这类样本微调 Qwen3.5-9B 和 Gemma4-12B-it 后，模型在训练中未见的广泛真实任务上 IH 表现显著提升。将这些样本加入现有 IH 训练集也能获得更好的指令遵循效果，且在八个 benchmark 的无冲突任务上平均效用变化极小。
正文 · 原文
Abstract:The recent incident in which OpenAI agents autonomously compromised Hugging Face's infrastructure illustrates the importance of instruction hierarchy (IH): an LLM should follow lower-priority instructions, such as user requests, only when they do not conflict with higher-priority instructions, such as system policies. IH can be effectively improved by fine-tuning LLMs on simulated examples of instruction conflicts, whose diversity is key to teaching LLMs to properly handle conflicts at test time. However, our decomposition of a task into goals and constraints reveals a conflict type -- goal-goal conflicts -- that is absent from existing IH training sets from OpenAI and NVIDIA. We show the importance of this conflict type: fine-tuning Qwen3.5-9B and Gemma4-12B-it on examples of this conflict type significantly improves IH on a wide range of real-world tasks unseen in training; adding these examples to existing IH training sets for fine-tuning also gives better IH adherence. The IH gains come with trivial mean utility changes on no-conflict tasks across eight benchmarks. Our work advocates including this new conflict type in IH training, even if these examples do not reflect typical LLM use.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.35093 [cs.AI] |
| (or arXiv:2610.35093v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35093 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sizhe Chen [view email]
[v1]
Mon, 26 Oct 2026 09:33:17 UTC (464 KB)
来源：arXiv cs.AI · arxiv.org