跳到正文
arXiv cs.AI· Peng Yu, Xiaoyu Wen, Zhida He, Ziyuan Zhou, Han Qi, Shao Zhang, Qiaosheng Zhang, Ying Wen, Chaochao Lu·· 4 小时前AI 评分32

DACE:面向鲁棒 LLM 安全对齐的多样性驱动对抗协同进化框架

DACE: Diversity-Driven Adversarial Co-Evolution for Robust LLM Safety Alignment

AI 导读

DACE 是一个多样性驱动的对抗协同进化框架,同时针对攻击策略坍缩与防御方对抗遗忘两个训练病理。攻击侧引入“风险类别×攻击风格”二维策略空间与归一化边际覆盖增益奖励,防御侧维护 Beta–Bernoulli 威胁后验的贝叶斯对抗回放池,按时间衰减刷新并用 Thompson sampling 采样。

正文 · 原文

View PDF HTML (experimental)

Abstract:Ensuring robust safety alignment of large language models (LLMs) is increasingly difficult as adversarial attacks evolve and outpace alignment pipelines built on pre-collected data. Existing co-evolutionary frameworks let the defender chase a moving attacker, yet their training dynamics expose two compounding pathologies: \emph{attack strategy collapse}, where the attacker overfits to a narrow set of high-reward rewrites, and \emph{defense adversarial forgetting}, where the defender loses competence against earlier attacks as the attack distribution drifts. Existing remedies are partial: attacker-side diversity rewards score textual novelty rather than strategy novelty, and defender-side replay relies on discrete judge scores that ignore estimation uncertainty, response stochasticity, and defender non-stationarity. We introduce \textbf{\method}, a diversity-driven adversarial co-evolution framework that targets both pathologies jointly. On the attacker side, \method couples an explicit two-dimensional strategy space (risk category $\times$ attack style) with a \emph{normalized marginal coverage gain} reward, providing a bounded, non-vanishing exploration signal at the strategy layer. On the defender side, \method maintains a unified \emph{Bayesian adversarial replay pool} whose Beta--Bernoulli threat posteriors are refreshed by time decay and sampled via Thompson sampling, anchoring defender training to an evolving threat landscape. Across standard safety, automated-attacker, and general-capability benchmarks, \method improves robustness to out-of-distribution attacks while preserving general capabilities, and yields broader attacker strategy coverage.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22278 [cs.AI]
  (or arXiv:2611.22278v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22278

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Peng Yu [view email]
[v1] Tue, 17 Nov 2026 06:27:03 UTC (466 KB)

来源:arXiv cs.AI · arxiv.org