跳到正文
arXiv cs.AI· Ananya Sutradhar, Fabien Roger·· 10 小时前AI 评分34

能否让模型相信提示词中的虚假事实?研究探索 prompt-conditioned SDF

Can you make models believe false facts in the prompt?

AI 导读

研究尝试用 prompt-conditioned SDF 训练单一模型,使其相信提示词标签中给出的任意事实,从而替代需为每组虚假信念单独微调模型的 SDF。模型对未见事实的相信深度随训练事实数量大致呈对数线性增长,但将该方法用于小型模型的对齐审计时初步结果为负面。

正文 · 原文

View PDF HTML (experimental)

Abstract:Making LLMs believe false facts is useful for alignment evaluations and for building model organisms. Previous work explored doing so with Synthetic Document Finetuning (SDF). But SDF requires fine-tuning one new model for each set of injected false beliefs. To enable larger scale SDF-powered alignment evaluations, we attempt to produce a single model that believes any fact placed in a <facts> tag in the prompt. To do so, we explore doing prompt-conditioned SDF: fine-tuning a model on synthetic documents covering thousands of facts, with each document conditioned on a fact presented in the prompt. We evaluate how deeply the model believes the facts provided in the prompt using evaluations from prior work and a new Jacobian lens evaluation. We find that the belief depth on unseen facts scales roughly log-linearly with the number of training facts, making this a cheap, promising avenue for making models believe false facts, though preliminary experiments on applying it to alignment audits on the small model we studied showed negative results due to these small models often already acting as if they believed facts provided in regular honeypot system prompts. Using RL to reduce the rate at which models verbalize following a false fact in the prompt, on top of prompt-conditioned SDF, generalizes to held-out evals and almost matches SDF on some evaluations. But training against one of our metrics for belief depth makes it harder to trust those evaluations.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23938 [cs.AI]
  (or arXiv:2611.23938v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.23938

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Ananya Sutradhar [view email]
[v1] Wed, 18 Nov 2026 06:26:12 UTC (505 KB)

来源:arXiv cs.AI · arxiv.org