arXiv cs.AI· Jinyuan Zhang·· 10 小时前AI 评分31
ReFrain：在保留已获得正确答案的前提下实现多模态弃答
Multimodal Abstention That Preserves Acquired Correct Answers
AI 导读
研究者提出 ReFrain（Regression-Free Abstention），通过首 token logits 线性探针、留出图像簇上的阈值校准和确定性否决机制，在弃答时保留其他回答。
正文 · 原文
Abstract:Adding abstention to a task-adapted multimodal model can discard correct answers, including capabilities acquired during adaptation. We formulate abstention repair as maximizing correct abstention on unanswerable questions under a budget on newly lost correct answers. We propose ReFrain (Regression-Free Abstention), which combines a linear probe on first-token logits, threshold calibration on held-out image clusters, and a deterministic veto that replaces rejected outputs with the displayed "none of the above" option while preserving all other responses. "Regression-free" denotes zero loss on the calibration set at the zero-budget target. All abstention methods in our main comparison start from the same answering-SFT checkpoint; we separately track lost correct answers, repaired errors, and retained SFT gains. On Qwen2.5-VL-3B, ReFrain at zero budget retains all 57 SFT-gained answers with no losses and improves correct abstention by 7.30 percentage points. At a 0.5% calibration budget, it reaches 74.05% correct abstention with 8 losses and retains 55 gains; CLIP-UP EmbLoRA reaches 64.65% with 62 losses, 41 repairs, and 40 retained gains. The same probe at its default threshold loses 298 correct answers. With models and thresholds frozen, confirmation on 228 independent question clusters yields 66.05% correct abstention and no losses at the 0.5% point, exceeding EmbLoRA by 32.72 percentage points. Ordinary refusal SFT achieves higher correct abstention at greater regression cost. Code and reproduction materials: this https URL.
| Comments: | 28 pages. Code and reproduction materials: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.14311 [cs.AI] |
| (or arXiv:2611.14311v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14311 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jinyuan Zhang [view email]
[v1]
Tue, 10 Nov 2026 16:41:49 UTC (36 KB)
来源：arXiv cs.AI · arxiv.org