arXiv cs.CL· Oscar Chew, Po-Yi Lu, Yi-Jie Cheng, Kunal Jain, Hsuan-Tien Lin, Zhengzhong Tu, Kuan-Hao Huang·· 3 天前AI 评分23
我们真的在测量遗忘吗？多模态 LLM 机器遗忘评估的陷阱
Are We Really Measuring Forgetting? Pitfalls in Evaluating Machine Unlearning for Multimodal LLMs
AI 导读
研究指出当前多模态大模型机器遗忘评估存在三大陷阱：缺少留出评估以区分概念级遗忘与实例级过拟合、依赖合成微调知识而非预训练获得的真实知识、对不同遗忘方法套用相同训练设置导致比较不公。
正文 · 原文
Abstract:As machine unlearning for multimodal large language models (MLLMs) rapidly advances, rigorous evaluation of its effectiveness becomes increasingly important. However, existing benchmarks largely inherit evaluation protocols from text-only unlearning and overlook the unique challenges of unlearning in multimodal settings. In this work, we identify three major pitfalls that may lead to misleading conclusions: the lack of heldout evaluation to distinguish genuine concept-wise forgetting from instance-wise overfitting, the reliance on synthetic, fine-tuned knowledge rather than real-world knowledge acquired during pre-training, and unfair method comparisons caused by applying the same training settings to different unlearning methods. To address these challenges, we introduce GEMMU (Generalizable Evaluation for Multimodal Machine Unlearning), a benchmark built on real-world knowledge and a principled evaluation protocol. Re-evaluating representative unlearning methods under GEMMU reveals substantial generalization gaps and challenges conclusions drawn from previous benchmarks. Our findings highlight the importance of rigorous evaluation for advancing multimodal machine unlearning and provide guidelines for future benchmark design.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19613 [cs.CL] |
| (or arXiv:2610.19613v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19613 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Oscar Chew [view email]
[v1]
Wed, 14 Oct 2026 05:36:03 UTC (10,828 KB)
来源：arXiv cs.CL · arxiv.org