arXiv cs.AI· Lisong He, Yunxuan Jiang, Haishan Ye, Xiangyu Chang·· 3 天前AI 评分26
ZOIF：仅用前向传播的零阶影响函数估计，实现 LLM 数据选择
ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes
AI 导读
ZOIF 是一个仅通过前向传播完成影响函数估计的零阶框架，用于大语言模型的数据选择，无需反向传播。它将影响估计分解为三步无 BP 流程：零阶估计平均代理梯度、以辅助零阶优化求解逆 Hessian 向量积、再沿参数更新方向用前向有限差分得出每样本影响分数。实验显示其选出的数据子集下游效用与基线相当甚至更优，同时峰值内存显著降低、总计算时间相近。
正文 · 原文
Abstract:Influence functions enable principled data selection for large language models (e.g., identifying influential training samples for model editing, debugging, or efficient fine-tuning). However, scaling IFs to LLMs is hindered by the intractability of billion-scale Hessian matrices and the prohibitive memory overhead of backpropagation (BP). While many classical IF approximations reduce Hessian complexity, they still rely on BP for gradient computation, requiring massive storage for intermediate activations that impedes practical data selection at scale. To address this, we propose ZOIF, a zeroth-order framework for influence estimation that operates entirely via forward passes, enabling memory-efficient data selection under resource constraints. ZOIF decomposes influence estimation into three BP-free stages: (1) approximating the average proxy gradient via zeroth-order estimation; (2) reformulating and solving the inverse Hessian-vector product as an auxiliary zeroth-order optimization to bypass explicit inversion; and (3) deriving per-sample influence scores via a forward finite difference along the previously computed parameter update direction. Experiments show that ZOIF selects data subsets with downstream utility competitive with, and in some cases exceeding, the evaluated baselines, yet at substantially lower peak memory cost and with comparable overall computation time.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15336 [cs.LG] |
| (or arXiv:2610.15336v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15336 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lisong He [view email]
[v1]
Sun, 11 Oct 2026 04:25:40 UTC (249 KB)
来源：arXiv cs.AI · arxiv.org