arXiv cs.AI· Shihua Yu, Yanzhi Wang, Tianchi Chen·· 3 天前AI 评分20
递归专家计算的粗粒化：延续相对接口与失效模式
Coarse-Graining Recursive Expert Computation: Continuation-Relative Interfaces and Failure Modes
AI 导读
研究提出延续相对评估框架，用下游任务质量与总执行成本联合判断能否用廉价模块替换一段计算，并区分边界表示、被跳过计算的摘要、实际执行操作与下游损失。在 17 亿参数模型上，用 next-token-loss 训练比重建被跳过模块输出使分布内负对数似然降低 0.1901 nats/token，但该排序在全部九组分布偏移配对比较中反转。
正文 · 原文
Abstract:Computational coarse-graining aims to replace a sequence of updates with a cheaper module while preserving task performance. We present a continuation-relative evaluation framework that assesses replacements through the computation that follows them, using downstream task quality and total execution cost as joint criteria. The framework distinguishes boundary representations, summaries of skipped computation, executed operations, and downstream loss. It makes interfaces and permitted deployment information explicit, combines fine-path references with directly trained controls using the same shortened architecture, and accounts for selection overhead. An exact loss decomposition separates replacement effects at a fixed downstream action from effects of changing that action. A finite construction shows how retained information can reduce the stipulated cost of zero-error prediction under fixed information and controller restrictions. Experiments in two frozen pretrained language models show that representation fidelity and task loss can favor different fitting objectives. In a prespecified comparison on a 1.7-billion-parameter model, next-token-loss training lowers in-distribution negative log-likelihood by 0.1901 nats per scored token compared with reconstructing the skipped blocks' output, using the same optimizer and update count. This ranking reverses in all nine paired comparisons under distribution shift. In a recursive expert prototype, the jointly adapted candidate lowers task error relative to frozen-bank and router-only baselines, but trails directly trained controls, and the tested input-only selectors fail the joint quality-cost criteria. Together, the protocol and studies provide explicit criteria and diagnostics for distinguishing useful computational replacement from local approximation gains within specified interfaces and evaluation settings.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computational Complexity (cs.CC); Computational Engineering, Finance, and Science (cs.CE) |
| Cite as: | arXiv:2610.14413 [cs.LG] |
| (or arXiv:2610.14413v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14413 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianchi Chen [view email]
[v1]
Sat, 10 Oct 2026 00:21:04 UTC (313 KB)
来源：arXiv cs.AI · arxiv.org