arXiv cs.AI· Tommy He, Thomas Jiralerspong·· 12 小时前AI 评分34
Context–Answer Metamodels：在首个答案 token 生成前预测 LLM 激活
Context--Answer Metamodels: Forecasting LLM Activations Before the First Answer Token
AI 导读
研究者提出 Context–Answer Metamodels，将上下文激活序列映射为答案激活的点预测或分布预测，无需大量自回归 rollout 即可刻画随机解码下的答案变化。
正文 · 原文
Abstract:Under stochastic decoding, a single context can elicit many different answers from a language model. Characterizing this variation typically requires many expensive autoregressive rollouts, each revealing only one realization. We ask whether the activations produced while reading the context can directly forecast the model's rollout activations. We train context--answer metamodels that map a context activation sequence to point or distribution forecasts of mean-pooled answer activations. We find that (i) the four metamodel families we study forecast mean-pooled answer activations with high fidelity, and the distribution metamodels also capture rollout variability; (ii) performance improves as the training set grows to 500,000 prompts, and the metamodels generalize zero-shot to out-of-domain datasets; (iii) their forecasts retain behavior-predictive signal; and (iv) context activations from Qwen2.5 predict mean-pooled answer activations from Qwen3.5 and Gemma 4, suggesting that activation forecasts can transfer across models. Together, these results suggest that context activation traces can support scalable forecasts of behaviors in LLM answers before the first answer token is generated.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.06837 [cs.LG] |
| (or arXiv:2611.06837v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06837 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tommy He [view email]
[v1]
Wed, 4 Nov 2026 17:53:52 UTC (1,465 KB)
来源：arXiv cs.AI · arxiv.org