arXiv cs.CL· Zehao Lin, Qianfeng Shen, Xixuan Hao, Shaobo Cui, Chunyu Li, Kai Chen, Bo Tang, Zhenjun Wang, Jin Yan, Zhiyu Li, Feiyu Xiong·· 13 小时前AI 评分31
将自回归 LLM 控制编译为策略精确的并行推理，用于分层 RAG
Compiling Autoregressive LLM Control into Policy-Exact Parallel Inference for Hierarchical RAG
AI 导读
研究者提出把分层 RAG 中 LLM 的串行语法约束解码编译为并行推理：一次根节点 prefill 加一次树因果探针即可评估所有前缀条件下的动作分布，在精确算术下与串行策略完全一致，关键路径仅两次前向、输出 token 为零。
正文 · 原文
Authors:Zehao Lin, Qianfeng Shen, Xixuan Hao, Shaobo Cui, Chunyu Li, Kai Chen, Bo Tang, Zhenjun Wang, Jin Yan, Zhiyu Li, Feiyu Xiong
Abstract:Large language models increasingly act as controllers in structured decision pipelines, yet their decisions are still executed by autoregressive text decoding. Hierarchical RAG is one such setting: at each frontier of a document tree, an LLM commits to $b$ of $k$ sections under an input-dependent action grammar, executed as either failure-prone free-form JSON or reliable but strictly serial grammar-constrained generation. We show that this serialization is unnecessary for bounded-horizon control: the reachable future forms a shallow action-prefix tree, which we compile --- one root prefill plus one tree-causal probe evaluate every prefix-conditioned action distribution with serial decoding's tokens, positions, and causal ancestry. A prefix-state lemma makes compiled control policy-exact in exact arithmetic: the same policy with a two-forward critical path and zero output tokens. One further probe pass certifies the exact order-marginalized MAP set. Across four document-disjoint splits and two scales on the primary MetaX device, compiled control matches the serial policy on $84$--$89\%$ of questions --- every instrumented divergence a floating-point near-tie --- and cuts control latency by $35$--$47\%$ versus token-wise serial decoding and $6$--$7\%$ versus an idealized chunked engine; an H800 repetition preserves the ordering.
| Comments: | 5 pages, 2 figures, 1 table |
| Subjects: | Computation and Language (cs.CL); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2610.30664 [cs.CL] |
| (or arXiv:2610.30664v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30664 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zehao Lin [view email]
[v1]
Thu, 22 Oct 2026 03:25:21 UTC (183 KB)
来源：arXiv cs.CL · arxiv.org