跳到正文
arXiv cs.AI· Vedant Jhaveri, Arashpreet Singh Mor, Satyam Kumar, Alex Tsun, Qing Lan, Shaunak Mishra, Sriram Vasudevan·· 4 小时前AI 评分22

Trie-Head:用模型原生 Trie 解码与共享上下文注意力实现高效宽束生成式检索

Efficient Wide-Beam Generative Retrieval with Model-Native Trie Decoding and Shared-Context Attention

AI 导读

Trie-Head 将标识符 Trie 以经过验证的 GPU 张量形式存储,作为模型原生输出接口,替代 Python 自定义 logit 处理器(CLP),减少 CPU-GPU 协调与内存流量。

正文 · 原文

View PDF

Abstract:Generative retrieval (GR) predicts catalog entries as short semantic identifiers (SIDs). Trie-constrained beam search keeps these identifiers valid, but wide beams expose two serving costs: vocabulary-wide score tensors that are mostly discarded, and repeated attention over the same prompt. We present Trie-Head, a model-native output interface that stores the identifier trie as validated GPU tensors. Compared with a Python custom logit processor (CLP), it reduces CPU--GPU coordination and memory traffic by avoiding Python constraint callbacks and vocabulary-wide candidate buffers. It scores compact codebook candidates and streams the full-vocabulary normalizer, preserving the original model's valid-token scores in real arithmetic without retraining. Shared-context attention complements this output path by reusing prompt key/value blocks across beams while preserving their private decode tails. We implement both mechanisms in SGLang and evaluate candidate sourcing for LinkedIn Hiring Assistant and qualification retrieval for the Qualification Retrieval Platform. The combined system improves closed-loop throughput by up to 5.57x over the CLP baseline and up to 2.63x over BF16 FlashRec at a common beam width of 200 on the Qualification Retrieval Platform workload.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22309 [cs.AI]
  (or arXiv:2611.22309v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22309

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Shaunak Mishra [view email]
[v1] Tue, 17 Nov 2026 07:07:00 UTC (2,354 KB)

来源:arXiv cs.AI · arxiv.org