arXiv cs.AI· Javier Porras-Valenzuela, Arush Arora, Alejandro Ribeiro·· 12 小时前AI 评分31
用可学习图位置编码扩展语言模型的图推理能力
Scaling Graph Reasoning in Language Models with Learnable Graph Positional Encodings
AI 导读
研究者提出用可学习的图位置编码增强语言推理模型（LRM），通过 GNN 加 Graph Transformer 计算位置编码并调制注意力 logits，省去序列化邻接表，将上下文负担降至 O(N)。位置编码器与 LRM 的 LoRA 适配器用图论求解器生成的最优解联合微调，在小图上训练后可在最多 1,000 个节点的图上提升尺寸泛化能力，且生成 token 少于传统 LRM，对节点重标记保持稳健。
正文 · 原文
Abstract:Solving reasoning tasks supported on graphs typically requires providing a Language Reasoning Model (LRM) with a serialized description of the graph. This approach results in the input length growing with the number of edges, which limits the size of the problems that can be solved. Furthermore, even when graphs fit in context, performance sharply decays as the graphs grow. We propose to address this by augmenting LRMs with scalable and permutation equivariant graph positional encodings that modulate attention logits. An additional post-fusion-style module incorporates structural information into hidden states. We compute these positional encodings using a GNN followed by a Graph Transformer. This eliminates the need for serialized adjacencies, reducing the context burden to $O(N)$. We jointly fine-tune the positional encoder and the LRM LoRA adapters using optimal solutions generated by graph-theoretic solvers, resulting in a computationally efficient method. Trained on small graphs, the model exhibits improved size-generalization across graph-theoretic tasks on graphs with up to 1,000 nodes, and produces fewer tokens than a conventional LRM. Additionally, we observe that our model remains robust to node relabelings. Evaluations on graph-theoretic benchmarks also show that performance on small graphs is preserved.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.27439 [cs.LG] |
| (or arXiv:2610.27439v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27439 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Javier Porras-Valenzuela [view email]
[v1]
Tue, 20 Oct 2026 00:39:49 UTC (400 KB)
来源：arXiv cs.AI · arxiv.org