arXiv cs.CL· Yuchen Guo, Zhuoyuan Mao, Yiwen Guo·· 12 小时前AI 评分31
扩散语言模型中的事实访问状态：从机制到知识编辑与检索
Factual Access States in Diffusion Language Models: From Mechanism to Knowledge Editing and Retrieval
AI 导读
研究揭示掩码扩散语言模型（DLM）的事实访问状态在首个注意力层即已组装完成：关系措辞被路由到可见实体位置，早期前馈层在该融合表示上完成事实查找，因此状态随关系表面形式和查询方向变化。基于此提出的 Multi-View Edit 知识编辑器按方向拟合共享值并写入早期 MLP 块，在释义与反向提问上显著优于 AR 编辑器。各实体位置的状态还可作为免训练的关系特定检索向量，返回双实体查询的两条证据。
正文 · 原文
Abstract:Factual recall is a lookup: some representation inside the network serves as the key, and what that key is made of decides which queries can retrieve the fact, what an edit must cover, and what a retriever can read. We call it the factual access state. In autoregressive (AR) models the subject position enriches attributes largely independently of the relation, and relation-specific selection happens later, at the readout position. In masked diffusion language models (DLMs) we find the state is assembled differently, and much earlier: in the first attention layer the wording of the relation is routed into the position of whichever entity is visible and the early feed-forward layers look the fact up on this fused representation. The state therefore varies with the relation's surface form and with the query direction, so the queries about one fact occupy a region of state space rather than a point, and an operation on the fact succeeds only for queries that enter it. We treat the state as an interface to a DLM's factual knowledge, and put it to two uses. Writing to the state gives a knowledge editor: from one declarative statement, Multi-View Edit fits one shared value per direction across that region and installs it in the early-MLP block, preserving source-prompt reliability while substantially improving paraphrase and reverse question answering, where AR editors gain near zero. Reading it suggests a complementary use: the state at each entity position can serve as a relation-specific retrieval vector with no training, returning both pieces of evidence for two-entity queries on which pooled, prompted and lexical retrievers almost never return both.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.27580 [cs.CL] |
| (or arXiv:2610.27580v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27580 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuchen Guo [view email]
[v1]
Tue, 20 Oct 2026 03:45:45 UTC (4,914 KB)
来源：arXiv cs.CL · arxiv.org