跳到正文
arXiv cs.CL· Jeonghoon Lee (Attractor Dynamics Inc.)·· 10 小时前AI 评分39

冻结语言模型如何通过事件读取实现精确状态追踪

Learning to Read Events for Exact State Tracking with Frozen Language Models

AI 导读

研究者用冻结语言模型训练选择器逐事件读取状态转移,配合外部整数寄存器按序组合,在43,687个事件的299个窗口上,0.8B模型全部读对;新机器复测298/299,未达单侧95%置信下界0.99的目标。

正文 · 原文

View PDF HTML (experimental)

Abstract:Long event streams require identifying each change and applying the changes in order. We train a selector on a frozen language model to read each transition. With API-only access, we ask the model to list the transitions. An external integer register composes them in order. A confidence rule can flag uncertain updates or retain several candidates. We study products of permutations of five items, a task that fixed-depth, log-precision transformers cannot solve on all input lengths under the standard conjecture that $\mathrm{TC}^0 \ne \mathrm{NC}^1$. On two-event windows, a 27B model answered 3 of 100 correctly without visible reasoning and 96 of 100 when it generated intermediate steps. A frozen 0.8B model with a trained selector read every event in 299 of 299 windows of 43,687 events each. The same package passed 298 of 299 new windows on a fresh machine, missing the target of 0.99 for the one-sided 95% lower confidence bound. A separate follow-up sized to allow one failure yielded 472 of 473. On 1,000-event room-swap documents, three of ten models solved all five documents by reasoning alone. All seven models tested both ways at this length solved all five with extraction and the register, with no chain of thought requested. With concurrent extraction, median wall-clock time fell from 136.1 to 10.4 s for Sonnet 5.5 and from 313.5 to 28.0 s for GPT-6.1 Sol. At fixed per-token rates, mean cost per document fell by 55% for Sonnet 5.5 and 44% for GPT-6.1 Sol. A regular expression with an executor also solved all 15 fixed-format documents. On 500 free-form issue-tracker comments, a separately adapted 4B reader identified the recorded value in 68.8% of cases, compared with 61.0% for a frequency baseline fitted on the same training projects.
Comments: 11 pages, 5 figures. Data and code: this https URL
Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as: arXiv:2611.25211 [cs.LG]
  (or arXiv:2611.25211v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.25211

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jeonghoon Lee [view email]
[v1] Wed, 18 Nov 2026 21:38:05 UTC (230 KB)

来源:arXiv cs.CL · arxiv.org