arXiv cs.AI· Hanfei Liu, Shuchang Feng, Yanxia Chen, Changzeng Fu, Shiqi Zhao·· 9 天前AI 评分22
Spora 如何重新权衡脉冲语言模型的时间编码与非线性计算
Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Models
AI 导读
脉冲语言模型研究提出 Spora，通过联合设计脉冲编码与注意力算子来平衡短时窗连续语义表示与非线性注意力开销。其二进制时间权重让 T 个脉冲承载最多 T bits 的组合值，优于脉冲计数读出的 O(log₂T) bits；UBS 用阈值与脉冲触发残差衰减生成非负整数编码，BBS 分离符号与幅值并学习有符号激活的缩放。
正文 · 原文
Abstract:Spiking language models face a tradeoff between representing continuous semantic features over short temporal windows and retaining costly nonlinear attention operations. We introduce Spora, which jointly designs spike encodings and attention operators. Binary temporal weights let $T$ spikes represent compositional values with up to $T$ bits of capacity, compared with $O(\log_2 T)$ bits for spike-count readout. Unipolar Binary Spiking (UBS) uses thresholds and spike-triggered residual decay to produce non-negative integer codes; Bipolar Binary Spiking (BBS) separates sign and magnitude and learns a scale for signed activations. These representations support accumulation-and-shift dot products and integer-exponent mappings in attention. With four time steps, Spora achieves 76.6 average GLUE score and 44.1 CoLA MCC, improving over SpikeLM by 1.2 and 6.2 points, respectively. Extending BBS to six steps raises these scores to 78.2 and 47.4. Conditional-decay analysis, matched-budget activation-quantization comparisons, event-workload statistics, and fixed-point evaluation further characterize the connection between encoding fidelity and computational cost.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.10933 [cs.LG] |
| (or arXiv:2610.10933v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10933 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hanfei Liu [view email]
[v1]
Wed, 7 Oct 2026 21:41:46 UTC (4,883 KB)
来源：arXiv cs.AI · arxiv.org