arXiv cs.AI· Maria Elkj{\ae}r Montgomery, Christian Igel, Mikkel Odgaard, Martin Sillesen, Mads Nielsen·· 12 小时前AI 评分27
Transformer 应如何在电子健康记录中编码数值？
How Should Transformers Encode Numeric Values in Electronic Health Records?
AI 导读
研究系统比较了离散、连续与混合三类数值编码策略在真实 EHR 数据上的表现：显式建模数值-概念交互的方法在精度敏感的算术任务上最优，但受架构约束限制。保留数值并在投影前分箱的混合 token 方案更稳健、适用面更广，最优分箱数随数据集规模呈经验幂律关系。模型普遍表现为可靠的“够用”数值计算而非精确算术，临床收益因任务而异。
正文 · 原文
Abstract:How do we encode numeric values in transformer-based sequence processing, particularly in electronic health record (EHR) data? We systematically compare discrete, continuous, and hybrid value encoding strategies using synthetic arithmetic tasks embedded within real-world EHR data, as well as real-world clinical prediction tasks. Our study reveals trade-offs between numeric precision, optimisation stability, and architectural flexibility. We find that approaches that explicitly model value-concept interactions perform best on precision-sensitive arithmetic tasks when architectural constraints permit. Hybrid token-based approaches that retain numeric values but apply binning prior to projection provide a more robust and broadly applicable alternative, with the optimal number of bins following a simple empirically derived power-law in dataset size. Across tasks, models consistently exhibit reliable "good enough" numeric computation rather than exact arithmetic, while clinical gains from incorporating laboratory values are task-dependent. This suggests that robustness and deployability often outweigh maximal numeric precision in practice, motivating hybrid token-based approaches as a practical default.
| Comments: | 16 pages, 15 figures, 3 tables, accepted to ICML 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.01391 [cs.LG] |
| (or arXiv:2607.01391v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01391 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 43rd International Conference on Machine Learning (2026) |
Submission history
From: Maria Elkjær Montgomery [view email]
[v1]
Wed, 1 Jul 2026 18:49:41 UTC (6,177 KB)
[v2]
Fri, 23 Oct 2026 13:14:27 UTC (6,177 KB)
来源：arXiv cs.AI · arxiv.org