arXiv cs.CL· Avichal Sahai (Ofbusiness), Nishant Raj (Ofbusiness), Animesh Srivastava (Ofbusiness)·· 3 天前AI 评分44
三值语言模型导出中的 bf16 转换问题：低学习率代码改动为何被回退
Lost in the bf16 Cast: Exporting Ternary Language Models Can Revert Most Low-Learning-Rate Code Changes
AI 导读
对 BitNet b1.58、Falcon-E 和 BitCPM 三个实验室的三值语言模型导出流程审计发现，文档化的 bf16 转换会大幅回退微调效果：Falcon-E-1B-Base 的 GSM8K 严格准确率从 58.79% 降至 0.78%。
正文 · 原文
Abstract:Ternary language models such as BitNet b1.58, Falcon-E and BitCPM are fine-tuned with higher-precision latent weights and deployed as ternary codes produced by an export step that, in the labs' documented pipelines, first casts the latents to bf16. We audit those pipelines across three labs. In released checkpoints, fp32 quantization of the shipped latents disagrees with the deployed codes on 0.83-1.77% of codes in Falcon-E and BitCPM and on 1.530% in BitNet 2B-4T; for Falcon-E and BitCPM most disagreements are products that bf16 rounding lands exactly on the threshold, which ties-to-even maps to zero, and the unmodified onebitllms exporter reproduces all four Falcon-E releases byte for byte. At fine-tuned endpoints, with learning rates selected to match a nominal learning-rate-to-bf16-ULP ratio, the documented export lowers greedy GSM8K strict accuracy from 58.79% to 0.78% for Falcon-E-1B-Base and from 36.13% to 0.39% for BitCPM-CANN-0.5B, and a bf16 save and reload lowers BitNet 2B-4T's strict accuracy by 27.54 points while its last-number accuracy rises. Two compatibility remedies, writing the training quantizer's codes directly or adjusting the bf16 inputs until the unchanged tools emit them, each met a 4-point strict-accuracy non-inferiority criterion against online evaluation in all three models. In a prospectively frozen study of bf16 latents during fine-tuning (one task, one recipe and trajectory per arm), native bf16 writes left 2B-4T's codes unchanged at the endpoint and changed one Falcon-E code, while emulated stochastic-rounding and compensated writers changed codes and were non-inferior to fp32 training at the same margin in both models. In two model families, randomized interventions on the initial distance from the threshold support distance-dependent selection of the codes that fine-tuning changes.
| Comments: | v2: adds Section 6, a pre-registered study of bf16 latent writes during fine-tuning (native bf16, and stochastic-rounding and compensated-write emulations, on BitNet 2B-4T and Falcon-E), and an updated audit of public fine-tuned checkpoints. 18 pages, 4 figures, 20 tables |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.07853 [cs.LG] |
| (or arXiv:2610.07853v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07853 arXiv-issued DOI via DataCite |
Submission history
From: Avichal Sahai [view email]
[v1]
Tue, 6 Oct 2026 06:58:35 UTC (585 KB)
[v2]
Wed, 14 Oct 2026 09:28:41 UTC (592 KB)
来源：arXiv cs.CL · arxiv.org