StreamDecisionBench：评估语言模型在动态语言流上的决策能力

View PDF HTML (experimental)

Abstract:Language models increasingly make real-time decisions in applications that apply the latest answer until a newer one arrives. A late answer can prolong an outdated decision, such as a call recorder still running while a customer reads out card details, an error offline accuracy misses. We make three contributions. First, we release StreamDecisionBench (SDB), a dataset of eight streaming scenarios in four application families, with executable reference decisions derived from public rules. Second, we propose an evaluation protocol and a metric, in-force accuracy: the share of time the applied decision is correct across update intervals of 0.5-8 s. It reflects accuracy and latency jointly, attributing each error to judgment, latency or both. Third, we evaluate fifteen single-model settings, and this attribution separates speed-limited from judgment-limited models: slower, more accurate models lose 42-47% of the time to outdated answers, a fast model 35% to wrong ones. We therefore test hybrids in which a slow model corrects a fast one; with the right pairing and configuration, a hybrid outperforms every single model. However, even the best evaluated system keeps a correct decision in force only about two-thirds of the time, leaving a substantial gap for real-time use.

Comments:	32 pages, 9 figures. Code and data: this https URL
Subjects:	Computation and Language (cs.CL)
Cite as:	arXiv:2609.38612 [cs.CL]
	(or arXiv:2609.38612v3 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2609.38612 arXiv-issued DOI via DataCite

Submission history

From: Jhen-Ke Lin [view email]
[v1] Tue, 29 Sep 2026 22:15:49 UTC (223 KB)
[v2] Thu, 1 Oct 2026 14:48:21 UTC (261 KB)
[v3] Fri, 9 Oct 2026 16:51:55 UTC (328 KB)