arXiv cs.AI· Darek Kleczek, Fuheng Zhao, Alexander W. Lee, Julien Tissier, Pawel Liskowski, Ugur Cetintemel, Anupam Datta·· 11 小时前AI 评分31
AvalancheBench：评估企业智能体驱动语义查询处理的潜世界恢复基准
AvalancheBench: Evaluating Enterprise Agent-Driven Semantic Query Processing Through Latent World Recovery
AI 导读
AvalancheBench 是一个多模态企业分析基准，通过从潜世界规格生成观测数据，评估数据智能体恢复该世界的能力。基准含 3 个验证任务、58 道覆盖文本图像文档的题目，以多部分分析简报形式给出并共享中间结果，暴露错误沿合法依赖传播的“雪崩效应”。
正文 · 原文
Abstract:As LLMs improve, data agents are emerging as a new interface for analytics, while semantic query processing engines use LLMs to operate over structured and unstructured data. Existing benchmarks largely evaluate agents and semantic query engines in isolation, leaving it unclear whether agents can decide when and how to use semantic operators in end-to-end analysis.
We introduce AvalancheBench, a multimodal enterprise analytics benchmark that generates observed data from latent world specifications and evaluates data agents on their ability to recover that world. The benchmark's 3 validated tasks contain 58 questions spanning text, images, and documents. Questions are provided together as a multi-part analytical brief and may share intermediate results, exposing the avalanche effect: errors can propagate through legitimate dependencies, and inappropriate reuse can cause otherwise unrelated questions to fail. AvalancheBench measures analytical fidelity, monetary cost, and capability-adjusted token cost (CATC), a new metric that weights token use by model capability to separate technical efficiency from provider pricing.
We evaluate 5 model-harness configurations across 3 agent harnesses using Snowflake Cortex AISQL as the engine, with the best configuration reaching macro analytical fidelity of 0.82. Agents fail primarily in constructing the analysis: failing to recover facts that require semantic computation costs 45.1% of lost points, yet incorrect operator outputs account for only 2.5%, versus 36.3% from agents misusing or avoiding semantic operators. Misinterpreting requirements costs the most, at 51.5%. Errors also spread between questions: 29% of lost points come from reusing another question's results. Finally, monetary cost and CATC produce different Pareto frontiers, distinguishing economically attractive configurations from those that are technically resource efficient.
| Subjects: | Databases (cs.DB); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.24183 [cs.DB] |
| (or arXiv:2605.24183v2 [cs.DB] for this version) | |
| https://doi.org/10.48550/arXiv.2605.24183 arXiv-issued DOI via DataCite |
Submission history
From: Dariusz Kleczek [view email]
[v1]
Fri, 22 May 2026 20:16:41 UTC (110 KB)
[v2]
Wed, 4 Nov 2026 18:20:29 UTC (9,533 KB)
来源：arXiv cs.AI · arxiv.org