arXiv cs.AI· Alexander Herzog, Aliai Eusebi, Lorenzo Cavallaro·· 13 小时前AI 评分31
Aurora 框架：评估恶意软件分类器在分布偏移下的置信可靠性与操作稳定性
On the Reliability and Stability of Selective Methods in Malware Classification Tasks
AI 导读
研究者提出 Aurora 框架，从置信质量和操作韧性两个维度评估恶意软件分类器，并对模型置信度剖面做验证以判断估计是否可靠。Aurora 还配套一组指标，用于在时间评估周期内衡量操作稳定性，而非只看单点性能。在漂移程度不同的数据集上，SOTA 框架均表现出脆弱性，作者认为或需重新审视其底层假设。
正文 · 原文
Abstract:The performance figures of modern drift-adaptive malware classifiers appear promising, but does this translate to genuine operational reliability? The standard evaluation paradigm primarily focuses on baseline performance metrics, neglecting confidence-error alignment and operational stability. While prior works established the importance of temporal evaluation and introduced selective classification in malware classification tasks, we take a complementary direction by investigating whether malware classifiers maintain reliable and stable confidence estimates under distribution shifts and exploring the tensions between scientific advancement and practical impacts when they do not. We propose Aurora, a framework to evaluate malware classifiers based on their confidence quality and operational resilience. Aurora subjects the confidence profile of a given model to verification to assess the reliability of its estimates. Unreliable confidence estimates erode operational trust, waste valuable annotation budgets on non-informative samples for active learning, and leave error-prone instances undetected in selective classification. Aurora is further complemented by a set of metrics designed to go beyond point-in-time performance, striving towards a more holistic assessment of operational stability throughout temporal evaluation periods. The fragility we observe in SOTA frameworks across datasets of varying drift severity suggests it may be time to revisit the underlying assumptions.
| Subjects: | Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2505.22843 [cs.CR] |
| (or arXiv:2505.22843v4 [cs.CR] for this version) | |
| https://doi.org/10.48550/arXiv.2505.22843 arXiv-issued DOI via DataCite |
Submission history
From: Aliai Eusebi [view email]
[v1]
Wed, 28 May 2025 20:22:43 UTC (4,778 KB)
[v2]
Wed, 25 Jun 2025 09:30:26 UTC (423 KB)
[v3]
Wed, 21 Jan 2026 18:26:18 UTC (212 KB)
[v4]
Tue, 27 Oct 2026 14:20:53 UTC (407 KB)
来源：arXiv cs.AI · arxiv.org