arXiv cs.AI· Kevin L Coakley, Odd Erik Gundersen·· 14 小时前AI 评分34
深度学习性能分布实证研究：跨架构与任务的鲁棒性差异
Measuring Deep Learning Performance - An Empirical Study of Performance Distributions Across Architectures and Tasks
AI 导读
一项针对图像分类与时间序列预测架构的 186 组实验（每组 100 个随机种子、共 18,600 次运行）发现，深度学习模型性能呈分布而非单一值，且常为非高斯分布，时间序列预测尤为明显。
正文 · 原文
Abstract:Context: Non-determinism in deep learning algorithm design and implementation leads to performance variation, meaning model performance is not a single value, but rather a distribution. These model performance distributions are underexplored despite their impact on robustness. Objective: We investigate the robustness of deep learning performance to sources of non-determinism, specifically focusing on how performance distributions differ across various architectures and tasks. Method: We conducted 186 experiments on image classification and time series forecasting architectures. Each experiment was run 100 times with different random seeds to generate performance distributions, resulting in 18,600 runs. Robustness was quantified using metrics for spread, symmetry, and tail risk. Results: Performance distributions are frequently non-Gaussian, particularly in time series forecasting. Model size does not systematically affect robustness -- larger image classification models show fewer outliers but not lower spread, while smaller time series models show lower spread but more extreme underperformers. Training duration does not scale linearly; early stopping effectively balances performance and robustness. Mean performance does not predict robustness - time series forecasting shows moderate correlation while image classification shows none. Time series models produce nearly three times more underperforming outliers than image classification models, indicating substantially higher tail risk. Conclusions: Tail risk poses serious concerns for Trustworthy AI in high-stakes applications. Models performing well on average may exhibit long tails and extreme outliers revealed only through distributional analysis. Mean performance alone should not guide model selection; assessment of spread, symmetry, and tail risk is essential for reliable model assessment where consistent performance is critical.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38037 [cs.LG] |
| (or arXiv:2610.38037v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38037 arXiv-issued DOI via DataCite |
|
| Journal reference: | Sci. Rep. 16, 20474 (2026) |
| Related DOI: | https://doi.org/10.1038/s41598-026-49656-z
DOI(s) linking to related resources |
Submission history
From: Kevin Coakley [view email]
[v1]
Tue, 27 Oct 2026 22:47:10 UTC (720 KB)
来源：arXiv cs.AI · arxiv.org