arXiv cs.AI· Joran Leest, Ilias Gerostathopoulos, Claudia Raibulet, Patricia Lago·· 12 小时前AI 评分31
分布偏移下的反事实模型测试方法
Counterfactual Model Testing Under Distribution Shift
AI 导读
针对部署后 ML 模型因输入分布偏移而性能退化的问题，研究者提出将分布偏移下的在线模型测试形式化，并给出一种方法：在标签已知的受控基线上分别复现来自总体变化与数据完整性问题两种机制的偏移，以估计当前性能及修复数据问题可带来的潜在收益。在带模拟故障的真实数据集实验中，区分这两类来源通常能改善两项估计，流失预测案例展示了该测试如何支持监控与修复优先级排序。
正文
Abstract:Deployed machine learning (ML) models can degrade as the distribution of their input data shifts. Such shifts can result from changes in the population or from data integrity issues in the pipeline that materializes model features, two sources that operate through different mechanisms. Because labels are often absent or delayed, it is common practice to track shifts in the data instead, but a detected shift does not indicate how it affects the model. We formulate this problem as online model testing under distribution shift. Testing requires reproducing the shift on a controlled baseline with known labels. Since the same shift can come from either source, doing so requires knowing which mechanism changed. We propose a methodology that reproduces observed changes in both mechanisms to estimate current performance and the potential gain from repairing observed data integrity issues. Experiments on real-world datasets with simulated faults show that separating the two sources generally improves both estimates, supporting this formulation. A churn prediction case illustrates how these tests can support monitoring and repair prioritization
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.33594 [cs.AI] |
| (or arXiv:2610.33594v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33594 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Joran Leest [view email]
[v1]
Sat, 24 Oct 2026 10:42:31 UTC (59 KB)
来源：arXiv cs.AI · arxiv.org