跳到正文
arXiv cs.AI· Dowling Wong·· 10 小时前AI 评分34

如何评估受治理的 LLM 驱动 ML 实验:生命周期、溯源与失败指标

Evaluating Governed LLM-Driven ML Experimentation: Lifecycle, Provenance, and Failure Metrics

AI 导读

研究者提出按实验过程而非仅最佳分数来评估自主 ML 实验智能体,并构建了由控制平面掌管每次试验生命周期、校验编辑范围、决定保留或丢弃并写入仅追加账本的治理框架。

正文 · 原文

View PDF HTML (experimental)

Abstract:Autonomous agents that run ML experiments are usually judged by the best score they reach. That score says nothing about how it was reached: whether the agent stayed within the files it was allowed to edit, whether failed trials were kept or silently dropped, and whether each keep or discard decision can be traced to its evidence. We propose evaluating such systems on their experimental process as well as their outcome, and we build a governed harness that makes this possible. A control plane, not the LLM that proposes changes, owns each trial's lifecycle, checks its edit scope, decides keep or discard, and appends one record per trial to an append-only ledger. Governance metrics such as the invalid-action rate, the repeated-bad rate and provenance completeness are computed from that ledger. Across six ML tasks, from synthetic logistic regression to language-model training, and 1,445 trials, the same contract applied unchanged. The ledger exposed failures that a task score hides: a no-memory arm in which all 120 trials failed, a benchmark adapter whose bounded proposals ran out, and a reset failure that reproducibility hashes revealed, after which the affected campaigns were rerun. In a matched counterfactual, a loop that logs only successful trials left no record of 43 of 50 trials, while the governed ledger recorded all 50. Memory ablations show that memory's effect on repeated bad proposals depends on the task. We release the ledgers of all 1,445 trials, so that every number reported for them can be recomputed.
Comments: 22 pages. Presented at the KDD 2026 Workshop on Evaluation and Trustworthiness of Agentic AI (ETAAI; non-archival). Code and trial ledgers: this https URL (DOI https://doi.org/10.5281/zenodo.24174991)
Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)
Cite as: arXiv:2611.23678 [cs.AI]
  (or arXiv:2611.23678v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.23678

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Dowling Wong [view email]
[v1] Wed, 18 Nov 2026 01:06:35 UTC (230 KB)

来源:arXiv cs.AI · arxiv.org