跳到正文
RadarAI· X:Rohan Paul (@rohanpaul_ai)·· 3 小时前AI 评分38

研究:LLM 评审拖累 AutoML 搜索效果

AI 导读

Amazon 与马萨诸塞大学的论文发现,在 AutoML 搜索循环中加入 LLM 评审收益极小、下行风险更大。在 96 种 pipeline 配置的表格模型搜索中,Llama-3.1-8B 评审在 8 个数据集中的 7 个上低于无评审基线,最多损失 0.245 AUC;理想评审最多仅提升约 0.005 AUC,且危害在 250 种配置、20 轮后大多消退。

正文 · 原文

Rohan Paul· @rohanpaul_ai · X·2026-11-18 03:23· 59 分钟前AI 评分49

AI 导读

Amazon 与马萨诸塞大学论文发现,在搜索循环中加入 LLM 评审收益极小、最多约 0.005 AUC,但下行风险更大。在 96 种 pipeline 配置的表格模型搜索中,Llama-3.1-8B 评审在 8 个数据集中的 7 个上低于无评审基线,最多损失 0.245 AUC;理想评审最多仅提升约 0.005 AUC,且危害在 250 种配置、20 轮后大多消退。

正文 · 原文

New Amazon + University of Massachusetts paper finds that adding an LLM critic to a search loop has small upside, about 0.005 AUC at best, and much larger downside.

Weak LLM critic can push AutoML search below plain random search

Agent systems often add an LLM critic that reads past results and tells the search what to try next. Teams check whether it helps, rarely whether it hurts.

They ran the same tabular model search with and without a critic, over a fixed pool of 96 pipeline setups. Random critic mistakes cost almost nothing since the loop keeps its best result, but advice that steadily points the wrong way wastes tries.

A Llama-3.1-8B critic scored below the no-critic baseline on 7 of 8 datasets, losing up to 0.245 AUC. An ideal critic gained at most about 0.005 AUC, and the harm mostly faded with 250 setups and 20 rounds.

Before trusting an LLM critic, even a large one, run the same budget without it and drop it if it loses.

来源:Rohan Paul · x.com

来源:RadarAI · aihot.news