跳到正文
RadarAI· X:Rohan Paul (@rohanpaul_ai)·· 5 小时前AI 评分55

论文发现低基准分可被精确制造,单看低分难证模型能力

论文:低基准分可被精确制造,难证模型能力

AI 导读

一篇论文发现低基准分可以被精确制造,因此单看低分几乎无法说明模型能力。研究者训练了一个 LoRA 旋钮,把 Qwen 模型拉向更小的 Qwen 模型,其对未见目标分数的误差在 1.9 分以内,而提示词诱导欠佳的偏差达 42 至 47 分。但在多选题上,该旋钮最多 97% 的新增错误集中在同一个字母选项上,这一比例远超其模型池中任何自然模型。

正文 · 原文

Rohan Paul· @rohanpaul_ai · X·2026-11-18 01:23· 56 分钟前AI 评分48

AI 导读

一篇论文发现低基准分可以被精确制造,因此单看低分几乎无法说明模型能力。研究者训练了一个 LoRA“旋钮”,把 Qwen 模型拉向更小的 Qwen 模型,其对未见目标分数的误差在 1.9 分以内,而提示词诱导欠佳的偏差达 42 至 47 分。但在多选题上,该旋钮最多 97% 的新增错误集中在同一个字母选项上,这一比例远超其模型池中任何自然模型。

正文 · 原文

This paper finds that a low benchmark score can be manufactured precisely, so a low score alone says little about what a model can do.

Safety tests often read a low score on a dangerous-capability probe as reassurance. But a low score can come from a model being held back, so the number alone proves little.

They trained a LoRA "dial" that pulls a Qwen model toward a smaller Qwen model, then compared its errors with the real small model's. They also tried prompting models to underperform, raising temperature, and adding weight noise.

The dial hit unseen target scores within 1.9 points, while prompting missed by 42 to 47 points. But on multiple choice, up to 97% of its new errors landed on 1 letter, a share no natural model in their pool approached.

Before reading a low score as low capability, check which items the model fails and how its wrong answers are spread.

来源:Rohan Paul · x.com

来源:RadarAI · aihot.news