跳到正文
Hacker News· algoth1·· 6 小时前AI 评分58

Astra 6.2 与 Sol 6.2 在 SimpleBench 基准上表现不佳

Astra 6.2 and Sol 6.2 flop at Simple Bench

AI 导读

SimpleBench 是一个面向 LLM 的多选题文本基准,包含 200 多道覆盖时空推理、社交智能和语言对抗鲁棒性(陷阱题)的题目,九名普通(高中知识水平)参与者基线为 83.7%,高于所有受测模型,包括得分 81.9% 的 Claude Fable。

正文 · 原文

We introduce SimpleBench, a multiple-choice text benchmark for LLMs where individuals with unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions). For the vast majority of text-based benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on our small sample of nine participants,

outperforming every tested LLM, including today's top model, Claude Fable, which scored 81.9%

. While we expect model performance to improve over time, the results of SimpleBench confirm that the memorized knowledge, and approximate reasoning retrieval, utilized by frontier LLMs is not always enough to answer basic questions just yet.

For a deeper dive into our results and our methods, check out the full technical report here.

Use all of these models on the Simple Bench app - LMcouncil.ai

来源:Hacker News · simple-bench.com