arXiv cs.AI· Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, Arman Cohan·· 3 天前AI 评分24
Physics：面向大学水平物理问题求解的基础模型基准测试
Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving
AI 导读
研究者推出 Physics 基准，包含来自六个核心物理领域 PhD 资格考试难度的 1034 道开放式题目，每题配有解答与独立核验的参考答案，并采用基于 LLM 的评分协议评估数值、符号和文本作答。
正文 · 原文
Abstract:We introduce Physics, a benchmark for evaluating foundation models on advanced physics problem solving. It contains 1034 annotated, open-ended problems from PhD-qualifying exams across six core physics domains. Solving these problems requires applying domain-specific knowledge to physical setups and carrying out multi-step mathematical reasoning. Each problem includes a worked solution and independently checked reference answers. An LLM-based grading protocol evaluates individual answers and complete multi-part problems, supporting numeric, symbolic, and text responses. We evaluate 53 proprietary and open-weight models. On test problems without figures, high accuracy is concentrated in a small group of thinking models, with the strongest open-weight models approaching proprietary performance. Most open-weight models still solve fewer than half of these problems. Our analysis of Qwen3.5-9B reveals failures in applying physics knowledge and interpreting figures, with misread figures affecting several subquestions.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2503.21821 [cs.AI] |
| (or arXiv:2503.21821v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2503.21821 arXiv-issued DOI via DataCite |
|
| Journal reference: | Findings of ACL 2025 |
Submission history
From: Kaiyue Feng [view email]
[v1]
Wed, 26 Mar 2025 06:21:56 UTC (1,815 KB)
[v2]
Mon, 12 Oct 2026 18:17:30 UTC (364 KB)
来源：arXiv cs.AI · arxiv.org