arXiv cs.AI· Evangelos Apostolakis, Erfan Hamdi, Sara Jahedazad, Emma Lejeune·· 14 小时前AI 评分31
TrussTutor-Bench：评测大语言模型在静力学中的解题准确率、谄媚与错误检测能力
TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics
AI 导读
TrussTutor-Bench 基准包含 200 道经核验的二维桁架题及自动生成的学生解答，用于评测 13 个大语言模型的解题准确率、谄媚与错误检测。基线准确率在 64.5% 至 97.0% 之间，各模型对合成错误解答的检出率达 90.4% 至 100%，未能区分模型；但当学生坚持正确答案有误时，模型否定自己已解出正确答案的比例为 1.6% 至 50.0%。
正文 · 原文
Abstract:Large language models (LLMs) have rapidly become a default source of help for students in science and engineering courses. In foundational engineering mechanics courses such as statics, students routinely use these tools to check answers, work through partial solutions, and ask for explanations. Effective tutoring in this setting requires more than a correct final answer. A model must also confirm correct work when a student doubts it, and identify errors in flawed work. Existing benchmarks evaluate LLMs primarily on final-answer accuracy, so it is unclear whether models that can solve mechanics problems also have the other technical capabilities that tutoring depends on. Here we introduce TrussTutor-Bench, a benchmark of 200 verified two-dimensional truss problems paired with automatically generated student solutions, and use it to evaluate thirteen LLMs on solution accuracy, sycophancy, and error detection. Baseline accuracy ranged from 64.5% to 97.0%. Every model detected nearly all injected errors in synthetic flawed student work (90.4% to 100%), so error detection did not distinguish models under these clean conditions. Sycophancy did. When a student insisted that correct work was wrong, models rejected correct solutions to problems they had themselves solved in 1.6% to 50.0% of cases. These results show that performance varies widely across models: the leading models solve, check, and defend student work nearly perfectly, while others are far less reliable, particularly when a student doubts correct work. TrussTutor-Bench provides a verified, extensible standard for determining which models are technically reliable enough to put in front of students. Follow-up work must now focus on teaching rather than technical capability, drawing on learning science to determine whether LLM help leads students to understand the material or simply to reach the right answer.
| Comments: | 25 pages, 7 figures. Code: this https URL |
| Subjects: | Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.38917 [cs.CY] |
| (or arXiv:2610.38917v1 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38917 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Evangelos Apostolakis [view email]
[v1]
Wed, 28 Oct 2026 21:07:58 UTC (2,263 KB)
来源：arXiv cs.AI · arxiv.org