arXiv cs.CY· Daniel Dom\'inguez Figaredo, Rafael Fern\'andez De la Cruz·· 9 天前AI 评分52
研究：按教学指标微调会让 LLM 辅导质量下降
The learner who does not learn: when optimizing a pedagogical metric degrades LLM tutoring
AI 导读
一项研究设计了教学适应性指标，在 2000 个学习者场景中审计一个前沿 LLM 辅导模型，并用开源权重代理模型微调修正其最弱案例。31 位受训教育者盲评显示，修正案例的指标从 +0.05 升到 +0.42，但专家评分从 4.46 降到 3.03，未修改的对照组保持不变。
正文 · 原文
Abstract:It is assumed that a natural way to improve the pedagogical quality of large language model tutors is to define a metric of instructional performance and fine-tune the model against it. To test this strategy, we designed a metric of pedagogical adaptivity that scores each instructional decision in a learning sequence against the conditions of the learning situation, which is the standard used for automated pedagogical scoring. We audited a frontier tutor across 2,000 learner scenarios, corrected its weakest cases by fine-tuning an open-weights proxy, and asked 31 trained educators to rate the pedagogical alignment of the outputs blind, before and after correction. The metric increased from +0.05 to +0.42 for the corrected cases, while the expert ratings decreased from 4.46 to 3.03, with the unmodified controls remaining unchanged and a base-proxy control ruling out the change of model. The tutor performed worse because any metric that scores decisions independently and averages them is maximized by repeating the single best decision, and the fine-tuned model collapsed to that exact optimum in every case, in and out of sample. Educators identified the repetition, which such metrics cannot represent, and preserving the learner's trajectory in the score reduced, but did not reverse, the metric's verdict. Weight analysis traced the correction to the model's output projection, where it had memorized its training strings rather than learned to adapt. We conclude that measurement validity does not imply optimization validity, and we derive design principles for benchmarks that assess or train AI-tutor instruction.
| Comments: | 33 pages, 7 figures, 5 tables |
| Subjects: | Computers and Society (cs.CY) |
| MSC classes: | 97U50, 68T50, 68T05 |
| ACM classes: | K.3.1; I.2.7; I.2.6 |
| Cite as: | arXiv:2610.12125 [cs.CY] |
| (or arXiv:2610.12125v1 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12125 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Daniel Domínguez Figaredo [view email]
[v1]
Thu, 8 Oct 2026 15:18:26 UTC (2,067 KB)
来源：arXiv cs.CY · arxiv.org