跳到正文
arXiv cs.AI· Jin Seong, Wencke Liermann, Minho Kim, Jong-hun Shin, Soojong Lim·· 4 小时前AI 评分37

研究揭示 VLM 在多行手写数学 OCR 中的“过度纠错”问题,并提出 PINK 评估指标

When VLMs 'Fix' Students: Identifying and Penalizing Over-Correction in the Evaluation of Multi-line Handwritten Math OCR

AI 导读

针对多行手写数学 OCR,研究首次系统揭示 VLM 的“过度纠错”失效模式:模型会擅自“修正”学生错误,掩盖教育评估本应检测的错漏。为此提出基于 LLM 评分标准并显式惩罚过度纠错的语义指标 PINK。

正文 · 原文

View PDF HTML (experimental)

Abstract:Accurate transcription of handwritten mathematics is crucial for educational AI systems, yet current benchmarks fail to evaluate this capability properly. Most prior studies focus on single-line expressions and rely on lexical metrics such as BLEU, which fail to assess the semantic reasoning across multi-line student solutions. In this paper, we present the first systematic study of multi-line handwritten math Optical Character Recognition (OCR), revealing a critical failure mode of Vision-Language Models (VLMs): over-correction. Instead of faithfully transcribing a student's work, these models often "fix" errors, thereby hiding the very mistakes an educational assessment aims to detect. To address this, we propose PINK (Penalized INK-based score), a semantic evaluation metric that leverages a Large Language Model (LLM) for rubric-based grading and explicitly penalizes over-correction. Our comprehensive evaluation of 15 state-of-the-art VLMs on the FERMAT dataset reveals substantial ranking reversals compared to BLEU: models like GPT-4o are heavily penalized for aggressive over-correction, whereas Gemini 2.5 Flash emerges as the most faithful transcriber. Furthermore, human expert studies show that PINK aligns significantly better with human judgment (55.0% preference over BLEU's 39.5%), providing a more reliable evaluation framework for handwritten math OCR in educational settings. Code is available at this https URL.
Comments: Accepted to EMNLP 2026 Main Conference (Oral). 22 pages. Code: this https URL
Subjects: Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
Cite as: arXiv:2604.22774 [cs.CY]
  (or arXiv:2604.22774v3 [cs.CY] for this version)
  https://doi.org/10.48550/arXiv.2604.22774

arXiv-issued DOI via DataCite

Journal reference: EMNLP 2026, pages 10932-10953
Related DOI: https://doi.org/10.18653/2026.emnlp-main.476

DOI(s) linking to related resources

Submission history

From: Jin Seong [view email]
[v1] Wed, 1 Apr 2026 05:27:52 UTC (2,486 KB)
[v2] Tue, 26 May 2026 06:37:15 UTC (2,494 KB)
[v3] Tue, 17 Nov 2026 06:11:15 UTC (2,115 KB)

来源:arXiv cs.AI · arxiv.org