arXiv cs.CL· Camilo Mari\~no (UDELAR, CB), Diego Belzarena (UDELAR, CB), Marina Gardella (CB), Ignacio Ram\'irez (UDELAR), Gregory Randall (UDELAR), Jean-Michel Morel (LU - Hong Kong)·· 12 小时前AI 评分24
面向历史文献转录的 OCR 系统实体感知评测
Entity-Aware Evaluation of OCR Systems for Historical Document Transcription
AI 导读
一项研究用 Berrutti Archive 的 175 份文档和 5,209 条人工标注实体提及，对序列识别 OCR、OCR 专用视觉语言模型和通用视觉语言模型做实体级评测。结果显示 OCR 排名随评价标准变化：Qwen 3.6 在所有测试容差下实体覆盖率最高，而部分精确匹配覆盖率较低的系统在容忍小字符误差后恢复明显。
正文
Abstract:Optical Character Recognition (OCR) is essential for digitizing historical archives, yet standard evaluation metrics such as Character Error Rate (CER) do not necessarily reflect whether meaningful information, such as names, dates, and identifiers, is preserved. In this work, we investigate entitylevel analysis of OCR errors as a complementary perspective for evaluating OCR quality in historical documents. Using a publicly available benchmark of 175 documents from the Berrutti Archive, augmented with 5,209 manually annotated entity mentions that will be released upon acceptance, we compare sequence-recognition OCR systems, OCR-specialized vision-language models, and general-purpose vision-language models. Beyond global transcription accuracy, we analyze entity preservation and recoverability under varying character-level edit tolerances. Our experiments show that OCR rankings depend on the evaluation criterion: while Qwen 3.6 achieves the highest entity coverage across all tested tolerances, several systems with lower exact-match coverage recover substantially when small errors are tolerated, suggesting that many of their errors correspond to localized character-level deviations rather than fundamentally incorrect entity transcriptions. Conversely, some systems remain unable to recover many entities even under relaxed matching. These findings show that OCR quality cannot be fully characterized by aggregate transcription metrics alone and highlight the importance of evaluating how OCR errors affect the preservation and recoverability of meaningful information.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Image and Video Processing (eess.IV) |
| Cite as: | arXiv:2610.33004 [cs.CL] |
| (or arXiv:2610.33004v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33004 arXiv-issued DOI via DataCite |
Submission history
From: Diego Belzarena [view email] [via CCSD proxy]
[v1]
Wed, 7 Oct 2026 12:12:01 UTC (469 KB)
来源：arXiv cs.CL · arxiv.org