arXiv cs.AI· Uddipan Basu Bir, Vincent Christlein, Andreas Maier, Mathias Zinnen·· 9 天前AI 评分22
从像素到结构：轻量级视觉语言模型用于文档 OCR 与结构化 JSON 抽取
From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction
AI 导读
一项对比研究评估了八款开源轻量级 VLM（最高 7B 参数）在三个高校遗产收藏上的 OCR-to-structure 表现，要求模型从文档图像中抽取文本并生成符合 schema 的 JSON。
正文 · 原文
Abstract:While massive, closed-source Vision-Language Models (VLMs) set strong benchmarks for document understanding, their dependence on commercial APIs limits adoption in institutional archives due to data autonomy concerns, recurring costs, and the environmental footprint of hyperscale computing. This is especially acute in heritage digitization, where documents include historical handwriting, domain-specific terminology (e.g., jewelry, prehistory, architecture), and non-standard layouts requiring high-dimensional structured extraction. We present a comparative study of eight open-source lightweight VLMs (up to 7B parameters) for Optical Character Recognition (OCR)-to-structure across three university heritage collections. Given a document image, models must extract text and generate schema-compliant JSON, enabling automatic validation and downstream use. We evaluate models under a constraint-aware protocol across zero-shot, few-shot, and fine-tuning settings, measuring extraction fidelity and structured-output quality using Character Error Rate (CER), Approximate Normalized Levenshtein Similarity (ANLS*), and mean Average Precision F1 (mAP-F1). Against a fine-tuning baseline, we further test the independent impact of (i) hyperparameter optimization, (ii) classical image preprocessing (illumination flattening, denoising, and CLAHE), and (iii) multi-stage training. Finally, we analyze the trade-off between dataset-specific fine-tuning and a single multi-dataset checkpoint, where joint training enables one model to operate across collections but can shift performance between datasets. Overall, we show that carefully adapted VLMs with up to 7B parameters can provide a sustainable, private, high-performing alternative to manual transcription or commercial black-box systems, and we offer actionable guidance for heritage institutions seeking institution-controlled OCR-to-JSON extraction.
| Comments: | 17 pages. Published in Document Analysis and Recognition - ICDAR 2026, LNCS vol. 16974, Springer. Code: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.11818 [cs.CV] |
| (or arXiv:2610.11818v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11818 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Document Analysis and Recognition - ICDAR 2026, Lecture Notes in Computer Science, vol. 16974, pp. 502-519, Springer, 2027 |
| Related DOI: | https://doi.org/10.1007/978-3-032-36039-7_30
DOI(s) linking to related resources |
Submission history
From: Uddipan Basu Bir [view email]
[v1]
Thu, 8 Oct 2026 12:21:52 UTC (866 KB)
来源：arXiv cs.AI · arxiv.org