arXiv cs.AI· Blagoj Mitrevski, Arina Rak, Julian Schnitzler, Chengkun Li, Andrii Maksai, Jesse Berent, Claudiu Musat·· 2 天前AI 评分33
InkSight：教视觉语言模型读写，实现离线手写到在线手写转换
InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write
AI 导读
InkSight 通过结合阅读与写作先验，将照片中的离线手写笔迹转换为可编辑的数字墨水（在线手写），无需大量配对样本即可训练。该模型首次能有效处理具有多样视觉特征和背景的任意照片手写文本，并可泛化到简单草图。在 HierText 数据集上，87% 的生成样本被视为对输入图像的有效描摹，67% 看起来像人类书写的笔迹轨迹。
正文 · 原文
Abstract:Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in a vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice that is still favored by a vast majority. Our work InkSight, aims to bridge the gap by empowering physical note-takers to effortlessly convert their work (offline handwriting) to digital ink (online handwriting), a process we refer to as derendering. Prior research on the topic has focused on the geometric properties of images, resulting in limited generalization beyond their training domains. Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds. Furthermore, it generalizes beyond its training domain into simple sketches. Our human evaluation reveals that 87% of the samples produced by our model on the challenging HierText dataset are considered as a valid tracing of the input image and 67% look like a pen trajectory traced by a human.
| Comments: | Accepted by Transactions on Machine Learning Research |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2402.05804 [cs.CV] |
| (or arXiv:2402.05804v5 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2402.05804 arXiv-issued DOI via DataCite |
Submission history
From: Chengkun Li [view email]
[v1]
Thu, 8 Feb 2024 16:41:41 UTC (42,732 KB)
[v2]
Wed, 21 Feb 2024 00:19:06 UTC (42,731 KB)
[v3]
Sun, 8 Dec 2024 21:05:43 UTC (43,136 KB)
[v4]
Tue, 17 Jun 2025 12:16:51 UTC (13,720 KB)
[v5]
Wed, 14 Oct 2026 16:50:11 UTC (13,728 KB)
来源：arXiv cs.AI · arxiv.org