arXiv cs.CL· Olga Loginova, Oleksandr Bezrukov, Frank Keller·· 13 小时前AI 评分18
PRISM：用逐步推理识别流程视频中的操作错误
Through a PRISM, Clearly: Step-Sequence Reasoning for Mistake Recognition in Procedural Videos
AI 导读
PRISM（Procedural mistake Recognition via Interpretable Semantic Modeling）将视频片段用语言描述锚定，再通过基于正确执行训练的轻量隐马尔可夫模型映射到流程状态，结合参考对齐与语义检查识别步骤内容及顺序错误。
正文 · 原文
Abstract:Procedural mistake recognition requires identifying deviations from valid multi-step activities. We study post-completion recognition from videos with benchmark-provided step boundaries. We introduce PRISM (Procedural mistake Recognition via Interpretable Semantic Modeling), which grounds video segments in language and maps their descriptions to procedural states using a lightweight hidden Markov model trained on correct executions. Reference alignment and semantic checks identify errors in step content and order. To support evaluation across heterogeneous benchmarks, we map dataset-specific annotations to four edit-based error types: insertion, deletion, substitution, and transposition. Across three benchmarks, we compare gold and VLM-generated descriptions in raw, semantic-role, and action-object forms. Decoder controls show consistent gains from the trained local decoder over nearest-candidate matching, with additional, dataset-dependent benefits from sequence decoding. Semantic role labeling improves Any-F1 in all six matched grounded comparisons, although the gains vary across datasets and do not translate into uniform improvements under separate configuration selection. These results establish language grounding as an effective interface for explicit procedural mistake reasoning and characterize the contributions of representation and decoding.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.31117 [cs.CV] |
| (or arXiv:2610.31117v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31117 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Olga Loginova [view email]
[v1]
Thu, 22 Oct 2026 11:32:37 UTC (441 KB)
来源：arXiv cs.CL · arxiv.org