arXiv cs.AI· Ardalan Aryashad, Yan Jin·· 10 天前AI 评分24
SEMAADB：面向连贯多图 SysML 模型的验证数据集与基准
A Validated Dataset and Benchmark for Coherent Multi-Diagram SysML Models
AI 导读
研究者发布 SEMAADB，一个包含 3,000 个工程上下文和 15,000 张图的 SysML 数据集，每个上下文含需求、块定义、活动、状态机、序列五种相互关联视图，其中 100 个上下文经人工校验构成基准测试集。在三个语言模型的两项任务评测中，最强模型修复了 64.3% 的语义错误，跨图更新最佳传播 F1 为 80.7%。结果表明语法修复已接近解决，语义修复与跨图一致性仍是难题。
正文 · 原文
Abstract:Systems engineers use several diagrams to describe the structure and behavior of systems. Engineers create these diagrams together to make sure that they use the same elements and remain consistent with one another. Large language models can generate diagrams as text or code, which makes it possible to create system diagrams automatically. However, their ability to generate coherent sets of diagrams is not well understood, and existing datasets and benchmarks do not directly measure this ability at scale. We introduce SEMAADB (Systems Engineering Modeling Assistant with AI Dataset and Benchmark), a dataset of 3,000 engineering contexts and 15,000 diagrams. Each context contains five connected SysML views: Requirement, Block Definition, Activity, State Machine, and Sequence. Here, a view is a diagram that presents one aspect of a system. We checked the diagram sets for consistency and valid rendering. A set of 100 contexts is also human-verified and forms the benchmark test set. We evaluate three language models on two tasks. In diagram repair, the strongest model repairs 64.3% of semantic errors . In cross-diagram update the best propagation F1 is 80.7% when a model applies one change across related diagrams. The results show that syntax repair is nearly solved, but semantic repair and consistency across diagrams are still challenging tasks for models. SEMAADB therefore provides both a large diagram resource and a set of benchmarks for measuring coherent multi-diagram SysML generation.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07356 [cs.SE] |
| (or arXiv:2610.07356v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07356 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ardalan Aryashad [view email]
[v1]
Mon, 5 Oct 2026 20:23:38 UTC (45 KB)
来源：arXiv cs.AI · arxiv.org