arXiv cs.AI· Niklas Wagner, Alexander M\"adche·· 12 小时前AI 评分31
面向不完美提示词的设计：基于知识图谱的 LLM 企业数据助手仿真评估
Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs
AI 导读
针对企业用户常提出含领域术语、缺少 schema 引用的不完美提示词，该研究设计并评估了两种语义中介方案：基于语义相似度的 schema 中介与基于数据湖自动构建知识图谱的中介。
正文 · 原文
Abstract:Enterprise data assistants using large language models (LLMs) promise democratized data access in organizations. Yet in our real-world deployment within the CFO unit's data lake at an industry partner, we observed that business users frequently pose imperfect prompts that include domain-specific vocabulary and lack schema references or structural information. This Design Science Research study examines automatic resolution using a semantic mediation layer. We design and evaluate two design variants: schema-based mediation via semantic similarity, and knowledge graph-based mediation leveraging an automatically constructed knowledge graph from the data lake. Our simulation-based evaluation using real-world data, supported by LLM-as-a-Judge assessments and human validation, shows that knowledge graph-based mediation outperforms the schema-based approach in Precision, Recall, F1, and preserving user intent. We propose a design principle for knowledge graph-based mediation, demonstrating that structured knowledge representation bridges user intent and database structure and guides the design of enterprise data assistants across skill levels.
| Comments: | Published in the Proceedings of the 34th European Conference on Information Systems (ECIS 2026) |
| Subjects: | Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Databases (cs.DB) |
| Cite as: | arXiv:2610.23304 [cs.HC] |
| (or arXiv:2610.23304v1 [cs.HC] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23304 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Proceedings of the 34th European Conference on Information Systems (ECIS 2026), 5 |
Submission history
From: Niklas Wagner [view email]
[v1]
Fri, 16 Oct 2026 11:07:04 UTC (2,039 KB)
来源：arXiv cs.AI · arxiv.org