arXiv cs.CL· Ziqi Liu, Ruiwei Xiao, Yuxuan Li, John Stamper·· 12 小时前AI 评分38
EduWild 研究：教师配置的教育聊天机器人在真实使用中如何失效及如何修复
EduWild: How Teacher-Configured Chatbots Fail in Real-World Use and How to Fix Them
AI 导读
研究发布 EduWild 数据集，包含 217 个教师配置聊天机器人的 1,726 段对话及对应配置说明，其中 900 段来自 35 个已部署机器人的真实终端用户。
正文 · 原文
Abstract:Teachers increasingly configure educational chatbots through natural-language instructions, but how these bots fail in real-world use remains poorly understood. We introduce EduWild, a dataset of 1,726 conversations with 217 teacher-configured chatbots, each paired with the instructions and configuration behind it, including 900 conversations from the intended end users of 35 deployed chatbots. We develop a 29-code taxonomy covering factual, workflow, diagnostic, and pedagogical failures and apply it using a human-calibrated LLM judge. The most frequent failures are lapses in following teacher intent --- bots skip required workflow steps (53.8% of end-user conversations), omit expectation setup (33.4%), and overload learners with information or questions (37.1%). Guided by these findings, we design a prompt-based harness targeting the six most frequent failure modes. In a study with 200 college students across ten bots, the harness reduces the mean number of target failure codes per conversation from 1.406 to 0.489, a 65.2% relative reduction with bots weighted equally. These findings show how real-world failure analysis can guide practical improvements to educational chatbots.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.23530 [cs.CL] |
| (or arXiv:2610.23530v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23530 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ziqi Liu [view email]
[v1]
Fri, 16 Oct 2026 14:28:06 UTC (2,746 KB)
来源：arXiv cs.CL · arxiv.org