arXiv cs.AI· Peter Steiner, Jan Hochweber·· 10 小时前AI 评分27
用知识空间理论评测 LLM 下一步推荐：KST 基准方法
Benchmarking LLM Next-Step Recommendations with Knowledge Space Theory
AI 导读
一项研究提出以知识空间理论（KST）为形式化评测基准，从任务目录和先修关系推导可允许知识状态及每个状态的完整即时下一步集合，再与模型推荐结果比对。在覆盖六个模型、四个数学领域、四种提示条件与两种目录顺序的 2400 条响应样本中，精确集合准确率从仅给描述的 61.3% 升至要求模型重建先修关系时的 62.8%，提供先修关系后达 96.3%，给出完整知识空间后达 98.2%。
正文 · 原文
Abstract:Selecting an LLM for next-step recommendation requires benchmarks that reflect the intended educational task and domain. We present a benchmarking method that uses Knowledge Space Theory (KST) as a formal evaluation reference. Given a task catalogue and an existing prerequisite relation, the method derives admissible knowledge states and the complete set of immediate next steps for each state. Recommendations generated by a model under a specified prompt are compared with these reference sets. A sample evaluation study uses 2,400 responses from six models across four mathematics domains, four prompt conditions, and two catalogue orders. Exact-set accuracy was 61.3% with descriptions only, 62.8% when models were asked to reconstruct prerequisites, 96.3% with supplied prerequisite relations, and 98.2% with the complete knowledge space. Performance varied across models and domains, and recommendation sets agreed across the two presentations in 85.25% of pairs. The benchmark supports comparisons of model-prompt combinations against a context-specific reference. Its scores describe structural recommendation accuracy under the evaluated conditions; they do not establish learning benefit or the necessity of using an LLM for selection.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13667 [cs.AI] |
| (or arXiv:2611.13667v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13667 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Peter Steiner [view email]
[v1]
Tue, 10 Nov 2026 07:15:25 UTC (47 KB)
来源：arXiv cs.AI · arxiv.org