arXiv cs.AI· Elham J. Barezi, Primus C Kabuo, Niveditha Nerella·· 12 小时前AI 评分23
用 Agentic LLM 助手把 HPC 用户支持建立在文档与已解决工单之上
Before the Ticket: Grounding HPC User Support in Documentation and Resolved Tickets with an Agentic LLM Assistant
AI 导读
研究团队构建了一个基于 Model Context Protocol 的 HPC 帮助台助手，通过编排器调用两个工具：对实时文档做 RAG 检索，以及用 QLoRA 在 11,244 条历史工单查询-解决对上微调的 Llama-3-8B，并合并两者答案。
正文 · 原文
Abstract:Most questions that reach an HPC help desk have already been answered in the center's user guides or in a ticket someone else filed last semester; users still file new tickets because the guides are hard to navigate and the ticket history is not theirs to search. We present an assistant that answers from both sources: an orchestrator calls two tools over the Model Context Protocol, retrieval-augmented generation (RAG) over the live documentation and a Llama-3-8B model fine-tuned with QLoRA on 11,244 query-resolution pairs mined from historical tickets, and merges their answers. Every model runs on-premises or on the university's hosted LLM service because ticket data may not leave the institution. On 400 held-out tickets from Purdue's Rosen Center for Advanced Computing, disjoint from training and free of near-duplicates, and evaluated by two judges from model families that generated none of the answers, fine-tuning on resolutions rewritten into self-service guidance is the clear winner in agreement with RCAC's resolutions (BERTScore F1 improves from 0.06 to 0.27 over the base model) and is judged most correct on average. The end-to-end system does not improve on its tools, although they are complementary: choosing the better answer per question would exceed either tool (judged correctness 3.41 versus 3.20, an oracle bound), while the merge follows the documentation tool in 78% of answers. Returning one tool's answer instead recovers the fine-tuned tool's quality (3.19 versus 2.66), but neither a learned selector nor an LLM verifier beats that tool alone.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.31703 [cs.SE] |
| (or arXiv:2610.31703v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31703 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Elham J. Barezi Ms [view email]
[v1]
Wed, 21 Oct 2026 19:01:41 UTC (383 KB)
来源：arXiv cs.AI · arxiv.org