arXiv cs.CL· Sachin Gupta·· 13 小时前AI 评分31
面向 LLM IT 工单分类的成本感知弃权评估框架
Cost-Aware Abstention for LLM-Based IT Ticket Classification
AI 导读
研究提出成本感知弃权作为离线、成本条件下的就绪度评估方法：在校准数据上按显式成本选定路由策略，再在留出测试集上冻结评估。在 47,837 条工单、八类语料的公开数据集上，TF-IDF 加逻辑回归以校准选出的 0.60 阈值将测试成本从 0.763 降至 0.472（相对降幅 38.1%），错误路由从 15.3% 降至 3.5%。
正文 · 原文
Abstract:Wrong automatic routing of an IT support ticket can cost more than human triage. We evaluate cost-aware abstention as an offline, cost-conditional readiness assessment: select a routing policy on calibration data under explicit costs, then evaluate that frozen policy once on held-out test data. On a public 47,837-ticket, eight-class corpus, we compare four trained non-LLM classifiers with four large language models under zero- and few-shot prompting. Under the illustrative cost model (c_ok, c_triage, c_wrong) = (0, 1, 5), TF-IDF with logistic regression and a threshold of 0.60 selected on calibration reduces test cost from 0.763 to 0.472, a 38.1% relative reduction (paired ratio-bootstrap 95% CI [35.7%, 40.4%]), while lowering wrong routes from 15.3% to 3.5%. Frozen calibration-selected policies also beat both always-route and always-defer on held-out test data in 14 of 16 settings in a post-hoc cost-parameter analysis. The strongest prompted configuration reaches 0.477 accuracy on a stratified 1,000-ticket test subset, versus 0.879 for fine-tuned DistilBERT on the same tickets. Isotonic lowers expected calibration error without reliably improving discrimination: GPT-4o-mini few-shot has one confidence value before and after recalibration and correctness AUROC 0.500. We therefore report correctness AUROC and tie-aware risk-coverage area alongside ECE. At the analytically derived break-even probability 0.80, seven prompted configurations defer all tickets; Haiku few-shot routes only 19/1,000. An LLM-only agreement control and margin/entropy rejection baselines further separate calibration from discrimination. The findings are scoped to this dataset, prompt design, model endpoints, and illustrative costs; the work is an evaluation framework, not a new abstention algorithm or universal deployment certification.
| Comments: | Accepted at the 38th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2026). 9 pages, 5 figures, 9 tables |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.31657 [cs.LG] |
| (or arXiv:2610.31657v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31657 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sachin Gupta [view email]
[v1]
Thu, 22 Oct 2026 17:58:52 UTC (332 KB)
来源：arXiv cs.CL · arxiv.org