arXiv cs.AI· Keegan Wang·· 3 天前AI 评分41
在前沿 LLM 上迭代，在 SLM 上部署：一种智能体任务蒸馏方法
Iterate on an LLM, Serve on an SLM
AI 导读
研究者提出先在前沿模型 GPT-5.4-nano 上用真实流量反馈迭代提示词、待行为收敛后再通过 SFT 将生产行为蒸馏进 SLM 的两步法。在家庭护理排班智能体案例中，基于 11.6k 条生产决策用 LoRA 微调的 Qwen3-8B 达到 0.913 一致率，而教师模型 GPT-5.4-nano 重跑自身记录提示词时自洽率仅 72.7%。
正文 · 原文
Abstract:As frontier LLMs gain more capabilities, the cost to run them also goes up. At the same time, small language models (SLMs) also gain more capabilities but have largely been able to keep their cost stable, offering 5-40 times cheaper inference for the narrow, repetitive subtasks that agents decompose work into. However, the standing limiting factor is getting the correct behavior out of a small model in a fast, efficient, and economical way. Unfortunately, in most cases, zero-shot small models fail narrow production tasks. Furthermore, retraining one every time the behavior changes is too slow to iterate on. We propose doing both in sequence. First, develop the desired behavior in prompt space, on a frontier model, with immediate feedback from live traffic. Then, once the behavior converges, distill the recorded production behavior (not synthetic imitations of it) into an SLM through supervised fine-tuning (SFT). We show a proper use case of this method through a scheduling agent for home-care agencies that fans out a single yes/no safety decision per caregiver-shift pair on GPT-5.4-nano. A LoRA-tuned Qwen3-8B trained on 11.6k recorded production decisions reaches 0.913 agreement, while the teacher (GPT-5.4-nano), re-run on its own recorded prompts, agrees with itself only 72.7% of the time, pointing to the fact that the SLM may also be more consistent. Furthermore, when the two disagreed, a blinded professional scheduler sided with the student 48-29. Finally, we find that the SLM has approximately 2.5 times lower median latency and is a fraction of the cost. We believe this lifecycle of quick building, live feedback, and cheap inference is a blueprint for others building similar systems.
| Comments: | Accepted to the NeurIPS 2026 2nd Workshop on Agentic AI Benchmarks and Applications for Enterprise Tasks (AABA4ET) |
| Subjects: | Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2610.14576 [cs.AI] |
| (or arXiv:2610.14576v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14576 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Keegan Wang [view email]
[v1]
Sat, 10 Oct 2026 04:51:24 UTC (168 KB)
来源：arXiv cs.AI · arxiv.org