arXiv cs.AI· Anil Srirangapatna Nagesh, Nitin Gupta, Anuj Taneja, Vivek Singh, Shane Eleniak, Theo Zhao, Moulidharan N, Shashank Keshetty·· 12 小时前AI 评分31
用 RSOD 构建电信语言模型：领域数据、专家后训练、内存感知蒸馏与网络评测
Building Telecom Language Models with RSOD: Domain Data, Specialist Post-Training, Memory-Conscious Distillation, and Networking Evaluation
AI 导读
研究提出端到端框架 RSOD，从网络分类体系构建 212,559 条路由训练记录，在共享继续预训练骨干上训练六个冻结的 LoRA 专家教师，再蒸馏为单一可部署学生适配器 Llama-CalixCLM v1。
正文 · 原文
Abstract:Telecom language models must combine broad networking knowledge with specialist behavior across routing, broadband services, access technologies, and operational diagnostics while remaining practical to deploy as a single model. We present an end-to-end framework that builds source-grounded synthetic training data from structured networking taxonomies and expert graphs, materializes 212,559 routed training records across fourteen networking routes, trains six frozen specialist LoRA teachers over one shared continued-pretrained backbone, and consolidates their behavior into one deployable student adapter using Routed Specialist On-Policy Distillation (RSOD). In RSOD, the committed student generates each trajectory, the route-assigned specialist teacher-forces the exact native token sequence, compact teacher hidden states are reconstructed through the shared frozen output head, and only the student LoRA is updated. A controlled reconstruction validation shows mean absolute logit error of 7.26 x 10^-6 with 100% top-1 agreement between native and reconstructed teacher logits. On four RTX 6000 Ada GPUs, the completed 2,000-step run produced 16,000 trajectories with zero observed student-version or token- ID mismatches. Under the primary route-conditioned evaluation over 2,850 held-out questions, the raw Meta-Llama-3.1-8B-Instruct model scores 12.8661/25 micro (13.0172/25 macro), the combined routed six-teacher system scores 21.7641/25 micro, and the consolidated RSOD student, released as Llama-CalixCLM v1, reaches 21.7112/25 micro (21.8006/25 macro) on the same five dimension LLM-judge rubric. Under a separate neutral deployment-like prompt, the same student reaches 20.0028/25 micro (20.0778/25 macro). The consolidated student remains within 0.0529 micro-score points of the combined routed-teacher system while retaining a numerically validated and auditable train-serve execution path.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| ACM classes: | I.2.6; I.2.7; C.2.3 |
| Cite as: | arXiv:2610.27993 [cs.AI] |
| (or arXiv:2610.27993v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27993 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: AnilKumar Srirangapatna Nagesh [view email]
[v1]
Tue, 20 Oct 2026 10:30:58 UTC (7,373 KB)
来源：arXiv cs.AI · arxiv.org