arXiv cs.AI· Sirisilp Kongsilp·· 10 小时前AI 评分36
小语言模型从用户反馈中学习:何时有效,何时衰减
Learning through Discernment: when a small language model learns from user feedback on the job, and when it decays
AI 导读
研究用 Qwen3 0.6B 和 1.7B 加 LoRA 适配器,在 WebShop 和 τ²-bench 上模拟最多一年的 127 次“夜间学习”,发现模型自评判会先提升后衰减。
正文 · 原文
Abstract:Small language models on devices could keep improving by learning, overnight, from their own daily work. Without a true outcome, however, something must decide which attempts are worth learning from, and both the model's own judgment and the user's reaction can be wrong. We study a small model (Qwen3 0.6B and 1.7B) with a LoRA adapter, fine-tuned each night on the attempts a judge accepts, over 127 lives of up to a year on WebShop and $\tau^2$-bench. We compare the true outcome, the user's stated verdict and the model's own judgment with a separate, frozen "feedback sense" that reads the user's reaction, built in three versions. Judging itself, the model improved and then decayed, over a year and whatever its adapter. A sense that took polite give-ups for success collapsed 3 of 5 lives; one built for discernment, which first asks whether the user gave a clear verdict, kept 4 of 5 near the oracle for two weeks, then faded on clean but scarce lessons, to the level of self-judging by day 60. When the user cannot see the outcome ($\tau^2$-bench), the sense read reactions correctly, yet 80% of its lessons were failures the customer had thanked. Across 97 lives, the share of failed attempts among the lessons predicted the outcome (Spearman $\rho$ = -0.81). In our experiments, learning from users held up only where the judge kept failures out and the user could see the outcome; lopsided lessons may also narrow the model.
| Comments: | 34 pages, 11 figures. Code and data: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| ACM classes: | I.2.6; I.2.7 |
| Cite as: | arXiv:2611.23716 [cs.LG] |
| (or arXiv:2611.23716v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23716 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sirisilp Kongsilp Ph.D. [view email]
[v1]
Wed, 18 Nov 2026 02:18:51 UTC (169 KB)
来源:arXiv cs.AI · arxiv.org