arXiv cs.CL· Federico Lucca·· 12 小时前AI 评分34
训练前先测量：类型化决策模型、经典基线与生成式 LLM 在个人多语言收件箱路由上的对比
Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox
AI 导读
自托管个人收件箱智能体 Foyer 在 739 封真实意大利语/英语邮件上对比了九套系统，生产关键词规则在邮件类型上仅 28% 准确率，而多数类基线为 58%。
正文 · 原文
Abstract:Personal email agents decide, for every message, what it is, whether it is urgent, whether to draft a reply, create a task, or ignore it. Their error rate is rarely measured: the stored labels were written by the system itself. We present Foyer, a self-hosted personal inbox agent in daily use, and a measurement protocol (stratified sampling, frozen snapshots, thread-grouped splits, pre-registered hypotheses, human labels in the agent's review loop) that yields ground truth before any training. On 739 real Italian/English emails labelled by the owner (231 unbiased, 500 adversarially sampled), we compare nine systems under thread-grouped cross-validation: the production keyword rules, a TF-IDF + logistic-regression baseline, local typed decision models (Tev1, Nimble), a hosted decision model (Nimble v4-max), and hosted generative LLMs (Qwen2.5-7B, Gemma 4 26B/31B). Four results. The production rules score 28% on message type against a 58% majority class: the field shown most prominently carried no information. With ~200 labels, logistic regression has the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90), though it never predicts the two rarest classes and is statistically tied with Gemma 4 on type; few-shot prompting does not close the gap. The sender's relationship (client, supplier, service) is recovered from other emails of the same contact with 98.6% accuracy where the contact has been seen, and is not inferable from a single message by any model: it belongs in the contact graph, not the prompt. Finally, accuracy is not the product metric: composed into actions, the most accurate system most often ignores emails needing action (18% vs 4% for the best LLM); a cost-chosen threshold on its noise probability halves that rate and makes it the cheapest system.
| Comments: | 8 pages, 7 tables. Single-user case study. Code and protocol: this https URL . The emails are private and not released |
| Subjects: | Computation and Language (cs.CL); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG) |
| ACM classes: | I.2.7; H.5.2 |
| Cite as: | arXiv:2610.32955 [cs.CL] |
| (or arXiv:2610.32955v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32955 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Federico Lucca [view email]
[v1]
Fri, 23 Oct 2026 17:49:43 UTC (19 KB)
来源：arXiv cs.CL · arxiv.org