arXiv cs.AI· Jiaqi Gan, Haoyuan Tang, Jamey Z. Liang, Siying Chen, Ankit Raj, Kidus Zewde, Yuchen Zhou, Yuxin Zhang, Simiao Ren·· 14 小时前AI 评分34
CallScreenBench：以小语言模型作为电话秘书的基准测试
CallScreenBench: Benchmarking Small Language Models as Phone Secretaries
AI 导读
CallScreenBench 提出用五个自动化通话与记录度量组评估小语言模型充当电话秘书的表现，聚焦文本域对话决策层，不涉及语音识别与端侧执行。测试覆盖三个模型家族、0.6-4B 的 4-bit 配对检查点，较大检查点在服务、召回与合理性等指标上点估计更高，但分诊区分度排序不同。基准不设 Q1-Q5 综合分或排行榜分数，并单独报告质量指标与戒备性通道，避免单一通过/失败分数掩盖权衡。
正文 · 原文
Abstract:Language models small enough to run on a handset, quantized to a few bits, are increasingly capable of acting on their user's behalf -- which makes on-device task automation newly plausible. One such task is answering the phone. A phone secretary takes an unknown inbound call on its owner's behalf, and unlike the agents most benchmarks evaluate, it has no cooperative caller-assigned task to complete: the caller holds the goal and may be an adversary, while the secretary must begin deciding how to respond without an oracle. What matters is not task success but whether the owner would endorse how their proxy handled the call. We evaluate only the text-domain conversational decision layer; speech recognition, audio interaction, end-to-end latency, and handset execution are outside scope. We present CallScreenBench, which reports five automated call-and-note measure groups motivated by owner endorsement. Each is paired, where available, with a counter-metric and an uncertainty estimate; no benchmark-wide Q1-Q5 composite or leaderboard score is defined. Three guardedness diagnostics identify candidate cases for a toolless proxy that holds no credentials and calls no tools. Across three model families represented by paired 4-bit checkpoints (0.6-4B), the primary scoring snapshot gives the larger checkpoint higher point estimates on several service, recall, and plausibility measures, while triage discrimination follows a different ordering. Bare scam-side TPR rewards universal suspicion, and pairwise separation changes when legitimate-side false positives are included and across judge snapshots. Scripted degenerate agents expose further floors, including a hangup-and-echo policy with entity recall 1.000. We report quality measures and guardedness channels separately so that a single pass/fail score does not hide their trade-offs.
| Comments: | 24 pages, 8 figures |
| Subjects: | Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.01033 [cs.CR] |
| (or arXiv:2608.01033v3 [cs.CR] for this version) | |
| https://doi.org/10.48550/arXiv.2608.01033 arXiv-issued DOI via DataCite |
Submission history
From: Simiao Ren [view email]
[v1]
Sun, 2 Aug 2026 06:31:25 UTC (256 KB)
[v2]
Fri, 21 Aug 2026 21:01:52 UTC (258 KB)
[v3]
Sat, 24 Oct 2026 20:53:19 UTC (259 KB)
来源：arXiv cs.AI · arxiv.org