arXiv cs.CL· Peidi Dong, Yunze Xiao, Yu-Lin Tsai, Yichen Zheng, Heyuan Huang, Weihao Xuan, Mark Dredze, Jen-tse Huang·· 11 小时前AI 评分31
TASC 基准评估对话助手的时间敏感安全感知能力
Before You Continue: Evaluating Time-Sensitive Safety Awareness in Conversational Assistants
AI 导读
研究提出 TASC 基准，用 3,418 条输入（源自 128 个模板）和 33 个场景评估 LLM 是否在对话中应用时间相关约束。在 13 个模型中，当建议动作违反规则时，助手仅在 30.5% 的情况下考虑了时间约束；62.1% 的失败案例中模型虽正确算出经过时间却未加以应用，明确给出时间信息也只能解决约一半的失败。
正文 · 原文
Abstract:Large language model (LLM) assistants increasingly help users with everyday activities that involve time-dependent constraints. However, users may not mention these constraints in later requests, requiring assistants to recognize and adjust their advice accordingly. To evaluate such capability, we introduce TASC (Temporal Action Safety in Conversation), a benchmark that evaluates whether LLMs apply time-dependent constraints when responding to conversational requests. TASC consists of 3,418 inputs derived from 128 templates, along with a separate set of 33 scenarios designed to compare models' ability to calculate elapsed time with their ability to apply that information. Across 13 models, assistants account for time-dependent constraints in only 30.5% of cases where the proposed action violates a rule, while rarely warning users unnecessarily. More importantly, models fail to apply these constraints in 62.1% of cases despite correctly calculating the elapsed time. Explicitly providing the timing resolves only about half of these failures, revealing a gap between temporal reasoning and its application in conversational advice.
| Comments: | Preprint |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.06839 [cs.CL] |
| (or arXiv:2611.06839v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06839 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jen-Tse Huang [view email]
[v1]
Wed, 4 Nov 2026 17:54:49 UTC (1,448 KB)
来源：arXiv cs.CL · arxiv.org