arXiv cs.AI· Kausar Moshood, Jace Bolante, Roshan Aditya Murali, Manish Motwani·· 3 天前AI 评分31
LEST：基于规范驱动测试的大语言模型安全评估框架
Specification-Based Testing of Large Language Models
AI 导读
研究者提出 LEST 框架，将模型卡与系统卡中的安全声明视为规范，对 LLM 进行黑盒测试。在 Claude Opus 4.8 和 GPT-5.5 的系统卡上，56 条安全声明中仅 29 条（51.8%）可被提示词测试，其中只有 17 条（58.6%）有对应基准提示词覆盖。
正文 · 原文
Abstract:Model cards, system cards, and safety reports increasingly document LLM safety behaviors, yet safety evaluations typically start from predefined benchmarks rather than asking which documented claims those benchmarks actually test. We introduce LEST (LLM Evaluation through Specification-driven Testing), a framework that treats documented safety claims as specifications for black-box testing. In LEST, each claim determines which benchmark prompt tests it and whether a Refuse-to-Comply transition, where a model refuses a prompt but complies with its mutated version, counts as evidence against it. LEST mutates these prompts to search for counterexamples, manually validates each candidate, and traces validated counterexamples back to the claims they contradict. We evaluate LEST using the system cards of Claude Opus 4.8 and GPT-5.5 with 97 seed prompts from SORRY-Bench across Mental Health and Self-Harm, Harmful and Violent Content, and Fairness and Bias. Of 56 identified safety claims, 29 (51.8%) are prompt-testable, but only 17 (58.6%) are exercised by an applicable benchmark prompt. Both models refuse >95% of benchmark prompts. Evaluating 1,836 mutated prompts on both models reveals 312 candidate Refuse-to-Comply transitions flagged by automated response classifiers. However, human validation confirms only 19 (6.1%) as both intent-preserving and genuinely compliant, showing automated evaluation substantially overestimates behavioral failures. Tracing these 19 transitions back to the claims reveals counterexamples for 6 of 9 covered claims for which refusal is an applicable oracle. Our results show that benchmark prompts may not be sufficient to validate all documented LLM safety behavior, and automated failure detection requires manual validation. LEST provides end-to-end traceability from natural-language safety documentation to executable tests and human-validated counterexamples.
| Comments: | 21 pages, 1 figure |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14443 [cs.SE] |
| (or arXiv:2610.14443v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14443 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Manish Motwani [view email]
[v1]
Sat, 10 Oct 2026 01:17:21 UTC (67 KB)
来源：arXiv cs.AI · arxiv.org