arXiv cs.CY· Huy Nghiem, Maria Kostylew, Noam Kolt·· 9 小时前AI 评分38
HireWatch:评估 LLM 在上下文压力下对美国雇佣法的合规性
HireWatch: Evaluating LLM Compliance with U.S. Employment Law Under Contextual Pressure
AI 导读
HireWatch 是评估 LLM 是否遵守美国雇佣法面试提问规则的基准,基于美国联邦法规构建了 200 条合法与违法面试问题库。对 18 个闭源与开源权重 LLM 进行约 10 万次试验显示,模型在低压力情境下通常能避开违法问题,但压力升高后违法率急剧上升;明确要求守法的提示词可提升合规率,却无法完全消除违法行为。代码与数据已公开。
正文 · 原文
Abstract:The deployment of large language models (LLMs) in hiring pipelines is growing, yet existing evaluations generally overlook their compliance with legal rules. To fill this gap, we introduce HireWatch, an evaluation that assesses whether LLMs comply with U.S. laws that govern the questions that job candidates may be asked. We construct a 200-item bank of lawful and unlawful interview questions, whose legal classification is based on U.S. federal statutes, regulations, and agency guidance. We instruct LLMs to select questions from this bank in preparing for job interviews, across a set of various occupations and contexts, and report the fraction of unlawful questions that LLMs select. Evaluation of 18 closed and open-weight LLMs across ~100,000 trials under different types of contextual pressure reveals that LLMs' legal compliance is brittle. While LLMs generally avoid selecting unlawful questions in low-pressure contexts, legal violations increase sharply under higher pressure. Explicit prompts to comply with law can improve compliance rates, but do not altogether eliminate legal violations. We publicly release our code and data at this https URL.
| Comments: | NeurIPS 2026 Evaluation and Dataset Track. Law and AI |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2611.24970 [cs.CL] |
| (or arXiv:2611.24970v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24970 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Huy Nghiem [view email]
[v1]
Mon, 16 Nov 2026 03:48:54 UTC (436 KB)
来源:arXiv cs.CY · arxiv.org