arXiv cs.CL· Volodymyr Ovcharov·· 12 小时前AI 评分40
法律多选题基准的“仅选项可解性”：模型不看题干也能答对多少
Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks
AI 导读
研究在乌克兰法官高等资格委员会发布的 UA-JudgeExam（11,990 道四选项题）上测量“仅给选项、不给题干”的可解性：Claude Haiku 4.5 得分 0.383，11.8% 的题目在全部八种选项顺序下都能盲答，而随机预期仅 0.2 题。
正文 · 原文
Abstract:Multiple-choice benchmarks are graded on whether a model picks the right option, not on whether it needed the question. Measuring that gap takes care: a model answering A to most items scores above chance wherever the key sits at A, and reads as recognition when it is not. We measure it on UA-JudgeExam: 11,990 four-option items with official keys, published by Ukraine's Higher Qualification Commission of Judges.
Shown the options and no question, Claude Haiku 4.5 scores 0.383 against chance, and the leak is concentrated: 11.8% of items are answered blind on all eight option orders, against 0.2 items expected by chance. It is not quotation: search over 280,059 editions of Ukrainian legislation recovers 0.128. Gating those out retains 8,128 items, on which the gating model itself now scores 0.204, while GPT-5.6, which took no part in the selection, still answers 0.515 of them with the question hidden. Scoring eighteen held-out models from eight vendors on the whole set and subtracting each one's position habit, four keep a sizeable excess: GPT-5.6 (+0.265), Gemini 3.1 Pro (+0.240), GPT-5.4 (+0.105), Sonnet 4.6 (+0.081); the rest stay within 0.031, and probes find no recall of the public bank behind it. Without the correction Llama 3.1 8B (0.292 blind, A on 92% of items) would read as a leaker.
The gate does select something real: on the items it rejected, eleven of twelve models score 0.518-0.789, every interval clear of what the same model scores on the items it kept. But that signal is one model's, and filtering on it does not transfer upward. Rewriting distractors instead overshoots to 0.168, below chance and as exploitable. The same probe on LEXam returns chance: every option there points into the stem, none longer than 33 characters. Item format decides whether the problem can arise; capability decides how much is extracted. We release the corpus, predictions and harness.
| Comments: | 23 pages, 4 figures, 12 tables. v2: five further models and a pre-publication GPT-4o control, recall probes, corrected answer parser, re-extracted text. Data and code: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2608.15428 [cs.CL] |
| (or arXiv:2608.15428v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.15428 arXiv-issued DOI via DataCite |
Submission history
From: Volodymyr Ovcharov [view email]
[v1]
Sat, 15 Aug 2026 22:11:00 UTC (32 KB)
[v2]
Sun, 1 Nov 2026 22:43:39 UTC (37 KB)
来源：arXiv cs.CL · arxiv.org