IBBench-Light：面向外部指令的任务条件响应配对评估

View PDF HTML (experimental)

Abstract:An external record may contain a procedure to apply or text to read, depending on the user's request. IBBench-Light tests both uses against the same record. Twelve semantic bases yield 144 matched pairs per model; four quantized instruction models produced 1,152 archived greedy responses. Paired exact-contract accuracy (PECA) requires both members to satisfy their output contracts. Qwen succeeds on 132 execute and 109 process prompts, but only 97 complete pairs, showing what marginal averages omit. We audit literal-target exposure and case normalization, then add 1,722 logged CPU generations to test directive-absent controls, twelve additional semantic bases, within-base wording changes, and generation stopping. In the pinned Phi rerun, changing the end-of-sequence (EOS) set changes exact paired success from 0/144 to 62/144. A bounded IHEval comparison uses the same SmolLM2 checkpoint and output budget while preserving its published instruction roles and scorer. The benchmark measures conditional task and output-contract success. Its task margins and paired count need to be read together with the stopping policy.

Comments:	6 pages, 2 figures, 5 table. Accepted at IEEE ACAIT 2026. Author accepted manuscript. Reproducibility artifact: this https URL
Subjects:	Artificial Intelligence (cs.AI); Logic in Computer Science (cs.LO)
Cite as:	arXiv:2609.13725 [cs.AI]
	(or arXiv:2609.13725v3 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2609.13725 arXiv-issued DOI via DataCite

Submission history

From: Kainan Zhou [view email]
[v1] Sat, 12 Sep 2026 05:27:20 UTC (108 KB)
[v2] Wed, 30 Sep 2026 23:59:14 UTC (162 KB)
[v3] Fri, 9 Oct 2026 04:54:13 UTC (162 KB)