arXiv cs.CL· S. M. Mahbub Uddin Ahmed, Abdullah Al Musaddiq·· 11 小时前AI 评分31
DMReranker：按子句拆分指令的零样本重排序方法
Every Clause Counts: Instruction-Following Reranking by Multiplying Small, Answerable Questions
AI 导读
DMReranker 是一种零样本重排序器，将带指令的查询拆成主题和若干带类型的子句，由小型评判模型逐条回答并把答案概率相乘得到分数，全程无需训练。在 InfoSearch 六个维度上其 macro WISE 达 50.9，原模式 nDCG@10 保持 82.9，但严格指令合规率 SICR 为 19.2，低于 GPT-4o 的 32.1；按排名计算则为 67.0。
正文 · 原文
Abstract:People often attach instructions to their search queries, asking for pages written for beginners, documents in a certain language or official sources only. Most rerankers were trained to judge topical relevance and treat such instructions as a few extra words. On the InfoSearch benchmark, even GPT-4o used as a list-wise reranker reaches an average WISE (Weighted Instruction Sensitivity Evaluation) of only 33.5. We present DMReranker (Decision-Model Reranker), a zero-shot reranker that splits an instructed query into a topic and a short list of typed clauses. A small judge model answers one narrow question per clause, and the answer probabilities are multiplied into a single score. Nothing is trained, and the configuration was frozen before the test runs. Reranking the same top-100 candidates as the published baselines, DMReranker reaches a macro WISE of 50.9 over all six InfoSearch dimensions while keeping original-mode nDCG@10 at 82.9. It trails GPT-4o on the stricter SICR (Strict Instruction Compliance Ratio) metric, 19.2 against 32.1. We show that this gap follows from how a product of probabilities behaves when SICR compares raw scores across runs. Computed from ranks, our SICR is 67.0. On the same judge, splitting the instruction raises WISE from 32.9 to 47.6 over a single combined question, while raw-score SICR falls. Scoring all six dimensions cost about $10 at list price.
| Subjects: | Information Retrieval (cs.IR); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.03179 [cs.IR] |
| (or arXiv:2611.03179v1 [cs.IR] for this version) | |
| https://doi.org/10.48550/arXiv.2611.03179 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: S. M. Mahbub Uddin Ahmed [view email]
[v1]
Mon, 2 Nov 2026 14:15:23 UTC (88 KB)
来源：arXiv cs.CL · arxiv.org