arXiv cs.CL· Jatin Agrawal, Radhika Mamidi·· 12 小时前AI 评分31
主观文本评估中类型化决策模型的呈现敏感性审计：Jev、Clef Flash 与 Perplexity Decider 研究
Presentation Sensitivity in Typed Decision Models for Subjective Text Evaluation
AI 导读
研究对 Jev、Clef Flash 和 Perplexity Decider 三个类型化决策模型做了受控审计，在六个数据集、9,000 条多重标注的幽默、有用性和摘要条目上收集了 702,000 次判断。
正文 · 原文
Abstract:Typed decision models offer structured choices, scores, and probabilities for low-cost automated evaluation. We present a controlled audit of presentation sensitivity in Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces. Across six datasets with 9,000 multiply annotated humor, helpfulness, and summarization items, we collect 702,000 judgments under interventions that vary positions, identifiers, authorship claims, endorsements, decision framing, and rating rubrics while preserving the evaluated texts. Jev's second-position probability advantage reaches 6.11 percentage points on summaries; reversing reader endorsements changes Perplexity's selected humor text on 40.40% of pairs; and rejection framing reduces Clef's summary agreement with human preferences by 23.36 points. Rating formats also shift score levels while often preserving rankings. Across these findings, repeatability, presentation robustness, and human alignment emerge as distinct properties of an evaluator. Reliable use therefore requires validating the decisions and score interpretations on which an evaluation pipeline depends.
| Comments: | 27 pages, 1 figure, includes appendices |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.06384 [cs.CL] |
| (or arXiv:2611.06384v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06384 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jatin Agrawal [view email]
[v1]
Wed, 4 Nov 2026 11:48:58 UTC (76 KB)
来源：arXiv cs.CL · arxiv.org