arXiv cs.CL· Md Rejaul Korim Sadi, Golam Mostofa Naeem, Toufiqur Rahman Tasin, Syed Mostofa Moosa, Mahmudul Hasan Emon, Mahmudur Rashid, Ferdus Ahmed·· 10 小时前AI 评分29
研究：语言模型与启发式规则对移动 UI 质量评分的一致性差异
Characterising divergence between heuristic and language-model ratings of mobile user interface quality
AI 导读
研究基于 RICO 数据集构建 15,000 屏语料，用文献启发式评分器与三个开放权重语言模型在两种输入表示下共产生 270,000 条评分，发现两类评分源一致性持续偏弱：18 个模型-表示-维度组合中有 17 个原始 kappa 低于 0.15。
正文 · 原文
Abstract:Automated assessment of mobile user interface quality is attractive because manual review does not scale, and large language models are increasingly proposed for the task. Whether their ratings correspond to those produced by established rule-based heuristics has not been characterised at scale. We construct a 15,000-screen corpus from the RICO dataset using six documented filters, score every screen with a literature-grounded heuristic scorer, and collect ratings from three open-weight language models across two input representations, yielding 270,000 ratings. We report linearly weighted Cohen's kappa with bootstrap confidence intervals, offset-corrected kappa, intraclass correlation, and Bland-Altman limits of agreement, together with a chance baseline for the tolerance-band agreement rule. Agreement between the two rating sources is consistently weak: raw kappa falls below 0.15 in seventeen of eighteen model-representation-dimension cells. However, fifteen of eighteen cells are statistically distinguishable from zero, and correcting for a constant offset raises five cells above 0.15, so the divergence is better described as a combination of systematic bias and weak covariation than as absence of relationship. Model ratings track structural properties of the interface, but substantially less strongly than the heuristic does, and more strongly from an untruncated structural element list than from a screenshot. We further show that dimensions elicited in a single generated response are correlated by construction, reaching rho = 0.97 in one condition, which constrains what such designs can measure. We report these results as a characterisation of divergence between two automated raters rather than as evidence that either is correct, and we set out the validation a stronger claim would require.
| Comments: | 15 pages, 4 figures, 12 tables (incl. appendix). Data and code in ancillary files. An earlier abstract was presented at EFAST 2026 under the title "Seeing Without Understanding: Large Language Model Evaluation of Mobile User Interface Quality, Failure Taxonomy, and Architectural Explanation" |
| Subjects: | Human-Computer Interaction (cs.HC); Computation and Language (cs.CL); Software Engineering (cs.SE) |
| Cite as: | arXiv:2609.05423 [cs.HC] |
| (or arXiv:2609.05423v2 [cs.HC] for this version) | |
| https://doi.org/10.48550/arXiv.2609.05423 arXiv-issued DOI via DataCite |
Submission history
From: Md. Rejaul Korim Sadi [view email]
[v1]
Sun, 24 May 2026 17:29:48 UTC (715 KB)
[v2]
Mon, 9 Nov 2026 21:15:50 UTC (1,327 KB)
来源：arXiv cs.CL · arxiv.org