IntentTwin：离线网页行为预测中的候选支持混淆与 Oracle 到部署的差距

View PDF HTML (experimental)

Abstract:Offline web-behavior benchmarks frequently insert the demonstrated action into a finite candidate set and then measure reranking. We show that this protocol can create a candidate-support confound: when the target is outside training support while all distractors are sampled from that support, support membership identifies the gold action without modeling behavior. We formalize this effect with support-identifiability rate (SIR), construct support-balanced candidate sets, and evaluate a target-free dual-path proposer that combines support retrieval with open-vocabulary DOM/LM proposals. On filtered OPeRA (4,282/469/992 train/dev/test steps), the original protocol has SIR 1.0000 on target-OOV rows and a support-only rule reaches 1.0000 OOV Exact@1. Support balancing reduces SIR to 0.0000; the strongest conditional reranker obtains 0.7214 overall Exact@1, 0.7988 on IV rows, and 0.5668 on OOV rows. The target-free dual-path system reaches Proposal Recall@16 0.6915 and end-to-end Exact@1 0.5212, compared with oracle conditional Exact@1 0.7450. On Mind2Web, the same audit yields SIR 0.2380 and an oracle-to-proposal gap of 0.2125. A 200-row semantic audit labels 0.7150 of exact-key OOV targets as semantically novel, 0.2450 as locator/string variation, and 0.0400 as ambiguous (Cohen's $\kappa=0.8173$). These measurements show why support composition must be controlled alongside candidate accuracy when offline results are used to reason about deployment.

Comments:	26 pages, 10 figures
Subjects:	Computation and Language (cs.CL); Information Retrieval (cs.IR); Machine Learning (cs.LG)
Cite as:	arXiv:2611.12918 [cs.CL]
	(or arXiv:2611.12918v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2611.12918 arXiv-issued DOI via DataCite

Submission history

From: Bokun Wang [view email]
[v1] Tue, 27 Oct 2026 18:39:23 UTC (1,841 KB)