arXiv cs.CL· Johann Emmanuel Li·· 2 天前AI 评分27
用小型宽松许可开源模型实现校准的证据推断
Calibrated Evidence Inference With a Small, Permissively Licensed Open Model
AI 导读
研究者用低秩适配器和分类头微调 Qwen3-4B-Base，用于证据推断任务，判断试验结果是显著增加、显著减少还是无显著差异。该模型仅用 2,657 篇训练文章中 1,431 篇 CC BY、CC0 或公有领域文章微调，以 Apache License 2.0 发布并可在笔记本上运行。
正文 · 原文
Abstract:Systematic reviewers who use a model to read randomized trials need each answer and a measure of how far to trust it. If the model is calibrated, so that its stated probabilities match how often it is right, a reviewer can accept its confident answers and check the rest by hand. This paper fine-tunes Qwen3-4B-Base with low-rank adapters and a classification head for evidence inference (Lehman et al., 2019; DeYoung et al., 2020): given a trial's article and a prompt that names an intervention, a comparator and an outcome, classify the result as a significant increase, a significant decrease or no significant difference. The articles of Evidence Inference 2.0 come from the open-access subset of PubMed Central, which includes articles under non-commercial and no-derivatives licenses, so the released model was fine-tuned only on the 1,431 of 2,657 training articles under CC BY, CC0 or the public domain; it is released under the Apache License 2.0 and runs on a laptop. On the test split, its expected calibration error is 0.017 and its log loss is lower than that of Gemma 4 31B-it given the same input zero-shot, meeting criteria registered on the Open Science Framework before testing. Its macro-F1, 0.925, is above every published result on that split that I found, though those systems used other inputs, scoring rules and prompt filters, and the test labels are public. Accepting its 80 percent most confident answers leaves an error rate of 2.6 percent among them. Context ablations show that the answers depend on the article, but the title alone carries some signal, which an informative title or recall of the trial could explain.
| Comments: | 22 pages, 2 figures, 15 tables. Code, results files and the paper's sources: this https URL. Model: this https URL. Registration: this https URL |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.07019 [cs.CL] |
| (or arXiv:2610.07019v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07019 arXiv-issued DOI via DataCite |
Submission history
From: Johann Li [view email]
[v1]
Sun, 4 Oct 2026 15:09:39 UTC (40 KB)
[v2]
Thu, 15 Oct 2026 12:12:43 UTC (42 KB)
来源：arXiv cs.CL · arxiv.org