arXiv cs.CL· Selen Erkan, Bastian Boll, Kristian Kersting, Bj\"orn Deiseroth, Letitia Parcalabescu·· 13 小时前AI 评分31
超越提示词与解析器工程：用软提示微调实现自动化 LLM 评测
Beyond Prompt and Parser Engineering: Soft-Prompt Tuning for Automated LLM Evaluation
AI 导读
研究者提出一种基于软提示微调的自动化 LLM 评测方法，仅优化 10 个软提示向量（7B 模型约 0.0006% 参数）即可让模型适配特定 benchmark 格式，避免格式遵循能力拖累知识得分。在 7 个模型、11 个数据集上，该方法约 150 步（约 1200 样本）即饱和格式遵循能力，效果接近强评测框架基线，且在长上下文数据集和输出风格不同的指令微调变体上评测信号更优。
正文 · 原文
Abstract:Benchmark scores often misrepresent a large language model's (LLM's) knowledge, because they rely, e.g., on the model's ability to follow specific formatting requirements. This especially penalizes base models that may know the correct answers but lack the ability -- typically introduced in post-training -- to structure them as instructed. To overcome this, we propose an efficient, automated, and architecture-agnostic method for model evaluation based on soft-prompt tuning. By optimizing only 10 soft-prompt vectors (~0.0006% parameters for a 7B model) over a short tuning period, we adapt models to specific benchmark formats, closing gaps in format-following and ensuring that underlying knowledge is accurately reflected in benchmark scores. We evaluate soft-prompt tuning across 7 models and 11 datasets. Our results show that (a) soft-prompt tuning saturates format-following within 150 steps (~1200 samples) while preserving knowledge accuracy, (b) it closely matches strong evaluation-harness baselines on well-studied benchmarks while requiring no manual benchmark-specific prompt or parser design, (c) it improves evaluation signal in settings where baselines are harder to transfer, including long-context datasets and instruction-tuned variants with different output styles, and (d) it produces cleaner outputs that improve agreement between exact match scoring and independent LLM judges. Overall, we propose a lightweight, automated, and flexible protocol based on limited-budget soft-prompt tuning for evaluating LLM knowledge.
| Comments: | 9 pages, 7 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2606.12117 [cs.CL] |
| (or arXiv:2606.12117v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.12117 arXiv-issued DOI via DataCite |
Submission history
From: Selen Erkan [view email]
[v1]
Wed, 10 Jun 2026 14:12:19 UTC (652 KB)
[v2]
Thu, 22 Oct 2026 12:34:38 UTC (691 KB)
来源：arXiv cs.CL · arxiv.org