跳到正文
arXiv cs.CL· Terry Yue Zhuo, Dingmin Wang, Hantian Ding, Varun Kumar, Zijian Wang·· 2 天前AI 评分40

CTF-Dojo:用可执行环境训练语言模型智能体发现漏洞

Training Language Model Agents to Find Vulnerabilities with CTF-Dojo

AI 导读

研究者推出 CTF-Dojo,首个面向 LLM 智能体训练的大规模可执行运行时,包含 658 个 Docker 容器化的 CTF 挑战并保证可复现。配套自动化流水线 CTF-Forge 可将公开素材在数分钟内转为可用环境。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large language models (LLMs) have demonstrated exceptional capabilities when trained within executable runtime environments, notably excelling at software engineering tasks through verified feedback loops. Yet, scalable and generalizable execution-grounded environments remain scarce, limiting progress in training more capable ML agents. We introduce CTF-Dojo, the first large-scale executable runtime tailored for training LLMs with verifiable feedback, featuring 658 fully functional Capture-The-Flag (CTF)-style challenges containerized in Docker with guaranteed reproducibility. To enable rapid scaling without manual intervention, we develop CTF-Forge, an automated pipeline that transforms publicly available artifacts into ready-to-use execution environments in minutes, eliminating weeks of expert configuration traditionally required. We trained LLM-based agents on just 486 high-quality, execution-verified trajectories from CTF-Dojo, achieving up to 9.9% absolute gains over strong baselines across three competitive benchmarks: InterCode-CTF, NYU CTF Bench, and Cybench. Our best-performing 32B model reaches 31.9% Pass@1, establishing a new open-weight state-of-the-art that rivals frontier models like DeepSeek-V3-0324 and Gemini-2.5-Flash. By framing CTF-style tasks as a benchmark for executable-agent learning, CTF-Dojo demonstrates that execution-grounded training signals are not only effective but pivotal in advancing high-performance ML agents without dependence on costly proprietary systems.
Subjects: Software Engineering (cs.SE); Computation and Language (cs.CL); Cryptography and Security (cs.CR); Machine Learning (cs.LG)
Cite as: arXiv:2508.18370 [cs.SE]
  (or arXiv:2508.18370v3 [cs.SE] for this version)
  https://doi.org/10.48550/arXiv.2508.18370

arXiv-issued DOI via DataCite

Submission history

From: Terry Yue Zhuo [view email]
[v1] Mon, 25 Aug 2025 18:02:23 UTC (532 KB)
[v2] Tue, 23 Sep 2025 03:30:33 UTC (534 KB)
[v3] Fri, 13 Nov 2026 11:44:46 UTC (537 KB)

来源:arXiv cs.CL · arxiv.org