arXiv cs.CL· Terry Yue Zhuo, Dingmin Wang, Hantian Ding, Varun Kumar, Zijian Wang·· 2 天前AI 评分40
CTF-Dojo:用可执行环境训练语言模型智能体发现漏洞
Training Language Model Agents to Find Vulnerabilities with CTF-Dojo
AI 导读
研究者推出 CTF-Dojo,首个面向 LLM 智能体训练的大规模可执行运行时,包含 658 个 Docker 容器化的 CTF 挑战并保证可复现。配套自动化流水线 CTF-Forge 可将公开素材在数分钟内转为可用环境。
正文 · 原文
Abstract:Large language models (LLMs) have demonstrated exceptional capabilities when trained within executable runtime environments, notably excelling at software engineering tasks through verified feedback loops. Yet, scalable and generalizable execution-grounded environments remain scarce, limiting progress in training more capable ML agents. We introduce CTF-Dojo, the first large-scale executable runtime tailored for training LLMs with verifiable feedback, featuring 658 fully functional Capture-The-Flag (CTF)-style challenges containerized in Docker with guaranteed reproducibility. To enable rapid scaling without manual intervention, we develop CTF-Forge, an automated pipeline that transforms publicly available artifacts into ready-to-use execution environments in minutes, eliminating weeks of expert configuration traditionally required. We trained LLM-based agents on just 486 high-quality, execution-verified trajectories from CTF-Dojo, achieving up to 9.9% absolute gains over strong baselines across three competitive benchmarks: InterCode-CTF, NYU CTF Bench, and Cybench. Our best-performing 32B model reaches 31.9% Pass@1, establishing a new open-weight state-of-the-art that rivals frontier models like DeepSeek-V3-0324 and Gemini-2.5-Flash. By framing CTF-style tasks as a benchmark for executable-agent learning, CTF-Dojo demonstrates that execution-grounded training signals are not only effective but pivotal in advancing high-performance ML agents without dependence on costly proprietary systems.
| Subjects: | Software Engineering (cs.SE); Computation and Language (cs.CL); Cryptography and Security (cs.CR); Machine Learning (cs.LG) |
| Cite as: | arXiv:2508.18370 [cs.SE] |
| (or arXiv:2508.18370v3 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2508.18370 arXiv-issued DOI via DataCite |
Submission history
From: Terry Yue Zhuo [view email]
[v1]
Mon, 25 Aug 2025 18:02:23 UTC (532 KB)
[v2]
Tue, 23 Sep 2025 03:30:33 UTC (534 KB)
[v3]
Fri, 13 Nov 2026 11:44:46 UTC (537 KB)
来源:arXiv cs.CL · arxiv.org