RiskyPy：面向 LLM 生成代码安全与功能联合评估的模型校准提示词生成

RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code

AI 导读

RiskyPy 是一套自动化流水线，利用功能与漏洞金丝雀样本生成按模型能力校准的安全导向代码生成数据集。其 Python 版本覆盖 2025 CWE Top 25 中的 18 类 CWE，共生成 101 条提示词；在 Qwen3-4B 和 Seed-Coder-8B 上，57-72% 的提示词产出可运行代码，仅 5-10% 的生成结果不含 Bandit 告警。

正文 · 原文

View PDF HTML (experimental)

Abstract:Evaluating approaches that improve the security of LLM code generation requires datasets that jointly assess security and functionality and remain adapted to the capabilities of the models being studied. However, existing benchmarks are costly to produce and are generally not calibrated to a chosen range of model capabilities. We introduce RiskyPy, an automated pipeline that generates security-oriented code-generation datasets calibrated to model proficiency using functionality and vulnerability canaries. We instantiate RiskyPy for Python using domains covering 18 CWEs from the 2025 CWE Top 25, producing 101 prompts. Experiments on Qwen3-4B and Seed-Coder-8B show that the generated tasks remain functionally accessible, with 57-72% of prompts yielding functional code, while only 5-10% of generations are free of Bandit findings, leaving room for joint evaluation of security and functionality of security improvement approaches.

Subjects:	Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
Cite as:	arXiv:2610.40735 [cs.CR]
	(or arXiv:2610.40735v1 [cs.CR] for this version)
	https://doi.org/10.48550/arXiv.2610.40735 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Daniele Lunghi [view email]
[v1] Thu, 29 Oct 2026 14:03:12 UTC (184 KB)

来源：arXiv cs.AI · arxiv.org