O'Reilly 调研报告：AI 安全的威胁、工具与技能缺口现状

O'Reilly Radar· Mike Loukides·2026-10-21 19:02· 2 小时前AI 评分57

The State of AI Security: Threats, Tools, and Skills Gaps

AI 导读

O'Reilly 基于 209 名安全从业者（总样本 351 人）的调查报告分析 AI 与安全的交叉现状。最常见攻击是 AI 生成钓鱼（55%）、漏洞扫描利用（30%）和 AI 辅助恶意软件生成（26%）；受访者最担心的风险是过度依赖 AI 未加人工验证和通过 AI 工具泄露敏感数据（各 49%）。

正文 · 原文

Artificial intelligence has changed cybersecurity permanently. While OpenAI’s inadvertent attack on Hugging Face brought AI-generated attacks to everyone’s attention, AI has been on security professionals’ radar for a long time. AI makes attacks possible at a scale that was hard to imagine a few years ago. Newly discovered zero-days can be exploited almost immediately. Deepfakes and their implications for phishing, vishing, and other attacks were nightmares that we knew were real; we just hoped we’d have a few more years before we needed to defend against them. And of course, inattention to security at OpenAI, Anthropic, Google, and other labs that we don’t yet know about has resulted in AI agents that attack other systems in an attempt to solve a problem.

While the headlines have been shouting about AI killing humanity, doomsday AI is more likely a smokescreen to divert attention away from real harms that are happening now. We don’t need panic. We need an assessment of where we are: What are the threats? Where are the skills gaps? Where is the budget? It’s hard to overstate the importance of these questions; Tomasz Tunguz, writing about the Theory 2026 AI Survey, reports that security is the biggest barrier to AI adoption. To start answering these questions, in September we ran a survey that asked infosec professionals about the intersection between artificial intelligence and security. 351 respondents completed the full survey, 209 of whom self-identified as being involved with security as part of their job. This report focuses on those 209 respondents.

Attacks, risks, and responses

52% of respondents said their work rarely (34%) or never (18%) involves AI-generated attacks. This seems unlikely. The most likely explanation is that the security profession includes a broad range of job responsibilities; many security professionals don’t deal directly with defense. That group includes auditors, security experts on software development teams, strategists, architects, and others. It’s also possible that most phishes are either blocked by an automated filter (itself an instance of AI) or deleted immediately by the recipient; many companies require their staff to take training courses that cover topics like recognizing phishes. Still, it’s surprising. Much simpler versions of AI have been used to generate spam and phishing email for years. AI’s ability to generate phishes was demonstrated at BlackHat in 2021 with GPT-3. It’s been part of the security picture for a long time.

The most commonly observed attacks are AI-generated phishing (55%), vulnerability scanning and exploitation (30%), and AI-assisted malware generation (26%). Respondents who frequently encountered AI attacks also reported more types of attacks. There’s no surprise here: If you see attacks more frequently, you are more likely to observe specific attacks. The strongest relationship was with AI-generated phishing, which was seen by 55% of all respondents and 86% of those who observed AI-generated attacks weekly. Among respondents who observed attacks daily, slightly fewer (78%) reported phishing. Given the small sample, the difference may not be meaningful. It’s also likely that respondents who observe attacks daily are seeing a much broader range of attacks and that phishing is genuinely less important to them.

Respondents reported seeing deepfake audio and video, which can be used for vishing and other impersonation attacks, less often than phishing, though it was far from nonexistent (25%). More respondents than we expected reported fully automated attacks, like OpenAI’s attack on Hugging Face, though the percentage was still low (16%). The percentage of respondents who observed adversarial inputs against models including prompt injection was similarly low (only 13%), but that’s high enough to indicate that prompt injection and other attacks directed against models are a real concern.

When we asked respondents what AI-related risks concerned them most, we got a different set of answers. The two most common concerns were overreliance on AI tools without human verification and leaking sensitive data via AI tools; 49% of respondents selected each answer. (The number of respondents selecting those answers happened to be the same, though there was little relationship between the two groups.) 

Although the quality of AI output has gotten a lot better in the past year, relying on unverified AI outputs remains problematic. However, AI can easily generate more output than humans are capable of reviewing, and software teams are now overwhelmed by the volume of AI-generated pull requests that need human review before they can be merged. This overload is causing many to rethink the review process and others to reject AI-generated PRs. Code review was never very good at finding bugs; it was better at familiarizing a team with developments in the codebase. It may be enough to rely on exhaustive testing.

Understanding how your software works when much of it is written by AI is a deeper problem. While AI is good at answering questions about a codebase, it usually doesn’t have access to the reasoning behind the code, which is often hidden in institutional knowledge—and institutional knowledge often means “Oh, yeah, that guy who retired three years ago.” In addition, software developers may lose their programming skills if writing code becomes a smaller part of their job. Security professionals themselves might wonder whether relying on AI for red-teaming, log analysis, and vulnerability detection leads to a degradation in their own skills.

Companies also fear losing sensitive data via AI. Most AI providers offer enterprise accounts that promise that they won’t train models on data in prompts or attached documents, but it’s not clear that these terms of service can be trusted. The frontier laboratories are getting increasingly desperate for training data, even going to lengths such as buying failing startups for their data. Read OpenAI’s description of its Zero Data Retention and Private Safety Processing cynically, and ask if this really protects corporate data or if it allows using data in ways that aren’t directly observable by humans—for example, automatically generating synthetic data. Security professionals are paid to be cynical; thinking about how software, protocols, and policies can be abused is part of the job.

Roughly a third (35%) of the respondents are concerned about attackers using AI to discover vulnerabilities faster than they can mitigate those vulnerabilities and apply patches. Updating software is usually a good practice, but it comes with its own risks: Software updates may introduce new bugs and can lead to incompatibilities with established tools and practices. And defenders can’t yet delegate vulnerability detection, patching, and updating to AI. So overall, is AI more of a help to defenders or attackers? Andrew Ng, Bruce Schneier, and others have argued that the attackers are ahead right now. It’s easier to find a vulnerability, use it to create a payload, and launch an attack than it is to stop one. But the balance will eventually shift to the defenders, as AI becomes more capable of finding and patching vulnerabilities before they can become a problem.

It’s surprising that only a quarter (25%) of the respondents listed AI-generated phishing as a top concern, especially since phishing is the attack most frequently seen in the wild. Perhaps we’ve become too used to phishing; we’ve all been to the security trainings, we’ve all experienced it directly, and some of us have even fallen victim. Phishing is also an attack that, in many cases, is handled automatically. Familiarity may not always breed contempt, but something you see almost daily usually isn’t your biggest worry. 

More exotic attacks are even less of a concern. Only 11% listed prompt injection as one of their top concerns, for instance. However, real-world examples of prompt injection are appearing with increasing frequency. A recent study observed that roughly 1% of résumés contained hidden text to trick an AI system. Google has noticed many kinds of prompt injection, ranging from benign to overtly hostile. And in 2025, OWASP recognized prompt injection as the greatest security risk for users of AI. It’s surprising and unfortunate that it doesn’t appear to be a greater concern for security professionals, especially since prompt injection is a vulnerability that might be lessened by careful filtering but can’t be eliminated.

Even fewer security professionals are concerned about data poisoning (4%). Data poisoning is a serious problem for organizations that train their own models, but today frontier labs conduct much of the most resource-intensive model training. However, this is changing as post-training on corporate data becomes more common. We believe the trend toward locally specialized models will continue, and that data poisoning, as distinct from data theft, will become more of a concern. 

We asked specifically about the use of agents, which exploded this year with the launch of OpenClaw and Gas Town in January 2026. Agents create another set of security concerns. We’ve all heard horror stories: agents that deleted production databases, handled data improperly, generated email that should never have been sent, and the like. Using an agent properly requires a lot of attention to detail, and many users aren’t used to that responsibility, nor do they understand security well enough to exercise responsibility.

The biggest concern among our respondents is excessive permissions or unclear ownership (28%). That’s hardly a surprise; many security problems trace back to permissions. Unclear ownership makes permissions issues more dangerous. When an agent is making many calls to a model, using tools, and delegating to other agents and models, it quickly becomes unclear who (or what) is the authority responsible for any action. This situation isn’t handled well by our current authentication and authorization systems, which assume that identities and authorizations are long-term, stable entities.

Lack of monitoring or auditability was the second highest concern (19%), followed by inability to verify whether the agent stayed within its role (18%). These concerns are closely related: Ownership is often unclear when agents are in use, and lack of monitoring makes the situation worse. Likewise, when you can’t monitor, it’s difficult, if not impossible, to verify whether the agents stayed in their lane—and we’ve seen all too many examples recently of agents that abandoned their authorized roles in inappropriate ways.

Data leakage appears again, though it’s surprisingly low on the list (15%) given that 49% of the respondents indicated that data leakage was one of their top concerns. It’s possible that we’re seeing the difference between single and multiple selection questions; many respondents are concerned about data leakage, but it’s not their top concern. 

Prompt injection doesn’t rank highly for either question: 9% selected it as their top concern about AI agents, and 11% selected it as one of their top concerns. 

Policies and funding

Although nearly three-quarters of the respondents reported that their companies have an AI use policy in place (72%), less than half (44%) reported that the security policy is “comprehensive,” while 28% said that the policy is basic and needs updating. Another 9% said that their employer doesn’t have a policy but is actively developing one. 19% said that there was no policy, or that they “didn’t know,” an answer we consider equivalent to “there is no policy.”

Organizations need clear AI policies that tell employees what AI tools they can use and how they should (and shouldn’t) use them. It’s important for employees to understand whether their company is sensitive to sending data offsite, whether the company’s chosen AI providers guarantee their data won’t be used for training, which practices for attribution are acceptable, and much more. We have some sympathy with organizations that have a basic policy that needs updating. AI’s capabilities are advancing at unprecedented speed; a policy that was adequate in 2023 certainly needs updating now, and a policy that’s only a year old still predates the widespread use of agents like OpenClaw, Claude Code, and Codex, along with the widespread use of skills and tools to extend an agent’s capabilities. Keeping a policy up-to-date with the current state of AI is an imposing task.

The existence of a policy appears to be related to whether employees use AI safely. We asked about “shadow AI” (the use of AI outside of any definitive policy) and found that 36% of the respondents thought that their teammates occasionally use “shadow AI” and 23% thought that the practice is widespread. About a quarter (24%) thought that shadow AI use is rare; and 16% were unsure. 

The relationship between shadow AI and the existence of an AI policy tells us a lot about whether policies are effective. Respondents with a comprehensive AI policy were more likely to say that shadow AI use is rare (37%) or that it’s only used occasionally (36%); only 17% said that shadow AI is widespread at their company. In contrast, 35% of the respondents who reported a basic policy said that shadow AI is widespread—almost the same percentage as those who reported that there was no policy at all (37%) and only slightly worse than those who said that a policy is in development but doesn’t exist yet (28%). Those who had a basic policy, a policy under development, or no policy were less likely to report that shadow AI was used “rarely or never” (13%, 6%, and 21%, respectively).

Granted that shadow AI is probably underreported across the board, our data clearly shows that a comprehensive AI policy is a powerful tool in limiting it. If the policy is poor, or it doesn’t exist, employees will use whatever tools they can to get their job done, which leads to AI use that’s inconsistent, insecure, and possibly counterproductive; it may even lead to legal, contractual, and reputational risks. When there’s no policy, it’s also likely that AI isn’t used as widely as it could be. Telling employees what is expected helps them to act responsibly. A good AI policy doesn’t just tell staff what not to do; it also communicates how to use AI in ways that are appropriate, and may even provide for training. 

When we asked whether organizations are adequately funding AI security, the picture was better than we expected. The largest group (32%) said that AI security is sufficiently funded, and 28% said it’s partially funded. Only 16% said it is severely underfunded, and 23% are unsure. I admit to being surprised by this result. Security is often underfunded, partly because a successful year is one in which “nothing bad happened,” and “we need to spend more so that nothing bad will continue to happen” isn’t a story that plays well at budget time. But executives have been catching on over the last few decades; high-profile attacks and data thefts lead to CIOs and CISOs losing their jobs, and that’s enough to get anyone’s attention. 

We saw a strong link between a comprehensive AI policy and adequate funding for security. Nearly half (49%) of the respondents who said their company has a comprehensive AI policy also said that AI security is sufficiently funded, and another 29% said that it’s partially funded. Only 3% of this group said that AI security is severely underfunded. That means nearly 80% (78%) are at least partially satisfied with funding.

Once again, the answers given by respondents who said that their organizations have a basic policy, a policy under development, or no policy were surprisingly similar. 16%, 22%, and 21%, respectively, said that AI security is sufficiently funded. That’s certainly not a vote of confidence, though we might be able to draw two more bits of information from it. Developing a policy looks like it’s slightly better than having an out-of-date policy; the company is putting in the effort and making an investment. That makes some intuitive sense, though it might also be wishful thinking. Among respondents whose organizations have a basic policy, the largest subgroup said that AI security is partially funded (42%). A middle-of-the-road policy is associated with middle-of-the-road funding. Is that fair, or is that what reality looks like?

Tools and skills

While AI gives security professionals powerful new tools, they aren’t yet fully taking advantage of them. Two-thirds of the respondents (68%) reported that they use a chat, copilot, or similar tool for research or documentation. But only slightly over a third use AI to scan for vulnerabilities (38%), just over a quarter use it for threat detection (27%), and a quarter use AI for incident response (25%). Security professionals widely use general-purpose AI. However, those using specialized tools like vulnerability scanners remain the minority.

To probe further, we asked where AI improved respondents’ workflows. More than half said that AI improved their workflows for research or documentation (61%) and that it reduced repetitive work (57%). But only 29% said that AI improved their workflow for triage, and 17% said that it improved detection accuracy. These results confirm what we saw for tool use.

The relationship between tool use and impact shows up even more clearly when we look at who’s benefiting most from AI. Those who are only using chat or a copilot see the greatest improvement in the speed of research or documentation. Chatbots and copilots save time, particularly when writing reports with a lot of boilerplate text, but they aren’t as helpful when actively defending against or recovering from attacks. Without specialized tools, that work has to be manual.

Respondents using security-specific AI tools were more likely to report improvements in alert triage (42% versus 18%) and detection accuracy (27% versus 5%) than those using only general-purpose chatbots or copilots. 

It’s far from surprising that respondents who didn’t use any AI tools were most likely to say that they saw no noticeable positive impact. The bigger question is why they saw any impact at all. 

Finally, we looked at the “skills gap” to determine what skills and tools would make respondents more effective. Only 7% of the respondents said that they don’t feel like they have a skills gap; that group was also the most likely to say that they aren’t using any AI tools. It’s clear that most of our respondents see AI as a necessary part of the future and want to be part of it. 

34% of the respondents want to use AI tools more effectively. That desire was evenly spread across all AI users, both chat/copilot only (36%) and users of security-specific tools (37%). 24% want to learn how to use AI to secure systems (including the use of AI for red-teaming); 19% want to learn AI-specific attack techniques; and 12% want to learn how to detect AI-generated attacks. 

The future of security

We started by claiming that AI has changed computer security forever. In addition to the many reports of research projects that inadvertently attacked other sites, AI has become a tool for attackers. They aren’t distracted by arguments about whether or not AI is taking jobs; their goal is to exploit their victims, and it’s clear that AI will help them. And as Ben Lorica points out in his newsletter, the economics of attack are changing: AI is infinitely patient, can try multiple attack vectors, can abandon failures, and can work at a scale that’s difficult for humans to match.

AI is also a tool for defenders. We saw more security professionals using AI than we expected, though not as many as we had hoped. Many defenders are taking advantage of chatbots and copilots but haven’t yet adopted more specialized tools, which are associated with improvements in workflows for triage and threat detection.

The tools for defenders undoubtedly need to mature, but tools aren’t the entire answer. Organizations that adopt AI successfully will have appropriate budgets and policies for AI use; they will understand the risks that come with AI adoption; they will proactively fill skills gaps; and they will recognize processes that are no longer productive and “turn scar tissue into infrastructure.” AI is, in the end, a “normal technology” that takes years to penetrate organizations, as Arvind Narayanan and Sayash Kapoor have argued. Tools, budgets, and thought processes all need to change, and that takes time. Getting ahead of the curve on AI won’t happen if individuals work in isolation, adopting tools and ideas as they see fit. It’s an effort that requires the entire organization.

来源：O'Reilly Radar · oreilly.com