Vulnerabilities in AI-Generated Code: A New Risk for IT Security?

Artificial intelligence has firmly established itself as a useful tool in software development — but it brings its own risks. Recently I generated a small script with ChatGPT myself. At first glance everything looked fine, but on closer inspection I noticed several variables and functions that served no real purpose. A quick analysis made it clear: such redundant code elements aren't just untidy — depending on where they end up, they can create additional attack surface and open up unexpected security holes.

The crucial point: I only noticed because I actually reviewed the code. And that is exactly where the problem lies.

Why AI Produces Insecure Code

One of the most common misconceptions about generative AI — particularly large language models (LLMs) like ChatGPT — is the assumption that these models understand and interpret a question the way a human does. In reality, they do nothing more than predict, based on their training data, what the most plausible next output is — whether answering a question or completing a block of code. Without genuine understanding of context or security, a model will happily suggest code with insecure functions, missing input validation, or hardcoded credentials.

This is not a fringe phenomenon. Veracode's 2025 GenAI Code Security Report tested more than 100 language models and found that roughly 45% of AI-generated code contained vulnerabilities from the OWASP Top 10 (Help Net Security). Other analyses report a vulnerability density around 2.7 times higher than human-written code. Notably, newer and larger models performed barely better — raw compute alone does not solve the security problem.

The Real Risk: a False Sense of Security

More dangerous than the insecure code itself is often the trust placed in it. Studies show that developers working with AI assistance tend to write less secure code — while feeling more secure about it. A widely cited Stanford University study found exactly this effect: those using AI produced more vulnerabilities and at the same time rated their own solutions as more secure. So AI doesn't just generate vulnerabilities; it also creates a false sense of confidence that undermines the critical review a developer would otherwise have applied.

This closes the loop back to the script I mentioned at the start: the redundant code would never have been spotted by anyone who copies it in unchecked.

A Genuinely New Risk: "Slopsquatting"

Some risks are old acquaintances in new clothing — others are genuinely new. A good example is "slopsquatting," a term coined by security researcher Seth Larson as a play on the well-known "typosquatting."

The background: LLMs have a tendency to hallucinate package names — that is, to suggest libraries that don't actually exist. A study covering 16 code models and hundreds of thousands of samples found that around 20% of the recommended packages were not real. The critical detail is that these hallucinations aren't random: across repeated prompts, many of the same invented names surfaced again and again. Attackers exploit precisely this predictability. They register frequently hallucinated package names — loaded with malicious code — on public registries such as PyPI or npm. Install the AI's suggestion unchecked, and you pull malware straight into your supply chain (BleepingComputer). MITRE ATT&CK tracks this path under "Compromise Software Supply Chain" (T1195).

A few years ago this attack vector simply didn't exist. It is a direct byproduct of widespread AI use — and a clear answer to the question in the title: yes, AI-generated code does introduce new risks.

What You Can Do About It

As a general rule, AI-generated code should be subjected to the same rigorous security checks as human-written code — including pentests and source code reviews. This Forbes article offers a worthwhile overview.

For the short scripts you generate quickly in day-to-day development, however, a full security review would defeat the very purpose of using AI — namely, getting working code quickly and efficiently. For those cases, a few simple but effective measures to reduce vulnerabilities right at generation time:

Demand security from the start. Explicitly instruct the AI to produce secure code and to follow the OWASP Secure Coding Practices.
Have the code explained. Ask the AI to walk through the generated code step by step — this quickly surfaces redundant or unclear elements.
Turn the AI against itself. Have the model review its own code for vulnerabilities and improve it according to security best practices.
Verify dependencies. Check every package the AI suggests before you install it: does it really exist, who publishes it, and how long has it been registered? This is the simplest defense against slopsquatting.

These steps don't replace a full security review like a pentest or a source code review — but they noticeably raise code quality during development.

When It Gets Serious

When it's time for the actual security review, this is where I come in. At HSEC.Consulting I provide technology- and industry-specific expertise — whether for a pentest, a source code review, or a well-grounded assessment of how to integrate AI safely into your development processes. 😉

#CyberSecurity #AI #KISicherheit #SecureCoding #Pentesting