# AI-Generated Code Security

> AI-generated code security (AI code security) checks the code, version pins, and package names that LLMs write before they run.

Also known as: AI code security, vibe coding security, LLM-generated code security, AI coding assistant security, secure AI-generated code

Source: https://socket.dev/glossary/ai-generated-code-security

Last updated: 2026-10-24

AI-generated code security, also called AI code security or vibe coding security, is the practice of verifying the code, version pins, and package names that large language models write before they run. Models predict code from training data frozen at a cutoff, so they repeat insecure patterns, pin vulnerable versions, and invent packages attackers can register.

The first-party code a model writes is the part teams know how to review, and it is the smaller problem. The larger one is the dependency layer, where model output meets a live registry the model last saw months ago. In a [May 2026 measurement of 10 models](https://arxiv.org/abs/2605.06279), 36.7% to 55.7% of tasks where a model pinned versions included a version with a known CVE. Between 72.27% and 91.37% of those CVEs were public before the model's training cutoff. Models also invent package names, which turns a hallucination into a [slopsquatting](https://socket.dev/glossary/slopsquatting) target the moment someone registers it.

Our position: every package name and version in AI output is an unverified claim about a registry, and the check that cannot be skipped has to run at install time. Prompts and pull request scans help, but neither sits on the path where a dependency first executes. The humans who accept AI output have largely stopped reading it. Andrej Karpathy, who coined "vibe coding" in February 2025, [described the workflow](https://x.com/karpathy/status/1886192184808149383) in one line: "I 'Accept All' always, I don't read the diffs anymore."

## How does AI-generated code become insecure?

AI-generated code becomes insecure because a code model optimizes for plausible code that runs, and security is mostly invisible to that objective. A model is trained to continue text so that it resembles its training corpus. The feedback it gets in use is whether the code runs and the tests pass. Neither signal measures whether the code is safe.

Three properties of how models work produce four distinct failure modes.

The training corpus contains insecure code. Public code includes years of string-formatted SQL, disabled TLS verification, and hardcoded keys in tutorials. A model reproduces common patterns, including the common bad ones. The result is **insecure logic** in first-party code: injection, path traversal, weak crypto.

The model cannot see your deployment. Authorization often lives in database policies, gateway config, and identity settings that are not in the prompt. A model asked to "add a profile page" writes the query and the UI. It does not know which policy should stop one user from reading another user's row. The result is **missing controls**: code that is correct and unprotected.

The model's knowledge of the package ecosystem is a snapshot. npm and PyPI change by the minute: new versions, new advisories, names registered and removed. A model without a registry lookup only knows what was true at its cutoff. The result is **stale pins**, versions that were current in training and vulnerable now, and **invented names**, packages that never existed.

The first two failure modes are old bugs written faster. The last two are new, because a human choosing a dependency at least looked at the registry first.

## Is AI-generated code safe?

AI-generated code is not safe by default. Controlled studies keep finding that a large share of it is exploitable, and that developers who use AI assistants trust their code more.

The evidence has been consistent across five years of models:

- In 2021, [researchers at NYU](https://arxiv.org/abs/2108.09293) prompted GitHub Copilot with 89 scenarios covering 18 weaknesses from MITRE's CWE Top 25. About 40% of the 1,689 programs it produced were vulnerable.
- In a [Stanford user study](https://arxiv.org/abs/2211.03622) of 47 participants, 36% of those with an AI assistant wrote SQL vulnerable to injection, against 7% of those without one. The authors also found that "participants with access to an AI assistant were also more likely to believe they wrote secure code."
- In 2025, [BaxBench](https://arxiv.org/abs/2502.11844) tested models on 392 backend tasks across 14 frameworks and 6 languages, using expert-written exploits. The best model produced correct code 62% of the time, and about half of the correct programs across models were exploitable.

Human-written code is not safe by default either. What changed is volume. In a [Risky Business episode](https://socket.dev/blog/risky-biz-podcast-ai-agents-open-source-risk) with Feross in February 2026, the discussion noted that some enterprises already report 40% to 50% of their code is AI-generated.

Missing controls are the failure mode scanners handle worst. [CVE-2025-48757](https://mattpalmer.io/posts/CVE-2025-48757/) covered apps generated by the Lovable app builder through April 15, 2025. The generated frontends queried Supabase directly with the public anon key, which is safe only if the database enforces Row Level Security (RLS). Projects that shipped without adequate RLS policies let unauthenticated visitors read, and sometimes write, tables holding user records, API keys, and payment status. Lovable disputes the CVE and says each customer is responsible for securing its own data. Either way, the root cause was not a line of generated code. It was a policy that did not exist, in a database a code scanner never sees:

```sql
-- The control the generated app depended on but did not create.
alter table public.profiles enable row level security;

create policy "users read their own profile"
  on public.profiles for select
  using (auth.uid() = user_id);
```

## AI-chosen dependencies are the larger attack surface

A model that writes `pip install` or a `requirements.txt` is choosing code you will execute, written by people you have not vetted, at a version the model remembers. That choice now happens at machine speed. In [our analysis of npm](https://socket.dev/blog/ai-has-taken-over-open-source), new packages with at least 100 weekly downloads began arriving at a rate unseen in 15 years starting in January 2026. Google searches for "date pickers" fell for four years while downloads of `react-datepicker` kept growing. Developers are no longer shopping for dependencies. Their assistants are. The same analysis found the packages AI tools pick are mostly good ones. The risk is that nobody checks the exceptions.

Stale pins come from what the model saw most in training, not from what is current. The [PinTrace study](https://arxiv.org/abs/2605.06279), posted in May 2026, gave 10 models 1,000 Python tasks. Among the tasks where models specified versions, 36.7% to 55.7% included at least one version with a known CVE. Between 62.75% and 74.51% of those CVEs were rated Critical or High. Models also converge: `django==6.0.1` was specified 400 times across all ten models, and the ten most common vulnerable pins made up 26.34% of all 6,378 vulnerable version assignments. Most of these CVEs were public before each model's cutoff, so the fixes were knowable at training time and the models still chose the old releases. The paper attributes this to a systemic bias toward the same small set of releases.

Convergence turns one stale version into a monoculture. We checked the paper's most common pins against PyPI's JSON API, which embeds OSV advisories for each release. The script is short enough to run before accepting any AI-written manifest:

```python
# vet.py: check package names and pins that an AI assistant wrote,
# against the live registry instead of the model's memory of it.
import json, sys, urllib.request, urllib.error
from datetime import datetime, timezone

def get(url, body=None):
    req = urllib.request.Request(url, data=body, headers={"content-type": "application/json"})
    try:
        with urllib.request.urlopen(req) as r:
            return json.load(r)
    except urllib.error.HTTPError as e:
        if e.code == 404:
            return None
        raise

def age_days(iso):
    t = datetime.fromisoformat(iso.replace("Z", "+00:00"))
    return (datetime.now(timezone.utc) - t).days

def pypi(name, version):
    doc = get(f"https://pypi.org/pypi/{name}/json")
    if doc is None:
        return "NOT ON REGISTRY"
    rel = get(f"https://pypi.org/pypi/{name}/{version}/json")
    cves = {a for v in rel["vulnerabilities"] for a in v["aliases"] if a.startswith("CVE-")}
    return f"{len(cves)} known CVEs, latest is {doc['info']['version']}"

def npm(name, version):
    doc = get(f"https://registry.npmjs.org/{name}")
    if doc is None:
        return "NOT ON REGISTRY"
    created = doc["time"]["created"]
    adv = get("https://registry.npmjs.org/-/npm/v1/security/advisories/bulk",
              json.dumps({name: [version or doc["dist-tags"]["latest"]]}).encode())
    return (f"created {created[:10]} ({age_days(created)} days ago), "
            f"{len(doc['versions'])} version(s), {len(adv.get(name, []))} advisories")

for arg in sys.argv[1:]:
    eco, spec = arg.split(":", 1)
    name, _, version = spec.partition("==")
    check = pypi if eco == "pypi" else npm
    print(f"{arg:<34} {check(name, version)}")
```

```text
$ python3 vet.py pypi:django==6.0.1 pypi:requests==2.31.0 \
    pypi:flask==2.3.3 pypi:djangorestframework==3.14.0
pypi:django==6.0.1                 26 known CVEs, latest is 6.1.2
pypi:requests==2.31.0              3 known CVEs, latest is 2.34.2
pypi:flask==2.3.3                  1 known CVEs, latest is 3.1.3
pypi:djangorestframework==3.14.0   6 known CVEs, latest is 3.18.3
```

As of October 24, 2026, the single pin models agreed on most often, `django==6.0.1`, carries 26 CVEs. Every one of those was fixed in a later release. A [software composition analysis](https://socket.dev/glossary/software-composition-analysis) tool catches these pins, because each one maps to a published advisory.

Removing the pin does not fix the problem. An unpinned or caret-ranged dependency resolves to whatever is newest at install time, and newest is where compromised releases land. On March 31, 2026, [the axios compromise](https://socket.dev/blog/axios-npm-package-compromised) published `axios@1.14.1` and `axios@0.30.4` with a malicious dependency, and any project on `^1.14.0` or `^0.30.0` would pull it on its next install. A [dependency cooldown](https://socket.dev/glossary/dependency-cooldown) closes that window by refusing versions younger than a set age. [pnpm 11 made a 24-hour minimum release age the default](https://socket.dev/blog/pnpm-11-adds-new-supply-chain-protection-defaults), and [npm added minimumReleaseAge](https://socket.dev/blog/npm-introduces-minimumreleaseage-and-bulk-oidc-configuration) in its 11.x releases. A cooldown also delays every security patch by the same amount, so it trades one exposure for another.

The AI-specific fix is to stop letting the model choose versions from memory. PinTrace tested this by injecting a registry-derived version for each library into the prompt. Task-level exposure fell from 36.7% to 55.7% down to 10.9% to 27.6%. It did not reach zero, because some libraries have no CVE-free release and models added libraries outside the injected list.

## How do LLM package hallucinations become slopsquatting attacks?

LLM package hallucinations become slopsquatting attacks when someone registers a name that models repeatedly invent. The next install of AI-written code then fetches the registered package instead of failing with a 404. Slopsquatting is a variant of [typosquatting](https://socket.dev/glossary/typosquatting) that relies on the model's mistake rather than the developer's.

Hallucinated names are frequent and repeatable. [The USENIX Security 2025 study](https://socket.dev/blog/slopsquatting-how-ai-hallucinations-are-fueling-a-new-class-of-supply-chain-attacks) generated 576,000 code samples from 16 models and found that 19.7% of recommended packages did not exist. It counted more than 205,000 unique invented names. When prompts that triggered a hallucination were rerun ten times, 43% of the invented names came back every time.

Newer models hallucinate less often, but they share their mistakes. [A 2026 replication](https://socket.dev/blog/slopsquatting-targets-across-frontier-llms) generated 199,845 responses from five frontier models and measured rates of 4.62% to 6.10%. It found 127 invented names that all five models produced. After review by PyPI Security and our team, 53 of them, 41 on PyPI and 12 on npm, were still open for registration as of April 2026. One registration could reach users of Anthropic, OpenAI, Google, and DeepSeek models at once.

We rechecked the three npm examples from the study, using the same script and `react-datepicker` as a control:

```text
$ python3 vet.py npm:dns-sd npm:dom-ains npm:css-color-stop npm:react-datepicker
npm:dns-sd                         NOT ON REGISTRY
npm:dom-ains                       NOT ON REGISTRY
npm:css-color-stop                 created 2026-07-28 (87 days ago), 1 version(s), 0 advisories
npm:react-datepicker               created 2014-09-10 (4427 days ago), 202 version(s), 0 advisories
```

`css-color-stop` was registered on July 28, 2026, six days after we reported on the replication. The only version, `0.0.0`, is a 53-byte tarball containing nothing but a `package.json`. No code runs. It may be a defensive reservation. The registry cannot tell you which, and that is the problem: a name that existed only in model output is now a package someone else controls, and the next version could contain anything.

> Four outcomes for a package name that appears in AI-written code. Registry state checked October 24, 2026.
>
> Four outcomes when AI-written code references a package name

The figure shows four outcomes when AI-written code references a package, ranked by risk. Safe failure: the name is not on the registry, as with `dns-sd` and `dom-ains` today, and the install fails loudly with a 404. False positive: the name is a valid framework import with no registry entry, such as `@ember/service`, which resolves through `ember-source`, so the code is correct and a naive registry check flags it anyway; our review removed four such Ember imports from the study's npm list. Latent risk: a third party has registered the name without code, as with `css-color-stop` (created July 28, 2026, one 53-byte file), so the install succeeds, nothing runs, and the owner can publish code at any time. Compromise: the name is registered with malware, so the install succeeds and the payload runs with the developer's or agent's privileges. We know of no case of the last outcome among the 53 names.

The registration check has to happen before the install, and the output above shows why the usual checks miss it. A CVE scanner reports 0 advisories for a placeholder and would report 0 for fresh malware, because malware does not get a CVE. A cooldown passes `css-color-stop` too, since the name is 87 days old. The useful signals come from the registry and from the package contents: a name with a single version and no history, appearing in code nobody on the team wrote, and the moment that name's next version adds code that runs.

## How do you secure AI-generated code?

You secure AI-generated code by checking each layer of model output with a control that can see that layer. The dependency checks belong at install time, because that is the first moment a model's choice executes.

> Where an AI-chosen dependency executes, and where each control sits.
>
> Where an AI-chosen dependency executes, and where each control sits

The figure traces an AI-chosen dependency from suggestion to deploy. Step 1: the model or agent writes an import, a manifest entry, or an install command. Step 2: the package manager fetches the package from the registry, and lifecycle scripts can run, though [npm v12 now skips dependency install scripts unless they are allowed](https://socket.dev/blog/npm-12). Step 3: the first import, build, or test run executes the package's code. Step 4: the change is committed. Step 5: pull request checks such as [static application security testing](https://socket.dev/glossary/static-application-security-testing) (SAST) and SCA run. Step 6: the change deploys. Dependency code has already executed by the end of step 3, before any commit exists. Instruction files and MCP lookups act at step 1, a package firewall at step 2, and SAST and SCA at step 5. Nothing checks step 3. An agent working in a throwaway sandbox often stops after step 3, so its dependencies never reach a repository scanner at all.

Each control covers part of the problem:

| Control | Catches | Misses | Cost |
| --- | --- | --- | --- |
| Instruction files (AGENTS.md, editor rules) | Nudges the model toward parameterized queries, popular libraries, current versions | Advisory only. BaxBench found a generic security reminder helped reasoning models and did little for others | Near zero |
| Registry grounding (MCP tools, injected versions) | Stale pins and invented names at suggestion time | Works only when the agent calls it. Models still add unlisted libraries | Low |
| SAST and secret scanning in CI | Injection, unsafe APIs, hardcoded keys in first-party code | Missing authorization in config or database policy. Anything a package does | Medium, mostly triage noise |
| SCA and CVE scanning | Pins with published advisories | Malware, invented names, fresh registrations: all have 0 advisories | Low |
| Install-time package firewall | Known-malicious packages, direct or transitive, before they are fetched | First-party logic. Packages not yet flagged. Installs that bypass it | Low |
| Targeted human review | Authorization, crypto, data access, new endpoints | Does not scale with AI output volume | High |

Verdict: use instruction files and registry grounding to make the model wrong less often. Use SAST and SCA to catch what it got wrong in code you will commit. Make the install-time check the one control that is not optional, because it is the only one on the path every dependency takes, including the ones that never reach a commit.

For grounding, the [OpenSSF guide to AI code assistant instructions](https://best.openssf.org/Security-Focused-Guide-for-AI-Code-Assistant-Instructions), published August 1, 2025, is a reasonable baseline. Rules work better when they name a tool the agent must call and a failure mode it must not paper over:

```markdown
## Dependencies
- Before adding any package, call the `depscore` tool on it and report the result.
- Never pin a version from memory. Resolve the current version from the registry.
- If a package is not on the registry, stop and say so.
  Do not try similar-looking names until one installs.
```

The last rule matters more than it looks. An agent that gets a 404 and retries with near-miss spellings is doing an attacker's typosquat search for them. The `depscore` tool comes from [Socket MCP](https://socket.dev/blog/socket-mcp), a Model Context Protocol server that returns supply chain, vulnerability, maintenance, quality, and license scores for a package before the agent adds it. Like any instruction, it only helps when the agent follows it.

For enforcement, a [package firewall](https://socket.dev/glossary/package-firewall) sits between the package manager and the registry. Socket Firewall Free wraps the package manager command (`sfw npm install`, `sfw pip install`, `sfw cargo fetch`, and likewise for yarn, pnpm, and uv), runs an ephemeral proxy for that process, and refuses to fetch any package, including any [transitive dependency](https://socket.dev/glossary/transitive-dependency), that Socket has confirmed as malware. Its limits are documented: the free tier [blocks only human-confirmed malware](https://docs.socket.dev/docs/socket-firewall-free), warns on AI-detected malware without blocking, lets unscanned versions through, and cannot see packages already in a local cache. Built into an AI app builder, the same check [was blocking around 8,000 packages per day across Replit's users as of June 2026](https://socket.dev/blog/socket-partners-with-replit-to-block-malicious-packages).

## What AI code security cannot see yet

AI code security tooling assumes code passes through a repository. A growing share does not. An agent asked to clean a spreadsheet can install packages into a sandbox, run them, and discard the environment, and [no scanner, proxy, or reviewer sees that supply chain event](https://socket.dev/blog/the-code-you-didnt-write-is-still-yours-to-defend) unless the control sits on the install path. That boundary, what agents execute rather than what models write, is the subject of [agentic AI security](https://socket.dev/glossary/agentic-ai-security).

The registry side has an easier fix that nobody has fully deployed. Hallucinated names are stable and shared across model providers, which makes them cheap for attackers to predict and just as cheap for defenders to precompute. PyPI's prohibited-name list and normalization rules already blocked 68 of the 109 Python names from the 2026 replication. The 12 remaining npm names had no security hold, and the researcher could not reach npm through its published security channels.

npm may be covering part of the gap by accident. Since December 2017, its [moniker rule](https://blog.npmjs.org/post/168978377570/new-package-moniker-rules) has refused new names that match an existing package once punctuation is stripped. `domains` and `dnssd` both exist on npm, so under that rule `dom-ains` and `dns-sd` should be refused. We did not attempt a publish to confirm it. `css-color-stop` has no punctuation-free twin, and it is the one that was claimed within a week of our report. Registries already know how to reserve names that collide with real packages. They should reserve the names models keep inventing, before someone else does.