Langfail: 82 Real Bugs in a Fake MLOps Platform

TL;DR

• We built Langfail, a working MLOps platform with 82 deliberately planted vulnerabilities and 50 safe look-alikes. It’s open source today.
• We pointed 12 scanners, LLM reviewers, and agentic pipelines at it. The best scored 57%. Nothing broke 60%.
• LLM reviewers out-recalled the scanners, which is what the literature already says. A fixture like ours flatters them, and we’ll show why.
• Nine bugs were missed by every run we have detailed records for. They share a trait: none of them contains a dangerous line of code.
• That set is the gap. Nothing in the current tooling landscape is shaped to find it.

Why we built it

Every deliberately vulnerable app we could find teaches 2015. SQL injection in a login form, XSS in a comment box, a file upload that accepts .php. Useful, and thoroughly solved by tools you already run in CI.

Meanwhile the applications people actually ship now load pickled models, run background jobs, hand tools to an LLM, and expose an MCP server. The bugs in those systems don’t look like a login form. They cross process boundaries, sit dormant in a database until a worker picks them up, or live entirely in a config default that nobody set.

We wanted to know what current tooling does with that. Not vendor claims: a fixed target, a written answer key, and the same test for everyone.

What Langfail is

A self-hosted MLOps platform. Model registry, dataset uploads, experiment tracking, a prediction API, an LLM assistant with tools, an MCP server. Flask and SQLite. Type hints, docstrings, passing tests. It looks like something you’d inherit from a colleague who left.

Inside it: 82 planted vulnerabilities across six difficulty tiers, from one disguised line to taint that crosses an HTTP request, a database write, a job queue, and a worker process before it detonates. About a third are AI-specific: prompt injection direct and indirect, agent memory poisoned across users, invisible Unicode smuggled past filters, tool descriptions swapped after approval.

Alongside them: 50 precision decoys. Safe functions parked next to their vulnerable twins, so a reviewer has to read the code instead of grepping for pickle. Flag one and you’ve scored a false positive.

Our favourite is a helper called sanitize_path. It does not sanitize paths. It strips ../ in a single non-overlapping pass, so ....// collapses politely into ../ and strolls out of the artifact directory. It has a reassuring name and a confident docstring. That is the entire trick, and it works on almost everything.

Scanners vs. AI agents

Twelve tools, one answer key. Ten of these numbers are self-reported from our own runs with no raw-output artifact you can audit, so treat them as directional rather than precise. Open-Rowan’s two rows are the exception: rerun against a fresh blind copy, raw output committed, scored by a script instead of typed in. We’ll come back to why that distinction matters more than it looks.

The interesting result isn’t the ranking. It’s that the two categories fail in opposite directions.

Static scanners are precise and structurally blind. They found the gnarly stuff: a command injection that survives a database round trip and fires in a worker process, a second-order template injection laundered through a cache in base64. Then they walked past a missing security header, because there is no sink to match and no taint to follow. Most of the table’s false positives came from this half, and the ones we checked by hand were the decoys doing their job: crying wolf at an allow-listed column safely bound inside an f-string.

AI reviewers see more and hold it less steadily. The top score is an LLM, which is not news. On benchmarks with planted bugs, LLM reviewers have been out-recalling SAST for a while, and for the obvious reason: they reason about intent instead of pattern-matching sinks, so a bug with no dangerous call is still visible to them.

Worth saying plainly that a fixture like ours flatters them. Give a model a dense, self-contained app with 82 findable bugs and an answer key and it does well. Published work on de-duplicated, chronologically-split real-world code has the same class of model dropping to roughly chance at telling a vulnerable function from its patched twin. Our table confirms the easy result, not the hard one.

Our false-positive column we trust least of all. It shows LLM reviewers as cleaner than the scanners, which is backwards from the published result, where models routinely flag half a codebase. That isn’t a finding, it’s a symptom: none of these rows has a per-finding mapping, so nobody ever counted them properly.

We graded our own tool on this too, and it’s a good illustration of both halves at once. Open-Rowan, our static engine, has a newer hunt mode that layers LLM triage on top: static findings go in, a second model tries to disprove each one, and a separate stage asks the model for bugs the rule corpus can’t express at all. We turned on its object-level authorization pass and ran hunt against Langfail. The static pass produced its usual candidates, unchanged. None of them reached the LLM stage: they land at medium severity or lower, and triage only opens up medium-severity findings when nothing higher severity exists anywhere else in the app, which on a real codebase is never. Turning the flag on cost nothing and, once triage was involved, gained nothing over the plain scan. The same run’s LLM layer did earn its keep elsewhere: it reversed one of the false positives from our static-only pass, correctly noticing that the call it flagged could only ever render one of two hardcoded filenames.

The durable difference is smaller and duller. A scanner is deterministic: same commit, same rules, same findings, and you can diff two runs. An LLM review is a performance. It varies by model far more than the tools do (a 40-point spread across six of them here, wider than the gap between the best and worst static engine, and three of those six are the same vendor), it varies between attempts, and unless you save the output it leaves nothing behind to check.

So the trade isn’t accuracy against noise. It’s coverage against reproducibility.

The gap

Nine bugs were missed by every run we kept finding-level records for:

• Full confidence vectors in the prediction API, so you can steal the model.
• A loss endpoint that scores training rows lower, so you can ask it who was in the training set.
• No cap on agent iterations, so anyone can spend your inference budget.
• User emails going to a third-party LLM API unredacted.
• Session tokens accepted in the URL query string.
• No framing headers on any dashboard page.
• Markdown images in assistant output, which is a GET request to anywhere.
• An MCP tool description swapped out after a human approved it.
• Ask the model to repeat a word forever and it returns memorized training data.

Every one is real and has a proof-of-concept test. Not one contains eval, subprocess, or a string-built query. There is nothing to grep for, nothing to taint, and nothing to match.

This is the actual hole in the landscape. SAST looks for dangerous operations. DAST looks for bad responses. Dependency scanners look for known CVEs. None of those categories has a home for “the model leaks its training set” or “this default was never set” or “the approval happened before the swap.” The bug isn’t in a line of code. It’s in a default, an absence, or a behavior.

That class is growing, and right now the only tool that reliably finds it is a person who already knows to look.

We got things wrong too

Grading your own homework is fine if you publish the homework, so: 20 of our 50 decoys turned out to have no callers and no routes. Safe, correct, and completely dead. You cannot catch a scanner crying wolf when there is no wolf, no field, and no sheep. Only 30 were testing anything.

A scanner’s authorization pass then flagged two dozen handlers with genuinely missing ownership checks that our answer key doesn’t list. The tool was right and the key was incomplete.

Both are logged in the repo. Expect more.

Try it

Don’t deploy it anywhere real: it writes outside its own directory, the SSRF plants will happily fetch your cloud credentials, and the MCP transport binds every interface with no auth by default. That last one is a feature, in the sense that it is a bug.

git clone https://github.com/hedgerow-dev/langfail
cd langfail
python scripts/export_blind_copy.py /tmp/langfail-blind

Point something at it, map your findings to benchmarks/ground_truth.yaml, and send a PR with your raw output attached.

That last part isn’t bureaucracy. A score nobody can re-derive is a rumour.