Langfail: 82 Real Bugs in a Fake MLOps Platform

TL;DR

• We built Langfail, a working MLOps platform with 82 deliberately planted vulnerabilities and 50 safe look-alikes. It’s open source today.
• We scored eight reproducible runs against it: four static scanners, three LLM reviewers, and an agentic pipeline. The best found 91%. Three cleared 60%.
• LLM reviewers out-recalled the scanners, which is what the literature already says. A fixture like ours flatters them, and we’ll show why.
• Four bugs were missed by every run on the board. They share a trait: none of them contains a dangerous line of code.
• That set is the gap. Nothing in the current tooling landscape is shaped to find it.

Why we built it

Every deliberately vulnerable app we could find teaches 2015. SQL injection in a login form, XSS in a comment box, a file upload that accepts .php. Useful, and thoroughly solved by tools you already run in CI.

Meanwhile the applications people actually ship now load pickled models, run background jobs, hand tools to an LLM, and expose an MCP server. The bugs in those systems don’t look like a login form. They cross process boundaries, sit dormant in a database until a worker picks them up, or live entirely in a config default that nobody set.

We wanted to know what current tooling does with that. Not vendor claims: a fixed target, a written answer key, and the same test for everyone.

What Langfail is

A self-hosted MLOps platform. Model registry, dataset uploads, experiment tracking, a prediction API, an LLM assistant with tools, an MCP server. Flask and SQLite. Type hints, docstrings, passing tests. It looks like something you’d inherit from a colleague who left.

Inside it: 82 planted vulnerabilities across six difficulty tiers, from one disguised line to taint that crosses an HTTP request, a database write, a job queue, and a worker process before it detonates. About a third are AI-specific: prompt injection direct and indirect, agent memory poisoned across users, invisible Unicode smuggled past filters, tool descriptions swapped after approval.

Alongside them: 50 precision decoys. Safe functions parked next to their vulnerable twins, so a reviewer has to read the code instead of grepping for pickle. Flag one and you’ve scored a false positive.

Our favourite is a helper called sanitize_path. It does not sanitize paths. It strips ../ in a single non-overlapping pass, so ....// collapses politely into ../ and strolls out of the artifact directory. It has a reassuring name and a confident docstring. That is the entire trick, and it works on almost everything.

Scanners vs. AI agents

One answer key, eight runs that leave an artifact behind. Every row here was rerun against a committed blind copy, with raw output saved and each finding mapped to a manifest id by a script rather than typed in. Earlier self-reported numbers from other models are deliberately off the board: a score nobody can re-derive is a rumour. The false-positive column counts decoy hits, where a run called one of the 50 safe look-alikes broken.

The interesting result isn’t the ranking. It’s that the two categories fail in opposite directions.

Static scanners are precise and structurally blind. They found the gnarly stuff: a command injection that survives a database round trip and fires in a worker process, a second-order template injection laundered through a cache in base64. Then they walked past a missing security header, because there is no sink to match and no taint to follow. Most of the table’s false positives came from this half, and the ones we checked by hand were the decoys doing their job: crying wolf at an allow-listed column safely bound inside an f-string.

AI reviewers see more and hold it less steadily. The top score is an LLM, which is not news. On benchmarks with planted bugs, LLM reviewers have been out-recalling SAST for a while, and for the obvious reason: they reason about intent instead of pattern-matching sinks, so a bug with no dangerous call is still visible to them.

Worth saying plainly that a fixture like ours flatters them. Give a model a dense, self-contained app with 82 findable bugs and an answer key and it does well. Published work on de-duplicated, chronologically-split real-world code has the same class of model dropping to roughly chance at telling a vulnerable function from its patched twin. Our table confirms the easy result, not the hard one.

The false-positive column still deserves a caveat. It shows the LLM reviewers as cleaner than the scanners, which is backwards from the published result, where models routinely flag half a codebase. Here every row is counted the same way, from a committed per-finding mapping, so the zero is real for this fixture. It is not a virtue of the models: give them a dense app with a fixed answer key and they stay disciplined; point them at a sprawling real codebase and the decoy count climbs.

We graded our own tool on this too, and it’s a good illustration of both halves at once. Rowan, our static engine, has a newer hunt mode that layers LLM triage on top: static findings go in, a second model tries to disprove each one, and a separate stage asks the model for bugs the rule corpus can’t express at all. We turned on its object-level authorization pass and ran hunt against Langfail. The static pass produced its usual candidates, unchanged. None of them reached the LLM stage: they land at medium severity or lower, and triage only opens up medium-severity findings when nothing higher severity exists anywhere else in the app, which on a real codebase is never. Turning the flag on cost nothing and, once triage was involved, gained nothing over the plain scan. The same run’s LLM layer did earn its keep elsewhere: it reversed one of the false positives from our static-only pass, correctly noticing that the call it flagged could only ever render one of two hardcoded filenames.

The durable difference is smaller and duller. A scanner is deterministic: same commit, same rules, same findings, and you can diff two runs. An LLM review is a performance. The three LLM runs here span 41 points, all of them one vendor’s models, a wider gap than separates the best static engine from the worst. It varies between attempts too, and unless you save the output it leaves nothing behind to check.

So the trade isn’t accuracy against noise. It’s coverage against reproducibility.

The gap

Four bugs were missed by every run on the board:

• Full confidence vectors in the prediction API, so you can steal the model.
• A loss endpoint that scores training rows lower, so you can ask it who was in the training set.
• An agent that lets content it just fetched pick its next tool call.
• No framing headers on any dashboard page, so it clickjacks.

Every one is real and has a proof-of-concept test. Not one contains eval, subprocess, or a string-built query. There is nothing to grep for, nothing to taint, and nothing to match.

This is the actual hole in the landscape. SAST looks for dangerous operations. DAST looks for bad responses. Dependency scanners look for known CVEs. None of those categories has a home for “the model leaks its training set” or “this header was never set” or “the agent obeys whatever it just fetched.” The bug isn’t in a line of code. It’s in a default, an absence, or a behavior.

That class is growing, and right now the only tool that reliably finds it is a person who already knows to look.

Try it

Don’t deploy it anywhere real: it writes outside its own directory, the SSRF plants will happily fetch your cloud credentials, and the MCP transport binds every interface with no auth by default. That last one is a feature, in the sense that it is a bug.

git clone https://github.com/hedgerow-dev/langfail
cd langfail
python scripts/export_blind_copy.py /tmp/langfail-blind

Point something at it, map your findings to benchmarks/ground_truth.yaml, and send a PR with your raw output attached.

That last part isn’t bureaucracy. A score nobody can re-derive is a rumour.