Open source

Two tools we gave away

A scanner for model files, and a vulnerable app to test scanners against. Both MIT licensed, both built out of work we needed done for ourselves.

Hayward

Security scanner for machine-learning model files. Know whether a checkpoint will run code on your machine, before you load it.

torch.load, joblib.load and numpy.load(allow_pickle=True) execute code from the file they read. That is not a bug, it is what pickle does. The file is named pytorch_model.bin, and the .bin is doing a lot of work in that sentence. It is a program, and loading it runs it, with your credentials, as you.

Hayward reads the file without executing it and tells you what it found.

pip install hayward
hayward scan ./models
Scans
PyTorch, joblib, NumPy, TorchServe, NeMo and skops on the pickle side. SafeTensors, GGUF and TFLite as tensor containers. ONNX, TensorFlow SavedModel, Keras and PMML as graph formats. 24 extensions in total, and format comes from magic bytes, so a payload renamed weights.safetensors does not walk past on its extension.
Who for
Teams pulling checkpoints from public hubs who want the check to run in a CI pipeline, on a laptop, or inside an environment where nothing is allowed to leave the network.
Approach
Unknown callables are judged by the arguments they were handed, not by an allowlist of names. A URL, a shell command, a host and port. That is what generalises past a deny list. A file it cannot parse produces an explicit finding rather than silence, because a clean report should mean the file was read.
Operating model
One dependency, no model framework, no native extensions, no outbound calls. Python 3.10 and up. CLI, HTML report, desktop window, or a Python API. Documented exit codes and a threshold you set, so it fits a build.
Scope
A clean result means Hayward read the files and recognised nothing dangerous in them, not that the model is safe. It is a smoke alarm, not a survey of the building. The repository documents what it does not cover.
MIT Python 3.10+ 42 rules 1 dependency No outbound calls
GitHub PyPI Try it in your browser

Langfail

A working MLOps platform with 82 vulnerabilities planted in it on purpose. Target practice for scanners, agents, and people who think they are good at code review.

Flask and SQLite, with a model registry, dataset uploads, experiment tracking, a prediction API and a built-in LLM assistant. The bugs are spread across web classes, LLM and agent attacks, authentication, supply chain, and several IDOR variants. Each one is verified exploitable.

The answer key is kept outside the code, so a scanner or an agent can be pointed at the app blind and graded afterwards against a fixed list.

git clone https://github.com/hedgerow-dev/langfail
pip install -e .
flask --app langfail seed
flask --app langfail run
Why it exists
We wanted to measure Rowan against something we had not written the rules for, and every benchmark we could find was either synthetic snippets or a real app whose full bug list nobody knew. So we built the app and the answer key together.
What we found
We ran 12 scanners, LLM reviewers and agentic pipelines against it, including our own. The best scored 57%. Nine bugs were missed by every single one, and none of those nine contain a dangerous line of code.
Use
Education and research only. It is deliberately insecure software: run it locally, not on anything reachable.
MIT Python 82 planted bugs 50 decoys Blind-copy export
GitHub Read the write-up

Licensing

Hayward and Langfail are MIT. Thicket and Rowan are commercially licensed and are not part of this. Report a vulnerability in either open-source project to hello@hedgerow.dev rather than in a public issue.