Clean, or Just Not Alarmed?

Two missing details

A scanner can report no findings at a chosen threshold without having inspected every file. It can also emit lower-severity findings that, at its default settings, still fail a build. Most published false-positive figures say nothing about either.

Disclosure. We build Hayward, one of the five scanners below, and we wrote the benchmark harness. Treat every number here as self-measured until someone else reproduces it. The raw per-file results are public and the last section shows how to rerun them.

We ran five open-source model-file scanners over the same 222 benign files and measured three things: how often each tool flagged a benign file, how often it returned a verdict at all, and what its default build gate would actually block.

• Counting alarms only, four of the five flag between 0% and 6% of the benign files they gave a verdict on.
• Counting warnings too, the range across all five is 0% to 96%. Among the three tools that gave a verdict on most files, it is 3%, 20% and 41%.
• Verdict coverage runs from 38% to 100% of the files handed to each tool.
• At their default gates, no tool is best at everything. The tools that block the most attacks also block the most benign files, and Hayward's default gate misses a large share of one external attack set.

This is a comparison of each tool's base installation on one corpus, not a like-for-like accuracy ranking. Each tool is scored on the files it gave a verdict on, and those subsets differ.

Result 1: how often each tool flags a benign file

We group each tool's findings into alarms, which the tool presents as dangerous, and warnings, its lower-severity or advisory findings. A warning on a benign file asks a person to look; it is not an accusation of malware. So when warnings are counted we call the result a benign flag rate, not a false-positive rate. The exact grouping for each tool follows this result.

Each percentage is over the files that tool gave a verdict on. A file counts once, however many findings it produces. A zero over a small denominator is weaker than it looks: ModelScan's 0 of 80 says nothing about the other 142 files.

fickling's 96% comes from strong verdicts, not warnings. Of the 90 benign files it flagged, it rated 67 LIKELY_UNSAFE and 23 LIKELY_OVERTLY_MALICIOUS. fickling focuses on pickle and pickle-based files and also offers automated safety checks and import hooks; this result describes its CLI as tested on this corpus, not every way it can be used.

Hayward's seven are one generated test file and six real models: four scikit-learn pickles, a Keras model and a skops pipeline. Each got an INFO finding, which Hayward uses for a reference it cannot verify, such as a class that is not on its known-safe list. INFO asks for review and does not fail its default build gate.

How we grouped each tool's findings

These are benchmark-defined groupings, written into the harness so anyone can disagree with them line by line. Severity and confidence are not interchangeable across tools, so the mapping is approximate.

For picklescan, ModelAudit, fickling and Hayward, the warning group is the level each tool itself presents as advisory or undecided; ModelAudit, for example, describes warnings as potential issues that require review. ModelScan has no such level. Its MEDIUM and LOW are lower-severity findings, not a documented uncertainty verdict, so for ModelScan the grouping is a severity cut only. It changes no benign number: ModelScan raised nothing on any benign file it read.

The grouping is for comparing flag rates. It is not what each tool blocks by default. That is the next result.

Result 2: what the default gates block

Each tool's command line decides pass or fail on its own rule. Using each tool's documented default, and the harness's records where the documentation is silent:

• Hayward fails on HIGH or CRITICAL.
• picklescan exits non-zero on dangerous globals only.
• ModelAudit exits non-zero on a warning or critical finding.
• ModelScan exits non-zero on any issue, and uses separate exit codes for errors and for "no supported files".
• fickling's CLI flags anything above LIKELY_SAFE.

Each cell counts only files that tool gave a verdict on. Files with no verdict are shown on the last row and are neither blocked nor passed here: what happens to them depends on how your pipeline treats each tool's exit codes.

Read across a column, not down a row. ModelAudit and fickling block every PickleCloak file, and also block 41% and 96% of benign files. Hayward blocks no benign file, but its default gate lets through 11 of 57 PickleCloak exploits and 41 of 97 chains. It reports 5 and 35 of those at LOW or MEDIUM and misses 6 of each entirely. With --fail-on low, Hayward blocks 51 and 91 of them and still no benign file in this set. That is a fact about this corpus, not a recommendation to change the default without testing on your own models.

The Hayward rows at HIGH and above were re-scored on 6 November from the same harness revision and corpora, on a second machine. Its counts from LOW up matched the first run exactly.

Result 3: verdict coverage

All 248 files (222 benign and 26 generated attacks), and every reason a tool gave no verdict:

A verdict is not proof of complete inspection. It means the tool reported a result for the file. The harness counts as no verdict: a parse failure, a crash, a timeout, a scan of zero files, a missing reader, and, for Hayward, a file whose only output is a coverage gap. A Hayward file with a real finding and a coverage gap counts as a verdict, because something was read; this run's data does not separate those partial reads out. For ModelAudit, the harness checks that at least one of its internal scanners ran on the file.

ModelScan was tested as a base PyPI installation, without extras. Its documentation lists h5py and TensorFlow extras for the formats behind 23 of its gaps, so these are not inherent format limits. We have not run it with extras. ModelScan also reports these cases: it has a separate exit code for "no supported files" and a --show-skipped option.

picklescan's gaps are quieter. When it cannot parse a file, or has no reader for the format, its summary still reports zero infected files. A pipeline that only reads that line records a pass for a file nobody looked at.

What limits the attack-set results

• PickleCloak has no benign half. A tool that flags everything scores 100%. Read those rows next to the benign row, never alone.
• picklescan's test files check a list Hayward inherited. Hayward's deny data starts from picklescan's, with credit in its NOTICE file.
• None of these sets is a blind test for Hayward. We have measured against them during development.
• We wrote the 26 generated attacks alongside Hayward, so they flatter it. Even so, its default gate blocks 22 of them, the same as ModelAudit.

How the benign set was chosen

• Source: 213 real files from Hugging Face, pinned by SHA-256, across 14 formats. Three could no longer be fetched (two now need authentication, one was changed by its owner), leaving 210. The harness adds 12 generated benign files, for 222.
• Admission rule: Hugging Face's overall label for the file had to be safe or caution. No sub-scanner (its pickle check, Protect AI, an antivirus scan, VirusTotal, JFrog) could report it unsafe, and no import could carry the pickle check's "dangerous" rating. The file had to come from a recognised organisation or a repository with meaningful downloads.
• What that does not mean: a sub-scanner that had not scanned a file, or was still queued, did not disqualify it. So the rule is "no scanner reported unsafe", not "every scanner inspected it and returned safe".
• Selection bias: excluding files that related scanners flagged conditions the set on detection policies like the ones being tested. It is a convenience sample of real files, not an independently labelled random sample. In August we also disassembled every "caution" file in an earlier version of the set and found no code-execution path.

What about Hugging Face's own labels?

Hugging Face combines several scanners and shows a label of safe, caution or unsafe from the results it has. Not every file gets a completed verdict from every scanner.

In this article's terms, "caution" is a warning. For 43 of the 46 "caution" files in our set, the manifest records that it came from Hugging Face's own pickle check, which marks ordinary imports such as scikit-learn's LogisticRegression as "suspicious". On one file we spot-checked live, every other scanner reported safe.

"Unsafe" is the alarm. We excluded every file with that label, so this article says nothing about its accuracy. Practically: "caution" means look closer, not malware. "Unsafe" means do not load the file.

Our own number, three months ago

In August, on a benign set of 215 models, our scanner (then part of Open-Rowan) raised INFO findings on 72 models, 33%. picklescan's suspicious tier flagged 57 of that set. At the time we were saying "zero false positives" because we raised nothing above INFO, which is exactly the habit this article is about.

Today that figure is 3%. The August set overlaps with this one but is not identical, so part of the change may come from the files. Since August, Hayward recognises more things by exact name: standard-library value types that model metadata carries, constructors of plain values, and the specific transformers objects that Trainer saves. Counted from LOW up, its results on the three external attack sets went from 34, 49 and 91 to 35, 51 and 91: no regression on those fixed sets, which says nothing about detection elsewhere.

Until this week the quickset README showed only the alarms-only row, listed "zero false positives" as a Hayward strength, and called quickset independent. It now shows both rows and states that we built it alongside Hayward.

What to take away if you use these tools

1. Know what your gate blocks. Run your scanner on a sample of your own models at the setting you will actually use. A gate that fires on a large share of normal models creates substantial triage work and encourages bypasses; one that is too quiet lets attacks through.
2. Never record "no verdict" as clean. Fail closed, or require an explicit, audited exception. Check picklescan's scanned-files count, ModelScan's exit code, and Hayward's --fail-on-coverage.
3. Check verdict coverage against the formats you ship, with the extras those formats need installed.
4. Ask any vendor: does the false-positive figure count warnings, what share of files got a verdict, against which named corpus, and what does the default configuration block?
5. Where possible, use safetensors to avoid pickle's executable deserialisation. It does not remove every risk in the model-loading stack.
6. Measure before you trust a number, ours included.

Conclusion

All five scanners catch the harness's direct-os-system case, a pickle that calls os.system directly. They differ on files they are unsure about, on files they cannot read, and on where their default gate sits. A single alarms-only false-positive rate shows none of the three, including the one we used to quote.

Limits

• Small, selected sample. Under a simple binomial model, zero events in 222 files puts the 95% upper bound near 1.4%. That assumes independent, correctly labelled files; it does not account for label error, selection bias or related files, and it is not a production guarantee.
• Different subsets. Each tool is scored on the files it gave a verdict on. We have not published a comparison restricted to files every tool read, or a per-format breakdown.
• Self-measured. We build one of the tools, wrote the harness, wrote 26 of the attacks and chose the groupings.
• Base installs, one main run. Optional extras would change ModelScan's coverage. Transitive dependencies of the first run were not locked.
• Open-source tools only. ModelScan is Protect AI's open-source scanner. Guardian, its commercial product, is not evaluated here, and neither is the Protect AI scan Hugging Face displays.

Reproducing this

The evidence folder holds the per-file results, the run log, the list of the 210 models that were available, the Hayward re-score at HIGH and above, and a setup script that builds one environment per scanner at the exact versions above:

git clone https://github.com/hedgerow-dev/quickset.git
cd quickset
git checkout a51d9f8
sh evidence/2026-11-05-scanner-comparison/setup.sh
python -m quickset.realmodels
python -m quickset.external
python -m quickset.run --json results.json

Expect small differences: models disappear from the Hub, and transitive dependencies were not locked. If your numbers differ from ours, we want to hear about it at hello@hedgerow.dev.