☰

Want more detail?This is the summary. The full paper, The Inference Threat Model, labels every category with an evidence class, names the adversary each one needs, and cites a public source for each.

The one idea

The parts of an inference stack built to make it fast are the parts that decide who can read whose data.

Caches, routers and model loaders were never meant to be security controls. But once two customers share a system, those components are what keeps them apart. Nobody designed them for that job, and mostly nobody checks whether they do it.

That is the whole argument. Everything below is a consequence.

How sure we are varies, and it is worth knowing which is which. A few of the ten have public break-ins behind them. Several have published advisories but no known incident. The rest follow from how these systems are built, and nobody has published an example yet. That last group is not less real, it is less documented, and it tends to be the part that is deployed widely and looked at least. The full version labels every item and cites a source for each.

What the stack looks like

Roughly this, whatever product names are on the boxes. The dashed line near the top is the only place the system reliably knows who is asking. Everything below it works from values computed out of the request, and the two dashed rails on the sides are the planes that touch every layer.

Reference architecture of an LLM serving deployment Clients cross a trust boundary into an ingress gateway, which authenticates the caller against an identity source and derives a signed authorization context. That context is carried down every internal hop: to the scheduler, to both model runners, to the peer-to-peer transfer plane and to the shared KV cache tiers. The runners also load artifacts and make outbound fetches. A platform control plane and an observability plane run alongside every layer of the stack. PLATFORM / CONTROL PLANE CRDS · RBAC · SECRETS · WORKLOAD IDENTITY · ADMISSION OBSERVABILITY METRICS · TRACES · LOGS Clients / tenants many principals, one endpoint TRUST BOUNDARY IDENTITY AND AUTHORIZATION CONTEXT BOUND HERE Ingress and gateway authenticate · authorize · derive signed context claims Identity source IdP, API tokens, mTLS, workload identity Scheduler / router dispatches on policy and isolation context Artifact store models, adapters, tokenizers, configs Model runner A prefill / decode Model runner B prefill / decode Outbound fetch remote artifacts, media, adapters Transfer plane authenticated KV movement KV cache tiers GPU · host RAM · disk · object store: physically shared, partitioned only if you partition it request and data path signed authorization context: tenant · project · cache-isolation namespace
The green dashed line is the thing most deployments are missing: the authorization context the gateway works out, carried down every hop that touches shared data. The cache tiers at the bottom hold every customer’s data in one space, on purpose, which is what makes them a boundary nobody designed as one. Both runners load artifacts and fetch remote files; only one of each is drawn, to keep the picture readable.

Why this is not just a web app with a GPU

The cache exists to share work between requests. That is its job, not a bug. So “reuse” and “leak” are the same mechanism seen from two sides.

Things get looked up by what they contain, not by who asked for them. Cache entries are found using a value computed from the text of the request. If who-you-are is not part of that value, the system cannot see it.

Model files and config files can contain instructions, not just data. Several common formats carry expressions or file paths that whatever opens them will run or follow.

A lot of the internals were written assuming a private network. Many components still listen with no password, because the original deployment was one team’s cluster.

Four rules that would prevent most of it

  1. Know who is asking, and keep knowing. Whoever the user is at the front door, every component that touches shared data should still know it.
  2. Put everything that matters into the lookup key. Not just the text. Also the customer, the model version, the adapter.
  3. Make components prove who they are to each other. An internal service should not believe a name another service simply tells it.
  4. Refuse rather than downgrade. If a component cannot apply a security setting, it should fail the request, not quietly serve it without protection.

The ten ways it breaks

Each one carries the single check that tells you whether your deployment has it. The code on the right opens the matching section of the full paper.

  1. 1 The cache key forgets the customer. INF01 → Two customers send the same text and get the same cache entry. CheckList everything in your cache key. Is any of it about who is asking?
  2. 2 Internal services have no locks. INF02 → Coordination ports, metrics, cache services and job submission, all open, all trusting the network. This is how the publicly reported cluster takeovers happened. CheckList every open port in a running system. What checks the caller on each one?
  3. 3 Services take each other’s word for it. INF03 → A request or a service says which peer it is or where its data lives, and it is believed. CheckFor every name a service receives, is it verified against something the sender could not have chosen?
  4. 4 Isolation turned off for speed. INF04 → Shared memory, host namespaces, privileged containers. CheckDoes anything run privileged, or share the host’s network, memory or process space?
  5. 5 A fast answer tells you it was cached. INF05 → If someone can guess text and time the reply, they learn what other people have sent. CheckCan a low-privilege user measure response time precisely, or read per-entry cache metrics?
  6. 6 Headers steer internal traffic. INF06 → A caller sets a header that picks the backend, the model, or a URL to fetch. CheckWhich headers survive from outside into your internal network, and when was that list last compared against the code that reads them?
  7. 7 Opening a file runs code. INF07 → Model checkpoints that execute on load. Config fields rendered by a template engine. Data files that name another file to read. CheckWhich components open files users supply, and for any template rendering, is the engine sandboxed? Look at the dependency, and at the pinned version.
  8. 8 Downloads escape their folder. INF08 → A file name or digest from outside gets joined onto a local path and lands somewhere it should not. CheckYou probably have several storage backends. Do all of them contain the path, or only some?
  9. 9 One pod becomes the whole cluster. INF09 → The break-in gives a shell in a container. What turns that into a disaster is the token mounted next to it, the metadata service it can reach, and the permission to launch a privileged pod. CheckFrom a shell in an inference pod, can you read the service account token, reach metadata, or create a privileged pod?
  10. 10 A setting is on, and does nothing. INF10 → Someone configured a protection. A component in the path ignores it. No error, no warning, nothing in the docs. CheckFor each protection you rely on, find the line of code that enforces it. Not the line that configures it.
The one people miss

Number 10 is the worst kind, because the system looks correct. A system that refused would be safe. A system that warned would be fixable. A system that silently accepts leaves someone believing they are protected.

Testing it for real

Prove your test can fail before you trust it passing.

If you check that two customers do not share a cache entry and the answer is “they do not”, that result is worthless until you have shown the same test says “they do” when sharing is expected. Otherwise you cannot tell a working boundary from a broken test.

Comparing keys is not enough. Two customers can have different keys and still leak, through timing, metrics or scheduling. The real test is that one customer’s activity changes nothing another customer can observe.

Get permission first. Some of these checks involve reading credentials, creating privileged pods, and making requests that look like an attack. Staging, not production, and tell the platform owner.

If you only do four things

  1. Find every open port and put a lock on it or close it. Cheapest, biggest effect.
  2. Assume one container gets compromised, and limit what that gets you. Token mounting, metadata access, admission rules. This caps the damage from everything else.
  3. Work out what is in your cache key, and whether it crosses a customer boundary.
  4. Decide, deliberately, whether customers share a cache. The decision matters more than which way it goes.

Being honest about this

This tells you where to look. It does not tell you what you will find, and none of it means any particular product is broken.

Real incidents cross several of these at once. A file parser reached through a routing header, escalated through container permissions. So do not use the list to file things neatly afterwards. Use it to decide where to look first.

The long version, with evidence classes, the adversary model, per-category controls and citations, is The Inference Threat Model. If you are reviewing a serving stack and want a second pair of eyes, write to hello@hedgerow.dev.