AI gateways: strong locks, open toolboxes

In security we have a habit of trusting the thing with the lock on it.

When companies started letting AI agents call models such as Claude and GPT, many put a gateway in between. It made sense. The agent never sees the real API key, every call is logged and billed, and you decide which parts of the provider's API the agent may touch. It looks like a checkpoint, and it is one.

But what passes through that checkpoint has changed. A model API is no longer a text box. One chat request can ask the provider to fetch a web page, search the web, run code, call an outside service, or reuse files stored on the provider's side. Those capabilities are not separate doors. They ride inside the request, usually as one more line in a list of "tools".

So we asked a narrow question. Give an agent an allowed route and a key, but don't trust it to choose what the provider does. Can you configure the gateway so that anything nobody approved never reaches the provider?

After testing nine products, our view is that the industry has built good locks around the wrong question. Gateways check who is calling and where. For agents, the question that matters is what the request makes the provider do.

What we tested

We tested nine products that can sit in that position. They are different kinds of product, and several never claim to inspect what is inside a request:

• Routers and LLM proxies: LiteLLM, Portkey
• AI and agent gateways: Agent Router (formerly Envoy AI Gateway), agentgateway
• API gateways: Kong, Azure API Management
• Egress firewall: Pipelock
• Credential proxy and sandbox: nono, NVIDIA OpenShell

So this is not a ranking, and it is not a charge of negligence. For some of them, our question is outside what they set out to do. But they all sit where an agent's requests pass, which is why people will reach for them.

What we found

In the configurations we tested, we found no explicit policy that reliably refused a request because it switched on a provider-hosted capability. The products did enforce useful controls at the edge of a request: eight of the nine refused nearly all non-chat routes, several refused wrong or missing keys, and two removed named top-level fields. None of those added up to a deny-by-default list for tools embedded in an allowed chat request. Rejections for schema or parsing reasons are not lasting feature controls, so we list them separately.

Each product ran twice: in a minimal working setup (its default or quickstart configuration, pointed at our test provider with a key), and in the most restrictive working setup we could build from its documented built-in settings while keeping the chat route open (scope below).

We tested three kinds of case:

• URL fetch: image and document links in a request that the provider would fetch.
• Hosted tools: tools the provider runs, such as web search, code execution and remote MCP servers, declared in the request.
• Unexpected replies: scripted replies from our fake provider to a plain request, carrying structures such as a web search result, a call to a tool the request never offered, or an error body with internal detail. We checked whether each reached the caller.

One of the 110 test requests, an ordinary chat request with one entry asking for a page fetch:

"tools": [{"type": "web_fetch_20250910", "name": "web_fetch", "max_uses": 1}],
"messages": [{"role": "user", "content": "fetch https://pcc-b002.invalid/"}]

LiteLLM, Portkey and Kong, each in its strictest setup, forwarded it. The tools entry arrived at our recording server with the fields the test looks for.

Strict setups, tests forwarded:

All nine received every test in these three areas, so the denominators are the same. "Forwarded" means the case reached our fake provider or the caller intact, not that a real provider would act on it. Why the counts drop below the maximum:

• (a) Three tests put the capability in a named top-level field (mcp_servers, web_search_options). The gateway removed it, so these count as stopped. Neither product could reach a hosted tool listed inside tools.
• (b) The gateway could not parse one of our scripted unexpected replies.
• (c) A schema check against Microsoft's API description, which did not know those fields. It also rejected some normal requests.

What this means if you use a gateway

Allowing a chat route is not the same as allowing only text generation.

• Make two decisions: which model endpoints an agent may use, and which provider-hosted capabilities.
• Switch off unneeded hosted tools at the provider or account level. That holds even when the gateway passes a request through.
• If the agent is not trusted to choose API fields, deny by default. Put a layer in front of the provider that only lets known fields through.
• Check what actually arrives. Point your deployed setup at a server that records requests, including retries, fallbacks and side endpoints such as token counting.
• Keep network egress controls as a second line, not a substitute.

One more finding: requests that went around

We pointed LiteLLM's quickstart configuration at our test server. Three test requests instead made it try to connect to api.anthropic.com or api.openai.com. A recorder on our network caught the connection attempts but not their contents. We saw no request body and no credentials, so we do not claim any data was disclosed.

Two of the requests named no model, and LiteLLM picked the provider's default API address. The third was a token-count request, a known public issue: LiteLLM ignored the configured api_base for that call (BerriAI/litellm#29764, opened June 2026). In a similar deployment, that call could go to Anthropic's default address instead of a configured self-hosted model.

LiteLLM's git history places a fix in v1.106.0, released 18 October 2026. We tested v1.103.2 only and have not verified the fix. LiteLLM's strict setup refused all three requests by route.

Scope and method

Scope. Built-in, documented settings only, in open-source builds and Azure's BasicV2 tier. No custom plugins or policy code (Lua, WASM, CEL), WAF rules, paid add-ons or network policies. Kong, agentgateway and Azure API Management support custom policy code; we did not test whether a custom policy could enforce the allowlist described here. No vendor has reviewed our configurations yet.

Method. Every gateway forwarded to a fake provider that records what it receives, and tests are scored only on that. A plain request had to get through first, so a broken setup could not score as a block. With no gateway, all 110 arrive. Versions were pinned; results are from 17 October 2026.

Briar, our reference gateway

We also ran Briar, the gateway we are building. It does not filter requests: it rebuilds each one from a fixed list of allowed fields, so anything else is refused or never copied across. In this suite, it refused 58 of the 61 tests that applied to it. In the other 3 it silently dropped the tested header, which keeps the boundary but tells the caller nothing.

That is evidence for these request shapes, not complete protection against future provider features or untested encodings. Of the 110 tests, 43 were skipped (42 use OpenAI's API, which Briar does not serve, and one needs a provider key it does not hold). The other 6 carry harmless content and count toward no score. Briar ran from commit 5f3fa99 (18 October 2026), without its network isolation. And we built it, so weigh it accordingly.

Where this leaves us

For agents, the old perimeter question was who gets in. The new one is what an allowed request can make the provider do. An agent with a key and a route can hand the provider a job, and the provider will do it: fetch, search, run, remember. The gateway sees an authorised caller on an authorised path and lets it through, because that is what it was built to check.

We don't think the answer is gateways that understand every provider feature. Providers add tools often, and a gateway that has to learn each one will always be a step behind. The answer is to flip the default: a request carries only the fields someone approved, and nothing else leaves. That is a design choice, not a product category. You can make it today, with provider-side settings, a strict schema in front of the provider, or a layer that rebuilds requests.

Until then, an allowed route is an allowed set of capabilities, whether you meant it or not. Check what yours lets through.

If you maintain one of these products and think we configured it unfairly, tell us at hello@hedgerow.dev and we will re-run it.

---