AI gateway for agent and employee LLM traffic

Your engineers’ coding agents already call api.anthropic.com from their laptops, and that traffic never passes through a gateway you configured. An AI gateway for security teams has to sit where those calls start, not only in front of the apps you build.

What is an AI gateway?

An AI gateway is a proxy between the clients that call large language models (apps, agents and employees) and the model providers’ APIs. Every request passes one control point, so you can authenticate it, apply policy, log the prompt and response, and track cost. An LLM gateway and an LLM proxy are the same thing under different names.

Most AI gateways in today’s search results are built for developers. A team points its app’s base URL at the gateway to switch between models, add caching and rate limits, and see token costs per app. That’s a real need, and it’s a different job from the one a security team has.

A secure AI gateway (or AI agent gateway, when agents send most of the traffic) answers security questions: which person or agent sent this prompt, what data was in it, which tools did the model call, and can you stop the request before it leaves? This page covers that side.

Why do security teams need a different AI gateway?

A gateway that depends on a base URL change only sees the clients someone reconfigured. Coding agents such as Claude Code, Cursor and Codex run on laptops and call their providers directly. Employees open chatgpt.com in a browser. None of that traffic crosses a gateway that lives in front of your own apps.

Agents also act. One session can read files, run shell commands and call MCP tools, and each tool call arrives inside the model’s response. To control that, the gateway has to parse the provider’s wire format, not just count requests and tokens.

Security also needs attribution. A provider log that shows an API key tells you which key was used, not which engineer started the agent. An enterprise AI gateway has to tie each request to a person, and to the agent that acted for them.

How does Formal’s AI gateway work?

Formal enforces LLM traffic in one of two places. On laptops and servers, the Formal Endpoint intercepts it with Transparent Mode on macOS and Linux, and the Formal Endpoint for Kubernetes does the same as a native Pod sidecar. For apps in your own infrastructure, a Formal Connector sits in the path. Both evaluate the same policies, then forward the request to the provider.

On the Endpoint, a network rule picks which flows to intercept. On macOS, Formal recognizes known agents by their code-signed process identity, including the processes they spawn. On Linux, it matches process names or known LLM provider hostnames. This rule from the AI governance docs intercepts LLM and MCP traffic from known agents on macOS:

{
  "condition": {
    "pre_tcp": process.is_agent || process.has_agent_ancestor,
    "pre_tls": true,
    "post_tls": technology in ["mcp", "llm"]
  }
}

Apps keep calling api.anthropic.com: no SDK or base URL change. Server-side apps take the Connector path instead. You link an LLM resource to a Connector listener and swap the provider hostname for the Connector’s, for example https://anthropic.<connector-hostname>/v1/messages. That’s the one setup with a hostname change. The Endpoint can also forward its LLM traffic to a Connector rather than evaluate it locally.

Here’s the catch: Transparent Mode terminates TLS with certificates from a local Formal CA. Browsers and Go programs trust it once it’s in the OS trust store. Node.js and most Python HTTP clients ship their own roots, so the Endpoint can set variables such as NODE_EXTRA_CA_CERTS and SSL_CERT_FILE for them. Transparent Mode runs on macOS and Linux, not Windows.

AI data loss prevention: mask sensitive data before it reaches the model

Request-stage policies run before Formal forwards the prompt. A rewrite action can add, replace or remove headers, or replace the body, so you strip a value instead of blocking the whole session. This example from the policy evaluation docs redacts US Social Security numbers in prompt text and tool results:

package formal.v2

import future.keywords.if
import future.keywords.in

ssn_pattern := `\b\d{3}-\d{2}-\d{4}\b`

request := {
  "action": "rewrite",
  "body": regex.replace(input.http.body, ssn_pattern, "[REDACTED-SSN]"),
  "reason": "Redact SSNs from LLM request messages",
} if {
  input.resource.technology == "llm"
  some msg in input.llm.messages
  some part in msg.content_parts
  part.type in {"text", "tool_result"}
  regex.match(ssn_pattern, part.text)
}

The rewrite replaces the whole HTTP body, so keep replacements JSON-safe. It catches what the pattern matches, nothing more, so run new policies in Dry-run first and check the logs before you set them to Active.

Gusto runs the server-side version of this pattern. It put the Formal HTTP Connector between Salesforce and an external LLM provider, so PII and PHI in customer support tickets are detected and masked before they reach the provider’s API.

Data also leaks through accounts. Anthropic’s tenant restrictions let a proxy allowlist your organization ID, and a Formal rewrite policy injects the anthropic-allowed-org-ids header on Claude traffic. Personal Claude accounts and other orgs then get a 403 from Anthropic (see the setup guide).

Can an LLM gateway detect prompt injection?

Formal detects it; it doesn’t block it inline. Insights, an AI engine, reviews new LLM traffic every 15 minutes against the OWASP Top 10 for LLM Applications (2025), including LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM06 Excessive Agency and LLM10 Unbounded Consumption.

Each insight names the risk, the entity involved (such as a user) and the log events behind it. Insights never change anything on their own. Formal forwards them to your log integrations, such as Datadog, Splunk, Amazon S3, Google Cloud Storage and Azure Blob Storage, so your SIEM can alert on new findings.

To enforce, write a policy. Response-stage rules read the tool calls in the model’s reply, so you can block a response that, for example, calls Bash with a specific command. That’s where prompt injection protection becomes a control rather than a report: you limit what a redirected agent can do.

LLM cost tracking: how do you cap spend per user?

Formal estimates the cost of every LLM request from token prices you set per model and provider, and each log carries that cost. Formal has no built-in price list, and the Token Pricing page lists models seen in logs without a price. That makes AI spend tracking per user, model and provider a log query.

Request policies read each user’s spend for today, this week and this month (UTC windows), as cost per currency and as a token count. A policy can block a user after a daily budget, cap tokens per week, or block any model that has no price yet, so a new model needs a price before anyone uses it. The spend docs include both policies.

Two limits: spend refreshes every few minutes, so a burst of requests can go slightly over a budget, and budgets need a Formal user on the request.

Discover shadow AI tools and models

Because the gateway sees the traffic, it also inventories it. The AI usage pages list every model, AI client application (such as Claude Code or Cursor) and provider in proxied traffic, with users, requests, tokens and policy hits. Formal only finds what it proxies, so coverage follows your network rules. To review what people already use, see shadow AI detection.

Who can read the prompts your gateway logs?

An AI gateway that logs full prompts gives you LLM observability, and a new store of sensitive data. Log schemas let you encrypt prompt text, tool inputs and model replies with an encryption key you configure, or drop prompts from logs entirely. You can scope a schema to Endpoint or Connector logs, so the LLM security controls apply to your own audit trail too.