LLM security covers the controls that stop an application built on a large language model (LLM) from leaking the data it handles, taking actions nobody approved or running up its bill, whatever text ends up in its context. It’s the part of generative AI security (GenAI security) that deals with text models and the software around them, and its reference list of risks is the OWASP Top 10 for LLM Applications, in an edition published on November 17, 2024.

What is LLM security?

One LLM feature spans four places: a model behind a provider’s API, an application holding the system prompt and the retrieval code, the network its requests cross, and whatever databases or tools the output can reach. LLM security covers all four. In our reading of the OWASP list, only two of its ten entries, supply chain and poisoning, are mainly about the model and its training data. The other eight are about what surrounds it, so the list reads more like an application security checklist than a machine learning paper.

That surrounding layer is where most security teams get pulled in. A developer wants a coding agent on every laptop, a product team wants to send support tickets to a model, someone has already connected an agent to a dozen MCP servers. Each of those is a stream of requests leaving a laptop, a server or an agent sandbox, carrying prompts, file contents and tool results, and each can be logged, limited and checked before it lands. For a company that doesn’t train its own models, that traffic is most of the attack surface, and most of what LLM cyber security means in practice.

Identity is the gap worth noticing in the list. None of the ten entries is about which person or agent sent a request, yet two of the mitigations depend on knowing it: LLM06 asks you to execute extensions “in user’s context”, and LLM10 asks for user quotas. A provider’s console records the API key behind a request, and one key is often shared by a whole team or an entire app. AI agent identity covers how to tie each request to a person.

AI model security is a neighbouring discipline with its own concerns: stolen weights, adversarial examples, model extraction, leaks of training data. Teams that fine-tune or host models need it. Teams that call a provider’s API mostly don’t.

LLM security risks: the OWASP Top 10 for LLM Applications (2025)

The OWASP Gen AI Security Project published its current list of LLM security risks on November 17, 2024, as the 2025 edition of work that, in OWASP’s words, “started in 2023 as a community-driven effort to highlight and address security issues specific to AI applications.” The LLM Top 10 index still presents it as the latest edition. Some guides still print the 2023 names, such as Insecure Output Handling where the 2025 list has Improper Output Handling.

The ten LLM vulnerabilities, under OWASP’s names:

  1. LLM01:2025 Prompt Injection. Text in a prompt changes what the model does. The user can type it, or it can arrive indirectly, inside a web page, a file or a tool result the application passes along. More on this one below.
  2. LLM02:2025 Sensitive Information Disclosure. Personal data, financial and health records, credentials, confidential business data or legal documents end up where they shouldn’t: in a completion shown to the wrong person, or in a prompt sent to a provider. Most LLM data security and data leakage work lands under this entry.
  3. LLM03:2025 Supply Chain. Something upstream is compromised. OWASP says these supply chains “can affect the integrity of training data, models, and deployment platforms,” and that includes third-party pre-trained models and datasets.
  4. LLM04:2025 Data and Model Poisoning. “Data poisoning occurs when pre-training, fine-tuning, or embedding data is manipulated to introduce vulnerabilities, backdoors, or biases.”
  5. LLM05:2025 Improper Output Handling. Model output goes into a browser, a SQL engine or a shell without being validated or encoded, so text the model produced runs as code.
  6. LLM06:2025 Excessive Agency. The model can call more tools, with more functions and wider permissions, than its task needs. A manipulated output, or just a wrong one, then does real damage.
  7. LLM07:2025 System Prompt Leakage. The system prompt gets revealed. That only hurts if something sensitive was in it, which is OWASP’s point: the system prompt “should not be considered a secret, nor should it be used as a security control.” Credentials and connection strings don’t belong there.
  8. LLM08:2025 Vector and Embedding Weaknesses. Flaws in the vector stores behind retrieval: chunks that cross tenant or permission boundaries, poisoned documents, embeddings that can be turned back into text. RAG vs MCP goes through retrieval security in detail.
  9. LLM09:2025 Misinformation. “Misinformation occurs when LLMs produce false or misleading information that appears credible,” and a person or a program acts on it.
  10. LLM10:2025 Unbounded Consumption. Nothing caps how many requests or tokens a user, agent or app can push through. OWASP ties it to “denial of service (DoS), economic losses, model theft, and service degradation.”

Prompt injection (LLM01): direct, indirect and jailbreaks

Two kinds of people can inject a prompt. The user can type instructions meant to override the system prompt, which is direct prompt injection. Indirect prompt injection comes from anyone who can write to content the application later hands the model: a retrieved document, a support ticket, a web page an agent browsed, the description of an MCP tool. The person using the app never sees those instructions, and the attacker never has to touch the app itself. OWASP treats jailbreaks as part of the same entry: “Jailbreaking is a form of prompt injection where the attacker provides inputs that cause the model to disregard its safety protocols entirely.”

OWASP’s entry also says retrieval and fine-tuning don’t fully fix it. So plan for injection to succeed some of the time, and decide in advance what an injected model can still do: which tools it can call, which hosts it can reach (the job of egress filtering), which data it holds. Agentic AI security covers those limits for agents. A proxy that logs prompts and tool results can flag likely injection after the fact, which helps you find it and clean up; stopping the next attempt is a policy’s job.

Which control addresses which LLM security risk?

OWASP’s own mitigations put most of the ten risks outside the model. Under LLM06, the item called “Complete mediation” reads: “Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not.” Under LLM07, OWASP rules out the system prompt as a security control. The table below sorts each risk by where it’s mainly controlled, and by whether a control on the traffic around the model can help.

By a control on the traffic we mean a proxy between whoever sends the request (a laptop, a server, an agent sandbox) and the model provider, plus the MCP and API calls an agent makes. It reads each request with the identity behind it, which for HTTPS takes TLS inspection.

OWASP 2025 riskMainly controlled inCan a control on the traffic help?
LLM01 Prompt InjectionApplication design: untrusted content kept apart from instructionsPartly. It can’t reliably recognize injected instructions. It can cap what follows (blocked tool calls, MCP tool allowlists, egress rules) and flag suspected injection in the logs afterwards
LLM02 Sensitive Information DisclosureData handling on both sides of the modelYes, for data leaving in prompts and tool results: redaction before the provider, secrets blocked in tool-call arguments, sign-ins held to the corporate account, an encrypted prompt log. No, for what a model memorized in training
LLM03 Supply ChainProvenance of models, datasets and packagesPartly. It can find and block the MCP servers and skills an agent loads without approval. It doesn’t scan models, weights or packages
LLM04 Data and Model PoisoningTraining and fine-tuning pipelinesNo. Data poisoning happens in the training data, before any request is sent
LLM05 Improper Output HandlingThe code that consumes model outputPartly. Response-stage rules can read a tool call before it runs, and the systems behind a proxy keep their own policies. Encoding output for a page or a query stays in the application
LLM06 Excessive AgencyPermissions of tools and of the identities they run asYes. Tool allowlists, each user’s own OAuth token per server, a log of every tool call and blocks on risky calls are complete mediation in practice
LLM07 System Prompt LeakageApplication design: no secrets or access rules in the promptMostly no. Keeping API keys on a proxy means they never sit on the machine that builds the prompt
LLM08 Vector and Embedding WeaknessesThe retriever and the vector store’s access controlPartly, through identity-aware access and masking at the database behind a retriever. The vector store needs its own controls
LLM09 MisinformationHuman review, grounding, evaluationNo. Full prompt and completion logs give you an audit trail and nothing more
LLM10 Unbounded ConsumptionRate limits and quotasYes, for consumption by your own users and agents: budgets on spend and tokens per person

That’s three yes, four partly, one mostly no and two no. We build a proxy, and we’d still put it plainly: nothing on the network repairs poisoned training data or catches a confident wrong answer. LLM04 belongs to whoever runs the training pipeline, and LLM09 to whoever reviews the output before it’s used.

LLM security best practices

When a team asks security to approve its first LLM feature or a coding agent rollout, this is the order we’d work in, with the OWASP entry each step answers. It also works as an LLM AI security and governance checklist.

  1. Build an inventory before writing policy: which models and providers are in use, from which laptops, servers and apps, and which MCP servers and skills your agents load (LLM03). The ones nobody approved are shadow AI.
  2. Take secrets out of prompts and system prompts. API keys, connection strings and access rules don’t belong in text a model can repeat (LLM07). Keep provider and tool keys on a proxy that adds them on the way out.
  3. Redact sensitive data before it reaches the provider, in prompts and in tool results (LLM02). AI data loss prevention covers the patterns and where they miss.
  4. Give each tool the least privilege its task needs, and run it as the user who asked, for example with that user’s own OAuth token at the minimum scope (LLM06).
  5. Authorize every action in the system that performs it: the database, the code host, the ticketing API. Not in the prompt, and not in the model’s judgement (LLM06).
  6. Ask a person before deleting data, moving money or merging to a protected branch, and show them the raw action rather than the model’s summary of it.
  7. Set quotas per person and per agent, on requests, tokens and spend (LLM10).
  8. Log prompts, completions and tool calls with the person or agent behind each one.
  9. Red-team before release (LLM01, LLM07), and keep monitoring afterwards against the same ten categories, so findings land where your security team already looks.

Step 8 creates a problem of its own. A log of every prompt is a copy of every customer record, credential and contract excerpt anyone pasted in, kept long after the conversation ended and readable by whoever runs the logging stack. Encrypt the content fields, or drop the prompt text and keep the metadata. The AI gateway page goes into that trade-off.

LLM guardrails vs controls outside the model

LLM guardrails are filters wrapped around a model call. Input validation looks for injection patterns or off-limits topics; an output check looks for toxic text, leaked personal data or a response that doesn’t match the expected schema; sometimes a second model grades the first. They run at the model or application layer, and they’re probabilistic. A classifier trained on known jailbreak phrasings will eventually meet one it hasn’t seen.

Controls outside the model work on facts the model can’t change: which person or agent sent the request, which tool is being called and with what arguments, how much that person has spent today, which host the request is going to. They give the same answer every time, and they run somewhere the injected text never reaches.

We’d run both. Guardrails reduce how often a model says or does the wrong thing, and the outside controls decide how much the wrong thing can cost. If you have to pick one first for an agent with tools, pick the outside controls, since a guardrail miss on an agent with broad access goes straight to the tool.

The word also gets used for other things. Formal’s own Guardrails setting, for example, decides what happens to MCP servers and skills nobody has approved (allow and monitor, or block), which is an LLM03 control rather than a content filter.

Types of LLM security tools

Most LLM security solutions combine tools from a few categories, and each category sees a different slice of the ten risks:

CategoryWhat it doesMainly addresses
Model and dependency scanningChecks model files, datasets and dependencies before they’re usedLLM03, LLM04
AI security posture management (AI-SPM)Inventories models, endpoints and datasets in your cloud and how exposed they areLLM03, parts of LLM02
Red teaming and evaluationAttacks the application before release and measures how it holds upLLM01, LLM07, LLM09
Guardrails and AI firewallsFilter what goes into and comes out of the model callLLM01, LLM02, LLM05
AI gateways and LLM proxiesSit between people, agents and apps and the model provider, adding identity, logging, policy and budgetsLLM02, LLM06, LLM10
DLP for promptsFinds and redacts sensitive data in prompts and tool resultsLLM02
MCP gatewaysPut a policy point between agents and the tools they callLLM03, LLM06

Of these, the AI gateway is the one that sits on the traffic around the model. Many gateways were built for developers, to switch models and track token costs per app; a security team needs the person behind each request, policy before the request leaves and a log it can trust. Our AI gateway page covers that version.

When you evaluate a vendor, these questions separate the categories faster than feature lists:

  • Where does it sit, and which traffic does it see: browser chat, CLI coding agents and the child processes they start, server-side apps?
  • Does each log line name a person or an agent, or only an API key?
  • What does it store, for how long, and can you encrypt or drop prompt text?
  • Which operating systems and runtimes doesn’t it cover?
  • Which of the ten OWASP entries doesn’t it address?

LLM security vs generative AI security and agentic AI security

OWASP’s naming shows how far the terms overlap: its project is called the Gen AI Security Project, and its index heads the list “2025 Top 10 Risk & Mitigations for LLMs and Gen AI Apps”. Generative AI security does reach further, into image, audio and video models and problems like deepfakes and content provenance. For text models and the software around them, the two names describe the same work.

Agentic AI security starts where model output becomes an action: a database write, a pushed commit, a call to an MCP tool, a message sent to a customer. OWASP gives agents a list of their own, the Top 10 for Agentic Applications, and its tool misuse entry builds on LLM06. Agentic AI security maps that list to controls; LLM06 is where the two lists meet.

How Formal controls the traffic around an LLM

Formal sits on the path between whoever sends a request and the system it reaches, and for LLMs that system is the provider’s API. Formal Endpoint in Transparent Mode intercepts LLM traffic on macOS, Linux and Kubernetes with no client changes. Server-side apps send their requests to a Formal Connector instead, at a base URL such as https://anthropic.<connector-hostname>/v1/messages, with headers that associate each request with a Formal identity (LLM resources). On laptops, network rules pick which flows to inspect, and they can match agent processes and the processes those agents spawn.

Nothing in the exchange gets summarized away: the text going in and coming out is kept whole, tool calls keep their arguments and results, and token counts sit next to the model and provider that produced them. Policies also know who sent the request: the user, the end user behind an application, and, for connections through the Endpoint, the agent.

Policies run at three stages. A session policy can block before anything reaches the provider. A request policy can rewrite the body, which is how the SSN example in our policy docs replaces Social Security numbers in prompts without ending the session. A response policy reads the tool calls the model wants to make. The same rewrite adds Anthropic’s anthropic-allowed-org-ids header so that only your organization’s accounts can sign in and use the API. Redaction in depth is on our AI DLP page.

At Gusto, the requests carried Salesforce support tickets to an LLM-powered API, and those tickets often held personal and health information. The HTTP Connector sits between Salesforce and the provider and masks those values in flight:

“Formal Satellites simplified ensuring any PII or PHI entities between Salesforce and our LLM provider were automatically detected and masked, without needing detailed data structure knowledge.”

— Ian Wardell, Data Privacy Lead @ Gusto

The rollout took 5 days, and the Gusto case study counts 100K+ requests secured.

Remote MCP servers sit behind the MCP gateway: agents connect to a Connector, which supplies the upstream credential, applies policy and writes every tool call to the log. With linked accounts, the token sent upstream is the user’s own OAuth token, so the server acts for that person rather than for a shared key, and access controls say which users may call which tools. Our MCP gateway page has the setup. Assessments put two numbers on each tool call the model proposes, one for risk and one for whether it fits the request the user actually made, and a response policy can stop the call on the risk number. A response with no tool call gets no score, so the numbers describe the model’s next action, whatever prompt led to it.

Budgets for LLM10 start from prices you enter per model, because Formal ships no price list of its own. With those in place, a request policy compares a user’s spend or token count for the day, week or month against a limit, and a separate rule can refuse any model that has no price yet. The totals update every few minutes, so a fast burst can overshoot a little (LLM spend).

Insights run an AI review over each new batch of LLM logs, every 15 minutes, sorted into the ten OWASP 2025 categories from this page, with a severity and a confidence on each finding. Findings go out through your log integrations, so they can land in your SIEM. They don’t block requests or edit configuration; only policies enforce. That leaves the log itself as something to protect, and log schemas let you encrypt the LLM content fields under your own key, or remove prompts from the logs with a single drop rule.

Some of it we don’t cover. Transparent Mode isn’t available on Windows yet, and LLM resources don’t work through Windows local listeners, so Windows laptops get no LLM interception today (Windows docs). We don’t scan models or training data, and we do nothing for LLM04 or LLM09. On MCP responses, a policy can allow or encrypt but not mask, and local MCP servers started over stdio never reach the gateway. Skills are detected in Claude Code traffic only. Our docs describe budgets for your own users and agents, not protection for a model API you host from outside traffic.

If you keep one rule from the OWASP list, keep LLM06’s: the systems the model calls decide what’s allowed, and the model is a caller that can be talked into anything. That’s the layer we work on, and if you’d like to see it on your own LLM and MCP traffic, book a walkthrough.