An application built on a large language model is still an application. The usual threat modeling methods apply. What changes is one property of the model itself, and that property shapes most of the threats below.
This guide uses the risk names from the OWASP Top 10 for LLM Applications, a widely used public reference for this class of system.
The property that changes the model
In conventional software, code and data travel in separate channels. A language model has one channel. The system prompt, the user's message, a retrieved document and a tool result all arrive as text in the same context window, and the model has no reliable way to tell instruction from content.
Two working assumptions follow:
- Any text that reaches the context window may be treated by the model as an instruction.
- Any output of the model may have been shaped by whoever wrote that text.
Draw the system and its trust boundaries
Start with a data flow diagram. Most LLM applications contain some or all of these components:
- the user interface and the back end that orchestrates calls
- the system prompt and prompt templates
- the model, reached through a hosted API or run on your own infrastructure
- a retrieval layer: document stores, embeddings and a vector database
- tools the model can call: functions, plugins, APIs, code execution
- conversation history and long-term memory
- downstream consumers of model output: browsers, databases, shells, other services
- logs, traces and evaluation data
Then mark the trust boundaries. Four matter most:
- User to application. The user controls their input in full.
- External content to context. Web pages, emails, uploaded files and retrieved documents are written by third parties and enter the prompt.
- Model to tools and downstream systems. Model output crosses into components that act on it.
- Your organization to the model provider. Prompts and outputs leave your environment if the model is hosted.
The threats
Prompt injection, direct and indirect
In direct prompt injection, the user writes input that overrides the intended behavior, for example to ignore the system prompt or reveal it. In indirect prompt injection, the instruction is hidden in content the application fetches on the user's behalf: a web page, a support ticket, a PDF, a code comment. The user may be the victim, not the attacker.
Input filters and carefully worded system prompts reduce the success rate. They do not remove the problem, and there is no known complete fix at the model level. The design goal is therefore to limit what a successful injection can achieve.
Sensitive information disclosure
Sensitive data can leak through several paths. The retrieval layer can return documents the current user is not entitled to see. Secrets placed in a system prompt can be extracted. Prompts and completions kept in logs become a new sensitive data store. With hosted models, the provider's retention and training terms also apply.
Excessive agency
Agency is what the model can do through tools. The risk grows with three things: more tools than the task needs, broader permissions than the task needs, and actions taken without a person confirming them. An injection that reaches a model with read-only search is a nuisance. The same injection reaching a model that can send email, merge code or issue refunds is an incident.
Insecure output handling
Model output is untrusted input to whatever consumes it. Rendered as HTML, it can carry script. Placed in a SQL query, a shell command or a template, it can carry injection. One known pattern is exfiltration through Markdown: the model is induced to emit an image link whose URL contains conversation data, and the browser fetches it without a click.
Supply chain
The supply chain includes model weights, datasets, embedding models, orchestration libraries, plugins and tool servers. Some model file formats are based on Python pickle and can execute code when loaded. Third-party tools run with whatever access you grant them, and their descriptions are themselves text that enters the context window.
Poisoning and unbounded consumption
Anyone who can write to a knowledge base that feeds retrieval can influence answers, so write access to that store is a security boundary. Unbounded consumption covers requests crafted to drive up token use, cost or latency, including agent loops that do not terminate.
Practical controls
| Threat | Controls that limit it |
|---|---|
| Prompt injection | Separate and label untrusted content in the prompt; restrict what fetched content can trigger; require approval for consequential actions; test with adversarial inputs before release |
| Sensitive information disclosure | Enforce document permissions at query time in the retrieval layer; keep secrets out of prompts; redact and restrict logs; review provider data terms |
| Excessive agency | Minimum set of tools; narrow scopes per tool; act with the end user's identity, not a broad service account; human confirmation for irreversible actions |
| Insecure output handling | Context-specific output encoding; parameterized queries; schema validation for structured output; URL allowlists for rendered links and images |
| Supply chain | Pin and hash model and library versions; prefer safe serialization formats such as safetensors; vet plugins and tool servers; keep an inventory of models and datasets |
| Poisoning and consumption | Control and review write access to knowledge sources; rate limits, token budgets, step limits and timeouts for agents |
Above all, authorization must be enforced outside the model. The tool layer should check whether this user may perform this action on this resource, using the same access control as the rest of the application. A system prompt that says "only do this for administrators" is a request, not a control.
Working the model in practice
Run the threat model at design time with the people who build the application. Revisit it whenever a tool, a data source or a model is added, because each addition changes what an injection can achieve.
For each flow across a trust boundary, ask three questions:
- What is the least trusted text that can reach the context window here?
- If the model follows that text, what is the worst action or disclosure that results?
- Which control outside the model stops it, and how do we test that control?
Turn the answers into test cases. Keep a regression set of injection attempts, retrieval permission checks and output handling tests, and run it in the pipeline alongside functional evaluations. Log tool calls with the user identity, arguments and outcome.
Some residual risk from prompt injection will remain in any useful system. Record it as an accepted risk with a named owner.
Onion Infosec covers this work under its AI security service.