LAB/02Research area
AI security: testing, evaluating and constraining AI systems
AI security research covers two directions: the security of AI systems, and the safe use of AI inside security operations. Both need test methods that are repeatable and results that can be measured.
- Area
- LAB/02 · AI Security
- Outputs
- Research notes on attack and defense techniques for LLM applications, Test case libraries and evaluation methods for the AI security service, Tools such as test harnesses, where they are useful outside Onion, Evaluation criteria for the Autonomous SOC Analyst, Talks on completed research
- Status
- Active research
01Open questions
What we are working on
How can prompt injection through retrieved documents and tool output be tested repeatably, in a form that fits a release pipeline?
Which permission model limits the damage an LLM agent can do when its instructions are hijacked?
How should an AI system that investigates security alerts be evaluated: against which ground truth, and with which failure modes counted?
Which parts of the OWASP Top 10 for LLM Applications and MITRE ATLAS translate into tests that can be run against a production system?
How can training data, embeddings and model artifacts be given provenance and integrity controls like any other software supply chain component?
What should an LLM application log so that misuse can be detected and investigated afterward?
02Method
How the work is done.
- Adversarial testing of LLM applications
- Structured test cases for direct and indirect prompt injection, data leakage, tool misuse and privilege escalation through agents, run against test deployments of common architectures such as retrieval-augmented generation and tool-calling agents.
- Evaluation harnesses
- Versioned test sets, scoring rules and regression runs, so a change to a prompt, model or retrieval index can be checked the way a code change is checked by unit tests.
- Threat modeling for AI systems
- Models are built with MITRE ATLAS, the OWASP Top 10 for LLM Applications and the NIST AI Risk Management Framework, and extend conventional application threat modeling to data pipelines, model supply chain and agent permissions.
- Guardrail engineering
- Input and output controls, tool permission boundaries, tenant-aware retrieval and logging designs are built and tested as reference implementations that delivery teams can reuse.