Re-Testing Your LLM App on Every Change with Garak, PyRIT & Promptfoo
The scanner I built for LLM apps exists for one reason: a single prompt tweak or model upgrade can quietly undo your defences. Manual red-teaming once a quarter doesn't catch that. Automated tests on every change do.
Pick tests that map to OWASP LLM Top 10
Organise test suites by risk so results mean something to the team.
- Prompt injection — direct and indirect (via documents and tool output)
- System prompt leakage
- Sensitive data disclosure
- Insecure output handling (HTML/JS/SQL in responses)
- Excessive agency in tools
Use each tool for what it's good at
Garak brings a wide library of known probes. PyRIT is strong for multi-turn attack conversations. Promptfoo makes it easy to write your own assertions and run them in CI. Custom payloads cover what's specific to your app.
Make failures actionable
Every failing test records the prompt, the response, the OWASP LLM and MITRE ATLAS mapping, and a suggested fix. A red X with no evidence just gets ignored.
Gate on regressions, not perfection
No LLM app passes everything. Track a baseline and fail the build when a previously-blocked attack starts working again.
Tools mentioned
LLM security is a regression problem. Automate the attacks you've already beaten so they stay beaten.