What Article 15 actually says
In plain terms, Article 15 requires that high-risk AI systems achieve an appropriate level of accuracy, robustness, and cybersecurity, and perform consistently across their lifecycle. The clause that drives security testing is 15(5): systems must be resilient against attempts by unauthorized third parties to alter their use, outputs, or performance by exploiting vulnerabilities — and it names the threat classes directly: data poisoning, model poisoning, adversarial examples (evasion), and confidentiality attacks. For an LLM agent, those four abstract classes have concrete shapes.
The four threat classes, translated to agent tests
| Article 15 threat | What it is for an agent | How you test it | evsec pack |
|---|---|---|---|
| Adversarial examples / evasion | Prompt injection and jailbreaks that make the agent ignore its instructions or policies | Adversarial inputs (direct + indirect) graded by an injection-resilience judge over real traffic | injection |
| Confidentiality attacks | Extracting other users' data, PII, or the system prompt / tool schemas | Exfiltration and system-prompt-leak attempts graded against a disclosure judge | pii system-prompt-leak data-exfiltration |
| Model evasion via tools | Coercing state-changing tools past their guardrails (refunds, deletes, transfers without checks) | Tool-abuse attacks graded by a sensitive-tool-use judge that knows the agent's real dependency rules | sensitive-tool-use tool-abuse |
| Data / model poisoning | Malicious content in retrieved documents or training/feedback data steering behavior | Indirect-injection tests on retrieved-context spans; groundedness checks that the answer traces to trusted sources | injection groundedness |
Why a generic checklist isn't evidence
Article 15 asks you to demonstrate resilience for your system. A generic prompt-injection test
tells you the model resists textbook payloads; it says nothing about whether your agent's
IssueRefund tool holds its ReadPolicy gate under pressure, because it doesn't
know that tool or that policy exists. Evidence has to be grounded in the system under assessment. That's
the difference between a checklist and a test: the test exercises the actual attack surface.
What your technical file needs (Annex IV)
- A defined evaluator per threat class above, with an explicit pass/fail criterion.
- Results over a representative sample of real traffic — dated, versioned, reproducible.
- The design measures you rely on (guardrails, tool gating) and how the tests exercise them.
- Ongoing results, because 15 pairs with Article 72: resilience has to hold after release, not just at launch.
How evsec generates the evidence for Article 15
Point evsec at a sample of your agent's traces. It reads the real tools, policies, and data shapes, then generates the four evaluator families above — each an LLM-judge prompt with pre-specified procedure steps and bias defenses, plus the span-selection config to run it continuously in Progress Observability. Every pack carries its OWASP LLM Top-10 tag and its Article 15 mapping. Run them, and the results become the resilience evidence your technical file needs. evsec can also emit a downloadable evidence summary that lists each evaluator, its criterion, and the article it supports.
Try it on a bundled sample agent — the packs come out tagged to Article 15 and OWASP, with a downloadable evidence summary. No signup, no key.