Agentic AI oversight
Model AI Governance Framework for Agentic AI v1.5, §2.3.2 — Before deploying, test agents
No instrument requires an agentic deployment to be stress-tested before it is allowed to act. The expectations here come from a framework that recommends rather than requires, and the single binding testing duty nearby reaches providers of general-purpose models above the systemic-risk threshold — not the organisation running the agent.
§ 1 — Who it binds
Who this reaches — and who Article 55 reaches
Deployer
Any deployment with some degree of independent planning, decision-making and action-taking over multiple steps towards a user-defined goal. The framework scopes itself to agents built on generative AI models and sets no sectoral, size or risk-tier gate. The one binding testing duty in the vicinity, Article 55(1)(a) of the EU AI Act, reaches a different party entirely: providers of general-purpose AI models with systemic risk.
§ 2 — In practice
Why one run tells you nothing
Start with what does not exist. No binding instrument requires an agent to be tested before it is given tools, and the word *agentic* appears exactly once across the EU AI Act and the Regulation that amends it — as a notified-body scope-of-designation code in the new Annex XIV, which creates no duty for anybody. The testing expectations below come from the IMDA Model AI Governance Framework for Agentic AI, version 1.5 of 20 May 2026, updated 5 June 2026. Its register is *organisations should* throughout; the framework calls itself a living document and invites feedback, and it never says shall. Nothing happens to an organisation that reads it and does nothing.
There is one genuinely binding testing duty in the neighbourhood, and it is worth stating precisely because it is routinely borrowed. Article 55(1)(a) of the EU AI Act requires providers of general-purpose AI models with systemic risk to perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks. It has applied since 2 August 2025 and the Digital Omnibus did not disturb it. It reaches model providers above the Article 51 threshold. It does not reach anyone deploying an agent. A deployer citing Article 55 in its testing policy has cited an obligation that binds somebody else, and the citation will not survive the first person who checks it.
The framework sets four test dimensions, and three of them are usually absent. Overall task execution — whether the agent can complete the task accurately — is the one everyone runs. Policy compliance asks whether the agent follows defined procedures and routes for human approval when it should. Tool calling asks whether it calls the right tools, with the right permissions, with the right inputs and in the right order. Robustness asks how it responds to errors and edge cases. An agent that reaches the correct answer by calling the wrong tool with over-broad permissions has passed the first dimension and failed the third, and an evaluation scoring only final outputs cannot tell those two runs apart. Where a permitted action envelope has been drawn at all, the tool-calling dimension is what tests whether it holds — see bounding an agent’s authority.
Single-run evaluation is not a weak measurement of an agent; it is not a measurement of one. Behaviour is stochastic and context-dependent, so the framework asks for runs repeated across varied datasets and minor perturbations, to check that the same inputs and contexts render consistent outputs. A benchmark reporting one pass per task cannot separate a reliable agent from an unreliable one that got lucky, and the low-probability, high-impact behaviour is exactly what a single pass will not surface. The discipline that transfers cleanly from the high-risk regime is the declared-metric one: Article 15 requires levels of accuracy and the relevant accuracy metrics to be declared in the instructions for use, and "task success rate" with no definition attached is the artefact most often offered where a metric should be.
Two further requirements are stated plainly and met rarely. Agents must be tested individually and together, to surface emergent behaviour when they collaborate and the effect on others when one has been compromised; compromise-propagation testing is uncommon enough that finding it is itself a positive finding. And testing should happen in an execution environment mirroring production as closely as possible — against which the framework sets its own caveat, that realism must be calibrated against the risk of prematurely letting an agent reach tools that affect the real world. That trade-off has no clean answer, so where an evaluation sits on it should be a recorded decision rather than a default. For a financial institution the draft MAS Guidelines would add independence on top: paragraph 4.19 proposes formal independent validation before deployment for high risk materiality, by people independent of the development and deployment teams and able to provide effective challenge. That draft is not in force.
§ 3 — What discharges it
What the test file has to show
The artefacts an assessor asks to see, and what makes each one sufficient rather than merely present.
01
A test plan mapped to the four dimensions, with case counts
One column per dimension and a count against each. The policy-compliance column is usually empty, which is the finding: nobody wrote cases asking whether the agent escalated when the procedure said it must.
02
Repetition statistics rather than point estimates
A distribution across repeated runs of the same cases, with the definition of the measure attached to every headline number. A single figure with no definition and no spread describes one execution, not a system.
03
A negative-control suite, with results
Cases the agent is supposed to refuse or escalate, run and scored like any other. Suites made only of tasks the agent should complete cannot distinguish a well-bounded agent from an eager one.
04
An injection test against this deployment’s own untrusted inputs
Instructions embedded in the documents, tickets, pages or messages this agent actually reads, not a generic jailbreak corpus aimed at the model. Indirect injection is a property of the system, so only the system can be tested for it.
05
The environment specification, and its delta from production
Which tools were live and which were mocked, which data was real, and what the difference means for the result. The framework treats realism as a calibrated trade-off, so the calibration ought to be written down somewhere.
06
The reviewer independence record, with one unaccepted finding
Who reviewed, what they were independent of, and at least one point the build team did not simply accept. Review that never disagreed with anything is difficult to distinguish from review that never happened.
§ 4 — Worked example
Worked example — 94% on a single pass
A logistics operator runs an exceptions agent. It reads carrier notifications and customer emails, reschedules deliveries through the transport system, issues goodwill credits up to a set amount and writes to the customer. Before go-live the team ran a 200-case benchmark once, scored 94% task success, and recorded that figure in the release note. The agent has been live four months and the benchmark has not been run again.
Does the benchmark support the decision to deploy?
It supports one of the four dimensions, weakly. Ninety-four per cent of what, measured how, is unanswered: no definition accompanies the figure, so nobody can say whether a case counted as a pass because the outcome was right or because a person downstream corrected it. It was run once against a stochastic system, so it carries no stability information, and the rare failure that matters most is the one a single pass is least likely to produce. Nothing tested whether the agent routes for approval when it should, whether the credit tool is called within its limit, or what happens when the transport system times out halfway through a task. The untrusted input is sitting in plain sight — customer emails are written by third parties, and no test has ever put an instruction inside one. Four months of quiet operation adds little, because no monitoring signal was defined that would have made a noise.
§ 5 — What a weak answer looks like
“We red-teamed it” — the model, not the system
“We red-teamed it.” Pressed, that usually turns out to mean the model was red-teamed, or a guardrail layer was, by the supplier, before any tools were attached. Indirect prompt injection surfaces only at system level, where the agent is reading documents, tickets or web pages that somebody else wrote — one government sandbox exercise found an agent navigating to arbitrary URLs because content on a third-party page told it to. A generic corpus fired at the model establishes nothing about the deployment, and is offered as though it establishes everything.
§ 6 — Elsewhere
The same requirement elsewhere
Where another instrument addresses the same obligation. These are correspondences, not comparisons — the Council does not rank one framework against another.
Article 15 requires levels of accuracy and the relevant metrics to be declared for a high-risk system. The declared-metric discipline is what turns an agent evaluation into evidence a third party can read.
NIST AI RMF
MEASURE 3 addresses measurement where established metrics do not yet exist, which is the situation an agentic evaluation is in. It supplies method for documenting that gap rather than working around it.
MAS instruments
The draft MAS Guidelines propose formal independent validation before deployment for high risk materiality, with effective challenge. Proposed, not in force; the consultation closed in January 2026.
A correspondence indicates that two instruments address the same underlying obligation. It is not a mapping endorsed by either body, not a statement that one satisfies the other, and not a judgement about which is more demanding.
§ 7 — When it applies
Editions of the framework, not deadlines
22 January 2026
Version 1.0 of the Model AI Governance Framework for Agentic AI, launched at Davos. Not an application date — the framework recommends and nothing attaches to an organisation that does not follow it.
20 May 2026
Version 1.5, published 20 May 2026 and updated 5 June 2026. The current edition, and the one to cite: it separates platform providers from system providers, adds automation-bias practices and change management, and introduces the section on systemic and multi-agent risks.
§ 8 — provenance
The provision itself
This page sets out what the instrument requires and what discharges it. The official text is the authority — these go straight to it.
- MDDI newsroom release, Singapore launches new Model AI Governance Framework for Agentic AI, 22 January 2026 — the framework’s launch announcement, which also records that IMDA was developing guidelines for testing agentic AI applications.
- AI Verify Foundation resources index — where the framework and its editions are published.
- Publication dates of the framework’s editions. These are edition dates, not application dates: the framework creates no deadline.
- Article 55(1)(a) and Annex XI Section 2 point 2 — obligations of providers of general-purpose AI models with systemic risk; Article 51 for the systemic-risk threshold; Article 113 for the 2 August 2025 date.
§ 9 — Also read
Also read
Agent audit trails — what a trajectory must capture
Certification
Assessed on the same standard of evidence
Every Council credential is examined on applied judgement against a published anchor, set out the way the obligations on this page are. The Certified Applied AI Associate, the Council’s flagship, is open to any adult today, its fee waived by a coupon code from the AI-Readiness Check, and the register lists what is open for enrolment.