Auditing AI systems
Standards 14.3, 15.1 and 15.2 — Evaluation of Findings · Final Engagement Communication · Confirming the Implementation of Recommendations or Action Plans
A finding is not finished when it is written. Standard 14.3 asks how significant it is, 15.1 governs what the final communication may be made to say, and 15.2 requires the function to confirm that agreed action was actually implemented. On an AI system the last of these is the hardest, because a control can be implemented and still stop working.
§ 1 — When it applies
Which edition this is, and what the date means
9 January 2024
The publication date printed on the Global Internal Audit Standards, 2024 edition — an edition date for a professional standard, not a compliance deadline. No separate effective date is asserted: the 2024 framework supersedes the 2017 IPPF, which is no longer effective for internal audit functions.
§ 2 — In practice
Grading what you saw, confirming what changed
Significance under Standard 14.3 is a product of likelihood and impact, and on a probabilistic system both terms are estimated from an observation rather than read off a control’s design. The instinct is to grade what was seen: eleven exceptions in a sample of two hundred becomes a medium. That understates the class of failure AI systems produce, because the control that failed did not fail once — it failed for every input sharing the property that defeated it, silently, for as long as the property was present. Grading the sample rather than the population is a common way an AI finding arrives at the audit committee smaller than it is. The structural work of stating criteria, condition, cause and effect belongs upstream, in writing an AI audit finding; what happens here is the judgement about how much it matters.
Standard 15.1 governs the final engagement communication, and it carries the sentence that decides most contested AI findings. Where the auditor and management cannot reach a mutual understanding of the engagement results, internal auditors “must not be obligated to change any portion of the engagement results unless there is a valid reason to do so”. Pressure on an AI finding rarely arrives as disagreement about the facts. It arrives as a methodological objection — the prompt was unrepresentative, the temperature was not production’s, the evaluation set was stale. Sometimes that is a valid reason and the finding must change. Sometimes it is an argument that no test could ever have produced a finding, which is not a reason at all. Telling the two apart is the auditor’s job, and the standard puts the burden of establishing the reason on the party asserting it.
Standard 15.2 requires the function to confirm implementation of agreed action plans, and the word doing the work is confirm. A change ticket marked done, a screenshot of a new configuration and an email from the owner are all assertions that something was put in place; none of them is a confirmation that it operates. That gap is ordinary in audit practice and acute here, because the controls placed on AI systems degrade without announcing it: a threshold tuned for last quarter’s traffic, a human-review step whose reviewers learn to agree with the model, a guardrail that quietly stops matching a changed output format. A closed action plan is therefore not a closed risk. What closes the loop is a re-test held to the same standard as the original work — see evidence sufficiency for AI controls.
§ 3 — Who it binds
Which engagements this reaches
Internal audit function
Any internal audit function conforming with the Global Internal Audit Standards, on any engagement — the reporting and follow-up standards are not AI-specific and carry no AI-specific exemption. What changes for an AI engagement is what counts as confirmation, because the controls being confirmed can be present and inoperative at the same time.
§ 4 — What discharges it
What the reporting and follow-up file must show
The artefacts an assessor asks to see, and what makes each one sufficient rather than merely present.
01
The grading rationale, against the function’s own methodology
Not the rating. The reasoning that produced it: what population the impact was estimated over, whether the estimate assumed the failure recurs, and which of the methodology’s thresholds it was measured against. A rating with no rationale cannot be challenged, and an AI grading is challenged.
02
The disagreement record, where mutual understanding was not reached
Standard 15.1 contemplates that it may not be. The record names what management contested, what the auditor changed and did not change, and the reason accepted or rejected — so a reader can see whether the finding moved on evidence or on pressure.
03
An action-plan tracker carrying re-test dates, not closure dates
A closure date records when management said the work was done. A re-test date records when the function intends to look again, and at what. For controls that decay, the second is the field that makes the tracker an audit artefact rather than a project log.
04
A post-implementation re-test on new measurement
Measurement taken after the release, over a period during which the remediation was live, drawn the same way as the original test. Deployment artefacts show that something changed; only new measurement shows that the behaviour did.
§ 5 — Worked example
Worked example — a control that was deployed, not confirmed
An internal audit function found that the human-review step on a fraud-scoring model was being bypassed: reviewers cleared 11% of flagged cases in under three seconds, which is less time than the case screen takes to load. Management agreed the finding and committed to a hard block preventing a decision within ten seconds of the screen opening. Six months later the follow-up file records the change ticket, the release note and the product owner’s confirmation that the block is live.
Has the function confirmed implementation under Standard 15.2?
It has confirmed deployment, which is a different thing. The three artefacts establish that a control was built and shipped; none of them establishes that the behaviour described in the finding has stopped. The obvious re-test is cheap — pull the same population for a month after the release and measure decision latency again — and it is the only thing in the file that reperforms. There is a second question nobody asked. The original grading rested on the 11% observed in a sample, and a timing block addresses the mechanism that sample revealed rather than the reason reviewers were rubber-stamping. If the cause was volume, the block converts a fast bypass into a slow one and the risk survives the closure. Follow-up is where that becomes visible, provided the re-test measures the outcome the finding was about rather than the fix management chose.
§ 6 — What a weak answer looks like
The tracker where everything is green
A tracker in which every row is green. Closure is evidenced by the date management supplied and the ticket reference management supplied, and the function has attested to nothing it observed itself. This survives until someone asks the obvious question about an AI control — is it still doing what it was put in place to do — and the file holds no measurement taken after the remediation to answer with. The finding was closed on an assurance from the party the finding was about.
§ 7 — Elsewhere
Where this sits alongside other instruments
Where another instrument addresses the same obligation. These are correspondences, not comparisons — the Council does not rank one framework against another.
Article 72 puts a continuing duty on the provider to collect and analyse field data over a system’s lifetime. Follow-up under 15.2 asks a narrower question of a single control — whether the remediation agreed after an engagement is still operating.
A correspondence indicates that two instruments address the same underlying obligation. It is not a mapping endorsed by either body, not a statement that one satisfies the other, and not a judgement about which is more demanding.
§ 8 — provenance
The provision itself
This page sets out what the instrument requires and what discharges it. The official text is the authority — these go straight to it.
§ 9 — Also read
Also read
Evidence sufficiency for AI controls
Certification
Assessed on the same standard of evidence
Every Council credential is examined on applied judgement against a published anchor, set out the way the obligations on this page are. The Certified Applied AI Associate, the Council’s flagship, is open to any adult today, its fee waived by a coupon code from the AI-Readiness Check, and the register lists what is open for enrolment.