Auditing AI systems

There is a widespread assumption that auditing an AI system is a method nobody has written down yet. It is not. The engagement is governed by the same instruments that govern every other internal audit — the IIA Global Internal Audit Standards for the conduct of the work, ISO 19011 for the programme around it — and those standards are specific at exactly the points practitioners believe they are silent. What is genuinely new is narrower than it looks: the evidence is harder to obtain and harder to reproduce, and the criteria to audit against frequently do not exist yet.

That second problem has a stated answer. Where the criteria management has set are inadequate, the auditor does not report that there was nothing to audit against — Standard 13.4 requires them to identify appropriate criteria with the board or senior management, and names authoritative industry frameworks as an acceptable source. And where evidence cannot be obtained at all, Standard 14.1 is explicit that the auditor must consider whether the absence is itself a finding. Both are routinely read the other way round.

The evidence problem is where AI departs from ordinary audit practice, and the departure is not the one people expect. Sufficiency, under both 14.1 and 14.6, is a reperformance test: could a competent person work from this file and reach the same conclusion? For a probabilistic system that makes the model version, the inference parameters and the date load-bearing evidence rather than metadata, because without them nothing in the file reperforms. These pages set out what each standard requires, who it binds, and the artefacts an assessor asks to see — which the standards themselves do not specify, because they state outcomes and leave the evidence to practice.