Skip to content
  • There are no suggestions because the search field is empty.

ISO 42001: 2023 - A.8.3 Documenting AI System Performance During Operation

This article provides guidance on how to implement the ISO 42001: 2023 A.8.3 Documenting AI System Performance During Operation

ISO 42001 Control Description

The organisation shall document the operational performance of AI systems on an ongoing basis, maintaining records that capture performance metrics, monitoring findings, identified issues, and responses to those issues throughout the system's operational life.

Control Objective

To ensure that the organisation maintains a continuous and accurate record of AI system performance in operation, enabling performance trends to be tracked, the validity of risk assessments to be evaluated against operational evidence, and accountability for system behaviour to be discharged on an informed and evidenced basis.

Purpose

The performance of an AI system in operation is not a fixed quantity. It changes over time as the system encounters inputs that diverge from the training distribution, as the world it models evolves, as user behaviours adapt to the system, and as the system itself is modified through maintenance and retraining activities. An organisation that monitors system performance but does not document the results of that monitoring systematically loses the ability to detect performance trends, to assess the significance of individual events in the context of operational history, and to demonstrate to external parties that it has exercised appropriate stewardship over its systems.

Documented performance records serve multiple governance functions. They provide the evidentiary foundation for periodic risk assessment reviews, enabling risk assessments to be grounded in operational experience rather than relying solely on pre-deployment assumptions. They enable auditors and regulators to assess whether the organisation has maintained adequate oversight of its AI systems. In the event of an incident or a regulatory enquiry, they provide the historical record needed to reconstruct system behaviour and to demonstrate that the organisation took appropriate steps to identify and respond to issues as they arose.

Guidance on Implementation

Performance Metrics and Documentation Requirements

The organisation shall define the performance metrics that will be documented for each AI system, aligned with the performance criteria established in the requirements specification and the monitoring framework established under A.7.5. Documentation shall capture metric values at defined intervals or continuously where real-time monitoring is in place, along with any reference thresholds and whether those thresholds were met.

Performance documentation shall also record the context in which performance measurements were made, including information about the volume and nature of operational inputs processed during the measurement period, so that performance figures can be interpreted accurately.

Recording Monitoring Findings

The results of all monitoring activities conducted under A.7.5 shall be recorded in a structured format that enables findings to be reviewed, compared over time, and used as evidence in governance and audit processes. Records shall capture not only performance measurements but also the outcome of reviews of those measurements — including assessments of whether performance is within acceptable bounds, whether any deterioration is apparent, and whether any further action was taken.

Where monitoring findings triggered alerts, escalations, or investigations, these events and their outcomes shall be documented as part of the performance record.

Trend Analysis and Periodic Review

Performance records shall be reviewed at defined intervals to identify trends, including gradual performance degradation, emerging patterns of error or unexpected behaviour, and changes in output distributions that may indicate model drift. The frequency and depth of trend analysis shall be commensurate with the risk profile of the system.

Trend analysis findings shall be documented and shall inform decisions about maintenance activities, risk assessment reviews, and the adequacy of current monitoring arrangements.

Documenting Performance Against Fairness Criteria

Where the AI system is subject to fairness requirements, operational documentation shall include records of performance disaggregated by relevant population groups, enabling ongoing assessment of whether the system continues to meet its fairness criteria in operation. Any detected differential in performance across groups shall be recorded along with the organisation's assessment of its significance and any remedial actions taken.

Integration with Risk and Governance Records

Operational performance documentation shall be integrated with the organisation's broader AI governance records, including the risk register, the AI risk assessment, and the incident management records. Performance documentation shall be explicitly referenced in periodic risk assessment reviews, ensuring that risk assessments incorporate the evidence base provided by operational monitoring.

Record Retention

Operational performance records shall be retained in accordance with the organisation's document retention policy and any applicable regulatory requirements. Retention periods shall be sufficient to support regulatory inspections, audits, and any legal proceedings that may arise from the system's operational history. Consideration shall be given to the need to retain records for the full period during which the organisation may face accountability for the system's operational performance.

Reporting to Governance Functions

The organisation shall establish a process for reporting AI system performance information to relevant governance functions, including senior management and, where applicable, the AI governance body established under the AIMS. Reporting shall provide a clear picture of system performance trends, material issues, and the adequacy of existing controls, enabling informed governance decisions.

Related Controls

  • A.7.5 – AI System Monitoring: Monitoring activities generate the performance data that is documented under this control, and the two controls operate as an integrated system.
  • A.8.2 – AI System Incident Management: Incident records complement performance documentation and shall be cross-referenced to provide a complete picture of operational history.
  • A.7.6 – AI System Change Management: Performance documentation provides the evidentiary basis for change decisions and shall be reviewed when changes are being assessed.
  • A.6.1.2 – AI Risk Assessment: Operational performance documentation shall be used as input to periodic reviews of the AI risk assessment, ensuring risk assessments remain grounded in operational evidence.
  • A.6.2.8 – AI System Documentation: Operational performance records form part of the comprehensive AI system documentation maintained throughout the lifecycle.