What AI KPIs should I set?

Key Performance Indicators (KPIs) are measurable values that show how well you are progressing toward your AI Management System (AIMS) objectives. Good KPIs help you understand how effectively your AI Management System is operating, where improvement is needed, and how well AI-related risks are being managed over time.

AI KPIs should be relevant to your objectives, measurable, and reviewed as part of ongoing monitoring and management review. ISO/IEC 42001 expects performance evaluation and continual improvement of your AIMS through metrics and evidence (see Clause 9.1 – Monitoring, measurement, analysis and evaluation and Clause 9.3 – Management review).

Below are example AI KPIs you may wish to use or adapt, including a section that shows how KPIs link back to common AI objectives.

Examples of AI KPIs

These KPIs provide a high-level view of how effectively the Artificial Intelligence Management System (AIMS) is operating overall.

Number of AI-related incidents
Indicates how often AI systems produce unintended, harmful, or unexpected outcomes.

Number of AI-related non-conformities
Shows how well the organisation complies with its AIMS processes and ISO/IEC 42001 requirements.

Number of stakeholder or customer complaints relating to AI systems
Provides insight into how AI systems are perceived and whether transparency or explainability issues exist.

Percentage of AI systems with a completed and reviewed AI System Impact Assessment
Demonstrates the extent to which AI-related risks and impacts are identified and assessed across AI systems.

Percentage of staff involved in AI activities who have completed required AI training
Helps demonstrate organisational competence and awareness related to responsible AI use.

Number of AI-related supplier or third-party issues identified
Indicates the effectiveness of oversight of AI-related suppliers and dependencies.

Examples of KPIs linked to AI objectives

The following examples show how KPIs can be linked to specific AIMS objectives:

Objective: Strengthen AI governance and oversight

KPI:
Number of AI systems with identified oversight issues during the reporting period

Objective: Improve AI risk assessment coverage

KPI:
Percentage of AI systems with a completed and reviewed AI System Impact Assessment

Objective: Increase transparency of AI use

KPI:
Number of stakeholder or user queries, complaints, or challenges relating to AI systems

Objective: Enhance explainability of AI outputs

KPI:
Number of stakeholder challenges or queries relating to AI decision explainability

Objective: Strengthen AI lifecycle management

KPI:
Percentage of AI systems reviewed during the reporting period for lifecycle or performance concerns

Objective: Improve monitoring of AI performance and behaviour

KPI:
Number of AI-related incidents or unintended outcomes identified

Objective: Improve AI incident management

KPI:
Average time to resolve AI-related incidents

Objective: Increase AI competence and awareness

KPI:
Percentage of staff involved in AI activities who have completed required AI training.