The Human Oversight Gap in AI Deployment

The AI Act does not regulate AI agents as a separate category. An agent may nevertheless constitute, or form part of, a high-risk AI system because of its intended purpose. Annex III captures specified uses in areas including recruitment and employment, access to essential services, creditworthiness, certain life and health insurance risk assessment, and AI used as a safety component in specified critical infrastructure.

Not every system that falls within an Annex III use case is necessarily high-risk. Article 6(3) provides an exception for systems that do not pose a significant risk of harm to health, safety or fundamental rights, subject to specified conditions. Where an organisation deploys a system for an Annex III use case, it therefore needs to understand whether the system is classified as high-risk and whether the associated obligations apply. Vendor documentation may support that understanding, but the organisation must still consider how the system is actually being used in its own operating environment.

In practice, effective oversight requires three elements.

Designation

Human oversight must be assigned to individuals with the competence and training needed to understand the system's capabilities, limitations and context of use.

Authority

Oversight personnel must have the ability, where appropriate and proportionate to the risk, to disregard or reverse the system's output and to interrupt its operation through a stop mechanism or equivalent procedure. Formal assignment must be matched with the authority and organisational support to intervene effectively.

Evidence

The organisation should be able to evidence how oversight operates in practice. System logs provide part of that evidence; internal records can capture oversight assignments, escalations, interventions and resulting decisions.

Article 26 places the obligation on the deployer, not specifically on the board. For the board, the governance question is therefore one of assurance: can management demonstrate that high-risk systems have been identified, oversight has been assigned to appropriately competent people, and those people can intervene effectively when necessary?

Provider documentation can explain how the system is designed to be overseen. It cannot demonstrate that the deployer has actually assigned competent people, given them appropriate authority and support, or embedded oversight into its own operations. A useful governance record should therefore show which deployed AI systems have been assessed as high-risk, who holds oversight responsibility for each, and whether monitoring, interventions and escalations indicate that the oversight arrangement is functioning as intended.