Human Oversight for Agentic AI Systems

Article 14 human oversight requirements take on a distinct character when applied to agentic AI systems. A human overseer cannot realistically review every action in a multi-step chain, because the agent's core value lies in its capacity to execute workflows autonomously. Oversight design must therefore balance the agent's operational utility against the regulatory requirement for meaningful human control.

This tension shapes every aspect of agentic oversight architecture. The ai governance lead defines action categories, reasoning chain review processes, and escalation thresholds that together maintain Human Oversight compliance without reducing the agent to a step-by-step approval queue. The solution is a tiered approval gate architecture that applies different levels of oversight based on the risk profile of each action the agent can take.

The approval gate architecture categorises every agent action into one of three risk tiers, each with a different oversight treatment. The AI Governance Lead defines these categories by mapping each action to the action schema's risk tiers. The gate configuration is derived from the fria and the risk register, and it is reviewed quarterly.

The three tiers are autonomous, notified, and gated. Autonomous actions are low-risk, reversible operations that the agent may take without human approval: reading a database record, performing a search, or retrieving a document. These actions are logged but not gated. Notified actions are medium-risk operations that the agent may take immediately but that generate a real-time notification to the human overseer. The overseer can review the action after the fact and reverse it if necessary within a defined window; examples include updating a candidate's status, generating a draft communication, or scheduling a meeting.

Gated actions are high-risk or irreversible operations that the agent may not take until a human overseer has reviewed and approved the specific action. The agent proposes the action and execution is paused. The overseer then reviews the proposal, the reasoning trace, and the supporting evidence before deciding to approve, modify, or reject the action. Examples include sending a communication to an affected person, making a recommendation that influences a consequential decision, or deleting data. The complete approval gate architecture is documented in AISDP Module 7.

When an agent encounters a gated action, execution pauses and the proposed action is presented to the human overseer for review. The overseer examines three elements: the proposed action itself, the reasoning trace that led the agent to propose it, and the supporting evidence the agent gathered during its workflow.

Based on this review, the overseer takes one of three decisions. Approval allows the agent to execute the action as proposed, with the approval recorded for audit purposes. Modification directs the agent to revise the proposed action based on the overseer's feedback and re-submit it for further review. Rejection blocks the action entirely, with the rejection logged; the agent then continues its workflow without executing the blocked action.

This three-decision model ensures that Agentic AI Governance maintains meaningful human control at the points of highest consequence. Low-risk and medium-risk actions flow through without creating bottlenecks, whilst irreversible or high-impact actions receive direct human scrutiny before execution.

Beyond individual action approval, the human overseer must be able to review the agent's reasoning chain as a whole. The oversight interface should provide five capabilities: a summary of the agent's current task and progress; the reasoning trace in a human-readable format; the actions taken so far and their results; the actions the agent plans to take next; and the ability to intervene at any point by pausing, redirecting, or terminating the execution.

For long-running agent workflows, the oversight interface should provide checkpoint summaries at defined intervals. These checkpoints enable the overseer to assess whether the agent is still operating within its intended purpose and making reasonable progress toward the intended outcome. Checkpoint-based oversight is particularly important for extended workflows where drift from the original objective becomes more likely over time.

Agentic systems can exhibit emergent behaviour: novel strategies, unexpected tool combinations, or reasoning patterns that were not anticipated during design. Emergent behaviour is not necessarily harmful and may represent effective problem-solving that the designers did not foresee. The risk assessment challenge lies in distinguishing beneficial emergence from harmful emergence.

The fundamental rights impact assessment must account for the possibility that an agentic system's behaviour in production may diverge from the behaviour observed during testing. This requires both pre-deployment characterisation to establish behavioural baselines and Post-Market Monitoring to detect deviations from those baselines once the system is operational.

Pre-deployment behavioural characterisation is conducted by the Technical SME using adversarial scenario testing. The testing presents the agent with scenarios designed to provoke edge-case behaviours, including conflicting objectives, ambiguous instructions, tool failures, and situations that the intended purpose does not clearly address. The agent's behaviour in these scenarios is documented in AISDP Module 6.

Post-deployment behavioural monitoring operates through the post-market monitoring programme. The monitoring tracks four indicators: the distribution of action sequences to identify whether new patterns are appearing; the frequency of tool combinations not observed during testing; the frequency of reasoning chains that exceed expected length or complexity; and the frequency of gated action requests. An increase in gated action requests may indicate the agent is attempting more consequential actions than originally anticipated.

When a novel behaviour is detected, the AI System Assessor evaluates three questions: whether the behaviour falls within the system's intended purpose, whether it introduces new risks not captured in the risk register, and whether it requires changes to the action schema, the approval gate configuration, or the monitoring thresholds. This evaluation is documented in AISDP Module 12.

Action space enforcement must be implemented at the infrastructure level, external to the agent's own reasoning. A three-layer enforcement architecture provides defence in depth. The first layer uses tool definitions and parameter schemas to constrain the agent's available actions. The second layer, implemented as middleware in the action execution service, validates every tool call against the action schema before forwarding it to the tool. The third layer, implemented as an OPA policy, enforces cross-cutting constraints including rate limits, permission boundaries, and approval gate requirements regardless of the specific tool being called. This three-layer architecture is documented in AISDP Module 3.

Iteration and timeout limits prevent runaway agent behaviour. The Technical SME sets a maximum iteration count, representing the number of tool calls per execution, and a maximum execution duration. Both limits are enforced at the orchestration level, not within the agent's code. When either limit is reached, the execution terminates gracefully: in-progress tool calls are completed, the reasoning trace is finalised, and the overseer is notified. These limits are calibrated to the system's intended workflows and documented in AISDP Module 3.

Adversarial goal injection testing is the agentic equivalent of prompt injection testing. The red team presents the agent with inputs designed to redirect it from its intended task toward a different goal, such as directing a candidate screening agent to search for unrelated confidential information instead. The testing evaluates whether the agent's goal boundary, derived from the system prompt and the action schema, holds under adversarial pressure. Results are documented in AISDP Module 9.

Without automated action space enforcement, the human overseer reviews every action before execution by placing all actions in the gated category. This maximum-oversight configuration means the agent functions as a proposal engine rather than an autonomous executor. The overseer reviews each proposed action, approves or rejects it, and the agent proceeds based on the overseer's decision.

This approach is feasible for agents with low action volume, specifically fewer than 50 actions per day. For high-volume agents, the approval burden on the overseer becomes unsustainable and operational efficiency collapses. In those cases, automated enforcement with selective gating, following the Approval Gate Architecture described above, becomes necessary to maintain both compliance and operational viability.