Model Selection for Agentic AI Systems

Agentic AI systems take actions in the world rather than simply producing predictions or recommendations. They may browse the web, execute code, send communications, modify databases, or interact with external services on behalf of a user or an automated process. Article 3(1) of the eu ai act references systems that generate outputs "such as predictions, content, recommendations, or decisions," and this definition encompasses agentic behaviour where those outputs trigger real-world actions.

An agent that sends an email, modifies a database record, or initiates a financial transaction has moved from the advisory space into the consequential space. The compliance controls must match this heightened risk. Model selection decisions for these systems cannot rely solely on accuracy metrics; they must account for the safety, reversibility, and traceability of the actions the system performs.

These systems occupy a distinct compliance position because their outputs are not merely predictions or recommendations: they are actions with consequences. The model selection process must therefore address a set of considerations that go beyond conventional prediction systems, including action space constraints, reversibility assessment, chain-of-action traceability, and failure mode analysis. Model Selection Under the EU AI Act covers the broader model selection framework within which agentic considerations sit.

The core mitigation for agentic risk is action space bounding. Before deployment, the Technical SME defines every action the agent can take in an explicit, typed schema. The schema specifies the action name, its parameters with type constraints, and the permitted range of each parameter. Any action request that falls outside the schema is rejected, regardless of how plausible or helpful the agent believes it to be. This is a hard constraint, not a soft preference.

In framework terms, LangChain and LangGraph implement this through tool definitions: each tool has a Pydantic model specifying its input schema, and the framework validates every tool call against the schema before execution. Anthropic's tool-use API works similarly, with structured JSON schemas defining what each tool accepts.

An agentic system operating in a high risk ai system domain must have a bounded action space that can be documented, tested, and monitored. Open-ended architectures that can theoretically take any action, such as general-purpose code execution agents, require extensive guardrails whose effectiveness the Technical SME demonstrates in the aisdp.

The practical challenge of action space bounding is drawing the boundary correctly. Too narrow, and the agent cannot accomplish its intended purpose. Too broad, and the agent has excessive agency (classified as OWASP LLM08), creating unnecessary risk surfaces. Boundary derivation begins from the system's documented intended purpose in AISDP Module 1.

Every action that is necessary for the intended purpose is permitted; every action that is not is excluded. The AI System Assessor documents the mapping from intended purpose to permitted actions, and this mapping is reviewed during conformity assessment. This documentation creates the evidentiary chain linking each permitted action to a legitimate operational need, which auditors can verify against the system's actual behaviour in production.

The risk assessment for agentic systems must consider the reversibility of the actions the system takes. A system that recommends a decision to a human operator, who then acts on it, has a natural reversibility point: the human can choose not to act. A system that autonomously modifies a database record, sends a communication, or adjusts a process parameter may take actions that are difficult or impossible to reverse.

The model selection should favour architectures that support human-in-the-loop checkpoints for irreversible actions. This is not simply a best practice recommendation; it follows directly from the risk proportionality principle. Actions with greater potential for harm require stronger safeguards, and irreversibility is a key factor in assessing harm potential.

Each permitted action is classified by the Technical SME into one of three categories based on the effort required to undo it. Reversible actions can be undone programmatically, such as a draft email that can be deleted before sending. Partially reversible actions can be undone with effort, such as a database update that can be rolled back from a backup. Irreversible actions cannot be undone at all, such as a sent communication, a disclosed piece of information, or a physical actuator command.

Irreversible actions require a human-in-the-loop checkpoint: the agent proposes the action, a human reviews and approves it, and only then does execution proceed. LangGraph supports this pattern through "interrupt" nodes that pause the graph execution and wait for human input.

For high risk ai system deployments, the AI System Assessor logs each approval step with the approver's identity, timestamp, and the full action proposal. This logging creates an audit trail that demonstrates human oversight was exercised at each irreversible decision point, supporting compliance with the oversight requirements. Human Oversight Under the EU AI Act addresses the broader framework for human oversight mechanisms that these checkpoints fit within.

Agentic systems often take sequences of actions where each action depends on the result of the preceding one, and the model architecture must support logging this entire chain. The chain-of-action log captures the reasoning and action sequence: the initial prompt or trigger, each intermediate observation the agent made, each decision point and the reasoning behind it, each tool call with its parameters and return value, and the final outcome. This log is essential for article 12 compliance with automatic recording and for diagnosing failures in post-market monitoring.

For a prediction system, the log captures the input and output. For an agentic system, the log must capture the complete execution trace. The engineering team structures the chain-of-action log as JSON with a defined schema, writing it to append-only storage for immutability and indexing it for efficient retrieval of a specific execution trace.

This chain-of-action log enables reconstruction of what the agent did and why during post-incident investigation. When an agentic system produces an unexpected outcome, the operations team can trace back through the entire sequence of observations, decisions, and tool calls to identify where the reasoning deviated from the expected path. Without this level of traceability, diagnosing failures in agentic systems becomes significantly more difficult than in conventional prediction systems. Logging and Monitoring for AI Systems provides detailed guidance on structuring these audit trails.

Agentic systems have more complex failure modes than prediction systems. A prediction system can fail by producing an incorrect output. An agentic system can fail in multiple distinct ways: by taking an incorrect action, by taking the correct action at the wrong time, by entering an infinite loop, by escalating its own permissions, or by interacting with external systems in unintended ways.

Model selection should assess the candidate architecture's susceptibility to each of these failure modes and the availability of safety mechanisms to address them. The key safety mechanisms are timeouts, action limits, permission boundaries, and human approval gates for high-consequence actions.

The candidate architecture must support these mechanisms natively or through integration with the orchestration framework. A model architecture that does not permit external enforcement of action limits introduces a risk that cannot be mitigated through operational controls alone. The Technical SME evaluates each candidate architecture against the full catalogue of agentic failure modes and documents the assessment in the AISDP.

Timeout and iteration limits are safety mechanisms that prevent runaway behaviour. An agent that enters an infinite loop, repeatedly retries a failed action, or escalates its own permissions can cause harm that accumulates over time. The timeout should be set at a level that allows the agent to complete its intended task but terminates execution if the task takes abnormally long.

The iteration limit caps the number of tool calls in a single execution. A reasonable starting point is ten to twenty for most workflows, though this is system-specific and should be calibrated during testing. Both limits must be hard-enforced at the infrastructure level, not within the agent's own code, which may be overridden by adversarial inputs. Enforcement at the infrastructure level means the orchestration framework or the deployment platform terminates execution, regardless of what the agent's own logic attempts.

When either limit is triggered, the event is logged for review by the operations team. This logging serves both operational and compliance purposes: operationally, it alerts the team to potential issues; for compliance, it demonstrates that safety mechanisms are functioning as designed. Safety Mechanisms for AI Systems covers additional safety controls that complement timeout and iteration limits.