An AI agent that works well in a demo impresses with its autonomy: it reads, searches, decides, acts. In production, the opposite is often what makes it valuable: its ability to recognize that it should not act alone. An agent that sends a wrong email to a customer, grants an unjustified commercial gesture or edits the wrong CRM record costs more than the time it saved.

“Human in the loop” does not mean having every action approved by a person, which would defeat the purpose of the agent. It means deciding, at design time, exactly when the agent must stop, ask, or hand over. This article presents the mechanisms we put in place to achieve that.

Not all actions are equal

The first step is to rank the actions the agent can take by their impact. A simple grid is often enough:

  • Read and search: consult a knowledge base, a history, a customer record. Low risk, the agent can act alone.
  • Prepare: draft a message, pre-fill a form, suggest an answer. The agent produces, a person decides whether to use it.
  • Act reversibly: create a task, add an internal note, classify a ticket. The agent can act alone if the action is logged and easy to undo.
  • Act irreversibly or visibly: send a message to a customer, trigger a payment, delete data, change a contract. Human approval required.

This classification is done with business teams, not only technical ones. They are the ones who know which errors are acceptable and which are not.

Approval points

For sensitive actions, the agent does not stop at a simple “are you sure?”. It gives the approver everything they need to decide quickly and well: the proposed action, the information it relies on, the sources consulted, and what will happen after approval. A good approval request takes seconds to handle; a bad one forces the person to redo all of the agent’s work.

Format matters too: the approval should arrive where the person already works, in their inbox, ticketing tool or CRM, rather than in an extra interface they will end up ignoring.

A good agent is not the one that does everything alone. It is the one that knows exactly when it should not.

Knowing when to escalate

Beyond sensitive actions, an agent must recognize situations outside its scope and pass them to the right person. The most useful escalation triggers are often simple:

  • the request matches no planned case;
  • the information found is contradictory or incomplete;
  • the customer expresses dissatisfaction, urgency or a threat to leave;
  • the agent has already tried several approaches without success;
  • an amount, volume or stake exceeds a defined threshold.

A successful escalation passes on the full context: what the agent understood, what it has already done, and why it is stopping. The person taking over should never have to start from scratch.

Confidence thresholds: useful but not enough

It is tempting to ask the model to rate its own confidence and escalate below a certain level. It helps, but handle it with care: a model can be very confident and wrong. The most reliable signals are often outside the model: was the source found, are the amounts consistent, is the customer known, is the business rule respected?

So we combine both: explicit checks in code, and the model’s self-assessment as an additional signal. Thresholds are set from real cases, then adjusted as we observe how the agent behaves.

Trace everything, and be able to undo

Every agent action must leave a readable trace: what it did, based on which information, when, and who approved it where relevant. This traceability is used to understand an incident, improve the agent, and be accountable when needed.

It goes hand in hand with the ability to undo. Whenever possible, we design the agent’s actions to be reversible: a draft rather than a send, a status change rather than a deletion, a versioned edit rather than an overwrite. A system whose mistakes can be undone is deployed with far more peace of mind.

These safeguards also protect against prompt injection: even if planted content manages to hijack the agent, a sensitive action cannot go through without a person seeing it.

A concrete example

A customer service agent handles refund requests for an e-commerce site. It checks the order, verifies the return conditions and prepares a reply. In a first version, it could grant every refund it judged compliant on its own.

The deployed version distinguishes three cases. Standard refunds, below a defined amount and within the conditions, are granted automatically and logged. Requests outside the conditions or above the threshold reach an advisor with a complete file: customer history, the agent’s analysis, a suggested reply. Messages expressing strong dissatisfaction are escalated immediately, with no automatic reply. Advisors now only handle the cases that deserve their attention, and every agent decision can be found and explained.

Our recommendation

Start with more human approval than you need, then gradually relax it. The first weeks are for observing the agent’s decisions, identifying the cases it handles reliably and those where it hesitates. Thresholds and approval points are adjusted on those observations, not on assumptions. An agent that earns the team’s trust step by step ends up handling far more than one launched in full autonomy and quickly switched off after its first visible mistake.