Testing traditional software is a well-established practice: for a given input, you expect a precise output, and the test passes or fails. With an AI agent, that framework is no longer enough. For the same request, the agent may take different paths, phrase its answer differently, call its tools in another order. Two identical runs can give two correct results… or one correct and one wrong.

This is often where agent projects slow down: everyone feels the agent “works pretty well”, but nobody can say how well, or whether it works better or worse after a change. This article presents the approach we use to test an agent before go-live, then to monitor it once in service.

Start from real scenarios

A good test set is not invented in an office: it is built from the requests the agent will actually receive. We start from a sample of real cases, tickets, emails, internal requests, which we anonymize, then complete them with business teams. Each scenario specifies the request, the available context, and what a good outcome looks like.

The test set must cover three families of cases:

  • Common cases, which make up most of the volume and must be handled reliably.
  • Hard cases, which require combining several sources or reasoning in several steps.
  • Cases where the agent should hold back: out-of-scope requests, missing information, situations that must be escalated to a person.

This last family is the one most often forgotten. Yet an agent that confidently answers a question it should not answer is more dangerous than an agent that gets a common case wrong.

Evaluate the outcome, and the path

For an agent, looking at the final answer is not enough. Two dimensions must be evaluated.

The outcome: is the task done? Is the answer accurate, complete, in the expected tone? Is the action taken the right one?

The path: which tools did the agent call, in what order, with what parameters? Did it consult the right source? Did it attempt an action it should not have, even if it was blocked? How many steps did it need? An agent that finds the right answer after fifteen useless calls is slow, costly and fragile.

An agent that gives the right answer for the wrong reasons will eventually give a wrong answer.

How to score: rules, people and an evaluator model

Not every check is done the same way. We combine three approaches:

  • Automatic checks in code: was the right tool called, is the output format respected, is the amount correct, did the agent escalate when it should have? This is the foundation, fast and objective.
  • An evaluator model: to judge more qualitative criteria, such as the relevance or clarity of an answer, a second model can score outputs against a precise rubric. Useful for evaluating many cases quickly, as long as you regularly check that its scores agree with a person’s.
  • Human review: essential to calibrate the other two, and for the most sensitive cases. Domain experts spot errors no rule anticipated.

Because an agent’s behavior varies from one run to the next, each important scenario is run several times. A case that passes one time out of two is not a passing case.

Edge cases and adversarial tests

Beyond business scenarios, you need to actively try to trip the agent up: ambiguous requests, contradictory information in sources, a tool that is down or returns an error, an empty database response, a user who changes their mind midway. You also need to test resistance to manipulation attempts, such as instructions hidden in a document or email the agent reads.

The goal is not for the agent to succeed at everything, but to fail cleanly: by stopping, flagging the problem or handing over, rather than improvising.

The regression set

An agent keeps evolving: new instructions, new tools, a new model version, new document sources. Each of these changes can improve some cases and degrade others, without anyone noticing.

The test set therefore becomes a regression set: it is rerun on every change, and results are compared with the previous version. No change goes to production if it sets back cases that used to work. Every error discovered in production joins this set, so it does not happen again.

After go-live: monitor

Pre-launch tests never cover the full diversity of real life. Once the agent is in service, we track a few simple indicators: task completion rate, escalation rate, steps per task, cost per task, explicit user feedback. A sudden change in any of them is often the first sign of a problem.

We complement these indicators with regular review of a sample of conversations, picked at random and from flagged cases. It is this review that reveals new types of requests, subtle errors and avenues for improvement.

A concrete example

An agent helps the teams of an industrial company find information in technical documentation and create maintenance requests. Before launch, the test set brings together several dozen scenarios built with technicians, a large share of which are cases where the agent must answer that it cannot find the information or pass the request to a manager.

The first tests show the agent almost always finds the right answer, but sometimes creates a duplicate maintenance request because it did not check existing ones. The problem did not show in the final answer, only in the path. A check is added, and the case joins the regression set. Three months later, when switching to a new model version, the test set is rerun in a few hours and confirms nothing regressed before the switch.

Our recommendation

Build the test set before building the agent, or at the latest at the same time. It serves as a specification, a measuring tool and a safety net. An agent without a test set is an agent whose real quality is unknown, and that nobody dares change anymore. An agent with a good test set can evolve quickly, with confidence, and prove its value with numbers.