There is a specific moment in most AI agent projects where enthusiasm meets reality. The demo has been shown to leadership, everyone is impressed, and then somebody asks what happens when the agent is wrong.
If the answer is a pause, the project is about to stall. Here is what tends to be missing.
The demo is the happy path, and production is not
A demo is built against documents someone chose, questions someone anticipated, and a system that happened to be responding normally. Production is a scanned form at an angle, a question phrased in a way nobody predicted, and a downstream API that is timing out.
The gap is not model quality. It is that nobody defined what the agent should do when it is uncertain. An agent that must always produce an answer will always produce an answer, including when it should have escalated.
The single highest-value design decision is giving the agent an explicit, first-class way to say it does not know, and a route to a human when it does.
Evaluation before implementation, not after
Teams routinely build the agent first and think about measurement later. This makes every subsequent change a matter of opinion: someone tries a new prompt, it feels better on three examples, and it ships.
Building a labelled evaluation set before the agent is the least glamorous and most valuable step in the project. A few hundred real examples with known-correct outputs turns prompt engineering from taste into measurement. When the model provider ships an update, you find out in an hour whether it helped or hurt.
The uncomfortable side effect is that you will discover the agent is worse than you thought. That is the point.
Confidence has to route, not decorate
Many systems produce a confidence score and then display it next to the answer, where it functions as decoration. A human reviewer glances at it, sees a high number and moves on.
Confidence earns its place only when it changes what happens. Above the threshold, the output flows through automatically. Below it, the item lands in a review queue. The threshold itself is a business decision, not a technical one: it encodes how costly an error is compared with how costly a human review is.
On one document processing engagement, ninety-one percent of extracted fields cleared the threshold and the remaining nine percent went to a reviewer. That split is what made the system trustworthy. Trying to automate the last nine percent would have destroyed confidence in the other ninety-one.
Grounding with citations, not just retrieval
Retrieval-augmented generation reduces fabrication but does not eliminate it. The model can still summarise retrieved context inaccurately, or blend two sources into a claim neither of them made.
Requiring the system to cite the specific source span for each claim changes the reviewer's job from re-reading the document to verifying a highlighted line. That is a difference of seconds versus minutes, and it is what makes human-in-the-loop economically viable at volume.
What this costs you
Doing all four adds meaningful time to the first delivery. An agent with an evaluation harness, guardrails, confidence routing and citation takes noticeably longer to build than one without.
It also tends to be the difference between a system that is still running in eighteen months and a prototype that quietly stopped being used. We have inherited enough of the second kind to have a strong opinion about which is cheaper.
If you are scoping an agent and want an honest conversation about whether it is the right tool for your problem, talk to our AI practice.