The Eval Harness Behind Our Voice Agent
How we test a voice agent on real calls — score every call, calibrate the judges, and replay real failures before shipping.
A caller told us her name. Our voice agent heard "Can I do onion rings?"
A script would never predict that failure.
1. The problem
Eve's voice agent answers intake calls for plaintiff law firms. The caller is usually someone recently hurt — a crash, a fall, a job injury — calling a law firm for the first time. When the agent mishears a name, loops on a question, or transfers to the wrong number, an injured person doesn't reach a lawyer. The firm loses a case it never knew called.
Intake is the sharpest case, but the same agent also answers firms' front-desk lines, calls back people who asked about hiring the firm, and makes outbound calls about ongoing cases — every failure mode in this post applies to all of them, and the same harness grades them all.
Voice breaks in ways nobody scripts for. And at thousands of calls, nobody can listen to them all. For months our only quality signal was anecdotes — a firm's complaint here, a painful recording there.
2. The numbers that told us
Scoring every call turned the anecdotes into rates, and the rates were worse than the stories. Transfers sometimes dialed hallucinated numbers — one caller bound for a law firm rang a carpet cleaner in the same area code. A closing line that was supposed to vary appeared, word for word, on a large share of calls. A firm's "she sounds robotic" report stopped being one firm's impression and became a pattern we could see across firms. A problem you can count is a problem you can fix — and prove fixed.
3. What we built
We grade our voice agent on the real calls where it breaks. Every confirmed failure becomes a permanent test. A prompt or model change must pass those tests before it ships.
A bug once caused some callers to be sent to the wrong number on transfer. Through our eval harness, we captured all instances of this happening and cut those misdials by two-thirds. We proved it on the exact real calls before anyone heard the change. This post shows how we score, judge, and replay real calls.
Grade every call
Our scorers grade every call the moment it ends. Scorers are our automated judges. Some are LLM judges. Some are simple rule checks. Seventeen run today.
One scorer judges whether the call met its goal. Another rates conversation quality across nine dimensions. Another measures barge-in — how often the agent talks over the caller. We score barge-in from the audio. Transcripts hide the overlap exactly where it happens.
Every call trace is stored. The scores roll up into a dashboard anyone on the team can slice. We tag every call with what the caller wanted — a quote, an appointment, a human. The dashboard ranks failures by those tags. So we see which caller requests break most, and fix what people ask for first.
The failures are often human and funny. Counting them turns a good story into a number we can track.
Calibrate the judges
An LLM judge can be wrong. Grade the graders, or your metrics will lie.
We run labeling parties. People score a batch of calls by hand. We compare each scorer to those human labels. We track how often the scorer agrees. Disagreement flags a scorer that needs work.
The first result humbled us. Half of our first "quality gain" came from deleting false positives. Those were good calls a scorer had flagged as bad. A metric that grades your judge's mistakes is worse than none.
Freeze the call, replay the next turn
This is Voice Agent Bench, our replay harness.
We take a real call. A turn is one reply in the conversation — the caller speaks, then the agent speaks. We stop at one turn. We freeze everything the agent knew at that moment — the transcript so far, its tools, its variables. We run the candidate agent on that frozen moment. We capture its next turn. A scorer grades that turn.
How does the scorer know what a good turn is? Each replayed call carries a one-line rule, written by the person who confirmed the failure. For example: "do not reconfirm contact details before the transfer." An LLM judge reads the transcript and the new turn. It grades the turn against the rule — pass or fail — and writes down why. The rule is the failure that motivated the test. A pass means the exact bug is gone.
Why the next turn, and not a whole conversation? A full simulated call drifts. Each made-up caller reply pulls the test further from reality. Freezing a real moment removes the drift. We test the one decision we changed. A replay costs ten times less than a simulated call. It runs thirty times faster. Cheap and fast means we run it on every change.
Turn failures into permanent tests
A firm reports "Jenny sounds robotic." Wall-E, our always-on engineering agent, finds the source call. We confirm the failure. We add it to Voice Agent Bench as a permanent test. We fix the prompt. The fix must pass that test and every other one. The bug cannot come back silently.
This is how we reduced transfers to the wrong number. We cut misdials by two-thirds on the replayed calls. The transfers that already worked kept working. That guardrail matters as much as the fix.
A quality win: reshaping how a call ends
The harness improves quality, too. Our agent used to end calls with a stiff, scripted goodbye. A scorer flagged the pattern. The dashboard showed it was routine. We rewrote the closing in the prompt. Voice Agent Bench replayed the exact calls where it happened and confirmed the new ending held. The scripted line dropped to zero. Closings now sound like a person.
Test every change the same way
Every change runs the bench before it ships. A one-word prompt edit. A new tool. A full model swap. Small edits break real calls as easily as big ones. The transfer fix and the closing rewrite above both passed through it.
A model swap is the scariest change of all. We once ran several models on the same set of replayed calls. No candidate beat the model we already ran. One model hung up early on most calls. The bench caught it before any customer did.
Good tests also say no. That is the point of the harness.
4. What improved
Three results, each proved on replayed real calls before anyone heard the change. Transfer misdials fell by two-thirds — and the transfers that already worked kept working. The scripted closing line fell to zero (Figure 3). And a model swap that would have hung up on callers never shipped, because the bench said no. The pattern is the same every time: the dashboard finds it, the bench replays it, and the fix proves itself on the exact calls that failed.
5. Takeaways for your own evals
We're hiring
We build the harness that keeps a live voice product honest. Scoring, calibrating, and replaying real calls is the daily work. If that sounds like your kind of problem, come build it with us. See our open roles.
