Building an evaluation harness for production agents

Traditional unit tests work because the same input always produces the same output. LLMs are non-deterministic even at temperature 0. The same prompt can produce subtly different responses across model versions, sampling runs, and context lengths. A test that passes today can fail after a model update you didn't ask for.

The second problem is that the failure surface is semantic, not syntactic. Your agent doesn't throw an exception when it hallucinates a refund policy. It returns a 200 with a confidently-worded lie. Standard assertion-based tests can't catch that. You need evaluation that understands meaning.

The third problem is coverage. The unhappy paths that matter in production, such as ambiguous questions, multi-turn context confusion, and users who push against policy, are exactly the paths that are hard to enumerate in advance. A good eval harness grows its coverage over time by capturing production failures and converting them into permanent regression tests.

A golden conversation is a complete multi-turn exchange with an expected outcome label: correct answer, correct escalation, or correct refusal. We seed golden sets two ways: manually curated cases from the support team, and automatically harvested from production conversations that were reviewed and rated by humans.

Each golden case includes the full conversation history, the expected final action (respond, escalate or refuse), a reference answer for response cases, and a set of must-not-contain strings for refusal cases. The eval runner replays the conversation up to the final turn, generates a new response, and scores it against the expected outcome.

Here's an example golden case in YAML:

We track four primary metrics. Factual accuracy: what percentage of response-case golden conversations produce an answer consistent with the source documents? We use an LLM-as-judge approach with a dedicated fact-check prompt, which is cheap, fast, and well-calibrated against human raters on our domain. Tone match: does the response match the brand voice rubric? Scored 1 to 5 by a secondary judge prompt.

Escalation precision and recall are the most operationally important. Precision: of the conversations where the agent escalated, what fraction actually needed human intervention? Low precision means agents are over-escalating and drowning your human queue. Recall: of the conversations that needed escalation, what fraction did the agent catch? Low recall means the agent is handling things it shouldn't. Both matter and they trade off against each other, so tune your escalation threshold intentionally rather than by accident.

We also track latency percentiles (p50, p95) and context utilization, meaning what fraction of the context window is actually needed, to catch regressions before they affect users.

Our eval suite runs as a GitHub Actions workflow triggered on any change to the agent configuration directory. The workflow spins up an ephemeral eval environment, loads the golden conversation set from S3, runs the conversation replay, scores results, and posts a summary comment to the PR with a table of metric deltas against the current production version.

A PR is blocked from merging if factual accuracy drops more than 2 percentage points, escalation recall drops below 90%, or latency p95 increases by more than 200ms. These thresholds are version-controlled alongside the agent config. If the team decides to accept a tradeoff, they do it explicitly with a config change, not implicitly by letting a test fail.

The total eval runtime for our current golden set (340 conversations) is 2 minutes 18 seconds. We run it in parallel across 8 workers. If your eval takes longer than 5 minutes, engineers stop running it locally, and the feedback loop breaks. Keep it fast.