Evaluating Agents You Cannot Unit Test

Traditional tests assert that a function returns an exact value. Agents produce different valid outputs on identical inputs, which breaks that model entirely and leaves teams shipping on intuition.

Build the evaluation set from real failures

Start collecting cases the moment the first prototype runs. Every time the agent gets something wrong, that input becomes a permanent test case with a note on what correct looks like. Within weeks you have a set grounded in reality rather than in imagined edge cases.

Grade on properties, not exact text

Assert the things that must be true: the right tool was called, the output validates against the schema, the required fields are present, no forbidden action was taken, the answer cites a real source. These are deterministic checks over a non-deterministic output.

Use a model as a judge, carefully

For subjective quality, a second model scoring against a written rubric works reasonably well, provided you validate the judge against human ratings on a sample first. An unvalidated judge measures its own preferences.

Run the full set on every prompt or model change and track the score over time. Without that, every improvement is a guess and every regression is discovered by a customer.