How many eval cases do you need for an LLM agent?

Enough that a score change you care about is larger than the run-to-run noise. In practice that is 200–500 scenarios per agent role. Below roughly a hundred, the confidence interval is wider than most real regressions, and the suite will tell you a change is fine when it is not.

Size the suite from the effect you need to detect rather than from a round number. If you want to catch a five-point drop in task success on a metric that varies by three points between identical runs, a few hundred cases gets you there; fifty does not.

Weight coverage toward the failure surface, not the happy path. Most of the value sits in ambiguous requests, missing context, tool timeouts, and adversarial inputs — the cases where the agent has to decide whether to act or ask.

Run the suite at two cadences. A fast subset of 30–50 cases on every pull request keeps the loop tight; the full suite runs nightly and before release.

Written by Binary AI Labs · Reviewed 2026-11-01