When should you start writing evals for an AI feature?

Before the second prompt change. The moment you make an edit and cannot say whether it helped, you needed evals already. Teams usually start after the first production incident, which is roughly three months later than the cheapest point to start.

The first version costs less than a day: twenty scenarios in a file, a rubric, and a script that prints a score. That is enough to stop the guess-and-ship loop, and it gives you something to grow rather than a blank page under deadline pressure.

Starting early also shapes the architecture. Systems built with evals in mind end up with seams — a retriever you can call in isolation, a planner you can score separately — that retrofitting rarely produces.

Written by Binary AI Labs · Reviewed 2026-11-01