A demo answers ‘can the model do this?’. Production answers ‘does this change the work?’. Those are different questions, and they need different numbers.
We ask teams to baseline the current process: minutes per case, error rate, rework, and how often a human still touches the file. After launch, we measure the same things, plus exception volume and time-to-correct when the model is wrong.
If those numbers do not move, the project is not a success — even if the interface looks impressive. Sometimes the fix is better retrieval, sometimes it is a simpler form, sometimes it is admitting the use case was a poor fit.
Treat evaluation as part of the operating cadence, not a one-off test set. Models, documents, and processes all drift. A quarterly review is less glamorous than a launch event. It is how AI stays honest.
