Evals, cost and latency
Know whether your AI is getting better or worse before your users do. Every change runs against an eval set, with accuracy, cost per answer and latency tracked run by run.
- Runs onEvery change, before release
- TracksAccuracy, cost, latency, refusals
- Lives inYour repository and CI
What we measure
The numbers that decide whether an AI feature is ready, agreed with you before the build.
- Accuracy
How often the answer is right, against answers your domain expert accepted.
- Faithfulness
Whether every claim is supported by the source it cites.
- Refusals
How often it says "I don't know" when it should, and only when it should.
- Cost per answer
What each request costs in model and search spend, against a budget.
- Latency
How long users wait, at the 95th percentile, not the average.
How the evals work
Built once, run on every change, and handed over with the code.
The eval set
Real questions with accepted answers, written with your team, including the awkward ones.
Scoring
Exact checks where possible, model-graded where not, calibrated against human labels.
In CI
The suite runs on every change, and a drop below target blocks the release.
In production
A sample of real traffic scored every week, and new failures join the eval set.
Bringing cost and latency down
Measured before and after, so you see what each change saved.
- Caching
Repeated questions and shared context served without a new model call.
- Routing
Smaller, cheaper models for the requests they handle as well, the larger one only when needed.
- Smaller prompts
Only the context that changes the answer, sent once.
- Streaming
The first words shown while the rest is written, for long answers.
Questions about evals
Including for AI features you already run.
We already have an AI feature. Can you add evals to it?
Yes. It starts as Discovery & Architecture: we build the eval set, measure where it stands, and write down what to fix first.
How big is an eval set?
Large enough to cover the real cases, usually a few hundred questions to start, and it grows as new failures appear.
Can a model grade another model?
For some checks, yes, once its grades agree with your team's on a labelled sample. Exact checks are used wherever they work.
Will evals slow our releases down?
They take minutes in CI, and they stop the release that would have cost you a week of support tickets.
See it built
A client project of this kind, anonymised under NDA: the architecture, the trade-offs and the results measured at launch.
The AI feature worked in the demo and broke in production
Instrumenting a shipped feature, building an eval set from real traffic, and gating every change in CI.
- 92%answer accuracy, now measured
- 2.3sp95 latency, from 4.1s
- 35%lower cost per answer