The most useful eval set your team will ever have is sitting in its own logs. Public benchmarks measure what researchers care about. Your logs record what your users actually ask, in their words, with their typos and their edge cases. Turning them into a test set is a week of careful work, and it repays itself on every release after that.
Why do your own logs beat a public benchmark?
Three reasons. Your logs match your traffic, so a score moves when the product gets better or worse for real users. They are private, so no model has trained on them. And they include the failures you already know about, which is where regressions tend to come back.
Public sets still have a place for choosing a base model. For deciding whether this week’s build can ship, your own set wins.
What do you take out before anyone reads them?
Do this first, before engineers or vendors look at a single row.
- Remove names, email addresses, phone numbers, account numbers and street addresses. Replace them with consistent placeholders so the item still reads naturally.
- Drop any conversation that contains material your terms do not allow you to reuse, or that came from a customer whose contract restricts it.
- Get sign-off from whoever owns privacy at your company on what the logs may be used for.
- Keep the set in your own environment. If a vendor will score it later, it moves under a signed contract that says where it is stored and that it will not train any model.
Watch your tooling’s retention too. Braintrust’s Starter plan keeps data for 14 days and Pro for 30, so traces you want to keep should be exported before they age out.
How should you sample?
Random sampling gives you hundreds of copies of your most common request and almost none of the ones that matter. Sample by bucket instead.
| Bucket | How to find it | Share of the set |
|---|---|---|
| Everyday requests | Cluster by intent and take a few from each cluster | About half |
| Unhappy users | Thumbs-down, retries, rephrasing, handoffs to a human | About a quarter |
| Rare but costly | Refunds, legal or safety topics, actions that change data | Most of the rest |
| Known past bugs | Your incident list and bug tracker | Every one you can reproduce |
The shares are a starting point, not a rule. If one bucket is where your product breaks, give it more room.
How do you write references and rubrics?
Each item needs a way to be scored. There are three kinds, and a good set mixes them.
Exact or checkable answers
Extraction, classification, code that must pass tests, tool calls with the right arguments. Write the expected answer or a test, and let code grade it.
Reference answers
For open questions with a clear right answer, write a short reference and grade by whether the output agrees with it on the facts that matter. Have a domain expert write these where the subject is specialist.
Rubrics
For tone, helpfulness and judgment, write three to five criteria with a plain description of a pass and a fail for each. Rubrics are what human raters and model graders both need. Test the rubric on twenty items with two people before scoring the whole set: if they disagree often, the rubric is the problem.
Open-source tools are enough to start. DeepEval is an Apache-2.0 framework for scoring outputs in Python or TypeScript, and Label Studio’s Community Edition handles human review.
How big should it be, and how do you split it?
A few hundred items per product area is plenty for a first version, as long as each bucket is represented. Size matters less than whether the items look like real use.
Split it in two. A development portion your team can look at while improving prompts, and a held-out portion nobody tunes against. Freeze the held-out portion for a quarter at a time. Once people start fixing individual held-out items, it stops telling you anything.
Version the set like code. Every score should say which version of the set produced it.
How do you keep the set fresh?
Logs keep arriving, and your product keeps changing. A short routine stops the set going stale:
- Each release, add every new failure that reached users.
- Each month, resample the unhappy-users bucket from recent logs.
- Each quarter, retire items that no longer reflect the product and refresh the held-out portion.
Watch your tooling limits as the set grows. LangSmith’s Plus plan includes up to 10k base traces a month before pay-as-you-go, which a large suite run on every build can pass quickly.
When should you bring in a vendor?
Bring one in when the set needs things your team cannot supply: expert reference answers in law, engineering or science, trained raters for preference scoring at volume, or an outside view on what you have missed. The types of AI evaluation page explains which kind of vendor fits which job, and the companies on our list of AI evaluation companies say on their own pages which they sell.
Hand over the rubric and a sample, not the whole log export, until the contract is signed.
Your logs already hold the test your product most needs to pass; the work is in choosing the items carefully and keeping them honest.