Blog
Working notes for teams that buy or run AI evaluation: what each method measures, how to set it up, and what to ask a vendor. Each post ends with its sources.
How to build an eval set from your own logs
How to build an eval set from your own production logs: what to strip out first, how to sample, writing references and rubrics, and keeping the set fresh.
Human preference evals or automated graders?
Human preference evals or automated graders: what each measures, what each costs from published prices, where model graders go wrong, how to calibrate.
Red-teaming vs benchmarking: what are you buying?
Red-teaming vs benchmarking for AI systems: what each one measures, what the deliverable looks like, when to buy which, and how they feed each other.
Get a quote
Company ZIP first, then a contact, then the evaluation itself. A few lines tell us which vendors are worth naming.
We are an introduction service, not an evaluation vendor, and we may be paid by the company we introduce you to.
Please do not paste prompts, logs, model outputs or any other data here. Describe the project in words; data only ever moves under a contract you sign with the vendor.
Brief received.
Here is what happens next:
- We read the brief and check it against the vendors' published offers and limits.
- We email you the names of the vendors that fit and why, with links to their own pages.
- Reply yes to the ones you want to hear from. Anyone you do not pick never sees your details.