Every team that evaluates a language model reaches the same fork. Paying people to judge outputs is slow and costs money every time. Asking a model to judge outputs is fast and nearly free per item, and nobody is quite sure whether to trust it. The answer is not one or the other. It is knowing which questions each one can answer and wiring them together.
What does each one measure?
A human preference eval shows raters two or more outputs for the same input and asks which is better against a rubric, or asks for a score on each criterion. It measures what people actually value: whether an answer is helpful, clear, appropriately cautious, on brand.
An automated grader is code or a model that scores outputs. Code graders check things with a right answer: does the JSON parse, does the code pass its tests, did the agent call the right tool. Model graders (often called LLM-as-judge) read an output and a rubric and return a score, which lets you automate subjective checks, approximately.
The word that matters is approximately. A model grader measures how a particular model reads your rubric. Whether that matches what your users think is an empirical question, and only people can answer it.
How do the costs compare?
| Human preference | Model grader | Code grader | |
|---|---|---|---|
| Cost per item | Rater minutes × hourly rate × raters per item | Model tokens, at your provider's price | Effectively nothing |
| Published reference | Prolific: $8 an hour minimum, $12 recommended, plus a platform fee | Your model provider's price list | None needed |
| Speed | Hours to days | Minutes | Seconds |
| Good for | Subjective quality, calibrating graders | Frequent runs of subjective checks | Anything with a checkable answer |
| Main risk | Rater disagreement, slow feedback | Systematic bias you cannot see | Only checks what you coded |
A worked number: 500 items, three raters each, two minutes per item is 50 rater hours. At Prolific’s published participant rates that is $400 to $600 before its platform fee. Managed services price rubric design, expert raters and reporting into a quote, and do not publish rates. The eval budget page lets you change every input.
Tooling has its own limits. Braintrust’s Pro plan includes 50k scores a month, so a grader that scores a large suite on every build can reach the cap faster than you expect.
Where do model graders go wrong?
Model graders have habits, and they are consistent, which makes them dangerous: a biased grader gives you stable, wrong numbers.
- Position. In side-by-side comparisons, the grader may favor whichever answer appears first (or second). Swap the order and score twice.
- Length. Longer answers can score higher regardless of quality. Check whether your scores track word count.
- Family resemblance. A grader may prefer outputs written in its own style, which matters if the grader and the system under test come from the same model family.
- Rubric drift. Vague criteria such as “helpful” get read differently from what you meant. The fix is a sharper rubric, not a smarter grader.
None of these mean model graders are useless. They mean a model grader is an instrument that needs calibrating, like any other.
How do you calibrate a grader against people?
- Pick 100 to 200 items that cover your buckets, including some hard cases.
- Have at least three people rate them with the rubric. Measure how often they agree with each other; that is your ceiling.
- Run the model grader on the same items and measure how often it agrees with the human majority.
- Where it disagrees, read the items. Fix the rubric, add examples to the grader prompt, or accept that this criterion needs people.
- Repeat after any change to the grader model, the rubric or the product.
If the grader agrees with people about as often as people agree with each other, you can let it score every build. If not, keep that criterion human.
What split works for most teams?
A pattern that holds up:
- Code graders for everything checkable, run on every build.
- A calibrated model grader for the subjective criteria it handles well, also on every build.
- A human preference round each major release, on a fresh sample, both to measure quality directly and to recheck the grader.
- Expert raters where judgment needs domain knowledge, from a managed service.
For the human part you can run raters yourself on a participant marketplace, use your own staff in a tool such as Label Studio (HumanSignal charges $99 monthly for Starter Cloud, which covers a dozen users), or hand the whole round to a managed service. Surge AI, for example, lists human evaluation, rubrics and verifiers among its products. Our list of AI evaluation companies shows which vendors publish what.
What should you ask a vendor offering either?
For human rounds: who the raters are, how they were qualified, how many rate each item, and the agreement figure for your project. For graders: which model does the grading, how it was calibrated against people, and whether the calibration data is yours to keep.
Trust the grader exactly as far as the people who checked it.