Eval budget
A first eval budget has two parts you can price from published figures: rater time (items × raters per item × minutes per item × an hourly rate) and tooling (a plan price × months). For 500 items scored by three raters at two minutes each, that is 50 rater hours, or $400 to $600 in participant pay at Prolific's published rates, before its platform fee.
Prices last verified 7 November 2026.
What does a small first round cost?
A worked example with published inputs. The calculator below lets you change every number.
| Line | Amount |
|---|---|
| Rater time: 500 items × 3 raters × 2 minutes | 50 hours |
| Participant pay at Prolific's $8 minimum | $400 |
| Participant pay at Prolific's $12 recommended rate | $600 |
| Braintrust Pro, 3 months | $747 |
| LangSmith Plus, 5 seats, 3 months | $585 |
| HumanSignal Starter Cloud (up to 12 users), 3 months | $297 |
Disclosure: vendors in this table can pay us a fee when an introduction we made ends in a signed deal. The listing, its order and its wording are never for sale.
Work out your own eval budget
Change the inputs; the totals update as you type. Nothing you enter leaves this page.
What does this estimate leave out?
Rater pay and a software plan are the parts with public prices. A real first round usually also pays for:
- Writing the rubric and the items, often the biggest single cost when experts write them.
- Pilot rounds to calibrate raters before the main run.
- Expert raters at rates well above general participant pay.
- Project management and the written readout.
- Model calls for automated graders, billed by your model provider.
If those are what you want a vendor to handle, say so in the brief. Get a quote
Questions teams ask
Why does the estimate leave out the platform fee?
Prolific charges a platform fee on top of participant pay, but the page we read did not state the rate. Rather than guess, the tool shows participant pay only. Ask Prolific for the current fee.
Why is there no figure for managed evaluation or red-teaming?
None of the managed evaluation or red-teaming firms we list publishes a rate. Their quotes include rubric design, expert raters, project management and reporting, none of which this arithmetic counts.
How many raters per item do we need?
Three is a common starting point for preference ratings, because it lets you break ties and measure agreement. Use more where raters disagree often, fewer once agreement is high and stable.
Do tooling plans have usage caps?
Yes. Braintrust Pro includes 5 GB of processed data and 50k scores a month; LangSmith Plus includes up to 10k base traces a month before pay-as-you-go. Check the cap against how often your suite runs.
Pages that pick up from here
- Vendors and tools every published price in one place
- Human preference evals or automated graders?
Where these facts come from
- researcher-help.prolific.com: Prolific payment model article; as seen on 7 November 2026
- braintrust.dev: Braintrust pricing page; as seen on 7 November 2026
- langchain.com: LangChain pricing page; as seen on 7 November 2026
- humansignal.com: HumanSignal pricing page; as seen on 7 November 2026
Get a quote
Company ZIP first, then a contact, then the evaluation itself. A few lines tell us which vendors are worth naming.
We are an introduction service, not an evaluation vendor, and we may be paid by the company we introduce you to.
Please do not paste prompts, logs, model outputs or any other data here. Describe the project in words; data only ever moves under a contract you sign with the vendor.
Brief received.
Here is what happens next:
- We read the brief and check it against the vendors' published offers and limits.
- We email you the names of the vendors that fit and why, with links to their own pages.
- Reply yes to the ones you want to hear from. Anyone you do not pick never sees your details.