AI Evaluation CompaniesGet a quote
Menu

AI Evaluation Companies

Find the team or the tool that tests your model before your users do. Some vendors send trained raters and red-teamers; others sell the software your engineers run on every build. Describe your release and we suggest who fits.

Get a quoteBrowse 20 vendors and tools
eval run: support-assistant, release candidate vs current

  1. 1 · Test set

    240 items from real requests

    • refund_17 Can I return a worn jacket?
    • tone_04 Angry customer, late order
    • pii_09 Asks for another user's address
    • + 237 more, held out

  2. 2 · Model run

    Both versions answer every item

    240 of 240 answered

    CandidateCurrent

  3. 3 · Graders

    Code, a model judge, then people

    • Codepolicy facts match
    • Judgerubric: grounded, polite
    • Human40 disputed items, 2 raters each

  4. 4 · Report

    Pass rate and what broke

    91%candidate

    Current: 87%. Three regressions to read, one on pii_09.

    Hold release for review

Invented example run. No real model, product or result is shown.

People or plumbing: what are you short of?

A managed service sells people: trained raters, domain experts or attackers, with rubrics and project management around them. Eval tooling sells software: test sets, CI runs, code and model graders, traces of live traffic. Most teams end up with tooling for the daily checks and a service for the work that needs judgment.

The four types of evaluation, explained

How the 20 vendors we track sell evaluation

  • Software you run6
  • Platform and services7
  • Managed service6
  • Participant marketplace1

Counted from each vendor's own description. Several sell both software and people.

Five ways vendors charge, and what makes the bill grow

Knowing which model a vendor uses tells you what to watch. Software usually prints a price; the people-heavy services quote per project.

Work out a first-round eval budget

  • Plan tiers

    Braintrust: $0 Starter, $249 a month Pro

    Passing a cap

  • Per seat

    LangSmith Plus: $39 per seat monthly

    Headcount

  • Usage units

    Labelbox: 20 LBU per data row, dollar rate by quote

    Volume

  • Participant hours

    Prolific: $8 an hour minimum, $12 recommended

    Minutes per item

  • Project quote

    Scale AI, Surge AI, Toloka, TELUS Digital

    Scope and reruns

The contract decides whether the results are worth the invoice

Ask for a written test plan before anything runs, item-level results rather than a headline score, and the price of rerunning the same set on your next release. A security badge on a page is a claim: ask for the actual SOC 2 or ISO 27001 report under NDA and check its scope.

Six lines to get in writing

  • Your outputs train nothing, theirs or another client's model
  • Private items stay private and are not reused across clients
  • No conflicted graders: whoever made your training data does not score it
  • Rater criteria and an agreement figure for every human-scored task
  • The eval set is yours: prompts, rubrics and grader code
  • Retention: where outputs are stored, and for how long

Four kinds of evaluation you can buy

They differ in who does the work and how often you need it. Most teams need at least two.

Did this release break what worked last week?

Regression

A fixed suite run in CI on every build or deploy, often scored by model graders. Your engineers run it on eval tooling.

What can someone make it do that it should not?

Safety and red-teaming

Adversarial prompting by people plus automated attack tools, before launch and after major changes. Who sells it.

Which answer do real people judge better?

Human preference

Side-by-side ratings against a rubric, per release or when automated graders disagree.

Can the model do the task, and how well?

Capability and benchmarks

Held-out test sets with reference answers or verifiers, built by your team or an expert-data vendor.

Start free, add people later

DeepEval is Apache-2.0 open source, Giskard's library runs locally for free, and Label Studio's Community Edition handles human review. What open source does not supply is trained raters and attackers.

Twelve vendors, and what each says about price

From each company's own page. Software tends to print a price; services quote. All 20, with security claims.

Software your team runs

People, platforms and red teams

  • Kili TechnologyFree trial; Grow and Enterprise customPlatform and services
  • Labelbox20 LBU per data row; no dollar price per LBUPlatform and services
  • Prolific$8/hr participant minimum, plus a platform feeParticipant marketplace
  • Scale AIQuote onlyPlatform and services
  • Surge AIQuote onlyManaged service
  • TolokaQuote onlyManaged service
  • Gray Swan AIQuote onlyPlatform and services

Disclosure: any vendor on this list can pay us a fee when an introduction we made ends in a signed deal. The listing, its order and its wording are never for sale.

We do not audit vendors or run trials. Plan limits and security badges are each company's own claims. Teams pay nothing; how this works.

A first engagement, start to rerun

The same five steps, whichever vendor you choose.

  1. 1

    Scope

    Agree the behaviors to test, the model version and the pass marks. This becomes the test plan.

  2. 2

    Pilot set

    A small set is written and scored to check the rubric reads the same to everyone.

  3. 3

    Calibrate

    Raters or model graders are tuned until agreement is acceptable; ambiguous items are fixed or dropped.

  4. 4

    Full run and readout

    You get item-level results and a written account of failures.

  5. 5

    Rerun

    The same set runs on each release, so you see movement rather than a snapshot.

Prices and numbers vendors print

  • $0Braintrust Starter, with 1 GB of processed data and 10k scores
  • $39LangSmith Plus, per seat monthly, 10k base traces
  • $8/hrProlific's participant minimum; $12 recommended
  • 23,000+People in Gray Swan's Arena red-teaming network, by its count

As printed on each company's own pages. None of these is an offer or a quote.

Questions teams ask before buying evals

More in the readouts.

What does an AI evaluation company do?

It measures how a model or AI product behaves before and after release. Some send trained people to rate outputs and attack the system (a managed service); others sell software your engineers use to run tests, score outputs and trace production traffic. Many sell both.

How much does LLM evaluation cost?

Software has public prices: Braintrust charges $249 a month for Pro, and LangSmith's Plus tier costs $39 per seat monthly. Human rating through Prolific starts at its $8 an hour participant minimum plus a platform fee. Managed evaluation and red-teaming firms do not publish prices; they quote per project. The eval budget page shows the arithmetic.

What are the best AI evaluation tools for production?

It depends on the job. For tests that run on every build and tracing of live traffic, Braintrust, LangSmith and Confident AI (built on the open-source DeepEval) are the platforms with published plans; Giskard adds continuous red-teaming with an open-source library, and Galileo, now owned by Cisco, sells evaluation, observability and guardrails. We do not test or rank them. All vendors and tools.

Should we use LLM as a judge or human raters?

Usually both, in different amounts. A model grader is cheap enough to score every build, but it only measures how a model reads your rubric. Human raters measure what people actually value and are the only way to check the judge, so most teams calibrate an LLM judge on a small human-rated set and repeat the human round each major release. LLM judge or human raters, in detail.

How do I get started with AI evals at our company?

Start from your own production logs, not a public benchmark. Remove personal details, sample a few hundred real requests, write a checkable answer or a rubric for each, and freeze a held-out portion. A free tool is enough for the first version: DeepEval is Apache-2.0 open source, Giskard has a free library that runs locally, and Label Studio has an open-source Community Edition for human review. Setting up evals from your logs.

Is red teaming the same as evaluation?

Red-teaming is one kind of evaluation: people (or tools) try to make the system fail on purpose, then report what worked. Benchmarks and regression suites measure known behavior repeatedly. Most teams need both; red-teaming services covers who sells the adversarial side.

Where these facts come from

Get a quote

Company ZIP first, then a contact, then the evaluation itself. A few lines tell us which vendors are worth naming.

We are an introduction service, not an evaluation vendor, and we may be paid by the company we introduce you to.

Please do not paste prompts, logs, model outputs or any other data here. Describe the project in words; data only ever moves under a contract you sign with the vendor.

Part one of three

Where is the company based?
Get a quote