AI Evaluation CompaniesGet a quote
Menu

Types of AI evaluation

There are four types of AI evaluation worth paying for: capability tests (can it do the task), safety and red-team tests (what can it be made to do), human preference ratings (which output people judge better) and regression suites (did this release break anything). Each one answers a different question, so a vendor strong in one is often average in another.

Which type of evaluation answers your question?

Start from the decision you need to make, not from the test you have heard of.

Match the decision to the evaluation type.
The decision in front of youType to runWhat you hand overWhat you get back
Which model should we build on?CapabilityYour task definitions and success criteriaScores per model on the same private set
Is it safe to launch?Safety and red-teamAccess to a staging endpoint and your policySuccessful attacks, severity, reproduction steps
Is the new prompt actually better?Human preferencePaired outputs and a rubricWin rates and rater comments
Can we ship this build?RegressionA fixed suite wired into CIPass or fail per item, every run

What is a capability evaluation?

A capability evaluation measures whether a model can do a defined task and how well, against a fixed bar. Public benchmarks are the familiar form, but the useful version for a product team is a private set built from your own tasks, because public sets end up in training data and stop measuring anything.

What to buy: expert-written items with reference answers or verifiers for hard domains (law, code, science), or tooling to run your own sets on every model you consider. Companies on our list whose own pages describe capability work: Scale AI, Surge AI, Snorkel AI, Toloka, Mercor, Turing.

Watch for

Items the vendor reuses across clients, and scoring that rewards format over correctness.

What is a safety and red-team evaluation?

Safety evaluation checks behavior you never want: harmful instructions, exposed system prompts, data exfiltration through tools, policy breaks. Red-teaming is the adversarial part, where people try on purpose to make the system fail. Companies on our list that describe it on their own pages: Giskard, Galileo, Toloka, TELUS Digital, Gray Swan AI, Centific.

The output that matters is a list of successful attacks with steps to reproduce, ranked by severity, not a single safety score. See red-teaming services for how engagements are scoped.

What is a human preference evaluation?

Raters compare two or more outputs for the same input and pick the better one against a rubric. It is the most direct way to test tone, helpfulness and judgment, and the standard way to check whether an automated grader can be trusted.

Cost is driven by items, raters per item and minutes per item. On a participant marketplace such as Prolific, the participant minimum is $8 an hour (recommended $12) plus a platform fee. Managed services price the rubric work and expert raters into a quote. Companies on our list that offer preference work: HumanSignal (Label Studio), Kili Technology, Labelbox, SuperAnnotate, Prolific, Scale AI, Surge AI, Toloka, TELUS Digital, Centific, Mercor.

What is regression testing for AI?

A regression suite is a fixed set of items run on every build, so a change that fixes one behavior and breaks another is caught before release. It is mostly code and model graders, owned by your engineers, on tooling such as Braintrust, LangSmith (LangChain), Confident AI (DeepEval), Giskard, Galileo.

Tooling caps matter here: Braintrust's Pro plan includes 50k scores a month, so a suite of 1,000 items run 50 times a month already reaches it.

Disclosure: vendors in this table can pay us a fee when an introduction we made ends in a signed deal. The listing, its order and its wording are never for sale.

Questions teams ask

What is the difference between an eval and a benchmark?

A benchmark is a fixed, usually public, test set with a scoring method. An eval is any structured test of a model or product, and can use a benchmark, a private set built from your own traffic, human raters, or all three.

Do we need human raters at all?

For tasks with a checkable answer (code that passes tests, a correct extraction), automated graders are cheaper and repeatable. For tone, helpfulness, safety judgment and anything subjective, you need people at least to calibrate the automated grader. Our readout on human preference evals or automated graders goes through the trade-off.

How big should a first eval set be?

Big enough to see the failures you care about more than once. Teams often start with a few hundred items per behavior, then grow the sets where scores move between releases. Size matters less than whether the items look like real use.

Can one vendor cover all four types?

A few come close: Toloka lists auto-verifiable evaluation tasks, preference data and red-teaming on its own pages. Many teams still split the work: tooling for regression, a service for human judgment, and a specialist for adversarial testing.

Where does agent evaluation fit?

Agent tests are capability and safety tests run over many steps with tools, often inside a simulated environment. The same four questions apply; the test harness is just heavier.

Pages that pick up from here

Where these facts come from

Get a quote

Company ZIP first, then a contact, then the evaluation itself. A few lines tell us which vendors are worth naming.

We are an introduction service, not an evaluation vendor, and we may be paid by the company we introduce you to.

Please do not paste prompts, logs, model outputs or any other data here. Describe the project in words; data only ever moves under a contract you sign with the vendor.

Part one of three

Where is the company based?
Get a quote