Who supplies expert human feedback, evaluations and red-teaming for LLMs and agents, how they differ, and what changed in 2025–26.
Updated 6 November 2026 · Indika AI editorial team
Human data for LLMs has moved on from simple preference ranking. In 2027, buyers want experts who can judge agent trajectories, write rubrics, red-team models and build evaluation sets in their own domain. The market also changed fast: Meta bought into Scale AI, several vendors raised large rounds, and at least one had a serious security incident.
We list Indika first because this is our guide; the rest follow no ranking. Many teams now run two vendors at once, so use the criteria below to pick a pair that covers each other's gaps.
| Company | Based | Best for | 2025–26 change |
|---|---|---|---|
| Indika AI | India | Independent domain-expert feedback | — |
| Scale AI | San Francisco | Very large and government programs | Meta 49% stake; new CEO |
| Surge AI | San Francisco | Premium RLHF, RL environments | — |
| Mercor | San Francisco | Individual expert contractors | Data breach, Mar 2026 |
| Turing | Palo Alto | Coding and reasoning data | Series E, Mar 2025 |
| Invisible | San Francisco | Platform plus expert marketplace | $100M raise, Sep 2025 |
| Toloka | Amsterdam | Multilingual preference data | Bezos Expeditions-led investment |
| Labelbox | San Francisco | Platform, Alignerr experts | Series E, Oct 2026 |
| Prolific | London | Participant panel, studies | — |
| Appen | Sydney / Kirkland | Crowd scale, 235+ languages | — |
Indika runs RLHF, preference ranking, evaluation and red-teaming with vetted domain experts (clinicians, lawyers, engineers, analysts and linguists), alongside annotation and data infrastructure work.
Founded in 2016, Scale sells data labelling and RLHF, runs the Outlier and Remotasks contributor platforms and builds AI applications for enterprise and government. Its security page lists SOC 2 Type II, ISO 27001:2022, FedRAMP High and DoD IL4.
Check: If you build models that compete with Meta's, ask how your data and feedback are kept separate.
Surge AI sells premium RLHF data, RL environments, benchmarks and access to an expert workforce.
Check: Its homepage does not list security certifications; ask for them.
Founded in 2023, Mercor connects AI labs with expert contractors who produce training and evaluation data.
Check: Ask what changed in security after the March 2026 incident and how contractor access to your data is controlled.
Founded in 2018, Turing supplies coding and reasoning data to LLM labs, builds GenAI systems for enterprises and runs its original remote-developer marketplace.
Founded in 2015, Invisible sells an enterprise AI platform whose modules include an RLHF expert marketplace and annotation and evaluation services.
Founded in 2014, Toloka runs a managed service for SFT, RLHF and DPO preference data, evaluations and red-teaming, and states 40+ languages with workers in 100+ countries.
Founded in 2018, Labelbox offers a labelling platform and managed services, now led by RL environments and evaluations for frontier labs. Its Alignerr network supplies expert trainers.
Prolific provides a vetted participant panel and an API for human feedback, evaluations and research studies.
Founded in 1996, Appen runs a crowd-sourced data service covering collection, annotation and evaluation. It states 1M+ contributors and 235+ languages, and its About page lists SOC 2 and ISO 27001.
Check: Ask which work runs through the open crowd and which through managed teams, and how contributors are vetted for your task.
If you already work with one large vendor, a second one gives you a quality benchmark and protects you if the first changes owners or priorities. Our guide to choosing a second RLHF vendor lists six checks, and our State of Human Feedback Data 2026 report shows how budgets are moving to specialist review.
A company that supplies people, usually vetted experts, to create training and evaluation data for AI models: preference rankings for RLHF, written demonstrations for fine-tuning, rubric-based evaluations and red-teaming.
Meta took a 49% non-voting stake for $14.3B in June 2025 and Scale's founder moved to Meta. OpenAI said it had been winding down its work with Scale, and Francis deSouza became Scale's CEO in August 2026.
Check ownership and data-use terms first, then expert vetting, then security certificates. Run the same paid or no-cost pilot with two vendors on your own tasks and compare agreement with your gold set.
Yes. Indika runs RLHF, preference ranking, evaluation and red-teaming with vetted domain experts, and offers a no-cost pilot.
Ask about a no-cost pilot on your own tasks and compare the results.
Request a pilot →The data foundation enterprises trust to build reliable AI, since 2021.