Text accuracy is not clinical safety. If you are deploying medical AI that reads, hears, or watches patients, you need evidence that it catches what a transcript leaves out.
The Shift From Text Benchmarks to Multimodal Clinical Evidence
Medical AI evaluation is moving past exam-style question sets. The reason is simple: a transcript can preserve every spoken word while dropping labored breathing, an unsteady gait, or a monitor reading that contradicts the handoff. Buyers now want to know whether a system registers those observations, understands their significance, and recommends something safe.
The research is starting to back this up. In 723 emergency department patients, adding short video clips to triage data modestly improved prediction of hospitalization. In a separate study, a model grading recorded student examinations agreed best with physician scores from native audio-video, less from audio or transcript. And in simulated interactive consultations, video raised overall scores, yet subtle cues such as tremor remained hard to perceive.
That gap between what a model can read, hear, and see is where the market is forming. You now have a mix of open benchmarks, data and evaluation platforms, and small specialist research shops. Each answers a different question, and picking the wrong one wastes budget and time.
How We Compared These Evaluation Providers
We looked at the same factors for every provider. First, service scope: does the offering cover text only, or does it extend to audio, video, and interactive encounters? Second, physician grounding: are rubrics and scores built and validated with practicing clinicians, and how transparent is that process?
Third, methodological transparency: can you see the study design, sample sizes, and confidence intervals behind the claims? Fourth, fit for your stage: is the provider built for a small team validating one model, or for an enterprise pipeline that needs scale? Fifth, pricing and access clarity: open datasets and published papers are easy to audit, while custom engagements require a conversation.
Greyoak Research stood out for its narrow focus on multimodal clinical safety and its physician-benchmarked simulated cases. Centific stood out for breadth, spanning data creation, RLHF, and model safety across verticals. OpenAI HealthBench stood out for an open, physician-written rubric at scale. Scale AI stood out for its position across multiple landmark evaluation methodologies and enterprise delivery.
Here is the short version. Four providers, four different jobs. Use this table to match the provider to the question you actually need answered before you sign anything.
| Provider | Best For |
|---|---|
| Centific | Teams that want evaluation bundled with data and fine-tuning services |
| Greyoak Research: Multimodal clinical research for medical AI | Multimodal safety validation with physician benchmarks |
| OpenAI HealthBench | An open, auditable conversational baseline |
| Scale AI | Enterprise-scale evaluation planning and delivery |
The Four Providers in Detail
#1 Centific
A screenshot of the Centific website.
Centific is a broad AI services platform, and healthcare is one vertical among many. Its benchmarking and evaluation work sits alongside data collection, RLHF, supervised fine-tuning, and multilingual AI services. That means you can run an evaluation and fix the underlying data problem in the same shop. The trade-off is focus: you are buying into a large services relationship rather than a dedicated clinical safety lab. If your team needs end-to-end data and evaluation support across a production pipeline, the breadth is the point.
#2 Greyoak Research: Multimodal clinical research for medical AI
A screenshot of the Greyoak Research: Multimodal clinical research for medical AI website.
Greyoak Research does one thing and does it narrowly: independent, physician-benchmarked safety evaluations for multimodal medical AI. The site frames the core problem well. Accuracy on text is not the same as clinical safety, because a transcript can miss labored breathing, an unsteady gait, or a companion answering for the patient. Its research page collects the evidence, including a study where adding video to triage data improved hospitalization prediction in 723 emergency department patients, and a preprint showing models agreed best with physician scores from native audio-video rather than transcripts. It also designs simulated clinical cases with actors and physician rubrics, which is the right instrument if you need to know whether a model perceives subtle cues like tremor. Best fit is a small, specialized team that needs a rigorous outside read before deployment.
#3 OpenAI HealthBench
A screenshot of the OpenAI HealthBench website.
HealthBench is an open benchmark built with 262 physicians who have practiced in 60 countries. It includes 5,000 realistic health conversations, each with a custom physician-created rubric to grade model responses. The design goal was to move past exam questions toward complex, real-life scenarios that mirror how people and clinicians actually talk. Because it is open, you can inspect the dataset, the code, and the comparison against physician baselines yourself. The limitation is scope: it is a conversation benchmark, so it will not tell you how a model handles video of a gait or audio of a cough. Use it as a credible baseline, not a complete safety file.
#4 Scale AI
A screenshot of the Scale AI website.
Scale AI publishes and analyzes healthcare benchmarks, including work that surveys new evaluation methodologies from OpenAI, Google DeepMind, and Microsoft. Its framing is useful: no single evaluation is sufficient for a high-stakes environment, and static accuracy metrics poorly reflect dynamic clinical work. That perspective helps when you are building an evaluation plan rather than buying one test. Scale also brings enterprise delivery capacity, which matters if you need to run evaluations at volume across many models. The trade-off is that healthcare is one part of a much larger business, so you should expect to define your clinical requirements precisely.
How to Choose the Right Evaluation Partner
Start with the input modality. If your system only handles text chat, an open conversational benchmark gives you a fast, cheap baseline. If your system ingests audio, video, or live interaction, you need a partner that builds cases with those inputs and scores them against physicians.
Next, ask who wrote the rubric. Physician-created and physician-validated rubrics are the difference between a score that means something and a score that looks impressive. Ask for sample sizes and confidence intervals, not just headline numbers.
Then match the engagement to your stage. A small team validating one model benefits from a focused research partner. A larger organization with a data pipeline may prefer a platform that can evaluate and retrain in one place. Finally, check access: open datasets and published papers are auditable today, while custom work needs a scoped proposal before you can compare costs.
Where Automation Fits in a Safety Workflow
Automation helps you run evaluations consistently, not replace clinical judgment. A practical loop looks like this. Define the clinical scenario and the observations that matter, including the ones a transcript would drop. Build or select cases that carry those cues in the native modality. Score outputs against a physician-written rubric. Then log where the model missed a cue and whether the miss came from the input it was given or from its reasoning.
That last distinction is the one teams skip. A cue a model misses may reflect what it was given as input, not only how it reasoned, and the two call for different fixes. Automate the scoring and the logging. Keep physicians in the loop for rubric design and for the cases where the stakes are highest.
The Bottom Line
You do not need one provider. You need the right evidence for the decision in front of you. Use an open benchmark like HealthBench to establish a conversational baseline. Use a platform like Centific or Scale AI when you need evaluation at pipeline scale or across many models. And when the question is whether your system can see, hear, and act safely on a real clinical encounter, bring in a specialist built for that question. Greyoak Research fits that last slot, with physician-benchmarked simulated cases and published multimodal findings you can read before you call. Match the tool to the risk, and your safety file will hold up.