How First, Do NOHARM (v2) works
Stanford/Harvard consortium · 1,100 consultation cases · updated August 17, 2026
First, Do NOHARM asks a narrower question than most medical benchmarks: not whether a model is right, but how often its recommendations could hurt someone, and how badly. The name is an acronym for Numerous Options Harm Assessment for Risk in Medicine. Version 2 was released in December 2025 by a consortium led out of Stanford and Harvard, with more than 50 researchers including 29 board certified physicians, documented in arXiv 2512.01241. It is hosted by ARISE.
| publisher | Stanford/Harvard-led consortium (50+ researchers incl. 29 board-certified physicians); hosted by ARISE |
|---|---|
| released | 2025-12 |
| size | 1,100 consultation cases, 10 specialties, 12,747 expert annotations on 4,249 management options |
| scale | percentage safety score, higher better |
The case set and the annotation layer
The set is 1,100 consultation cases across 10 specialties, written in the primary care to specialist format. In that format the useful output is a set of management options rather than a single diagnosis, so the grading works at the level of options.
Reviewers extracted 4,249 management options from model responses and annotated them 12,747 times, an average of three annotations per option. Each annotation records whether an option is problematic and how severe the consequence would be if a clinician applied it directly.
From annotations to a safety score
The headline is a percentage safety score where higher is better, and it combines frequency with severity. Keeping severity in the score matters, because many small slips and one catastrophic slip are not the same failure.
Two error types are tracked. An error of commission is a recommendation that should not be there. An error of omission is a recommendation that should be there and is missing. Omissions are harder to catch, because nothing on the page looks wrong when you read it. The v1 study found errors of omission behind more than 80 percent of severe errors, and potential for severe harm in up to 24.6 percent of recommendations if applied directly.
Where the number is published
Version 2 is one of six components inside the MAST composite. MAST publishes component level breakdowns only for First, Do NOHARM v2, so it is the one part of that composite you can read on its own. Keep the two numbers separate: a MAST score blends safety with reasoning, imaging, and agentic work, while this one is about harm alone. See /benchmarks/mast for how the composite is assembled.
Limits
The score measures potential harm as judged by expert annotators, not harm that reached a patient. Annotators imagine a recommendation applied directly, a deliberate worst case; in practice a clinician reads the output first and catches some errors. Coverage is 10 specialties in one consult format, so results do not transfer automatically to triage or inpatient work. The MAST board that hosts v2 is marked as a preview and was last updated August 15, 2026, so scores may move. Because frequency and severity fold into one percentage, two models with the same score can have different error profiles.
Current model results for First, Do NOHARM v2 are tracked on the results index at clinicalbenchmarks.ai.
The grading machinery behind pages like this one is covered in the rubric grading and model graders guides.