How HealthBench Hard works
OpenAI · 1,000 conversations · updated August 17, 2026
HealthBench Hard is the bottom fifth of HealthBench. OpenAI scored all 5,000 conversations with five frontier models in May 2025, then kept the 1,000 those models handled worst. The conversations and rubrics are unchanged; only the selection is different. It was built to restore headroom, and at release the best score on it was 0.320 against 0.60 on the full set.
| publisher | OpenAI |
|---|---|
| released | 2025-05 |
| size | 1,000 conversations |
| scale | 0 to 1, higher is better |
How the subset was selected
Five models scored every conversation in the parent set: o3, Grok 3, Gemini 2.5 Pro, Claude 3.7 Sonnet with extended thinking, and Llama 4 Maverick. Conversations where no model earned positive credit were removed first, about 1.5 percent of the set, on the reasoning that those were broken rather than hard. The 1,000 conversations with the lowest average score across the five panel models became Hard.
Two things follow. The subset is defined relative to a fixed panel of May 2025 models, so it encodes what was hard for those systems. And no task was rewritten to be harder; the difficulty is entirely a selection effect.
How grading works
Grading is inherited from the parent benchmark. The default grader is GPT-4.1. It reads the conversation, the response, and the physician-written rubric, then decides which criteria are met. Criteria carry weights from -10 to +10, and negative criteria subtract points when a response does something harmful.
An example's score is earned points over the maximum positive points for that example, so a single example can fall below zero. The reported figure is the mean across examples, clipped to the 0 to 1 range.
Why scores here look low
Compression is the point of the set. At the May 2025 release, o3 led at 0.320 while scoring 0.60 on the full benchmark, and GPT-3.5 Turbo, GPT-4o from August 2024, and Llama 4 Maverick all scored 0.00. A model can be genuinely useful and still score poorly here, because every task in the set was kept for being one that models failed.
Because the subset was frozen against 2025-era models, improvement over time mixes two effects: models getting better in general, and models getting better at that panel's failure modes.
Limits
Hard is a subset rather than an independent benchmark, so it inherits every property of HealthBench, including its grader. The selection was made once against a fixed panel, and that panel has aged, so failure modes that only appear in newer systems are not represented; nothing put them there. Dropping conversations where no model scored positive also discards some genuinely hard items along with the broken ones. And low scores compress the range, which puts small gaps between rows closer to run noise than they would be on the parent set.
Current results are tracked at healthbenchhard.ai, and it sits alongside its parent set at clinicalbenchmarks.ai.
The grading machinery behind pages like this one is covered in the rubric grading and model graders guides.