The Grader Problem

Start with what a rubric benchmark contains. HealthBench, published by OpenAI in May 2025, is 5,000 multi-turn health conversations scored against 48,562 rubric criteria written by 262 physicians from 60 countries, covering 26 specialties and 49 languages. A single conversation carries a median of 11 criteria, and some carry as many as 48. Each criterion has a weight between -10 and +10. Negative criteria subtract points for harmful content, so one example can score below zero.

Scoring one model on that set means making tens of thousands of criterion-level judgments. Scoring twenty models means doing it twenty times over. Physicians cannot work at that cadence. So the rubric is written by humans once, and a model applies it on every run.

The obvious objection is that the grader might not apply the rubric the way a physician would. HealthBench's authors treated that as something to measure. They ran a meta-evaluation: candidate graders scored responses physicians had also graded, and each was judged on how closely it matched. GPT-4.1 came out highest at macro-F1 0.709, ahead of o4-mini at 0.692, o3 at 0.681, GPT-4.1-mini at 0.661, and GPT-4.1-nano at 0.580. GPT-4.1 became the default grader.

The result worth sitting with is the next one. On 5 of the 7 themes in the benchmark, the chosen grader agreed with physician consensus more closely than the average individual physician did. That is the case for model grading, and it needs stating carefully. It does not say the grader is correct. It says that on this set, one physician's grades differ from the consensus by about as much as the model's do. Human grading has variance too.

Model grading also buys stability. Repeated runs of HealthBench vary with a standard deviation near 0.002 over 16 repeats. That is a different property from being right. A grader can be consistently wrong.

Two things vary between grading setups: which model reads the rubric, and what arithmetic is applied afterward.

HealthBench Professional, released in April 2026, replaced the grading stack. It uses GPT-5.4 at low reasoning effort, and it applies a length adjustment that penalizes verbosity. That adjustment is not a rounding detail. On HealthBench it removes roughly 2.99 points per 500 characters past 2,000; on Professional the rate is 1.47. The same response text, graded on the same rubric, yields one number with the adjustment and a different number without it. Later OpenAI system cards lead with length-adjusted variants, so the arithmetic behind a HealthBench figure depends on where it was published.

Score construction matters too. On HealthBench, an example's score is points earned divided by the maximum positive points available, which is why an example can go negative, and the overall score is the mean clipped to a 0 to 1 range. Clipping compresses the bottom.

Human baselines run through the same machinery. Physician-written responses score 0.437 on HealthBench Professional's rubrics. On the parent set, physicians writing unaided scored 0.13, and physicians given April 2025 model references scored 0.48, at which point they were no longer improving on the references themselves. Those anchors are grader outputs, not ground truth.

Consider how HealthBench gets reported across sources. Anthropic reports a raw score under its own protocol. OpenAI leads with length-adjusted numbers at maximum reasoning effort and reports its production Instant settings separately. Baichuan ran its competitors itself. Those rows share a name and are not the same measurement.

Subset is the second trap. HealthBench Hard is the bottom fifth of the parent set. It was built by scoring every conversation with five frontier models of the time, discarding the roughly 1.5 percent where no model scored positive, and keeping the 1,000 lowest-average conversations. The rubrics are the original physician-written ones; only the difficulty changed. At the May 2025 release the top score on Hard was o3's 0.320, against 0.60 for that same model on the full set. A figure quoted as "HealthBench" without naming the subset can be off by that much.

Some metrics are relative by construction. MedHELM ranks by mean win rate against the evaluated cohort, so adding or removing models moves every number even when no model changed. Vendor-reported figures use the vendor's own configuration, which often lands points away from an independent run of the same benchmark. More on that in reading a leaderboard.

Rubric grading is not the only option, and the alternatives show what it costs. EHR-Complex scores exact matches against answers derived from SQL and Python over a real EHR database. PhysicianBench verifies 670 structured checkpoints by executing them against actual EHR systems through commercial APIs. CHI-Bench runs deterministic unit tests first and calls an LLM judge only for what a unit test cannot check, such as evidence grounding and consent. Deterministic checks remove grader disagreement, but they only work where correctness is mechanically checkable, which rules out most clinical advice.

Where the judgment cannot be mechanized, the other lever is who judges. Health Optimization Bench grades each task blind with a panel drawn from three model families, and the family that authored a task never grades it. Every published score carries a 95 percent bootstrap confidence interval.

Removing grader variance does not remove run variance. Agentic boards make this visible by requiring repeated success: EHR-Complex reports consistency below 50 percent at Pass^4 for nearly every model, and CHI-Bench's launch reported no agent staying above 20 percent across three identical runs. See consistency and Pass^k.

Which model graded it, at what settings. If the page does not name the grader, its numbers cannot be compared with anyone else's or reproduced.

Whether the grader was checked against human graders, and how closely it matched. Agreement with expert consensus is measurable, and benchmarks that did the work publish the figure.

What arithmetic was applied after grading. Length adjustment, clipping, and normalization all change the number without changing the response.

Which subset, and how large. Parent set or hard slice, full task list or a sample.

Who ran it. A vendor grading its own model under its own configuration is a different artifact from an independent run. Both are worth reading when they are labeled.

How many runs, and how far apart they landed. One run with no interval tells you the model can do the task once.

Whether the score is absolute or relative to the cohort. Relative metrics move when the field moves.

A score is a claim about a procedure, not a fact about a model. When the procedure is documented, the number is useful. When it is not, the number is a brand. Current standings for the benchmarks discussed here are at clinicalbenchmarks.ai.