Consistency and Pass^k in health benchmarks

What a single run measures

A pass rate is an average over attempts. The benchmark runs each task once, or a few times, and reports the fraction that succeeded. Read literally, it estimates how often a fresh attempt at a task drawn from that set will succeed. That is a real quantity, and it is the right one if you only care about the average case.

How stable the average is depends on the kind of task. On a fixed set of single-turn conversations graded by a rubric, repeat runs land close together. HealthBench reports a run-to-run standard deviation of about 0.002 over 16 repeats of the same evaluation. At that level, one run is close to the long-run mean, and the score you get is the score you would keep getting.

Long-horizon agent tasks behave differently. PhysicianBench tasks run about 27 tool calls each and are checked against 670 structured checkpoints across 100 tasks. CHI-Bench workflows run 60 to 80 steps across four to six stages. Every step is a place where two runs of the same model on the same task can diverge, and that divergence does not average out inside a single trajectory.

Pass^k: the probability of repeating

Two metrics look alike and mean opposite things. pass@k asks whether at least one of k attempts succeeded, so it rises with k. Pass^k asks whether all k attempts succeeded, so it falls with k. When a benchmark reports a consistency number, it is usually the second one.

If attempts failed independently of one another, Pass^k would be the per-attempt success rate raised to the power k, and you could compute it from the headline. Attempts are not independent. A model that misreads a task tends to misread it the same way again, and a harness that drops a tool result tends to drop it again. So benchmarks measure Pass^k directly, by running each task k times in an identical configuration.

That is why trial counts belong in the setup, not the appendix. HealthAgentBench runs 3 attempts per task, giving 162 trials over 54 tasks in 7 environments. PhysicianBench reports pass@1 averaged over 3 independent runs and also reports Pass^3.

The consistency collapse in health agent benchmarks

Published consistency numbers sit far below the headline pass rates. EHR-Complex, which puts agents on MIMIC-IV databases through SQL and Python over a 3,915-task test set, reports consistency below 50 percent at Pass^4 for nearly every model it evaluated.

CHI-Bench led its launch report with reliability rather than capability: no agent stayed above 20 percent across three identical runs. The board was released May 20, 2026 and updated August 12, 2026 across 45 harness configurations, and its authors note that harness choice matters as much as model choice. HealthAgentBench makes a related point in its row labels, where the rows are agent harnesses evaluated end to end rather than bare models.

PhysicianBench shows the same shape inside one table: its lower half falls steeply, with several agents near 1 percent on Pass^3. A system can be useful at pass@1 and unusable at Pass^3, and nothing in the pass@1 column tells you which one you are looking at.

How to read a board with this in mind

Find the trial count before you read the scores. One attempt per task means the score carries sampling noise the board is not showing you. Then ask whether Pass^k is reported at all. If it is not, the board is describing average-case behavior, and you should not read it as a reliability claim.

Two other columns change what a row means. Check whether the row names a model or a harness, because a harness score bundles scaffolding, tool definitions and retry behavior together with the model. Check whether the result was run by the board's maintainers or submitted by someone else; CHI-Bench accepts community submissions, so its rows mix author-run and submitted results.

Current standings for the benchmarks named here live at clinicalbenchmarks.ai, which keeps one page per benchmark. For what happens when a benchmark stops separating models at all, see benchmark saturation.

what is Pass^k in AI benchmarks

Pass^k is the fraction of tasks a system solves on all k attempts, run in an identical configuration. It is not the same as pass@k, which counts a task as solved if at least one of k attempts succeeds. pass@k rises as k grows; Pass^k falls. Benchmarks that publish a consistency number are usually publishing Pass^k, and they measure it by running the task k times rather than deriving it from the single-run score, because repeated failures are correlated.

why do agentic benchmark scores drop when the same task is run several times

Long-horizon tasks give a run many places to diverge. PhysicianBench tasks average about 27 tool calls, and CHI-Bench workflows run 60 to 80 steps across four to six stages. A system that succeeds on average may take a different path each time, so requiring every attempt to succeed removes the runs that got there by luck. EHR-Complex reports consistency below 50 percent at Pass^4 for nearly every model it evaluated, and the CHI-Bench launch report found no agent above 20 percent across three identical runs.