How PhysicianBench works
academic team · 100 clinical tasks · updated August 17, 2026
PhysicianBench measures whether an agent can finish a physician's work, not whether it can describe the work. An academic team published it in May 2026 (arXiv 2605.02240). It puts agents inside real electronic health record environments and asks them to complete 100 real-world clinical tasks drawn from 21 specialties. The results also feed the ARISE MAST composite.
| publisher | Academic team (Ruoqi Liu, Imran Q. Mohiuddin et al., arXiv 2605.02240); also a MAST component |
|---|---|
| released | 2026-05 |
| size | 100 real-world clinical tasks, 21 specialties, 670 structured checkpoints (~27 tool calls per task) |
| scale | pass@1 success rate %, higher better (3 independent runs; Pass^3 also reported) |
Tasks, tool calls, and checkpoints
Each of the 100 tasks is a composite workflow rather than a single question. The suite defines 670 structured checkpoints across those tasks, and a run averages about 27 tool calls, so one task can involve dozens of steps against the record system.
Checkpoints are the grading unit. Instead of reading the agent's prose and judging it, the benchmark checks specific states that should exist if the work was actually done. That keeps grading deterministic and takes the judge model out of the loop.
Execution-grounded verification
Agents reach the record system through standard commercial APIs, the same interfaces a real integration would use. Verification is execution grounded: what counts is what changed in the system, not what the agent reported. An agent that says it placed an order gets credit only if the order exists.
This closes a gap that text-graded benchmarks leave open. A fluent description of the right action and the right action itself can score identically against a rubric. Here they do not.
pass@1 and Pass^3
Every task is run three independent times. The headline metric is pass@1, the share of runs that succeed. The paper also reports Pass^3, which requires the same task to succeed in all three runs.
The two numbers answer different questions. pass@1 asks whether the agent can do the job. Pass^3 asks whether it does the job every time. The gap between them is wide, and the lower half of the table falls steeply, with several agents near 1 percent on Pass^3.
Limits
The numbers come from the paper, and there is no standalone public leaderboard, so the model set is whatever the authors ran at publication and it does not refresh when new models ship. With 100 tasks, a difference of a few percentage points is a handful of tasks, so ordering near the top is fragile. PhysicianBench results also feed the MAST composite, which means the same evidence can reach you twice: once on its own and once folded into a composite score.
Current PhysicianBench figures, and the MAST composite that includes them, are tracked at clinicalbenchmarks.ai.
The grading machinery behind pages like this one is covered in the rubric grading and model graders guides.