How Healthcare AI Benchmarks Work

6 guides · 18 benchmark profiles · updated August 17, 2026

Every week someone cites a healthcare AI score without knowing how it was produced. This site explains the machinery: how rubric grading works, why the graders are themselves models, what a leaderboard number can and cannot tell you, and how each benchmark that still tests frontier models is actually constructed. When you want the current standings rather than the mechanics, they live at clinicalbenchmarks.ai.

Guides

• How Rubric Grading Works in Health AI Evals
  Rubric grading scores a model answer against a written checklist of physician-authored criteria, each with a weight. Here is how the scoring formula works.
• Model Graders in Health AI Evaluations
  Health AI rubrics are scored by another model. How graders are chosen against physician judgment, where they go wrong, and what changing one does.
• How to Read an AI Health Leaderboard
  A leaderboard row is a model plus a configuration plus a date. Four checks that keep you from comparing two numbers that were made differently.
• Consistency and Pass^k in health benchmarks
  Single-run benchmark scores measure one attempt. Pass^k measures repeat success, and health agent benchmarks report sharp drops when it is required.
• Benchmark saturation and what replaces it
  How medical benchmarks saturate: MedQA above 95 percent, HealthBench nearing its noise ceiling, and the hard subsets and successors built in response.
• Running your own rubric health evaluation
  How to run a rubric health evaluation yourself: what is released, which grader configuration to match, and why your number will differ from published ones.

Benchmark mechanics

A profile of each benchmark with current frontier activity: how it was built, how it grades, and what its score means. Profiles explain mechanisms; current results live on the boards each profile links.

Writing

The Grader Problem Health AI benchmarks are graded by language models. How grader choice, length adjustment, and subset selection change published scores, and what to ask before believing one.