How MedCode works
Vals AI · 2,755 patient records · updated August 17, 2026
MedCode tests one job that has a defined right answer: assigning ICD-10-CM diagnosis codes to a hospital stay. Vals AI built the board with Protege and released it in February 2026. Coding is unusual among clinical tasks because trained humans produce a reference that can be checked directly, which makes accuracy scoring possible without a rubric or a judge model.
| publisher | Vals AI (dataset with Protege) |
|---|---|
| released | 2026-02 |
| size | 2,755 patient records |
| scale | percentage accuracy 0-100, higher better |
The unit is a stay, not a note
Each of the 2,755 records is a full patient stay. The model reads the discharge summary along with progress and consult notes, then assigns a primary code and the secondary codes.
Working at stay level is what makes this hard. The primary code should reflect the condition that drove the admission, and you can only establish that by reading across the record, not from any single note. The secondary codes then have to capture the rest of what was treated without adding conditions the notes do not support.
Ground truth and scoring
Ground truth was double-annotated by certified professional coders, meaning two credentialed people coded each record and disagreements were resolved before an answer was fixed. That step matters because coders do disagree, and a single annotator's habits would otherwise become the standard. Scores are reported as percentage accuracy from 0 to 100.
One runner, one configuration
Vals AI runs every model itself rather than accepting vendor submissions, and the board carried 85 models as of its August 15, 2026 update. Because one party runs everything under one configuration, the rows are comparable to each other in a way that mixed source boards are not. That comparability stops at the edge of the board: a MedCode number cannot be lined up against a number from a benchmark with a different grader or prompt. Vals AI pairs this board with /benchmarks/medscribe and notes that coding accuracy lags documentation quality, so a model that writes a good note is not automatically a good coder.
Limits
Accuracy is measured against codes humans assigned, and coding carries real ambiguity even among certified coders, so the reference is a strong consensus rather than a physical fact. A single accuracy percentage also does not tell you where the misses were, and a wrong primary code is a different kind of error from a missing secondary one. The record set is fixed, which keeps runs comparable but means the board measures a static slice of coding practice. Coding rules and payer conventions change over time, so a score is tied to the rules in force when the reference was built.
Current MedCode scores live on the results index at clinicalbenchmarks.ai.
The grading machinery behind pages like this one is covered in the rubric grading and model graders guides.