Static medical benchmarks are lying to you

Why the exam score doesn't predict the encounter

The multiple-choice format quietly does most of the work. It hands the model a curated, complete history, five clean options, and the guarantee that exactly one is correct. A sequential encounter removes all three crutches at once. Now the model has to decide what to ask before it can reason about the answer, tolerate the fact that the informative test may not have been ordered yet, and hold a moving differential together across turns without the case ever declaring itself finished.

These are different skills, and models are far better at the second than the first. CRAFT-MD's authors traced the collapse to exactly this: models fail to synthesize information across turns, so accuracy that looks solid on a static vignette evaporates the moment the same facts have to be actively gathered. A leaderboard built on static MCQs measures recall over a laid-out case. It tells you almost nothing about how a model behaves in the encounter you actually care about, the one where a wrong first question sends everything downstream in the wrong direction.

The consequence is that static accuracy overstates interactive accuracy instead of conservatively approximating it. A number near 90% and a number near 50% measure two different abilities, and only the second resembles the deployment task.

"Just add a simulator" is not the fix

The obvious response is to make the benchmark interactive: wrap the questions in a simulated patient and an automated grader and call it dynamic. That is exactly what the leading interactive benchmarks did, and it introduces a fidelity problem their own authors flag.

In AgentClinic the patient, the measurement system, and the moderator are all language models, and the paper notes the errors that follow, including a self-preference effect where a model grading its own kind inflates the score. CRAFT-MD is built on an AI patient-simulator and an AI grader, both of which the authors describe as unfaithful to how real patients present and how real clinicians judge. Microsoft's sequential-diagnosis work leans on a language-model "gatekeeper" that has been shown to fabricate findings the case never contained (SDBench / MAI-DxO, Microsoft AI, 2025).

So the format is right and the substrate is wrong. A simulated encounter graded by a model reproduces the same circularity that made static benchmarks unreliable: the thing being measured and the thing doing the measuring are the same kind of system, with the same blind spots. You have made the task look like medicine without making the signal trustworthy.

What faithful interactive evaluation requires

Fidelity is the whole game, and it has two halves. First, the environment has to respond truthfully to whatever the model does: a hidden ground-truth state with faithful transitions for every clinically sensible action, so ordering the right test surfaces the right finding and ordering the wrong one costs what it should. Second, the reward has to be trustworthy over the whole trajectory, not just the final answer, because a model can reach the right diagnosis down an unsafe or wasteful path and a static check will never notice.

Neither half comes from a language model. Both come from physicians: clinicians author the case and its state, and clinicians grade the decision process against an adjudicated rubric. That removes the self-preference and fabrication failures the simulator-plus-judge approach carries, and it produces a score that means what a leaderboard number is supposed to mean: that the model can actually do the job.

The static-to-sequential collapse is not a curiosity at the margins. It marks the difference between a model that passes the exam and a model you would let near a patient. If your evaluation still stops at the exam, you are measuring the wrong number.

See our interactive-cases track for physician-authored sequential cases with physician trajectory grading.