Kaelio Health
Editorial
Benchmark contamination is inflating medical AI scores
At a glance
A benchmark only works if the model has not already seen the answers, and in medicine that assumption is breaking down. The classic text benchmarks were built from published exams, case reports, and abstracts, the same public corpus that trains frontier models, so part of every high score is memorization rather than reasoning. The ground-truth labels are noisy too: one audit found up to 57% label errors in a single subset. The durable fix is net-new, physician-authored cases that did not exist to be memorized.
A benchmark works only if the model has not seen the answers. That single assumption underwrites every leaderboard, and in medicine it is increasingly false. The classic text benchmarks (MedQA, MedMCQA, PubMedQA, the medical slices of MMLU) were assembled from published exams, case reports, and abstracts. The same public corpus is exactly what frontier models are trained on. So the test set and the training set overlap, and a chunk of every high score is memorization wearing the costume of reasoning.
You can see saturation as the symptom. o1-preview scores around 96% on MedQA (MedQA / USMLE, Jin et al., 2020). A number that high on a hard clinical exam should invite suspicion, not a press release. It is the reading you would expect if the benchmark had leaked into pretraining and the model were partly recalling rather than reasoning.
The tells of contamination
Two experiments make the leakage visible rather than hypothetical.
The first is the drug-name swap. Take a MedQA or MedMCQA question and replace a drug with a clinically equivalent alternative, a change that leaves the medicine identical and should not move a reasoning model at all. Accuracy drops by 1 to 10% (Gema et al., 2024). A model reasoning from pharmacology is unmoved; a model that has memorized the surface form of the question is not. The size of the drop is a direct read on how much of the score was pattern-matching to seen text.
The second shows up in the multimodal benchmarks, where "seen before" leaves an even blunter fingerprint: on SLAKE, 19.8% of the English images were flagged for overlap with model pretraining data (SLAKE, Liu et al., 2021). When a fifth of the test images may already be in the training set, the benchmark is grading familiarity as much as diagnosis.
Contamination isn't the only rot, the labels are wrong too
Even setting leakage aside, the ground truth these benchmarks are scored against is not clean. An audit of the MMLU medical subsets found a 6.49% overall ground-truth error rate, and in the Virology subset up to 57% of the items had label errors (Gema et al., 2024). PubMedQA's human ceiling is only 78.0%, set by a single annotator, which means the "correct" answers themselves carry meaningful noise (PubMedQA, Jin et al., 2019).
Put those two problems together and the picture is bleak. A large share of a model's score can come from having seen the test, and a meaningful share of the remaining "wrong" answers are the benchmark's mistakes, not the model's. You cannot build a trustworthy leaderboard, let alone a training signal, on top of data that is both leaked and mislabeled. Optimizing against it teaches a model to reproduce contamination and inherit errors.
The only durable fix is net-new data
You cannot decontaminate a public benchmark after the fact; once the questions are on the internet, they are in the next model's training set. The only reliable way to know a model has not seen the answers is for the answers not to exist until you make them. That means net-new cases, authored by physicians, that have never been published: cases that cannot already sit in a model's training data because they were written after it.
Authoring, not scraping, also fixes the label problem at the source. When the physician who writes the case also fixes the ground truth and the reasoning behind it, and other physicians adjudicate it, there is no upstream corpus of noisy labels to inherit. The step-by-step reasoning that comes with a written case is the signal process-supervision and verifier training most needs, and today that signal is mostly generated by models grading models, which reintroduces exactly the circularity you were trying to escape.
A saturated, contaminated, error-ridden benchmark is not a high bar that models have finally cleared. It is a broken ruler. Uncontaminated, physician-authored data is the only way to measure the thing you actually meant to measure.
See our reasoning-traces track for net-new, physician-authored cases and reasoning traces.
Sources
- MedQA / USMLE, Jin et al., 2020.
- Gema et al., 2024.
- SLAKE, Liu et al., 2021.
- PubMedQA, Jin et al., 2019.