Data catalog
Every track is net-new, consented, and graded by practicing physicians, then mapped to the gold-standard benchmark it improves on.
Physician-authored sequential patient cases with a hidden ground-truth state and faithful transitions, plus physician grading of a model's whole decision trajectory.
We replace the LM patient-simulator with physician-authored state, and the model-judge with physician grading, on novel cases that cannot be contaminated. Those are the two fidelity gaps the benchmarks' own authors flag.
Physician-authored cases with their state, action space, and transitions; an adjudicated rubric with each criterion written and checked by at least three physicians; an error taxonomy separating reasoning-process failures from factual and safety ones; a held-out split; inter-rater reliability against targets fixed in the pilot spec; and optional baseline rollouts for current frontier models.
The same cases with step-by-step physician reasoning traces and physician grading of model trajectories (rubric scores and pairwise preferences), in a form you can use for supervised fine-tuning, preference and RLHF/RLAIF training, and process-reward or verifier training.
Net-new consented multi-turn clinical conversations graded by physicians against adjudicated rubrics for correctness, completeness, safety, and calibration.
Human rubric grading is the ground truth model-graders are calibrated against. We deliver that human signal directly, on real physician-written conversations rather than synthetic ones.
Vignettes, differentials, and management plans with step-wise physician reasoning traces. This is supervised signal for medical verifiers and process-reward models.
The classics are saturated and contaminated, and they grade only the final multiple-choice answer. We produce uncontaminated, process-supervised reasoning traces: the physician-graded step labels PRM work currently lacks.
Physician diagnostic grounding and reasoning across radiology, dermatology, pathology, ECG, and clinical photographs.
We take on all three documented failure modes at once: auto-label noise, demographic and skin-tone bias, and language shortcuts. Our data is physician-labeled, balanced against those biases, and built to resist shortcuts.
Consented first-person procedural video with dense hierarchical expert annotation (procedure → phase → step → action), operator rationale, and an error taxonomy.
A large multi-site, multi-procedure surgeon-annotated video corpus does not exist. Cholec80 is one procedure at one hospital, and Ego-Exo4D has no medical content. This is genuine whitespace for VLA and world-model labs.
Physician-authored adversarial cases with a severity-graded harm/error taxonomy that separates reasoning-process failures from factual and safety ones.
Most residual medical hallucinations are reasoning failures, yet benchmarks rarely measure real patient harm and red teams stay small and ad-hoc. Physicians author graded harm cases at quality and scale.
And more, on request: longitudinal management, decisions under resource limits, and other axes as they prove useful.
Book a demo and we'll build one case end to end: the environment, the hidden ground-truth state, the grading rubric, and a graded model rollout, in your evaluation format, at no cost. Your team judges the fidelity first.