Quality & methodology
The hard part of dynamic clinical evaluation is fidelity: faithful state for whatever a model does, and a reward over the trajectory you can trust. Physicians who author the case and grade it clear that bar.
Each case is an interactive environment with an initial presentation, a hidden ground-truth state, and faithful transitions for every clinically sensible action across history, examination, investigation, treatment, and referral. If a model acts outside the authored space, the authoring physician adjudicates the step and folds it back in.
Every rubric criterion is written and checked by at least three physicians, with an error taxonomy that separates reasoning-process failures from factual and safety ones, each carrying a severity grade.
Held-out splits and inter-rater reliability targets are fixed in the pilot spec up front, so quality is measurable rather than asserted.
Physicians write the cases, so nothing comes from identifiable records. Fold in real cases later under the usual consent, ethics approval, de-identification, and HIPAA/GDPR-aware licensing.
Why not model-judges
On clinical answers, automated judges barely beat chance. That is why grading, not just authoring, has to be physician work.
AUC 0.49-0.66
LLM judges separate complete from incomplete clinical answers only marginally above chance. Even when they agree with a clinician, they cite the same reasoning just 24.6% of the time.
Independent evaluations of LLM clinical graders, 2025 to 2026
Book a demo and we'll build one case end to end: the environment, the hidden ground-truth state, the grading rubric, and a graded model rollout, in your evaluation format, at no cost. Your team judges the fidelity first.