Kaelio supplies physician-generated and de-identified real-patient data for training and evaluating frontier models. Choose from ready-to-license Kaelio datasets or commission a custom program, from multimodal clinical records to longitudinal and interactive environments. Every deliverable includes comprehensive clinician-authored ground truth.
What we provide
Two product formats for model training and evaluation, available from the Kaelio catalog or built to your specification.
Physician-generated and de-identified real-patient data across clinical conversations, records, documentation, imaging, video, reasoning, safety, and research workflows. Delivered with the clinical ground truth required for training, post-training, or evaluation.
Stateful cases in which a model gathers information, requests artifacts, uses tools, and manages a patient over multiple steps or visits. The environment records the full trajectory, not only the final answer.
Both formats can be licensed ready to use or developed as a custom program for a specific capability, specialty, modality, or failure mode.
Data sources
Our catalog is built through controlled clinical data programs, not by relabeling public benchmark datasets. We produce new data through practicing physicians and source de-identified real-patient data through clinical partners.
Practicing physicians create clinical conversations, cases, documents, reasoning traces, adversarial prompts, and interactive environments to a defined model-development specification.
Clinical records, imaging, procedural video, and longitudinal episodes sourced from real practice under the appropriate consent, ethics, de-identification, and data-use controls.
Specialty-matched clinicians comprehensively annotate every deliverable with the relevant findings, ideal outputs, rubrics, decisions, safety errors, severity, and adjudication for its intended use.
Frontier clinical tasks
The medical frontier has moved beyond exam questions. We support real clinician workflows, multimodal and longitudinal reasoning, safety-critical decisions, and agentic tool use.
Differential diagnosis, workup, management, and treatment decisions under incomplete information.
HealthBench Professional · care consultation
Notes, summaries, referrals, coding, patient messages, and structured documentation grounded in the source record.
HealthBench Professional · writing and documentation
Questions that require finding, weighing, and synthesizing current clinical evidence.
HealthBench Professional · medical research
Cases combining dialogue with imaging, ECGs, pathology, documents, laboratory results, audio, or video.
AMIE · Med-Gemini
Multi-visit episodes that track disease progression, treatment response, medication changes, adverse effects, and escalation decisions over time.
AMIE · longitudinal dialogue
Red flags, unsafe reassurance, contraindications, hallucinations, and the decision to reassure, investigate, refer, or escalate.
HealthBench · clinical safety
Models retrieve records, use clinical tools, place orders, and maintain state across complex workflows.
MedAgentBench · FHIR-AgentBench
Why new data
Frontier models have saturated many exam-style tests. The remaining failures appear in realistic, open-ended, multimodal, and sequential work, where missing context, unsafe actions, and inefficient decisions matter.
→ 51.6%
GPT-4 falls from ~90% on the MedQA exam to 51.6% once the same medicine is played out as a sequential encounter; Llama-2-70B drops from ~60% to 4.5%.
AgentClinic, npj Digital Medicine, 2026
AUC 0.49-0.66
LLM judges separate complete from incomplete clinical answers only marginally above chance. Even when they agree with a clinician, they cite the same reasoning just 24.6% of the time.
Independent evaluations of LLM clinical graders, 2025 to 2026
< 32%
No frontier model scored above 32% on HealthBench Hard at release, a benchmark that took 262 physicians writing 48,562 rubric criteria to build.
HealthBench, OpenAI, 2025
Flagship format
A static prompt captures an answer. An environment captures the decisions that produced it. Each case lets a model gather information, request artifacts, use tools, and manage the patient as the clinical state evolves.
A practicing physician specifies the hidden patient state, available observations and artifacts, valid actions, state transitions, escalation criteria, and stopping conditions.
The model asks questions, orders and interprets tests, requests records or images, revises its assessment, and decides what to do next.
Information appears only when the model takes the relevant action. The patient can change with time, treatment, and decisions across a single encounter or multiple visits.
Clinicians evaluate the complete path for accuracy, context gathering, calibration, safety, efficiency, and outcome. The trajectories can support evaluation, supervised fine-tuning, preference optimization, and process or verifier training.
Multimodal and longitudinal
We build datasets and environments around the artifacts models must interpret together, while preserving how a patient and their care change over time.
Artifacts a case can carry
~54%
The best model scores only about 54% on comprehensive medical multimodal benchmarks, and medical-specialized models often underperform general ones.
GMAI-MMBench (NeurIPS 2024); OmniMedVQA (CVPR 2024)
~81%
Popular medical image-QA benchmarks can be gamed without the image: models retain ~81% of their VQA-RAD accuracy with the image blanked out.
HSCR, MICCAI, 2025
Data catalog
Start with a Kaelio dataset available for licensing, or commission data for a specific specialty, modality, workflow, failure mode, or model capability. Each engagement is scoped around the format, volume, provenance, clinical ground truth, and training or evaluation target your team needs.
Stateful, multimodal cases where models gather information, use tools, and manage care over time, with the complete trajectory captured for training and evaluation.
Physician-generated and real-world clinical conversations for care consultation, documentation, research, and patient communication, with comprehensive clinical ground truth.
Novel cases, differentials, workups, and management plans that expose and supervise clinical reasoning under uncertainty.
De-identified real-patient and physician-generated imaging, records, ECGs, pathology, clinical photography, and video.
Difficult and adversarial clinical data targeting unsafe reassurance, missed escalation, hallucinations, contraindications, and other high-severity failures.
Technical diligence
Kaelio prepares complete dataset, evaluation, ground-truth, and delivery evidence for every data program. Exact results and representative materials are shared after a technical scoping conversation.
Exact sample counts, schemas, formats, splits, provenance, licensing, versioning, and availability.
Evaluation harness, current-model performance, pre-training and post-training scaling where applicable, difficulty subsets, and failure-case analysis.
Ground-truth audit results, inter-rater reliability, adjudication records, and generalist and specialist clinician baselines.
Validated weekly throughput, ramp plan, staffing model, dedicated queue owner, scalable quality control, and adversarial workforce controls.
Technical packs are matched to the dataset, model capability, and intended training or evaluation use your team is assessing.
Review technical diligenceReview ready-to-license Kaelio datasets or scope a custom program with us. Tell us the capability, specialty, modality, and format you are working on, and we will show you what is available or what we can build.