Methodology

Overview

Context

Our goal with this benchmark is to show how well AI models plan treatment in functional medicine, measured against the clinicians who treat these patients. Many strong medical AI benchmarks already exist. We wanted one that tests the decision that matters most in practice: the treatment plan itself.

Answer keys

We started with 17 treatment plans for 16 patients. Diadia drafted each plan, and the patient's own clinicians then reviewed, edited and signed it off. The signed-off plan is the correct answer for that patient: a ranked list of root causes, plus a full protocol listing each treatment's agent, dose, priority and timing.

Inputs

Every model gets exactly the data the plan was built from: lab values with the lab's reference ranges, symptoms, intake notes and, where the practice uses them, genetic results. The frontier models never see the clinicians' own target ranges or anything else specific to the practice.

Models

We tested five frontier models: Claude Opus 5, Grok 4.5, Gemini 3.6, GPT-5.6 Sol and GPT-5.6 Terra. Each received a plain physician prompt with reasoning effort set to high. We compared them with Diadia's production system, which has the clinical panel's knowledge built in. Every model answers every case three times, because a language model can give a different answer each time it runs.

Judges

Two language models judge the plans, and neither comes from the same family as any model we tested. They answer narrow yes-or-no questions and never assign a score. Code computes every number from their answers.

Evaluation mechanism

We split the work in two. Language-model judges answer four narrow questions, and code does all the math on top of their answers.

How it works

1. Freeze the case. We take the patient data and the clinician-reviewed plan and lock them. Nothing else from the practice comes with the case.
2. Build the answer key. We break the plan into two lists: ranked causes and treatments. Each treatment records its agent, category, dose, priority (core or support) and timing (start now, or wait for a named prerequisite).
3. Generate plans. Every model gets the same patient data and returns the same two lists in a fixed format, three times per case.
4. Match items. A judge pairs each cause and treatment in the model's plan with the clinicians' item that is the same clinical thing. Items without a match stay unmatched and still count toward the scores.
5. Check the rules. A judge answers every checklist rule that applies to the case with Yes or No, plus the evidence.
6. Compare doses. For treatments both plans share, a judge says whether the model's dose is the same, lower or higher.
7. Compute. Code turns the matches into scores and safety measures, and the rule answers into points. It averages across judges first, then across the three runs, then across cases.

Scores

Every rule uses the same 3-to-9 scale. The most important factor in a case (+9) outweighs any two secondary habits. A bad move costs about as much as a decisive good habit earns, so one good habit can't cancel out one dangerous step. The shared, practice and case rules carry roughly equal total weight, so no single layer dominates.

Safety measures

We track five safety measures, all computed by code from the matched treatment lists. Because they count clinical decisions, the way a plan is worded has no effect on them. Each is a percentage, averaged across judges, runs and cases.

Confidence and limits

We average the judges for each item, then the three runs for each case, then the cases. When a patient has more than one plan, we pool them. Every interval is a 95% paired bootstrap across patients with 2,000 resamples; the thin lines on the charts show these intervals. When a frontier model's interval doesn't overlap with Diadia's, the gap is very unlikely to be due to chance. That holds for all five frontier models on every measure on the results page.

• A small sample. Seventeen plans from sixteen patients means wide intervals. Treat models within a few points of each other as tied.
• One standard of care. Scores measure agreement with the clinicians who signed off each plan. That is a narrower thing than overall medical quality.
• Different information. The frontier models saw the patient's data and lab reference ranges only. The clinical panel's target ranges and treatment knowledge are part of what this benchmark measures, so we kept them from the frontier models.
• Diadia drafted the answer keys. Even so, Diadia doesn't reproduce the clinicians' final plans perfectly. Its score is the ceiling the frontier models are compared against, and we report it that way.