Presenting a version of our internal benchmark for evaluating large language models (LLMs) on complex patient treatment tasks
Today we are sharing the first results of the Functional Medicine Benchmark, the internal benchmark we use to evaluate how well AI systems make treatment decisions in functional medicine. The benchmark is still evolving: we are adding cases as our clinicians review more treatment plans, writing rules for every practice we work with, and refining how each decision is scored, so the results in this post describe the benchmark as it stands today.
The benchmark asks a practical question, which is whether an AI treatment plan makes the same decisions that a practicing clinician makes for the same patient: which underlying causes to treat, which treatments to include and which to leave out, at what dose, and in what order. Every case starts from a real patient. For each case, the treating clinicians reviewed, edited and approved a complete treatment plan, and that approved plan serves as the answer key. We gave the same patient data to five leading general-purpose AI models, which we refer to as frontier models, and to the Diadia pipeline, and we compared every plan they wrote with the approved plan.
Diadia scored 77.5 out of 100 for overall agreement with the approved plans, while the five frontier models averaged 47.4, and the largest differences appeared in the decisions that carry the most risk for a patient, such as starting a treatment before the patient is ready for it.
Most benchmarks for clinical AI test medical knowledge through multiple-choice exams, question answering, or checks on whether a model cites the right evidence. These tasks have clear right answers, and frontier models perform generally well on them.
The decisions that determine whether a plan helps or harms a patient receive much less attention: whether a treatment should start now or wait for a test result or an earlier step, whether it belongs in the plan at all, what dose to use, and in what order the treatments should begin. In functional medicine these decisions carry much of the clinical risk, because a typical plan combines supplements, diet, lifestyle changes, prescriptions and referrals, and the order in which they begin matters as much as the choice of each one. A model can cite the right evidence and still start the wrong treatment first, so we designed this benchmark to measure the decisions themselves.
Each plan is compared with the approved plan in two ways: four scores that measure agreement with the clinicians, and five measures that focus on patient safety.
The four scores, each out of 100, are:
The five patient-safety measures are:
A separate safety review looks for specific hazards in each plan, including problems with timing, drug interactions and contraindications, dosing, and claims that the patient's data does not support. Every hazard it records must quote the plan and name the fact about the patient that makes it a problem.
Every model received the same patient data with identifying details removed: lab values with the lab's own reference ranges, symptoms, intake notes, and genetic results where the practice uses them. The five frontier models, Claude Opus 5, Grok 4.5, Gemini 3.6, GPT-5.6 Sol and GPT-5.6 Terra, received a short prompt asking for a physician's treatment plan, with no information about Diadia or its methods, and each ran with its reasoning setting at high. None of the models saw the optimal lab ranges that Diadia uses. Every model wrote a plan for every case three times, because the same model can produce noticeably different plans for the same patient from one run to the next.
Two judge models from companies whose models were not part of the test, Kimi K3 and GLM 5.2, grade the plans, and they answer only narrow questions: whether two items are the same treatment, whether a plan meets a given rule, whether a dose is the same, lower or higher, and what in a plan could harm the patient. The judges never assign scores. Code calculates every number from their answers, averages the results across both judges and all three runs, and reports 95% confidence intervals calculated by resampling patients.
We use two checks to make sure that the results mean what they appear to mean. First, we also score each plan against the rules of other practices, where the scores fall by 22 to 47 points, which shows that the rules capture the standard of care of a specific practice and measure more than general quality. Second, we run the safety review on the approved plans themselves, which tells us how often the judges flag care that clinicians have already approved and gives us a baseline for reading every other result.
Diadia scored 77.5 out of 100 for overall agreement. The five frontier models averaged 47.4, and the strongest of them, Grok 4.5, scored 53.1. The largest gap was in the choice of treatments, where Diadia scored 73.3 and the frontier models averaged 37.6. On underlying causes Diadia scored 88.6 against 59.7, and on the clinicians' rules it scored 70.5 against 44.9.
The patient-safety measures show where the frontier plans depart from the approved plans. The frontier models started 71% of the treatments that the clinicians had deliberately held back, compared with 16% for Diadia. On average, 61% of the items in a frontier plan were treatments that the clinicians did not prescribe, compared with 20% for Diadia, and the frontier plans left out 52% of the clinicians' core treatments, compared with 21% for Diadia. Diadia matched the clinicians' dose on 70% of shared treatments and their order on 84%, while the frontier models averaged 45% and 57%. Each of the five frontier models departed from the approved plans more often than Diadia on all five measures. The chart below shows the same five measures as agreement with the approved plans, so that a higher bar is always better: the grey bars show the average of the five frontier models, and each tick marks one model.
The frontier models usually recognize what is wrong with a patient and rarely invent findings. They identified the clinicians' leading cause in 84% of plans, compared with 94% for Diadia, and fewer than 1% of the items in any model's plans rested on claims that the patient's data does not support. Their errors appear in the decisions that follow the diagnosis: starting treatments too early, adding treatments that the clinicians did not prescribe, leaving out core treatments, and choosing different doses.
To understand where Diadia's advantage comes from, we ran the Diadia pipeline with parts of its clinical knowledge removed. With no clinician knowledge loaded, the pipeline scored 45 on treatments chosen, which is close to the best frontier model at 47. Adding shared clinical reasoning guidelines raised that score by 2 points, and adding the practice-specific knowledge from our clinicians raised it by 30, which shows that almost all of the improvement comes from capturing how specific clinicians treat their patients.
The safety measures are averages, and the following five cases, one from each frontier model, show the kinds of decisions behind them. The first three were flagged as hazards by the judge models. GPT-5.6 Sol and GPT-5.6 Terra received almost no hazard flags, so their examples come from comparing their plans directly with the approved plans. We have described the patients in general terms.
A dose that could cause harm (Claude Opus 5). For an older patient who was severely underweight, the clinicians prescribed a modest increase of 300 to 700 calories a day above the patient's current intake, with 1.2 to 1.5 g of protein per kg of body weight. In one of its runs, Claude Opus 5 set a target of 30 to 35 calories per kg of body weight a day from the start, with no gradual increase, and the judge flagged the risk of refeeding syndrome, a dangerous shift in phosphate and potassium that can affect the heart and breathing when a malnourished person is fed too quickly.
A treatment the clinicians did not prescribe (Grok 4.5). A patient in their mid-70s came in for longevity screening, with an inherited pattern of high-risk cholesterol particles and no cardiovascular disease. The clinicians treated the cholesterol risk with a statin and ezetimibe. In all three of its runs, Grok 4.5 added daily low-dose aspirin to prevent a first heart attack or stroke. Current guidance advises against starting aspirin for this purpose after age 60, because the risk of bleeding in the stomach or brain outweighs the benefit, and both judges flagged it.
A drug interaction the plan did not address (Gemini 3.6). A patient in their late 60s was taking atorvastatin, amlodipine and other medicines that lower blood pressure. In one of its three runs, Gemini 3.6 planned a 30-day course of berberine and oregano oil for suspected fungal overgrowth in the gut, without mentioning those medicines. Berberine blocks the liver enzyme that clears atorvastatin and amlodipine from the body, which can raise their levels and increase the risk of muscle damage and low blood pressure. The clinicians' plan did not include berberine.
A higher statin dose (GPT-5.6 Sol). For an endurance athlete in their late 40s with a high level of cholesterol particles and early plaque in the arteries, the clinicians prescribed rosuvastatin 10 mg together with ezetimibe 10 mg. In all three of its runs, GPT-5.6 Sol started rosuvastatin at 20 mg with a planned increase to 40 mg. A lower statin dose combined with ezetimibe lowers cholesterol particles effectively with less statin, which matters for someone who trains hard, because muscle side effects from statins become more likely at higher doses and with heavy exercise.
A treatment started too early (GPT-5.6 Terra). For a patient whose zinc level was below the lab's reference range, the clinicians started zinc, magnesium and iron first and added psyllium fiber after four weeks, to be taken two to four hours apart from the minerals, since fiber taken at the same time as minerals can reduce how much of them the body absorbs. In two of its three runs, GPT-5.6 Terra started psyllium on the first day, alongside zinc.
Each of these is a decision about a specific patient that a test of medical knowledge would not measure, and in each case the clinicians made a different choice.
The benchmark is still small, so differences of a few points between models fall within the range of chance, which is why we report confidence intervals. The approved plans began as drafts written by Diadia that the clinicians then edited, which could favor Diadia's style of plan; the experiment with clinical knowledge removed suggests that the main effect comes from the clinical knowledge itself, because the same pipeline without that knowledge scores close to the frontier models on treatments. Judge models add their own variation, and the safety judges tend to favor cautious plans, which is why we compare their findings with how they rate the approved plans.
Diadia also makes mistakes: the safety review flagged serious issues in some of our own plans, such as describing an HbA1c of 6.7% as markedly elevated in order to justify a glucose-lowering prescription, and we track and correct these.
The Functional Medicine Benchmark remains an internal tool for now, and we will continue to develop it as our clinicians review more plans. Our next steps are to add approved rules for every practice, extend the benchmark to more areas of care, and run it again as new models are released.
Most evaluations of clinical AI stop at medical knowledge, which leaves the decisions that matter most for patients largely untested. Decisions about order, dose, and what to hold back or leave out shape what happens to a patient, and although they are harder to measure than exam answers, measuring them is essential for any AI system that helps plan treatment.
© 2026 Diadia. All rights reserved.
© 2026 Diadia. All rights reserved.
