Medical AI needs a six-dimension evidence matrix
In short: Microsoft's diagnostic benchmark is strong evidence for one clinical capability, but the blank cells still matter before anyone claims the whole care arc.
Microsoft reported on 22 October that MAI-DxO reached 83% diagnostic accuracy on 304 complex NEJM cases. Generalist physicians reached 22% without reference material and 36% with it. On emergency-department cases, the reported comparison was 80% versus 47%.
That is strong evidence about sequential diagnosis inside a controlled benchmark. It is not evidence about the whole care journey. Every broad medical-AI claim should carry a six-dimension evidence matrix, with the untested cells left visibly blank.
Microsoft tested a richer diagnostic loop
The Microsoft release describes SDBench as an alternative to static medical quizzes. A physician or AI starts with limited information, asks questions, orders tests and eventually commits to a diagnosis. The benchmark measures accuracy, requested evidence and estimated test cost.
The open SDBench repository makes the machinery inspectable: one model can act as gatekeeper, another judge scores the submitted diagnosis, and a costing module estimates the tests ordered. It also states an important limit. The five bundled synthetic cases are for development, not performance reporting; clinically sourced NEJM and MIMIC data require separate access or licensing.
| Reported result | Microsoft figure |
|---|---|
| NEJM diagnostic accuracy | 83% MAI-DxO; 22% physicians without references; 36% with references |
| Emergency-department diagnostic accuracy | 80% MAI-DxO; 47% physicians |
| Cost-optimised NEJM configuration | 81% accuracy at less than half the comparative estimated cost |
These are Microsoft-reported results. I did not reproduce the experiment.
The companion framework makes the missing scope explicit
Microsoft published a companion medical-superintelligence framework at the same time. It defines six interdependent dimensions: medical knowledge; diagnostic reasoning and triage; therapeutic decisions; longitudinal context; clinical coordination; and correspondence with a patient's values and agency.
That list changes how the benchmark should be read. SDBench directly supplies evidence for a sequential diagnostic task. The inspected sources do not report patient outcomes across all six dimensions, nor results from Microsoft's announced prospective Beth Israel Deaconess study.
A six-column matrix would make the boundary hard to miss:
| Dimension | Evidence in this release |
|---|---|
| Medical knowledge | Used within the task; not separately isolated |
| Diagnosis and triage | Controlled sequential-diagnosis benchmark |
| Therapeutic decisions | No reported outcome |
| Longitudinal context | No reported outcome |
| Clinical coordination | No reported outcome |
| Patient values and agency | No reported outcome |
Blank does not mean failed. It means untested here.
My practitioner rule is to price the blank cells
This is editorial judgment, not a claim that I have deployed MAI-DxO at Trigent or for a client. Before a product, procurement or governance team accepts a label such as “medical superintelligence”, I would ask for one row per claim and six evidence columns.
Each filled cell should name the population, workflow, comparator, outcome, study date and accountable reviewer. Each blank cell should carry an owner and the next test. That turns missing evidence into planned work instead of marketing whitespace.
The matrix also stops one valid result from inheriting authority it never earned. An accurate diagnosis can still meet a poor treatment decision, stale history, broken hand-off or recommendation that ignores what a patient can access and accept.
What's in it for you
- Separate the capability demonstrated from the care outcome implied.
- Require blank dimensions to stay visible in review decks and procurement claims.
- Reopen the matrix when the model, workflow, population or evaluator changes.
A diagnostic score can fill one evidence cell; it should never be allowed to colour in the whole care pathway.
