Medical AI: New system outperforms generalists on complex cases
Microsoft's multi-agent diagnostic tool shows significant accuracy gains over individual physicians in simulated hard cases, though cost and model dependency remain key variables.
By Rebecca Dawson ·
For patients facing complex medical issues, a new diagnostic tool from Microsoft AI promises a more accurate second opinion, with simulations showing it outperformed generalist physicians significantly. The system, known as MAI-DxO, correctly identified diagnoses in difficult cases more than twice as often as human doctors working individually. While this does not yet replace specialist care, it suggests a shift in how medical expertise might be distributed, addressing a growing gap between patient demand and the available supply of skilled clinicians.
The technology operates by coordinating multiple AI agents that function like a medical panel. These agents collaborate to question patients, order necessary tests, and debate potential diagnoses before reaching a conclusion. In a study published in Science on Thursday, the system achieved an 83% accuracy rate on complex cases derived from the New England Journal of Medicine. By comparison, primary care doctors and general internists, working with reference materials, were correct only 36% of the time. It is important to note that such complex cases are typically handled by specialists or teams, who were not included in this specific comparison.
The performance of the system is heavily dependent on the underlying AI models it utilizes. Microsoft tested the orchestration layer with models from various companies, including OpenAI, Google, Anthropic, Meta, and DeepSeek. The most significant results were obtained using OpenAI’s GPT-5 and GPT-5.5 models. However, the study indicates that much of the overall improvement stems from the power of these base models rather than the orchestration alone. For instance, when using the newer GPT-5.5 model, the system boosted accuracy from 72% to nearly 87% on complex cases, but offered little advantage on standard emergency department cases.
Cost efficiency is another factor in the analysis. Running the system with GPT-5 on complex cases cost approximately $8,400 per case, a figure that was reduced by about $3,000 when using the multi-agent approach, albeit with only a modest increase in accuracy. The benefits of the orchestration layer are most pronounced when paired with less powerful models; for example, a weaker model’s accuracy rose from 46.4% to nearly 69% with the assistance. Researchers noted that as base models approach theoretical limits, the additional gains from orchestration naturally diminish.
Microsoft AI CEO Mustafa Suleyman described the findings as a step toward 'medical superintelligence.' He stated that the company is collaborating with the Mayo Clinic to develop a foundation model for healthcare, with a release expected next year. Suleyman predicted that by 2027, the technology will enable significant breakthroughs in processing medical records, allowing for highly accurate predictions of how a patient’s condition might unfold. He believes this foresight will be invaluable to clinicians.
To validate these capabilities in real-world settings, Microsoft is partnering with Beth Israel Deaconess Medical Center in Boston. The evaluation will compare the AI’s recommendations against human doctors across various hospital environments, including emergency and urgent care. This follows a similar recent study by Google, which evaluated its own diagnostic AI in urgent care visits. Microsoft aims to use these real-world evaluations to determine if the system can be deployed at a larger scale, moving beyond simple benchmarks to integrated clinical decision support.
The launch of this study coincides with a broader manifesto from Microsoft on building medical superintelligence, defined as surpassing human-led systems in reasoning and coordination. Suleyman emphasized that the company is committed to accelerating the introduction of these tools, viewing them as a key differentiator in the competitive frontier AI landscape. He expects medical models to become an order of magnitude better within the next two to three years, driven by a feedback loop of real-world usage and iterative improvements.
Source: TechCrunch
