AI
The Access Gap in Cancer Care
20 million people are diagnosed with cancer worldwide each year. In the U.S., 2 million new cases arrive annually, but only 15% reach a comprehensive cancer center with a tumor board. The other 85% are treated in community or rural hospitals with no specialist panel.
Tumor boards are effective but slow and expensive. A top cancer center can review only a handful of patients per session. Clinical trial enrollment compounds the problem: most qualifying patients never find a matching trial, and many trials collapse for lack of participants.
Research Building Blocks: GigaPath and BiomedParse
Microsoft Research built two models that feed directly into the orchestrator. GigaPath, developed with Providence Health and the University of Washington, is “the world’s first pathology foundation model” capable of scaling transformers to gigapixel whole-slide images. Published in Nature, it has been downloaded over one million times. BiomedParse unifies nine imaging modalities and 64 object types through a natural language interface, letting a clinician query an image in plain text and receive segmentation output across radiology, pathology, and dermatology in one call.
Mapping Research to Agents
The Data Organizer agent applies Universal Medical Abstraction to pull diagnosis, staging, prior treatments, genetic biomarkers, and treatment preferences from clinical notes. Every extracted element is grounded back to the source note, so a clinician can verify any claim without leaving the interface. The Clinical Trials agent takes the same structured patient profile and checks it against current inclusion and exclusion criteria on clinicaltrials.gov, surfacing trials the patient may qualify for in real time. Agents can also field ad hoc questions from live tumor board discussions that the initial summary did not anticipate.
Deploying and Fine-Tuning Domain-Specific Models in Azure AI Foundry
Azure AI Foundry at ai.azure.com includes a health and life sciences model filter alongside models from OpenAI and Meta. Will Guyman ran the MedImageParse pathology and endoscope demo (10:50): the model segmented cancerous cells in red from a pathology slide, then identified neoplastic polyps in an endoscope image from a plain-language query. Deployed models get a HIPAA-compliant API endpoint in your Azure subscription.
The GastroVision fine-tuning notebook (12:32) fine-tunes MedImageParse on colon polyp data in Azure ML. Mean AUC plateaued at 0.98, with sensitivity and specificity adjustable per use case.
Running a Full Tumor Board in Microsoft Teams
One azd deploy command and a scripts/uploadpackage.sh call push all agents into a Teams meeting. In the Biomed orchestrator executing full tumor board in Teams (18:05), Biomed planned a six-agent run for patient_4, accepted a mid-session request to add a Research Agent, then executed. Each output links to the exact clinical note sentence it was drawn from.
“But maybe I don’t trust the AI model.” — Shrey Jain
Radiology used CXRReportGen. Pathology came from Page AI’s agent Alba. The Treatment Agent cited NCCN guidelines page by page and linked matched clinicaltrials.gov studies. The run produced a Word document with clinical summary, radiology, pathology, and history.
Customizing and Extending the Orchestrator
Every agent is defined in agents.yaml with a name, system prompt, tool list, and description. Tools are Python plugins with a one-to-one filename-to-tool-name mapping. The CXRReportGen plugin holds a decorator flagging it as callable and a pointer to the Foundry endpoint deployed earlier.
Setting healthcare_agent_service: true in agents.yaml connects to Microsoft’s Healthcare agent service portal, adding clinical safeguards and grounding to approved data sources. Pre-built templates for Radiology and Pathology are available there, reducing the code a developer needs to write.
Notable Quotes
there are 20 million new cancer patients worldwide Hoifung Poon · ▶ 4:15
the world’s first pathology foundation model Hoifung Poon · ▶ 5:29
But maybe I don’t trust the AI model. Shrey Jain · ▶ 21:06
Key Takeaways
- 85% of U.S. cancer patients are treated in community hospitals with no tumor board access.
- MedImageParse reaches 0.98 AUC after fine-tuning on GastroVision and deploys to a HIPAA-compliant endpoint in one click.
- The Biomed orchestrator accepts mid-plan feedback and routes dynamically across six specialized agents inside Microsoft Teams.