Explainable Medical AI - Dr. Rich Caruana | Ep. 6
Dr. Rich Caruana — Principal Researcher, Microsoft Research
Dr. Rich Caruana on models a doctor can read: explainable boosting machines that match black boxes on patient data, what their graphs reveal about care, and why accurate is not correct.
Key Takeaways
Accurate is not the same as correct
A model scores well by repeating its data's quirks. It can learn that very high creatinine is safe, because those patients got dialysis. 43:12
A glass box can match a black box
On tables of numbers, explainable boosting machines are as accurate as deep nets and boosted trees, and every effect is a graph. 7:58
The graphs show how care is given
In the pneumonia data, risk jumps at 85 and falls at 100: the very old were treated less aggressively, centenarians more. 17:56
Round numbers set thresholds
Treatment starts at 50, 100, 180. Giving dialysis at 85 instead of 100 might save lives, which only a trial could confirm. 25:14
Let a language model read first
Given the graphs as tables, an LLM found nearly every surprise the doctors found, and some false ones. It tells clinicians where to look. 40:52
“There's nothing more dangerous in life and in science than a false assumption.”
“How do you not drown? Well, you develop the ability to swim.”
Chapters
- 0:00Intro2 min
- 1:42Why interpretability matters2 min
- 3:15A short history of machine learning3 min
- 6:30Glass-box models for tabular data2 min
- 8:20What an EBM is7 min
- 14:52Pneumonia risk by age8 min
- 23:09Treatment effects in blood tests6 min
- 29:26Blood pressure and round numbers2 min
- 31:41Risk in childbirth5 min
- 36:30LLMs that read the graphs4 min
- 40:26Four doctors and an LLM3 min
- 43:12Accuracy is not correctness7 min
- 50:40Where it fits in a hospital10 min
- 1:00:16Connecting EBMs to LLMs10 min
- 1:10:27Intelligible, the company11 min
- 1:21:34Who the customers are5 min
- 1:26:59Keeping the detail in the graphs6 min
- 1:33:26Between labs and universities8 min
- 1:41:38Advice for young scientists9 min
- 1:50:43Healthcare in ten years15 min
Mentioned in This Episode
About this Guest
Dr. Rich Caruana
Principal Researcher, Microsoft Research
A leading voice for intelligible models — GA2M and Explainable Boosting Machines — and for why black-box models are dangerous in clinical decisions. His pneumonia-mortality work is a canonical example of a model learning a deadly shortcut.