Tomorrow's Medicine, Ep. 6: Explainable Medical AI - Dr. Rich Caruana

With Dr. Rich Caruana, Principal Researcher, Microsoft Research. Hosted by Arseniy Arsentyev. Published August 24, 2025.

https://aarsentyev.com/podcast/ep6-rich-caruana

Machine-transcribed and lightly edited: filler words and false starts are removed, and the wording is otherwise as spoken. Where the two differ, the recording is the record. Times are minutes into the recording. Not medical advice.

[0:00] Arseniy Arsentyev: It's the sixth episode of Tomorrow's Medicine Podcast. My name is Arseniy, and today I'm talking to Dr. Rich Caruana. Rich is an esteemed scientist in the fields of explainability in machine learning and intelligible machine learning models. He's currently a chief science officer of Intelligible. Intelligible is a startup aiming to bridge the gap between tabular data and large language models. Prior to this, Rich was a senior principal researcher at Microsoft. He did his PhD at Carnegie Mellon University under the supervision of machine learning legend Tom Mitchell. In his thesis about multitask learning, he created a new branch of machine learning called the transfer learning. Dr. Caruana, thank you for being here. I appreciate your time and effort that we put into preparing this episode.

[0:56] And so, guys, today is a special episode because we're using a lot of visual information. And for those who listen on Apple Podcasts, I'd highly recommend to go to my YouTube channel or my Spotify page to see the video, to see the graphs that we're about to discuss, to fully grasp the depth and the most interesting parts of the topics we discuss. Without any further delay, Dr. Caruana, it's a word for you.

[1:30] Dr. Rich Caruana: Thanks a lot, Arseniy. Hey, so I'm going to talk about the importance of interpretable machine learning, especially when you're using machine learning for critical problems like healthcare. And we're going to do something unusual in the podcast. We're going to actually start with a very short slide presentation. And that's because the models are interpretable. And they're interpretable, meaning they're visualizable. You can see what the model has learned. So we want to be able to show you what that looks like. And then after that, we'll go back into a long discussion about different aspects of machine learning and AI.

[2:06] Okay, so, hey, we're going to talk about why interpretability is so important in machine learning for healthcare. This is something I've been working on for 15 years now. And I've recently retired from Microsoft and joined a company called Intelligible Inc. that is focusing on bringing this kind of technology to market. But I want to give credit where credit is due. So the top line here in red are a bunch of interns who have worked on this with me at Microsoft over the last 15 years. The green line is a bunch of other researchers at Microsoft who had worked on this. And the blue line at the bottom is additional researchers at Microsoft who have worked on this.

[2:50] So, in fact, two-thirds of the people who have worked on this project with me over the last 15 years have all been at Microsoft. And all of this work has been done because Microsoft allowed us to do it. So I want to make sure I give credit to Microsoft. But now I've moved to Intelligible and we're trying to create products that bring this to the market. Before we start, let's do a brief history of machine learning. So back in the 50s, we had perceptrons. Those were linear perceptrons. Then statisticians got interested in things like K-nearest neighbor in the 60s. In the 70s, decision trees happened both in the U.S. and they were co-developed in Australia.

[3:31] And then in the 80s, we saw neural nets get added to the mix. Now, these are neural nets that are different from perceptrons because they now have a hidden layer. And then around the 90s, we started seeing more principled methods, more mathematically sophisticated methods like Bayes nets and support vector machines. Then around 2000, sort of ensembles of trees happened. We started seeing bagged trees, boosted trees, random forests. But SVMs were still very, very popular. Then a miracle sort of happened and neural nets hit again in 2010. These are deep neural nets. And they're amazing, right? They're much more powerful than the neural nets before them.

[4:11] Interestingly, though, if you worked on things like tabular data, that is, data that looked like columns of numbers, gradient-boosted trees and random forests were still king of the hill around 2010 for that kind of data set. So we start to see a separation of the kind of models that we use for different kinds of purposes. And then, of course, let's say five, six years ago, large language models happened. And that's really amazing. They are far more powerful than even the deep neural nets that happened before them. And an interesting question is, if you work in tabular data, what are we going to see going forward? Are large language models going to be king of the hill for tabular data? Or are still like ensembles of trees going to be king of the hill?

[4:59] What's going to happen? Before I do that, though, I just want to talk about neural nets, because neural nets are really, really exciting, particularly in their LLMs. So the first coming of neural nets is these linear perceptrons, which then grew out of favor because it was proved that linear models couldn't do that much. Then the second coming of neural nets happens when neural nets get a nonlinear hidden layer and are trained with backpropagation. So that's an exciting stage of development in neural nets. Then there's a third coming of neural nets. These are the deep neural nets. We all tried, by the way, in the 90s to get neural nets to be deep, but we didn't have the technology to make it work in the 90s.

[5:42] But somehow by around 2008, 2009, suddenly we could get deep nets to work. And that's amazing because suddenly they were far more powerful. And then there's the fourth coming of neural nets, which is these large language models that are now trained on things like the entire Internet. So it's really exciting that neural nets sort of keep coming back. And every time they come back, they're more powerful than before. And now perhaps the most exciting thing is that these large language models are definitely here to stay. So there's no doubt that we're going to see large language models in health care and in lots of other problems. And now these large language models are not just text. They can do images. They can do video. They can do things like that.

[6:30] Interesting question is, are they going to be the model of choice for tabular data? For things like tabular data, I mean things like numerical characteristics of patients. So this would be things like a patient's blood pressure or their age or the blood urea nitrogen in a blood test, things like that. And what we think is going to happen is that glass box machine learning is actually going to become the method of choice, particularly in health care, but in a lot of other domains as well going forward. And you might think, boy, that's kind of crazy that glass box machine learning would take over tabular data as opposed to, say, large language models or as opposed to gradient boosted trees and random forests. So let's talk about that.

[7:15] So a lot of people believe that there is an accuracy intelligibility tradeoff. That is, the black box models like deep neural nets, boosted trees and random forests are just inherently more accurate than the glass box methods that you can understand, things like logistic regression or naive Bayes. And the good news is that this is no longer true if you're working on tabular data. In fact, what we're going to talk about today is these EBMs. These are explainable boosting machines. So these are still boosted trees, but they're boosted trees that have been designed in such a careful way that what they learn is interpretable. And I'll show you how that's interpretable.

[7:58] And it turns out that these explainable boosting machines, these EBMs, are just as accurate as deep neural nets and boosted trees, gradient boosted trees and random forests on tabular data, on data that's numerical. But it turns out they're even more interpretable than things we're used to using, like linear and logistic regression. Okay, let me tell you a little bit more about what these EBMs are. They're a kind of what's known as generalized additive models, which have been around for 30 plus years. So now the simplest additive model is linear regression or logistic regression. Now it's very interpretable because it's very simple. It's linear.

[8:41] Sadly, it's not very accurate on complex data sets. And that's, again, because it's linear. It can't model nonlinearities. And you really have to be careful of these models. Sometimes they'll actually get the sign wrong on some of the coefficients in order to get the sign right on other more important coefficients. So if this coefficient is really important and it wants to get that slope right, it'll actually cheat and get the coefficient sign wrong on some of the other coefficients. So you really can't trust these models.

[9:10] And that means we should stop using them because there are now better models to use. Now at the other extreme, we've, of course, got black box machine learning. This is deep neural nets, random forests, boosted trees. Hey, they're black box models. So, of course, they're not very interpretable. But they can be very accurate. And that's because they can learn very complex functions, including nonlinearities. One thing that's important about these models, though, is they're so complicated that they have a tendency to learn spurious effects. That is, if you use LIME or SHAP to open up one of these black boxes, you will often see that some of the effects that these models have learned are not actually true statistics of the data. They've done some overfitting because of their complexity.

[9:55] In the middle, we have, what might be called the Goldilocks models. These generalized additive models of which EBMs, explainable boosting machines, are currently the best ones that are available. Now, these are more interpretable than linear and logistic regression, even though they're in the same family of models. And that's because they can model nonlinearities. So this is an example. Instead of this straight blue line that we have here on the left, here's a blue line learned by an EBM. And notice that low risk is here in the middle, risk goes up a little on the left, then risk rises rapidly on the right, but then it flattens off. And if you think about it, a straight line can't do this. In particular, a straight line can't go up on the left and up on the right.

[10:40] A straight line would be just sort of like the average of this kind of graph. It wouldn't show you all this detail. So the good thing is, because it can model this kind of detail, these EBMs can be much more accurate than linear models. And then the other good thing is, because they can be graphed in this way, it turns out they're extremely interpretable. That is, human experts love looking at these graphs because they see detail that makes them understand what the model has learned. And they will also see detail that makes them confident the model is learning the right thing.

[11:14] Another nice property of these models, because their complexity is restrained to be less than these black boxes, which can learn almost anything inside of them. Because of this, when these models learn something, it's more likely to be a real effect that's in the statistics of the data. So that's comforting that when these models learn something, it's almost always a real effect in the data. Okay, a really important thing about those models is that machine learning people didn't invent them. Although we invented the algorithm for EBMs, generalized additive models were invented by Trevor Hastie and Rob Tibshirani, two very famous statisticians down at Stanford, invented by them in the late 80s. So statisticians actually invented this class of generalized additive models, GAMs, and they were very, very popular in statistics in the early 90s.

[12:09] But then machine learning sort of started happening in the mid to late 90s, and machine learning kind of got all of the excitement because it was even more accurate than the early GAMs. Our contribution is going to be to take these GAMs that Trevor Hastie, this was Trevor's PhD thesis. It's going to be to take that method of learning, and what we're going to do is find even better algorithms by using all of the modern machine learning tricks. So we're going to use boosted trees.

[12:40] We're going to use ensembling, all those kinds of things. Okay, let me show you what this looks like. This is a model.

[12:49] Arseniy Arsentyev: Can I interrupt you for a second?

[12:50] Dr. Rich Caruana: Yeah, yeah, go ahead, Arseniy.

[12:51] Arseniy Arsentyev: I'm absolutely sorry. I just need to give the people a little overview on this because at this point, I understand that many of you who are listening or watching to this podcast, you may be overburnt by this ML technicalities and this level of description. So just for you, this generalized additive models is a mix between simple statistical tools. It's not a mix, but it's a mix in a way that it stands in the middle between simple statistical tools like regression and our LLMs. LLMs, as you all know, imagine GPT, imagine Grok, imagine anything you want to. It is the huge models that do accurate decisions based on data, but not correct decisions. About this topic, Rich will talk later, as I know.

[13:47] And for you to understand, these generalized additive models are so important to the healthcare data because they help clinicians to understand where certain biases in data are affecting the predictions and affecting the models working. So it is super important in the future of medicine. And at this point, I want to just keep silent and let Rich talk.

[14:19] Dr. Rich Caruana: Thank you. And that's exactly right. These EBMs represent a sort of middle ground that takes some of the best things that were developed in statistics 30, 40 years ago and combines them with some of the newer techniques that are in machine learning to sort of come up with an algorithm that's very powerful and as accurate as any of the black box methods are. And then what we're going to do later is we're going to connect these things directly to large language models so that the two of them can work together.

[14:50] Arseniy Arsentyev: That's great. Thank you.

[14:52] Dr. Rich Caruana: Let me show you what these things actually look like. So here's a data set where we're trying to predict which patients are at higher risk from pneumonia. So everyone in this data set has pneumonia. And we're just trying to predict the patients who are very high risk and should be admitted to the hospital versus the lower risk patients who can be treated as outpatient. And this is what the model has learned. It's got about 50 features. But this is what the model has learned about risk as a function of the patient's age. So on the x-axis here, we have the patient's age. And on the bottom, we just have a histogram of the patients that are in our data set. The data set has about 15,000 patients.

[15:31] And you can see most patients are in their 60s, 70s, 80s, and 90s. There are a few patients who are older, out near 100, and a modest number of patients who are in their 20s, 30s, 40s, and 50s. And the top is what the model has learned, the EBM has learned, about risk as a function of age. And this is on a log scale on the vertical axis. So down means a big contribution to low risk. And up means a large contribution to high risk. So the patients at the top here are much higher risk than the patients at the bottom, in part because this is a log scale.

[16:09] And then what you would do with this model is you would have 50 of these graphs because you've got 50 features. And you would go to each graph, and you'd find some contribution to risk. For example, suppose the patient is 60 years old. You would see that that adds minus 0.2 to the risk. And you would go to each of the 50 graphs, add this number up. And eventually, if the number is very high, it means the patient is in total high risk. And if the number is low and even very negative, it means the patient is low risk. Well, let's look at this graph. The graph is pretty much flat from age 18 to age 50. And basically, the graph is saying, well, it's good to be young.

[16:48] Everybody who's under 50 is sort of equally low risk as far as pneumonia is concerned. And then risk starts to go up slowly as you go from your 50s into your 60s. But it's not too bad. So this is a modest increase in risk. And then we see a pretty dramatic increase in risk right around 67, 68, 69. This is data from the United States. So this is a retirement effect. A lot of things happen to people in the United States when they retire. Not only do they stop working, but it turns out in the United States, it turns out their access to health care changes, their insurance provider changes. Perhaps now they're on Medicare instead of insurance provided by their company. Lots of things happen.

[17:31] And they might even move where they live and not have a doctor lined up yet. So a lot of things change when a patient retires in the United States with respect to their health care. And then we see risk continue to go up and climb as the patient goes in their 70s into their 80s. That's to be expected. The patient is getting elderly. And that means risk of many things in health care is increasing. And then we see a surprising step function jump in risk right at 85.

[18:01] That's unexpected. We see a surprising flatness in risk, actually, through the upper 80s and 90s. And then we see a surprising drop in risk that happens at 100, 101. This rise in risk at 85, we think what's happening is patients who are older than 85 are considered very elderly, and they're getting less aggressive care because they're so elderly. The average life expectancy in this data set is around 80. So patients over 85 are definitely on the elderly side of being old. And we think what's happening is something like, well, grandpa is 86 years old. Grandpa has a number of comorbidities associated with being this old. And now because of those comorbidities, grandpa has sadly got pneumonia. And the first rounds of treatment for pneumonia aren't working.

[18:55] Perhaps we should be less aggressive in our care and allow nature to take its course, allow grandpa to pass, because he has these other illnesses and his time is approaching. And 30 years ago, when this data set was collected, that was sometimes considered an appropriate care. And pneumonia was sometimes called the old man's best friend. The idea being that you have the illnesses associated with being elderly, and then you would get pneumonia because of those other illnesses. And then pneumonia would sort of take you out within a few weeks or months. And that maybe that was a good way to end. We no longer think that. In modern data sets, we don't see this jump at 85.

[19:44] But we believe this effect that we're seeing in the model, and the model is very confident of this jump, is because of a social effect, the way doctors are deciding to give care. We also think this drop in risk that's very surprising that's happening at 100 and 101 is a similar social effect that's caused by the way care is being delivered. And what we think is happening here is you've made it to being 100. Like you're a centenarian. Maybe your 101st or 102nd birthday is two or three months away, and you've got pneumonia. Now, pneumonia is a reasonably treatable illness.

[20:23] And if we fight hard enough, we have a reasonable chance, even if you're elderly, of getting you past pneumonia. And we think what's happening is these patients may or may not have other comorbidities that cause the pneumonia. It could be their age alone has caused the pneumonia. And because they're 100, 101, the doctors just don't want to give up on this patient. And they just want to basically go for the record. And maybe the patient wants you to go for the record. They want to get to that 101st birthday. Maybe the children or the grandchildren also want you to go for the record. So we think what we're seeing here, this reduction in risk that happens right at 100, 101, a very round number, right? And a very important round number in people's age, right?

[21:10] You're a centenarian now. We think that this is because the care is particularly aggressive at this point because you've made it to being a centenarian. You've made it to being 100. So interesting thing about this. Now, this model has 50 graphs. So you have to use all of them. We don't have time to look at all of them. We can see amazing detail here, right? Everybody under 50 is low risk. 50, 60 is not too bad. We see a retirement effect in U.S. data. Getting old adds to your risk. We see a round number effect that occurs at 85 as patients start to be considered very elderly. And we see an interesting effect at another round number, 100, because doctors just don't want to give up on patients who have made it to being 100.

[21:53] This model, it turns out, is the most accurate model anyone can train on this data. There are other models, such as deep neural nets and boosted trees, which are similarly accurate, but none of them are more accurate. And yet with this model, we can see all this detail. There's also something else. Doctors, although they kind of smile and understand why the risk goes down at age 100, they don't actually want that to be in the model. They don't want to predict that a 105-year-old is lower risk than a 95-year-old, even though the statistics in the data suggest that the treatments these patients are receiving make that true. So the doctors want to correct this in the model.

[22:32] And with this kind of model, because it is represented as a series of graphs, we actually can correct it. We can redraw this part of the graph and just make risk monotonically go up, even though the statistics of the data suggest that there might be lower risk because of treatment effects out here at 100. So this is really valuable. It means that not only can doctors see the detail of the model, try to understand why that detail is in the statistics of the data, but then the doctors can find details they're not happy with and they can change it. So that's very, very important.

[23:09] Now, these treatment effects that we're talking about happen all over the place. This is pneumonia. This is your blood urea nitrogen. So this is a blood test that pretty much all of us have had. And low blood urea nitrogen is low risk. And it is correct that risk starts to climb rapidly as your blood urea nitrogen sort of passes 30, 35, 40. Notice it flattens up right at 50. Then it starts to go up again as your blood urea nitrogen goes up. And then it peaks at 100. So notice it flattens at 50, a very round number, and peaks at 100, another very round number. And then the risk comes down a little bit and stays relatively flat.

[23:49] So what's happening is at 50, if you have this high of a blood urea nitrogen, you start to get a pill. You start to get medication. And that medication is effective at mitigating risk. So even if your blood urea nitrogen is higher at 60 or 70, the pill is very effective. Then eventually the pill isn't as effective as you would like. And as your blood urea nitrogen starts hitting, say, 80 or 90, your risk is going back up again because the medication is not strong enough anymore. And then notice the risk peaks at 100.

[24:21] That's dialysis. So we start giving the patients who have this high of a BUN, we start giving them dialysis, and that's very effective. And what happens is the patients who are all getting dialysis, say, at 110 or 120, are actually surprisingly a little lower risk than the patients, say, at 90 who are healthier who aren't getting dialysis. So it turns out that these treatments are incredibly effective. We can see the benefit of these treatments in the graph. The model has learned amazing detail. And it's even figured out exactly where the treatments have occurred. What's interesting with these sorts of additive models is we can also try to estimate what would happen if we were to change the treatment threshold.

[25:06] So it's kind of a shame that patients who have a BUN of 120 are lower risk than patients at 90 who are healthier. So we can play the game of suppose we move the treatment threshold to, say, 85 or 90, gave dialysis to patients earlier. And then we can conservatively estimate that the new treatment graph would look actually probably lower than this red line. We have reasons for being able to think that this is a conservative redraw of what the graph probably would look like if we change the treatment threshold from 100 to 85. And now the area between this new red line, if we were able to achieve that by changing the treatment threshold, and the original line is lives saved. So this is reduced mortality risk.

[25:53] And in the United States, we estimate that we would save something like 2,500 to 5,000 lives per year. And worldwide, we would save more than 50,000 lives. So I'm making this change in this treatment of when we apply dialysis to patients. Now, there are other considerations in dialysis.

[26:12] Arseniy Arsentyev: And it's only from one graph. So only from one test related to pneumonia risk, we can save more than 50,000 people worldwide. Just I want everybody to think about other 50 graphs that Rich talked about that we have no time to look at. And imagine how much inference and how much surprises we can extract from this, as it seemed, uncorrelated medical data.

[26:45] Dr. Rich Caruana: Thanks, Arseniy. And this is absolutely true. So just this one blood test, blood urea nitrogen, and just one treatment, dialysis, if we thought it was appropriate to change the treatment threshold. Now, there are lots of things that go into deciding whether it's appropriate and when it's appropriate to give a treatment. There's the cost of the treatment. There's the comfort and discomfort of the treatment. There's the other risks associated with treatment, which here perhaps are not mortality risks that show up in the short term, but maybe a treatment, for example, chemotherapy and radiation therapy and cancer have long-term risks associated with them that wouldn't show up in this graph.

[27:27] So it's important that clinicians would be the ones who would look at this sort of graph, decide whether it was appropriate to change the treatment threshold, what other risks and costs that might or might not incur. So we would never suggest that from one data set and one graph that you should sort of change healthcare worldwide. But definitely, immediately when the clinicians saw this, they started wondering, oh, this is interesting. I wonder if we should consider changing this treatment threshold instead of round numbers. What are the odds 50 and 100 happen to be the right places to be deciding to give care? Doctors are just using round numbers like everybody else. So they started wondering, well, maybe a round number of 85 or 90 would be more appropriate.

[28:15] And then also you start to think about, oh, maybe we have to do a randomized clinical trial in the future, do research in order to investigate whether this is an efficacious change or whether it's a change that isn't warranted for other reasons. So this sort of graph immediately raised all sorts of really interesting questions in the clinicians' minds. And they immediately wanted to do additional studies to follow up on this sorts of thing. And we do see this sort of thing in other graphs.

[28:44] So I'll show you, this graph says if our interpretation is right, if this data set generalizes to the world population, and if it would be appropriate to change this treatment threshold to 85 or 90 for dialysis, that's a big if, by the way, then we would be able to save perhaps 50,000 or more lives worldwide. But only if all of those things are true. Let me show you, though, that this sort of peak, most things, as you get worse on something, your risk just continues to go up. It doesn't go up, flatten off, then go up and then peak and then come back down. Let me show you that this is really common. This is the pneumonia data set. This is your systolic blood pressure. That's the higher of your two blood pressure numbers.

[29:30] You might be 120, 130. Notice there are interesting jumps in this graph. And notice it jumps here, and then it comes down a little bit. That's because most of the patients are starting to get treatment. But then it starts to go up. It peaks. And I was really surprised. It peaks not at 175, but it peaks at 180. That struck me as weird because 175 is a rounder number. Then it comes down. Then it goes up around 200, and it comes down. Then it goes up around 225 and comes down. This is not the case that as your blood pressure were to keep getting higher, your risk would go up, and then your risk would get lower, and then your risk would go up, and then it would get lower.

[30:11] This is not true at all biologically. What's happening is once you hit, 200, the doctor is thinking, well, your blood pressure is over 200. We have to treat it. They're going to treat it very aggressively. And because you've heard that your blood pressure is over 200, you're much more likely to obey the regimen of treatment that the doctor gives you, like reducing salt exercise and things like that, in addition to the medication that they're going to give you. Okay, so that explains why we get these jumps at round numbers. Now the question is, why do we get a jump right here at 180 as opposed to 175? Because I thought it should be at 175.

[30:50] It turns out, a doctor explained to me, the definition of hypertensive urgency is systolic blood pressure above 180, not above 175. So that's why we're seeing a peak right here at 180. It's because if you're at 175, you're below hypertensive urgency. You're probably not going to get certain kinds of treatments. But if you're at 185, you're past 180, you're now in, by definition, you're hypertensively urgent, and you're likely to get a more aggressive treatment. So that's why we see this round number peak happen at 180 as opposed to 175. And this model is accurate enough to learn this sort of detail and even effectively learn what the definition of hypertensive urgency is in this data set.

[31:38] Let me now go to a different data set. We're going to talk, instead of pneumonia, now we're going to talk about maternal risk during delivery. And this is risk to both the mother during delivery, and it turns out what's risky for the mother can also be risky for the baby that's being born. And this is a model that's trained on risk data. So we have outcomes that have happened to the mother. This would be things like the mother has had a hemorrhage, has needed a blood transfusion, has needed a surgery afterwards, possibly has even needed a hysterectomy. And in very, very rare cases, perhaps the mother has died.

[32:17] And the model, if you were to ask doctors, the doctors would say, well, a mother is at risk during labor if the mother has hypertension, that is high blood pressure. And that could be high blood pressure that's preeclampsia, or it could be they had high blood pressure even before they got pregnant. They're also at high risk if they have high body mass index or if they have any of a number of conditions like that. Those are the things a doctor will typically explain as being the highest risk. When we train our model, it agrees, by the way, that those are very important risk factors. But it says the two most important risk factors are these.

[32:56] It turns out the weight of the baby at the time the baby is going to be born is the most important risk factor. Remember, this is a log scale. So going from one to two is a tripling, it turns out, of risk. That's an amazing increase in risk. So as the baby gets bigger, the risk to the mother and it turns out to the baby increases dramatically. And that just makes sense, right? The larger the baby that has to come through the birth canal, the more stress that is put on both the mother and the baby in the process of delivery. The second most important risk factor, surprisingly, is the height of the mother. And you might think, well, why is the height of the mother important?

[33:40] And notice what the model has learned, which is that the taller the mother in centimeters, the lower the risk. Well, it turns out the height of the mother is the only measure we have in this data set, which is an estimate of the frame size of the mother. Taller women have larger frames. They have larger uterine openings. And because of this, a baby of any size is more easily going to fit through the birth canal if the mother has a larger frame. And the model has basically recognized that the physics of birth are really, really important to risk during delivery. That is, the larger the baby, the higher the risk. And the larger the mother in terms of frame size, the lower the risk. And it kind of makes sense.

[34:27] If you think about trying to put, to be indelicate about it, a large object, a large basketball, through a small hoop, you're going to have to struggle to get the ball through the hoop. However, if the basketball were smaller and if the hoop were larger, then you're going to have less pressure needed to do delivery. So, we think this is what the model has picked up on and has said is even more important than things like hypertension, preeclampsia, and body mass index. And it immediately makes you want to form a new feature, which is a ratio of the size of the baby to the size of the mother's frame. And we did that. So, body mass index is your weight divided by your height squared. So, that's what BMI is.

[35:20] We created a BMI for pregnancy, which is the baby's weight divided by the mother's height squared because it's the only feature we had. And immediately, this became the most important feature in this data set for predicting maternal risk and also risk to the baby. And you can see that basically, as this ratio gets higher, as the baby gets larger relative to the size of the birth canal, risk goes up dramatically. And this quickly became the most important feature. And one of the things this tells us is the height of the mother is a very poor way of measuring the size of the frame of the mother. It turns out by doing something very simple, like using a ruler, you can actually measure, say, from hip bone to hip bone.

[36:07] You can measure in a doctor's office in an exam a more accurate measure of the size of the birth canal. And this is what we think we should start doing going forward. So, here's another case where an interpretable model has basically suggested changes that we should make to healthcare to improve the quality of healthcare. Let's go on. Now let's talk about large language models and these amazing things that have happened in the last five years. So, this is basically GPT-4, 5, Llama, models like that. Here's a surprise. These graphs are really interpretable to humans. It turns out what's interpretable to humans is also surprisingly interpretable to large language models. So, let me just show you an example of that. So, here's a graph that we saw a version of before.

[37:03] This is risk as a function of age. And the model is able to understand this. What we do is we feed this graph as a table of information into the model. We explain this is a risk on a log-odds scale as a function of the patient's age. We tell it about the condition. And then we can ask the model to summarize it. Does risk increase monotonically with age? And it can say, yes, the graph looks monotonic. We can say, what's the average risk of a patient who falls somewhere on the graph? And it's able to read the graph and tell us that. We can actually even ask an LLM, surprisingly, to simplify it and say, well, give us a best linear trend line to represent this graph.

[37:43] And it does a pretty good job of that. So, this is true, these kinds of questions, that the LLM really does understand the graph and is not hallucinating. So, that's wonderful. But we can do more than this. So, not only does this mean that some people could interact with the model by asking questions in text, which is nice. It's another kind of user interface. So, you don't have to look at the graph. You could actually do a textual interface. But now we can do other things, like we can ask the LLM, do you see any surprises in the graphs? And sure enough, we can do that. So, this is risk as a function of the patient's blood creatinine. That's another blood test almost all of us have had.

[38:28] And I want you to look at the graph first. And notice the graph kind of peaks around 3.5, comes down. Then it peaks at 5 and then comes down. This is what the LLM says. The graph for creatinine shows a surprising decrease in risk and predicted log odds as creatinine levels increase above 5. And then again, another surprise here at 10. And it's counterintuitive because higher creatinine levels usually indicate worse health outcomes. The LLM is absolutely right. It turns out the healthy region for creatinine is 0 to 2, 0 to 3. And the reason why the graph peaks at 3.5 is because they start to give a medication at 3.5.

[39:15] And the reason why it then peaks at 5.5 is because they start to give dialysis at 5, just like we saw with blood urea nitrogen. And those treatments are very effective, which is why the graph peaks and then risk goes down and then it goes back up. Because that treatment, the medication wasn't strong enough and dialysis even more powerful. And here's the cool thing is that now we can actually have an LLM look at the model even before any humans have looked at the model. And if we give an LLM that knows healthcare enough detail about the model and the graphs, it can help us find surprises in these graphs. And it can focus our attention. And this is very important because some data sets have hundreds of features.

[39:59] And to get clinicians to look at hundreds of features, clinicians are expensive. Their time is very valuable. And it takes a long time to look at hundreds of graphs. So if the LLM can try to pre-process the graphs and find the most important surprises, then we can have the clinicians focus on those, which is great. And here the LLM has even given the clinicians a hint of what it thinks is surprising. So we did a study two years ago, which we call, jokingly, four doctors and a large language model walk into a bar. Sorry if that's not a funny joke in the EU, but it's a funny joke in the United States.

[40:42] So there's lots of jokes, which are: a bishop and a rabbi walk into a bar, things like that. So this is four doctors and a large language model walk into a bar. And what we did was we got four doctors who didn't know that the fifth doctor was actually a large language model. We had them look at 50 of these graphs for a data set, and we had them find surprises in the graphs. And we also had the large language model find surprises, just like what we did on that previous slide. And then we compared the results of the doctors to the large language model. And this is using GPT-4, which is not as good as the newer ones. And we found that the GPT finds more surprises. Some of them are spurious.

[41:25] It finds more surprises than the doctors, but there's a lot of overlap. Almost all surprises found by the doctors were found by the LLM model. So that's great. We found there was a lot of agreement. That is, doctors agreed with themselves. When we asked them a month later to review things, they agreed with themselves a lot. And they agreed with other clinicians a lot. So this is the inter-doctor agreement. And they tended to agree a lot with GPT, but you notice the agreement is less. One of the reasons the agreement is less is because the GPT is overproducing surprises.

[42:01] So they don't agree with some of the excess surprises that the GPT has found. And that's something that we can now work on, especially using better large language models. And then when we asked them to look at the explanations, the doctors, again, tended to agree with their own explanations. They tended to agree with the explanations from the other doctors. And again, because of the spurious overproduction, they didn't agree with the explanations for all of the things that the large language model found. But they tended to agree a lot with the explanations that came for the surprises that they also agreed were surprising.

[42:36] And what's interesting is that those results, which are two years old, and if you're into large language models, you know that two years old is like ancient history. It turns out those results suggested that GPT-4 was already good enough to start acting as a useful safety net to focus our attention. And now with newer models, we're getting even better results. It's hard, though, to redo the study, and that's because we have to get multiple clinicians to do a lot of work. It takes clinicians actually a day or two to do the work we're asking. So this is a hard study for us to replicate. So we only do this kind of work occasionally.

[43:12] The most important thing, though, that I want you to take home from all the graphs we've looked at and from the surprises that we see in these graphs, this is perhaps the single most important take-home message of what we're learning from our interpretable models, is that accuracy is not the same as correctness. And almost always we want our model to be correct. And why do I say accuracy is not equal to correctness? Well, usually our test set looks just like our training set. Like it's true in our test set that the patients over 100 are probably getting extra aggressive care, too. And the patients in our test set are also getting dialysis when their blood urea nitrogen hits 100 or when their creatinine hits 5.

[43:59] So that means the risk profiles we see for the test data are very similar to the risk profiles that the model has learned from the training data. And the model is rewarded with high accuracy by predicting on the test set things that it learned from the training set because those data sets look so similar. But correctness is different. Correctness, depending on how we're going to use the model, we might want the model to predict that no risk keeps going up as your creatinine gets higher. It keeps going up as your age goes higher. It keeps going up as your blood urea nitrogen goes higher. And it is not the case that risk actually comes down.

[44:40] In fact, if you think about it, a model that thinks that high creatinine is good for you might think that the treatment you should give the patient is to drive their creatinine high or to drive their blood pressure just over 200 to 205 because that would be a good thing. And we don't want a model that might ultimately be affecting the interventions we do in healthcare to think that patients are low risk for the wrong reason. We want correctness, and that's critical. So I'm going to stop there. That took a while, but hopefully that's an interesting introduction to how these interpretable models can help us guide healthcare towards better treatment and outcomes in the future.

[45:27] Arseniy Arsentyev: I appreciate your effort, and I think the audience will also like it. I have only two questions. First of them, whether the doctors in the latest studies that we discussed, whether they knew that the fifth doctor was an LLM?

[45:46] Dr. Rich Caruana: It's a really good question. And no, we did not tell them. We told them there were five doctors, and we were careful. We did – you have to be careful when doing these studies. We were careful to say, but there is a surprise in how we are doing this study, and we don't want to explain the surprise to you in advance because the surprise might influence the way you make decisions. So we'll only reveal the surprise to you after the study is over. So we were honest with them that they were part of a study, but we didn't give them any hint of how the study was being run until afterwards.

[46:29] Arseniy Arsentyev: Yeah. That's great. That's exactly the way I thought it should be done in my brain, and I'm glad that you did it in the same way. And the second question is, how do you think – what is the field that will first adopt this kind of interpretability and interpretable ML? What changes are the first to come in the medicine related to the technology you're discovering?

[46:55] Dr. Rich Caruana: That's a really interesting question. So now these EBMs that we started talking about are specifically designed for tables of numbers and patient characteristics like a gender, male, female, or insurance type in the United States or things like that. So tabular data. So this particular kind of interpretability that we're talking about will have the most impact in domains where there are tables of numbers. So, for example, it will not have impact directly in reading radiology, in reading x-rays, CAT scans, and things like that. Although we are working with a group who likes to derive features from their x-rays, CT scans, and MRIs. And then they do, because those features are now numerical, they do actually feed those features into a tabular explainable model here. To try to learn thresholds for treatments that they want to give.

[47:59] But these models will never understand a picture. They're not designed to work on arrays of pixels. Similarly, although they can work a little bit on text, they're not designed to work on text. So they're not going to work on doctor's notes. They're not going to search the web and find the appropriate study or result for a patient who has done a query in their search engine because they have certain symptoms and they're wondering, is this something I should worry about? This kind of model is not really designed for that.

[48:32] However, there's lots of tabular data in medicine. The first thing medicine does is measure it. The first thing that happens when you walk into a doctor's office is they measure your blood pressure, they measure your weight, they measure your height, they measure your pulse. They start doing things like that. And then as a result of your visit, the next thing they often do is they run certain blood tests. And then they might send you off to get certain kinds of imaging, which then often yields numbers like a percent fraction of blood that's flowing through your heart. If you might have a blockage in some arteries. So they do start creating lots of numbers and your electronic health record, the EHR, is full of numbers.

[49:15] And this model is really good at working with the numbers in a way that LLMs are not that good at working with the numbers yet. So this kind of interpretable model can help us take the numbers, compile them into these interpretable graphs, which the LLM can understand. And now the LLM can both understand the graphs that correspond to you as a patient, your numbers, and the doctor's notes, which are text, that correspond to you. The LLM can handle both of those just like a human clinician would. They can understand both the numbers and the text. And now the LLM, which can be attached at the end of the process, is capable of understanding both of these things and improving healthcare by doing the combination of both text and numbers.

[50:06] And that's the main thing we're trying to focus on with this new company, Intelligible Inc., is this combination of learning from numbers, something that's as interpretable as possible, and then combining that with an LLM, not only to help us understand the graphs better, the interpretable model better, but then also to be able to take other features perhaps into account, like text, that the EBM isn't able to deal with itself. So we're trying to combine these two modalities into a more powerful, a total capability.

[50:40] Arseniy Arsentyev: Next question that sparks in my head is: where is the place of this technology in the current healthcare system? For example, if you've got this magical hand to influence or maybe coerce any politician and to put this into healthcare practice, where do you see it fit? In a doctor's job? So doctor would interact with this kind of symbiosis of LLMs and the interpretable ML, or only scientists and healthcare economists who are making decisions about the healthcare system? I just want to hear your vision. Where is the place for this technology?

[51:23] Dr. Rich Caruana: Well, it's a wonderful, wonderful question. Okay. So doctors are very busy people, and they don't have the time to look at 50 graphs while you, the patient, are sitting in their office. So we're not going to have the doctors in real time looking at these graphs. We might, however, have a prediction from the model available to the doctor, like the model predicts your risk of readmission to the hospital or your risk of cardiac failure or your risk of mortality, in the worst case, is high, low, or medium. We might have that available to the doctor. And the doctors would tend to treat that as a second opinion.

[52:08] So, in fact, there's an interesting question of whether the doctor should be allowed to see that before they come up with their own opinion or only afterwards, because maybe we don't want to bias the doctor. Maybe it's a second opinion that happens after they've entered their own information into the record. The cool thing is the model can also, maybe it has 50 or 500 graphs in it, the model can also zero in on the three, four, five graphs that it thinks are most relevant to each patient. And it could if the doctor wanted. Like the doctor sees, oh, you're predicting this patient's high risk. Why? And the doctor could click on that. And then it would bring up just a few graphs that look highest risk.

[52:51] Now, with the LLMs, it actually could, if we're successful, it actually could just explain, oh, what we are seeing on the creatinine graph is the following. And that suggests that this patient is actually high risk despite the treatment. So the doctor could get a very terse explanation, but only if they ask for it. Okay, so you mentioned scientists and economists. Our first use of interpretability is definitely lots of doctors want to collect data and train models on the data and see what's in the data. So our first use of this interpretable model, after we train it on the data, is for research clinicians, clinicians who are willing to spend the time before they're looking at any one patient to look at the model, see what it learned from their data.

[53:49] Maybe they're doing a clinical trial, a study of some sort. This is a great kind of model to use if you're doing a study. So if you're a science research doctor, it's also great if you're an economist in healthcare, because we can often show that a threshold is actually slightly suboptimal. And we could improve healthcare. And by improving healthcare, if you really have the numbers for it, that's often economically viable to do that. That is, if you really can dramatically improve healthcare by making a change in the way you deliver healthcare, often you can cut the costs dramatically of healthcare by actually intervening more appropriately.

[54:34] So economists are interested in this. In fact, in the U.S., we've had people at the federal level ask us, should we sort of require that we use these interpretable models wherever possible so that we're making proper data-based decisions about how to adjust healthcare? And we think it's an important part of the puzzle, but we wouldn't say that everything should be based on this alone. So our first use of the interpretable model is to sort of just understand what's in the data, understand how healthcare is currently being delivered, look for surprises. The LLM helps us with those surprises.

[55:16] And then we can decide, is the model safe to deploy? Or, as is often the case, the doctors, the research doctors find a few changes they need to make to the graphs before they deploy them to make them safer. And then we deploy the modified, the edited model, and now we believe it's making safer predictions. And then those predictions can be used clinically, either in a screening way before you ever get to a doctor's office, or as a second opinion after you've been at a doctor's office to make sure that there's a safety net.

[55:52] One thing that's important is these models are comparable in accuracy to black box models. They're not more accurate. They're not less accurate. They're similar in accuracy. And what that means is the black box models have learned very similar things inside of them as well. Like the black box models have learned that being over 100 lowers risk of pneumonia, that being over 100 on blood urea nitrogen reduces risk. Creatinine over 5 is lower risk. Those black box models have learned very similar things. You just can't see it in those models. You don't know they've learned it. And they're getting rewarded with high accuracy for predicting these things for reasons I already explained.

[56:34] With these sorts of models, you can actually see these things, and you can make a decision about whether you need to edit the model. And so deploying a black box model is kind of scary because you now know that the models, maybe 10% of what they've learned is questionable and probably should be corrected. But you're going to deploy it anyway just because the accuracy looks high. And now with the glass box models, you can see what the models have learned. And because you can edit the models, the doctors can repair them and be more comfortable with how they're deployed.

[57:11] We do have – there are probably now, I don't know, 100-plus papers written in healthcare alone that use these models. The models are also used in other fields of science and insurance industry and economics, so credit lending and things like that. They're actually very, very popular in the credit lending and insurance companies because those businesses have been prevented from using black box models for ethical, legal reasons. And now we have accurate models that are glass box, so they're suddenly allowed to get high accuracy by using these interpretable models. But in healthcare, we're starting to see an uptick in these things as well.

[57:56] And we found that once clinicians start to use an interpretable model, they never want to go back. And that's because they now understand the black box model is making mistakes they can't see. And doctors don't like to make mistakes. More than anything else, doctors do not like to make mistakes. And if they think a model that they've trained is making mistakes, they want to fix that model. And there's no way to fix a model's mistakes if you don't see the mistakes first, if you don't know they're there.

[58:24] Arseniy Arsentyev: Then the model is useless if there is no way to fix the mistakes it makes. And, Rich, thank you for this wonderful interview. I think we can proceed to the second part of our discussion. More podcast-like.

[58:38] Dr. Rich Caruana: So I stop sharing the screen?

[58:39] Arseniy Arsentyev: Yeah, I would love if you do that. And let's go again with LLMs and interpretable models. I'll call them – what is the most popular one, EBM or GAM?

[58:58] Dr. Rich Caruana: Yeah, GAM is the general class of models. And it turns out every statistician would know what a GAM is. But it turns out EBMs are the most popular GAM now. So EBMs might be the thing to use now. It's the thing that most people in machine learning would know about. And most of the papers that are being written now are now using EBMs, explainable boosting machines. But there are other ways to do that. I just want to point out you can train one of these models using neural nets. And there we call them NAMs, neural additive models. So we have figured out how to train these things using lots of different machine learning tricks. Neural nets, boosted trees, things like that.

[59:43] Arseniy Arsentyev: This podcast is brought to you by my will and desire. I truly put the efforts to do the best quality of the content that I can. The best thing you can do to support my efforts is to share this video with your friend, like it, subscribe to my channel and most importantly support me on my Patreon page. Thank you in advance and let's drive the science, the medicine and the future of healthcare step by step. So let me use EBM and if it will be too general, correct me. So right now I want to take a stage and outline a little bit my understanding of how you want to connect this EBMs with LLMs. And feel free to stop me at any time and fix the possible errors I can make.

[1:00:37] From everything I read from Intelligible.ai scientific base and also listening to certain of your talks, you're using EBMs, you're isolating certain features and making this graph probabilistic predictions on them. And then we have this huge problem of how to connect each feature and use this graphical data. Because when I look at the graph, I can certainly understand what does it mean and with some thought and with some effort, I can understand the surprise that we talked about. For example, drop due to the treatment or some inconsistency in the data. However, LLMs cannot. And so you're building this kind of connection phase to kind of explain these graphs for the LLM. And it reminds me of divide and conquer in computer science, if you heard it. It's kind of similar.

[1:01:46] We're just dividing our ML model into different features, represent them, kind of embedding. We can draw analogy here. We're embed this into a different format and feed it into LLM. Am I correct?

[1:02:05] Dr. Rich Caruana: That's a great analogy, by the way, which I haven't used and I'm going to use going forward, which is that the EBM graphs serve as an embedding for the tabular data. And strangely enough, that embedding is now accessible to an LLM in a way that the original raw data wasn't accessible. So the embedding actually helps the LLM, which is kind of funny because the LLM, of course, has its own embeddings. But no, that's exactly right. So imagine two different situations. The simplest one is all of your data to begin with is tables.

[1:02:42] And that's how most machine learning was done for 20, 30 years. It's certainly how statistics has been done for 100 years. So if all of your data is in tables, it's tabular data, then the EBM helps us ingest that table. And that table could have hundreds of features, hundreds of columns, and it could be hundreds of thousands, millions, tens of millions of records. So it could be very, very large. And there's no way for an LLM to understand that table of data. So an LLM is not capable.

[1:03:23] You can't just feed it the huge table of data and say, oh, do you see any surprises? They're not capable of doing that. Certainly not yet. Although progress is being made on how LLMs might directly work with tables of data. But we're not very far along that yet. So what we do is we use the EBM almost as a preprocessor, which ingests a huge table of data, generates these graphs, and then, to be honest, if there's only 10 graphs and you've got a person who understands graphs, just let them look at the 10 graphs because it's easy.

[1:04:06] But if you've got 50, 100, 1,000 features, hey, looking at 100 graphs takes a long time. I know. I've done it many, many times. It's very time-consuming. And you can make mistakes. This is where an LLM really can help us. Now, maybe the LLM is going to be the interface to the end user anyway. If you're going to have people on the internet using their browser to ask questions, and you want the answer to their question to be informed by tables of data that were available to researchers, how do we do that? Well, one way to do that is we train EBMs, interpretable glass box models, on those tables of data. And then we make sure that those interpretable glass box models are now accessible in a way to the LLM that it understands.

[1:05:05] And the LLM can now do a lot of things. It can help us just find surprises. It can preprocess that interpretable model. Maybe it's just going to help the end users go back and look at the interpretable model. Maybe your goal isn't to create a text interface. Your goal is to just have the LLM help you focus in on the most important things. Maybe you see a surprise and you're wondering if the LLM can explain the surprise. This happens more often than you might think because often I'm a data scientist looking at graphs from a glass box model trained in a domain that I do not have that much expertise in. So I see things that look, well, that's interesting. Is that bump normal or not?

[1:05:51] And sometimes the LLM knows more about that domain than I do. In fact, one nice description of an LLM from a colleague is that for the things I'm an expert in, I know more than the LLM. But for the 99.99% of things in the world I'm not an expert in, the LLM actually seems to know more than I do. So this can be very useful if the LLM can come to the data scientist's aid and possibly help explain and detect anomalies in these graphs when the data scientist is working in a domain that they're not an expert in. And then it can serve, if you want, also as an interface. That is, there are some end users who would prefer not to look at graphs.

[1:06:36] Or if they're looking at the graph, they'd like to have a text summary of the graph next to the graph. That is, they want the multimodal description of the phenomena, of what's learned from the data, sitting side by side. And they want to be able to ask questions about the graphs. They don't want to have to use a cursor and try to do things. We can even imagine, for example, Google has just released a photo editing software or is about to release photo editing software where you can say things to edit your photos, like increase the contrast, remove the truck that's in the back of the image, things like that. Get rid of the telephone poles, whatever.

[1:07:22] You could also imagine in the future that we would be able to do some of this editing of the graph, like we want risk to go up monotonically increasing after age 90 in that pneumonia graph. You might imagine that a person, instead of having to use a manual editor that we've already created, could actually talk to it and just say, we would like risk to go up monotonically above age 90, use a straight line, make the slope be 0.5. Oh, that looks too steep. Make the slope be 0.4. Still too steep, 0.3. Oh, that looks good.

[1:07:58] Stop. So you could imagine that these LLMs could help us not only pre-process the graphs, provide a parallel explanation of the graphs, could act as a queryable interface to the graphs, could give doctors summaries of why it thinks a patient is high risk without the doctor ever having to look at a graph, and might even be part of the interface that helps a doctor edit the graphs. Now, that's the simple case where all of your data originated as a table of numbers. There's now another universe where some of your data is tables of numbers and some of your data is text, let's say, or images.

[1:08:47] Now that LLMs are getting good at understanding images. Now you can imagine the EBM helps process the tabular data, does what you said. I really like your way of framing it. It creates an embedding for the tabular data, an embedding that is interpretable to the LLM. And then the LLM, because LLMs are amazing, can understand things like the text that's in the doctor's notes, can understand perhaps the radiologic image that came from an x-ray or from a CT scan, can maybe now reason about both the embedding created for the tabular patient record, plus the information that's contained in the other modalities of the patient record that's not tabular. And now the LLM might be able to sort of reason about both of these together.

[1:09:42] All of the original things would still apply that apply to only the tabular setting, but now they would apply to both tabular plus additional modalities. So that's the kind of vision of the future. And if LLMs could ingest a table of a billion numbers and make sense of it right now and explain to you what the surprises were, well, maybe we wouldn't need EBMs anymore, but it turns out that's a hard thing for LLMs to do. So we're working on this. And I love your suggestion that it's like creating an embedding for the tabular data.

[1:10:27] Arseniy Arsentyev: Yeah, thanks. And as I understand, this is exactly the work you do at Intelligible. Should we call it just Intelligible or Intelligible.ai?

[1:10:40] Dr. Rich Caruana: Intelligible.ai is sort of our official web presence. Intelligible Inc. is the name of the company, Inc. meaning incorporated.

[1:10:50] Arseniy Arsentyev: Incorporated, I see. Can you tell more?

[1:10:54] Dr. Rich Caruana: It's a new company. And some of the people involved, like that first slide that had the list of all the people who had worked on it, and that first line was a bunch of interns, graduate students who worked. Well, it turns out the company was started by one of those students. That student graduated, then went on to do a postdoc at MIT for a few years, and is now a professor at Madison in Wisconsin. So that student created the company, and they created the company to sort of take things that we had worked on and to carry them into the future, to push them into what we think is possible and will be important. And our first focus is going to be in healthcare.

[1:11:41] So Intelligible Incorporated doesn't specifically say that it has to be healthcare. But we think healthcare is perhaps the first and largest domain that needs this capability. So that's the one we're focusing on. Also, it's one we have a lot of experience with. And to be honest, we're just strong believers in how important modern machine learning, modern statistics, and now modern AI are to this domain, to improving the quality of healthcare and even improving the economics of healthcare. So we're just strong believers, I guess, in the importance of this technology in healthcare. So it's always more fun to do all of your development in service of something that you're a very strong believer in, and that you think will save lives ultimately.

[1:12:37] Arseniy Arsentyev: And also, starting a company helps to put it into concrete objects, and it's easier for governments, for certain agencies to interact with it. And it is kind of a club for pushing the interests and for pushing this technology. And I do understand why you joined, because it's relative to a field, and I can see in your eyes that you're hugely inspired by the technologies you're doing it. So your role as a chief science officer is to oversee the scientific side of it, and the scientific side of providing this bridge that we were talking about between tabular data and insights that we can get from it using the LLMs and stuff.

[1:13:22] Dr. Rich Caruana: That's exactly right. And there are definitely things you can do as a company, even a small new company, that are harder to do as researchers, and vice versa. So that's part of the reason for forming the company. We've been publishing papers in this area. I've personally been working in machine learning for healthcare for a long, long time, since my PhD work at Carnegie Mellon in the 90s. And by the way, I want to give credit there to my advisor. One of my advisors was Tom Mitchell, who some younger people may not recognize that name, but Tom Mitchell was one of the most famous people in machine learning back in the 90s. Everyone would know back then who Tom was.

[1:14:08] He was one of my advisors, and he's the one who got me involved in doing machine learning for healthcare. So a big call out and thank you to Tom and to Carnegie Mellon. So I've been doing machine learning for healthcare for a long time, partly because I like working on real problems. And if you're going to work on real problems, you might as well work on important real problems. So as I develop new machine learning techniques, I like to be also working on very important real problems. And healthcare just seemed like it's kind of Turing-complete in the sense that every problem you're ever going to have in AI and machine learning is going to show up in healthcare. And there it's going to be critical to get it right.

[1:14:53] So I really like a domain where you can't finesse the things you don't know how to do, where you actually really have to get it right. And if you don't get enough things right, you shouldn't proceed. Like you don't deploy the model unless you really think it's right. And that's how I got involved in glass box machine learning. I was not originally an interpretable machine learning researcher at all. And then I sort of, as we started to see the mistakes that black box models could make on healthcare data, we started to realize we needed something high accuracy, but interpretable.

[1:15:27] And that's where we learned about generalized additive models that statisticians had invented, but I didn't like the way they fit them. So we used all these new modern machine learning techniques to make them better. And that's how we got involved in creating these EBMs. And the good news is they're working better than we ever expected. I really hadn't anticipated that they could completely match the accuracy of black box models. We were surprised by that. We're pleasantly pleased with that. It means there's no reason to use a black box model anymore on tabular data because you can get the same accuracy from something interpretable. Why would you intentionally use something black box, when you can get the same accuracy with something glass box?

[1:16:16] So, and then the company that Ben started, because Ben believed in this vision himself, and his own thesis work also was along these lines of personalized machine learning for healthcare. So when Ben was interested in starting the company, originally I just signed on as an advisor. I was still at Microsoft Research, so it would have been inappropriate to do anything more than just advise the company. And then eventually I retired from Microsoft and Ben asked me if I wanted to join, which, of course, I decided made a lot of sense. So I'm the chief science officer, as you said, the CSO.

[1:17:01] And my job as one of the inventors of the glass box techniques, my job is to help make sure that all of that part is dealt with appropriately, and as we need to create new tricks, that it's easy for us to do it. And then I've also worked in LLMs because they're just too exciting to ignore for the last, three, four, five years. So it's kind of really fun that we're marrying these things together to get the best of both worlds. They're going to be better than A and B alone. C, the combination of the two, is going to be stronger than either of those individually.

[1:17:40] Maybe on some problems it'll be 75% A and 25% B. On other problems it might be 75% B and 25% A. But the combination of the two is going to be more than 100%. It's going to be, 150% or something like that.

[1:17:55] Arseniy Arsentyev: It sounds like a really cute challenge to work on because it's not this overly covered by media. It's kind of techie and really into science. Not so many people work on it. And I just wish you good luck. And I'm just curious, do you still code a lot? Like how much code do you write during your job? Or is it for you right now purely like scientific endeavor?

[1:18:22] Dr. Rich Caruana: It's a really good question. And by the way, you mentioned that not too many companies are focusing on this. As far as we know, there are many, many companies who are focusing, of course, on the LLMs, right?

[1:18:35] Arseniy Arsentyev: It's a gold rush.

[1:18:36] Dr. Rich Caruana: Of course. And many of those companies are interested in LLMs for healthcare because that's a huge and important domain. But we are pretty sure we're the only company who's focusing on glass box, interpretable, intelligible machine learning and AI for healthcare. So we're probably the only ones who are doing that. And then we think that LLMs and tabular data are just two critical facets that have to work together or else you're missing sort of it. In terms of coding, let's see. So some of the early code for EBMs is code that I wrote. There were parts of the problem that I wanted to solve the problems in and write the code for.

[1:19:29] And so in particular, when we were doing statistical analysis, error bars, when we were going to create user interfaces of the graphs, when you've got hundreds of features, how do we sort the graphs, all those sorts of things. Those were things that I really wanted to do myself because I thought I had fun insights into how best to do those things. And there's no better way to take insights and put them into practice than to write the code yourself. At some point, the open source package that we created while at Microsoft, that open source package started to have multiple developers working in multiple languages. And at some point, I started doing less coding on specific things and more using the package and then inventing a new technique.

[1:20:20] As we run into a new problem, we would invent a new technique. And then I would maybe help do the demo of that technique to prove that it worked. But then other people might take it and write better, more efficient code than I would have written. And now that I'm CSO, I've only been CSO for a short period of time. And already I'm starting to think of some things where, oh, maybe I'll go write the code for that. We don't need it like really fast. We don't need it next week, but we know we're going to need it in the next three to six months. Maybe that's a fun thing for me to go write the code and go do.

[1:20:58] And now that I don't have all the responsibilities that I had when I was at Microsoft Research, now that I'm focusing more on this very specific problem, maybe I'll have the time actually to do more coding again, which would be great. Coding, in some ways you don't really understand a problem until you can write the code for it. Even if the code you write is just the first pass and it's not very efficient. Writing the code really makes the virtual real in a way. And as you said, it's fun.

[1:21:34] Arseniy Arsentyev: And the last question about Intelligible. It's more a question for Ben or some other person working more with the business side. But I'm curious, who are your customers? Are you planning to work with the government, with private organizations, with private clinicians maybe? What is the audience to which you're planning to provide or even sell your services?

[1:22:01] Dr. Rich Caruana: It's a really fun question. And it's one that we don't even have fully resolved yet in these early stages. Mainly we're approaching hospital systems. So we're approaching hospitals. And hospitals have lots of problems that they're interested in getting help with. And some of these are clinical, have to do with patient care. Some of these are what I might call subclinical. They have to do with how the clinicians and the hospital deal with insurance companies, which sadly is very important in the United States in a way that it isn't necessarily in the EU. And some of it has to do with even things like scheduling the emergency room. It turns out there's a whole part of hospital operations called utilization in the United States.

[1:22:56] And it turns out having efficient utilization is critical to not only better patient care, but it's also critical to more efficient running of the hospital. So it turns out being able to properly estimate how long each individual patient might be in an operating room based on not only the procedure that you're going to do to the patient, but the characteristics of the patient. There's a difference between doing a procedure on a patient who's male or female, a patient who's young, middle-aged, or old, a patient who is very thin, which can make things harder, who is middleweight or very heavy, which can also make things harder. So there's a difference in the time it takes to do those procedures. And an operating room is a very important scheduled thing. It's like scheduling gates at an airport.

[1:23:53] That's also very important scheduling at an airport is the runways and the gates. So there are many things at a hospital, including 30-day readmission is very important. And one of the reasons I like 30-day readmission is what's good for reducing 30-day readmission is not only good for the hospital. It's usually also good for billing, but it's also usually good for the patient. I like it when all these things line up. So that's what's good for the patient is also good for the hospital.

[1:24:24] So we're interested initially in all of those kinds of customers. However, we're talking with someone right now who is applying for a grant with the Veterans Administration, which is a government agency in the United States that provides a tremendous amount of health care for veterans, so the VA, and there's a chance that we'll be involved with them in pushing that work forward. And in that case, we're now talking to a government agency as part of the process. And then in other cases, there are clinical trials that are so large that having someone like us involved in the analysis of the clinical trial data would make sense also.

[1:25:18] And that's because there's a requirement when you're doing a clinical trial, maybe you're comparing two different treatments. As early as possible, if the statistics sort of strongly suggest treatment A is better than B or vice versa, as early as possible, if you know that, you're actually supposed to stop doing the clinical trial and start giving the preferred treatment to the patients. You don't keep experimenting on patients when you actually think you know the answer. So it could be that on larger clinical trials, it would also make sense to kind of hire us as collaborators to help in the analysis of the data. And that's because data keeps getting more and more complex.

[1:26:02] And the analysis that you need to do on the data keeps getting more and more difficult. And if you have just one in-house statistician, as the data gets very complicated, they may not have the resources or expertise to be able to do everything that you want to do. And these modern techniques are more powerful. In fact, one way to think of EBMs is they're almost like an automated statistician, in that they ingest the data and, as you said, come up with a statistical model. And then the LLM can help interpret that statistical model. So it becomes a very powerful statistical tool for just trying to understand what's in your data. Even if you never plan to field a predictive model, it's just a powerful way of understanding your data.

[1:26:52] So anyway, those are the kinds of, quote, customers that I think we're looking for first.

[1:26:59] Arseniy Arsentyev: And I just looked into more technical details. And here is the question. What about the compression of those graphs? Because as I understood, there are certain features that have many different slopes, many transition points. And LLMs have short context, not short, but pretty limited context window. And what tricks do you use to compress these statisticians or EBMs and put them into LLM? Especially I'm interested in your opinion on whether it reduces accuracy and the correctness of the statistical engine.

[1:27:43] Dr. Rich Caruana: It's a very interesting issue. And there are other researchers in glass box machine learning who believe the model should be as simple as possible as long as it doesn't hurt accuracy too much. So, for example, Dr. Cynthia Rudin, who's at Duke University, she's another very well-known researcher in glass box machine learning. So I'd like to call out her work. But she and I differ on this spectrum. What we have found is that if your model is reliable enough, the little tiny detail that we see in the graphs often tells us interesting things that are happening in the data. And also makes experts, clinicians looking at the graphs more engaged and even trust the model more.

[1:28:39] Like I've seen one graph on a data set where every round number divisible by five had a little tiny tick, a very tiny, tiny jump in it. And that's because humans, and this was age, so every age divisible by five had a bump. And then there were the important ages that everybody thinks about, like 65. 55 comes out to be important. If you're talking about pregnancy in the United States, age 35 is considered a very important number. So you had bigger jumps at these major round numbers. But you had little tiny jumps at all the other round numbers. And that's because clinicians are always reasoning like, well, you're over 50 now. We've got to take this seriously. Whereas if you were 49, the doctor wouldn't have said, oh, you're over 50 now.

[1:29:27] The doctor would have said, well, you're in your 40s. Let's just pay attention to this and see in the future if we should do something about it. And the model had learned this little tiny detail. And the clinicians found that really interesting that the model was that sensitive. And then in other cases, the model, you learn, like the jump that happened right at 180 in systolic blood pressure, that really bothered me for months. Until a doctor looking at it said, Rich, the definition of hypertensive urgency is 180. It's not 175. And that's why the model has learned that. Now, if we simplified and smoothed the models a lot, it turns out we start to lose that detail.

[1:30:15] And while the detail might not be critical in terms of accuracy, the detail really provides nice places for experts to look at the model, trust it, understand what it's doing, understand why it has done something. The detail really provides clues about what's happened. It's very important to realize this is a supervised learning model, EBMs. It is not a causal model. Causality is very, very difficult. So whenever we put a causal interpretation on some detail we see in a graph in the model, that's because we as humans think we have enough expertise to understand where the data came from and how the world works. We have enough expertise to come up with a causal explanation for why there's that bump.

[1:31:14] Sometimes, by the way, the LLM can even come up with that causal explanation. Sometimes we can even say, oh, but the reason why this is dropping above 5 is because dialysis is – 5 is often a treatment threshold for dialysis. Sometimes it can do that. Sometimes it can't. But humans definitely can often do that or they believe they can do it. They're not – humans are not 100% perfect. Sometimes we get our causal explanation wrong. And the little bit of detail we see gives the clinicians more opportunity to try to understand causally what has affected the data and how the real world works. And that's just very, very important when interpreting these models and trying to understand, do I trust the model? Do I trust it?

[1:32:01] Even before you trust the model, you want to trust the data. But like data – it's amazing how often the model tells us that the data is messed up. And we actually have to go back and redo something for the data because the model has summarized the data in a way that we now realize, oh, the data has a problem we need to fix. So – and I hate to lose detail. So I like to see all that kind of detail.

[1:32:29] Certain kinds of detail, like if error bars are large and we're seeing lots of little jumps and things like that. Well, if the error bars are large and there's lots of little jumps, the little jumps don't mean anything probably, right, because the error bars are large. And in those cases, we can imagine simplifying those parts of the graph. But I like especially the first pass to always be as detailed as possible. And it's amazing the stories we sometimes figure out for these details. It's so much fun to actually be able to fully see the statistics that are in your data for the first time.

[1:33:07] Arseniy Arsentyev: It is always fun to uncover these hidden insights that were always there, but nobody looked at them.

[1:33:14] Dr. Rich Caruana: It's like being Sherlock Holmes. And you've got this powerful new magnifying lens, so you can really look at the crime scene and figure out the detail. It's wonderful.

[1:33:26] Arseniy Arsentyev: In addition to you being a Microsoft researcher and right now being a chief science officer at Intelligible, you've done a purely astonishing amount of contribution into different scientific fields, into papers. You're the founder of Multitask Learning. And I even read out the stats. 55,000 citations. An h-index of 74. At this point, I want to ask you to tell me more about your scientific career and advice for aspiring scientists, because I have many friends who do truly see themselves working in academia and who want to provide as much impact as possible.

[1:34:15] Dr. Rich Caruana: Yeah, no, it's been a fun journey so far. And I've gone back and forth between research labs and academia multiple times. And we're seeing now a new mode. If you think about, oh, Yann LeCun, Geoff Hinton, Yoshua Bengio, there's an awful lot of people now who have their foot in both camps. That is, they are professors somewhere, but they also have like another full-time job essentially, at a company, right? So Yann is connected with Facebook. Geoff Hinton was actually at Microsoft Research when the very first signs of deep learning happened. And then Geoff moved to Google.

[1:35:12] So, and Yoshua has been connected with Microsoft Research. So, this is common. And what I've tended to do, instead of like trying to do both at the same time, I've tended to sort of do one for a while, then do the other for a while and go back and forth. So, in fact, I was at research labs even before I had the PhD. And at some point, I kind of realized, I have to go get the PhD if I want to keep doing this. So, then I went to Carnegie Mellon and got my PhD there.

[1:35:50] Big shout out to machine learning at Carnegie Mellon. Ben Lengerich, who's the head of Intelligible Incorporated, he's also a Carnegie Mellon graduate student. So, I've gone back and forth. And I guess most recently, I was a professor at Cornell in computer science. But I was also a professor for a couple of years at UCLA in their med school. And that's because I was working on machine learning for healthcare. So, I've gone back and forth. And I have to admit, like the grass is greener on both sides. It's kind of they each have advantages and disadvantages.

[1:36:30] It's easier to make things real sometimes at a company. But it is easier to work on crazy ideas that not necessarily anyone else believes in yet at a university. And it's easier to focus long term at a university as long as you can keep getting funding for what you're doing. Because ultimately, you can't really work on things in the United States, at least, if you can't get funding for it. And also, by the way, if you can't get students. So, when deep learning happened, a lot of my friends who were professors, all of a sudden, couldn't find good students if they weren't also willing to work on deep learning. So, just because the graduate students thought deep learning was very exciting and wanted to work on deep learning.

[1:37:17] So, we're seeing a similar sort of thing now with LLMs. So, basically, you do, as a professor, have to be able to get funding and students in order to do your research. But as long as you can do that, you tend to have a little more freedom to do crazy things that are further into the future as a researcher. And it's no accident, I guess, that the stuff that we're focusing on going forward at Intelligible Inc. is all stuff that happened at Microsoft Research. And you can think of Microsoft Research, for the last 15 years as being a huge academic research department that happens to be in a company.

[1:38:03] So, and those things were all developed initially at Microsoft Research. Microsoft gave me tremendous freedom to focus on the problems that I wanted to focus on and to work on them year after year. And to do things like get an intern or two every summer for years to work on different aspects of these problems. And now, in part, that's because we could show the relevance of this work to Microsoft as a company. If you think about it, Excel and Power BI are tables of numbers.

[1:38:38] So Microsoft understands tables of numbers and their importance, both internally and as products. But Microsoft gave us great freedom to do that kind of work. But then at some point, we really wanted to focus on a very specific connection between tabular learning and LLM learning. And I was also at the stage of my career where maybe I was thinking about retirement. And Microsoft gave me that opportunity to retire. So, Ben had already formed the company. And I knew a little bit about the company. And sort of one thing led to the other. And now I'm sort of continuing this mission, but in a very focused way. That as a researcher, I probably wouldn't be that focused. And in fact, there's an interesting thing as a researcher.

[1:39:36] Perhaps your number one job is working with students, with graduate students who need to do independent research and launch their careers and then go out into the world. And because of that, you wouldn't want your graduate students to be as narrowly focused on a piece of technology that you've been working on for 10 years. Because then they're working in your shadow and they're working on something very narrow. And that narrow thing might not seem exciting enough to the research community for them to then establish themselves as a researcher. And they may want to go on and have an academic career.

[1:40:13] So, when you're working with students, you're definitely committed to helping that student thrive, work on research problems that, of course, you're interested in, but which are somewhat independent of the things that you've already worked on. So, you want your graduate students to sort of develop their own interests and learn to do their own research. And as a company, we don't do that, right? As a company, we're focusing very specifically on trying to bring certain kinds of capabilities to practical implementation and then market. And that's a different kind of focus, which would be less appropriate. Now, we can have students helping us, but when the students are working with us, they're working on something that's very narrow within the confines of a company that is not a research organization, but a startup company.

[1:41:13] Whereas, at Microsoft Research, I would have the researchers, students come in and work on maybe different aspects of intelligibility. But always, it was some new thing. We didn't want to do the same thing, just better and better. We want to do something new. And that's how Ben got interested. And it turned out that what he was interested in doing also aligns well with what we're now interested in doing going forward.

[1:41:38] Arseniy Arsentyev: And I want to, again, say the second part of the question. I have many friends who want to become a scientist. What advice can you give them? And just imagine yourself being like 20, 25 years old, just in the beginning of your career. And what to focus on if you want to go into science and give an impact in the realm of human knowledge?

[1:42:03] Dr. Rich Caruana: It's a good question. And my answer won't be the full answer. It'll just be one of the paths. One of my other advisors, by the way, was Herb Simon, who many people won't recognize. But he was a Nobel laureate in economics and also one of the founders of the AI in the very early days. So in his book, Herb Simon said something like the halls of science have many doors. And I want to emphasize that there are many ways to succeed in science. So I'll give you a little bit of my experience. But don't think that this is the only way to do it.

[1:42:45] Personally, I found it important to work on a problem I believed in that I thought was important to mankind, to humanity. So that's part of the focus on health care. But I also believe machine learning and AI are important to mankind. So I could do that, too. I also thought it was very important that the things I developed worked in practice. That is, I was not what you might think of as a theoretical researcher or an abstract researcher. To me, the research had to really work in practice. And that was an extra challenge. That is, it needed to be new and exciting research that maybe pushed the edge of the envelope in terms of ideas.

[1:43:34] So, and for example, multitask learning, which was my thesis, helped do that. But it was really important to me that multitask learning worked in practice. And if it didn't work in practice, I wouldn't have continued it as a thesis because I would have been less excited by it. Now, a fair amount of academic research never works well in practice, but is still important and interesting. Let me give you one crazy example of that. For people who are familiar with support vector machines, which were very important in AI and machine learning 20, 25 years ago, they are no longer considered the method of choice. Although around 2000, there were many people in machine learning who thought they were the final perfect method.

[1:44:21] They were exciting because they worked pretty well in practice. They're usually not at the very top, but they worked pretty well in practice. And their math and theory is absolutely beautiful. Gorgeously beautiful. So, they like had both theory and good practice, so they were kind of amazing. But other machine learning methods eventually moved past them in terms of accuracy. So, they got to be less exciting perhaps. But, hey, I would be really proud of myself if I was the person who had invented SVMs because they're so beautiful. And the concepts that are in SVMs are very important to any person who knows machine learning deeply.

[1:45:09] If you know machine learning well, you should understand SVMs even if you never use them. And that's because the theory is valuable to the way you think. And the intuitions you have ultimately are what will guide a lot of your decision-making and research. But anyway, I guess for me the most important thing was it had to be a problem I thought was important. If I didn't think it was important, I wasn't that interested in it. And it had to be something which was both advancing ideas but which I thought would work in practice. There are a number of times I've had crazy ideas which I probably could have published and would have been successful perhaps as ideas.

[1:45:51] But which I knew for one reason or another were not going to succeed in practice because other things could do the thing they were trying to do better in practice. So, for me it was very important to do both. And then another thing is I guess I believed in sweating the details a lot. And the reason why is when you worry about the detail, that's when you find new opportunities. And sometimes worrying about a detail opens a door into a huge room that you never would have seen before.

[1:46:36] So, early on in multitask learning, which was done with neural nets, I was worried about what size neural net generalizes well. And there were reasons if you're into multitask learning like that issue was very important. And back in the early 90s, we all believed that small neural nets were the most accurate. Everyone, including Geoff Hinton, who is, if anyone, is the father of deep learning, it's Geoff Hinton. But even Geoff believed in the early days that small neural nets were important for accurate generalization. And in the process of being worried about this, I did some experiments before proposing this as a thesis topic. And those experiments suggested that actually if you allowed the neural net to get bigger, it often worked better.

[1:47:20] And that was a very important finding at the time, which not only enabled my own work, but also eventually, the community came to understand that a large number of parameters, excess parameters, was not necessarily a bad thing in a neural net model. Now, Geoff gets the main credit for that when they came up with deep learning in around 2008. But, yeah, so sometimes worrying about a detail, and that was a detail, man, it was just like, oh, I'll do some experiments for a couple weeks and check this detail. And for me, that opened an entire door that I just never imagined.

[1:48:03] So worrying about the detail can be valuable. It takes time. It not only makes you more of an expert, but it also creates opportunities that you wouldn't imagine. If you knew all the detail, well, you wouldn't need to do it. And the odds are, even if you think you know the detail, you don't know how to do it. I guess one summary of that kind of work is there's nothing more dangerous in life and in science than a false assumption. Because a false assumption means there are many things that you believe to be true, and therefore there are avenues you will not explore because you believe this, because of this assumption, that they don't make sense.

[1:48:47] And the false assumption we all believed in the early 90s was that the smaller the neural net, the better it generalized. And that's obviously not true, right? As our LLMs hit a trillion parameters, we obviously don't believe small is a critical part of the puzzle anymore. So it's very important to distinguish between things that you're absolutely confident are true and things which everybody just thinks are true. But, boy, it'd be good to run a quick experiment to verify that they're true.

[1:49:22] Anyway, that's been my path to success. And I've liked bouncing back and forth between real problems and abstract new ideas and real problems. And I find they feed each other. The more I work on a real problem, I discover new research topics that I can then abstract and do more abstract work on. And then that abstract work creates new capabilities so I can now tackle a new real problem. And then that new real problem, like no real problem is solved 100%. That always creates new opportunities to do research. So that's been my kind of focus. And I think there are others, for example, Yann LeCun and Geoff Hinton, who would agree that this sort of interplay between working on real things and then working on abstract ideas can be very valuable in the long run.

[1:50:12] Now, if you're a mathematician, your way of approaching things could be completely different. And that's also completely fine.

[1:50:22] Arseniy Arsentyev: I love your way of describing it. And I know that many people, many young people will find your advice extremely valuable. And it reminded me of this diagram I saw somewhere. When you have things that you love, things that you can, and things that bring money. And you need to strike right in the middle. Your speech reminded me of that. And if you allow, let's proceed to the main topic of this podcast, Tomorrow's Medicine. I want to just clear out the previous discussion. No interpretable machine learning. No ML. No intelligible. Just pure healthcare system as it is right now. How do you see it changing, let's say, in the following 10 years? And what do you wish to see?

[1:51:08] What is the best hospital experience in a kind of utopian world or maybe not in utopia, in the real world that we're heading to? I want to just hear your thoughts on it.

[1:51:21] Dr. Rich Caruana: This is fun. And of course, those of us who work in AI for healthcare, as researchers or in companies, of course, are always trying to imagine what could healthcare be 5, 10 years from now? And how could that be better than what it is now? Let's see. Healthcare, it's important to understand it's a very conservative field. It's very slow to adopt changes. And this is both a blessing and a curse. And it's a blessing because many things that look in the short term like they're better are not actually better in practice.

[1:52:03] And there are many times when healthcare has made mistakes, has adopted treatments, and those treatments proved to be, after 5, 10, 15 years, proved to be not the right treatment. Most recently, a lot of people believed 10, 15 years ago that if you were a cardiac patient admitted to a hospital, you almost certainly should be on low-dose aspirin. And to the point where there were automatic rules in electronic healthcare systems to flag if the patient was not being given low-dose aspirin and if one of the ICD codes was that. So, but now, as more evidence has accumulated, they're starting to rethink that. And it's no longer clear that low-dose aspirin is appropriate treatment. Now, in this case, it's not like the treatment was incredibly harmful and they made a huge mistake.

[1:53:08] It's more like when they do the cost benefits, it's like the low-dose aspirin helps in some ways, but it also creates risk in other ways. And when they do the trade-offs, it turns out it's not quite as good as they originally thought. But healthcare tends to move slowly so that it doesn't make too many mistakes like this. And that can be very, very good. And it's almost good that healthcare didn't adopt AI and machine learning too quickly, say, in the 80s and 90s. And that's because it's only in the 2000s and 2010s that we recognize that most data sets are learning more about treatment effects than in some cases about underlying health.

[1:53:57] And that can be very dangerous if your model is predicting a patient is low-risk because of a treatment you could have given them. But because you predict they're low-risk, you don't give them that treatment. Therefore, they're not actually low-risk because they're not getting the treatment because the model predicted they were low-risk in the first place. So, now they're high-risk. That's a terrible feedback loop. And it turns out this is a very important issue going forward in AI and machine learning in healthcare. So, healthcare has been slow to adopt things.

[1:54:29] They're very suspicious of new technology and almost always slow to adopt. And the randomized clinical trial historically has been the gold standard. We're not really going to believe it until there are a couple randomized clinical trials, possibly in multiple countries with different patient populations, that all kind of suggest the same thing is true. So, then maybe we're going to go ahead and do it. Healthcare is huge, though. And we are collecting ever better and better data, including now we're collecting genomic data.

[1:55:08] We can now do entire body scans of patients. Things like MRIs are not actually high-risk to patients. They might be expensive, but they're not high-risk. Unlike a CAT scan or an x-ray, which actually, gives you a dose of radiation, MRIs are believed to be completely safe. So, we now have new kinds of information. So, that includes genomic kind of information. We have all sorts of new kinds of blood tests that are created by our ability to work with genes. So, we can create new tests faster just because we have a better understanding of biology and chemistry.

[1:55:50] And then we have new modalities, like MRIs, which are just relatively safe. And now we have, everyday tracking, like your smartwatch can know things perhaps about your pulse, your sleep, sleep apnea, blood pressure. We know things if we want, like how reliably are you typing? When people are walking, we can measure tremor in gait. There's all sorts of information suddenly exploding that was not visible, possible 30 years ago. 30 years ago, we were in the dark days of healthcare where we had many fewer tests and modalities. Now we are in this, we are drowning in data.

[1:56:44] And the future means that's just going to be even more. And now we need, so think about COVID. So, COVID happened. Why did COVID happen at a world's scale? It happened because of travel. It was so easy for people to fly around the world. When's the last time we had a supermassive world pandemic? Well, it was at the end of a world war. And that's because a huge number of people were traveling around the world because of the end of the war. And now COVID happened worldwide because of travel. This is going to be the future of infectious diseases, respiratory infectious diseases at least.

[1:57:26] And in the early days of COVID, New York City was ground zero in the United States for COVID. It's where the first high density population of COVID patients occurred. They were desperate for help because nobody understood COVID and how it was different or the same as other respiratory illnesses. And we fortunately had the explainable boosting machines working. And we went to a hospital in New York City and said, we have this new technique. I don't know if you're interested in looking at it, but we might be able to help you.

[1:58:09] They were only supposed to meet with us for half an hour. They met with us for two hours. The next day they met with us for six hours. And in the night between those two meetings, they downloaded our open source software and ran it on their own data. And then they got our help interpreting the results Friday. And then we spent all day Saturday again interpreting the results. And they were using this new technique, new technique, by the way, which is invented by statisticians and then improved by machine learning researchers. So statisticians are the ones.

[1:58:48] Tibshirani and Trevor Hastie, these are just brilliant people. They really invented GAMs, a beautiful, beautiful thing. And all we've done is sort of improve them by using modern machine learning techniques, most of which we didn't invent. We combine these things. We had the ability to combine these things in a way that made it better. And then in the COVID days, the hospitals saw the value of this right away because it was able to show the ways in which COVID was like respiratory illnesses and then the ways in which it was surprisingly different.

[1:59:23] And they were able to confirm some of their own findings that they were starting to believe themselves from looking at patients and doing statistical analysis. But this automatic statistician sort of did it overnight. Now, back then, we didn't have LLMs to help us. LLMs would have accelerated that work even further. We would have trained the model and the LLM in the morning would have already had findings for us to look at, focused us on important things. As opposed to us having to spend multiple days with clinicians looking at graphs in what was a fairly modest data set with not too many features.

[2:00:03] So anyway, healthcare going forward is going to be, we're just going to drown in data. And how do you not drown? Well, you develop the ability to swim. You develop technology that keeps your head above water. And there's no doubt that AI, machine learning, modern statistics, and LLMs, which I guess are part of AI, are all part of the puzzle. And they may be useful for different parts of the puzzle. And it's critical in the short term, foreseeable future, that might mean five to 25 years, that these things as much as possible are assistants, that they're providing a second opinion, that they're providing a safety net, that they're helping to explain something when a clinician has a question.

[2:00:59] In the short term, we do not believe that these things are reliable enough, trustworthy enough, to be honest, accurate enough, in fact, correct enough, right? Their accuracy is surprisingly high, but their correctness is not always high. We don't believe these things are correct enough that they should be flying the plane by themselves. They shouldn't be in the driver's seat all by themselves. There should be human clinicians absolutely in the driver's seat. And these things should be there to amplify the power of the human clinicians. Humans aren't perfect. Humans either have or haven't had caffeine in a while. They didn't have their coffee yet. Yet they're seeing a patient there. They've been on call for 36 hours.

[2:01:46] Are they just as accurate at all times during a 36 hour call as they might have been at the beginning when they were fresh? No, humans have problems that AIs don't have. The AIs are sort of consistent, every time of day. And they don't need coffee. So I think we should, as much as possible, be using these new data-driven AI capabilities to assist clinicians to make the entire system both more correct. More correct ultimately means more accurate. Accuracy doesn't imply correctness, but correctness implies accuracy. Right?

[2:02:34] At least the maximum accuracy you can achieve given correctness. And you always want to be correct. Like 100% accuracy scares me. It's like, what's wrong with that model? But if the clinician says, oh, that model's correct, I trust it. So you want to be correct. You want to make as efficacious decisions as possible. You want to make them as early as possible. You want to reduce cost as much as possible. And that's because affordable health care gives us more opportunity to spend in other things that we don't know how to make affordable yet. So we can shift cost from something that's now easier, more affordable and easier to get right. We can now shift that cost to something else that's still important and not so easy to solve yet.

[2:03:24] So I think ultimately AI, machine learning, which basically means data, these huge new data modalities that are coming online, I think all of this is the future of health care for the next 25 years. Hey, 100 years from now, maybe an AI system will be the only doctor you have to see. I don't want to predict 100 years from now. But in the short term, it's clear it has to be a combination of human expertise, clinically trained people, and data-driven AI systems. And that's clearly the best way to do it in the short term.

[2:04:09] Arseniy Arsentyev: I will say no more or no less than you made me a huge proponent of this explainable and intelligible ML models in health care. I was just fascinated by the recent 10 minutes and your outline of the vision how to put this technology to work. I think it's a great point to conclude our episode. With all my heart, thank you, Rich, for providing us with the best information, with the cutting-edge information we can get. So thank you.

[2:04:44] Dr. Rich Caruana: Thank you very much, Arseniy. This was a lot of fun. And if people have questions, they should definitely reach out to you. And if necessary, reach out to me. That's perfectly fine. It's a very important mission going forward. And it's going to take tens of thousands of us to do it. It's not something any one or two of us can do. And I think your podcast is an important tool in helping us make progress.

[2:05:13] Arseniy Arsentyev: Thank you, everyone. A message for you also. Subscribe to my channel. Follow Tomorrow's Medicine on Apple Podcasts, Spotify, YouTube, everywhere. And the best thing you can do is to share this episode and support me on Patreon. That's it for today. Thank you, guys. Bye.