September 24, 2021: Examine. Diagnose. Prescribe. Is predictive analytics the magic pill for healthcare? Angelique Russell, Senior Clinical Data Scientist / Informaticist joins us today to share her advanced data science and deep healthcare expertise. How can we use visualization and modeling techniques to solve healthcare’s most challenging problems? What kind of insights are we looking to derive from the data? Have the tools changed much over the last five years? Are we still focused on provider data or are we starting to pull in some other data? And what is the future of machine learning, AI and NLP?
Building Effective Clinical Data Models with Angelique Russell
Episode 446: Transcript – September 24, 2021
This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.
[00:00:00] Bill Russell: Today on This Week in Health IT.
[00:00:01] Angelique Russell: There’s an idea in healthcare data science that we can apply deep learning basically in an unsupervised way, just take a big, vast database and say, we’re going to let the algorithms find the signals. And the risk there is that the signals you’re choosing are not going to be consistent over time. And that kind of black box approach, I don’t think it works at all in healthcare. [00:00:23]
[00:00:28] Bill Russell: Thanks for joining us on This [00:00:30] Week in Health IT influence. My name is Bill Russell. I’m a former CIO for a 16 hospital system and creator of This Week in health IT. A channel dedicated to keeping health it staff current and engaged.
[00:00:40] Special thanks to our influence show sponsors Sirius Healthcare and Health Lyrics for choosing to invest in our mission to develop the next generation of health IT leaders. If you want to be a part of our mission, you can become a show sponsor as well. The first step is to send an email to [email protected]
[00:00:55] I ran into someone and they were asking me about my show. They are [00:01:00] a new masters in health administration student and we started having a conversation and I said you know we’ve recorded about 350 of these shows and he was shocked. He asked me who I’d spoken with. And I said Oh you know just CEOs of Providence and of Jefferson health. And CIO’s from Cedars Sinai, Mayo Clinic, Cleveland Clinic and all these phenomenal organizations, all this phenomenal content. And he was just dumbfounded. He’s like I don’t know how I’m going to find time to listen to all these episodes. I have so much to [00:01:30] learn. And that was such an exciting moment for me to have that conversation with somebody to realize we have built up such a great amount of content that you can learn from and your team can learn from. We did the COVID series. Talked to so many brilliant people who are actively working in healthcare and in health IT addressing the biggest challenges that we have to face. We have all of those out on our website and we’ve put a search in there and makes it very easy to find things. All the stuff is curated really well. You can go out on a YouTube as well. You can actually pick out some episodes, share it [00:02:00] with your team, have a conversation. We hope you’ll take advantage of our website, take advantage of our YouTube channel as well.
[00:02:07] Today we have Angelique Russell with us. She is a healthcare data expert, formerly with Providence and several I’ll let her give her bio, but good afternoon Angelique. Welcome to the show.
[00:02:16] Angelique Russell: Thanks Bill. So happy to be here.
[00:02:18] Bill Russell: Yeah, I’m looking forward to the conversation. You have so many great posts on LinkedIn, but you and I have also already shared the stage at one point before. And it was, I think about maybe even five years ago, you, I and Darren [00:02:30] Dworkin in Southern California at a university, we were talking data.
[00:02:33] That was an interesting day. People were just firing their questions at us and we had to field all their questions. That was a lot of students and whatnot. It’s interesting. The perceptions of healthcare data out in the out, amongst patients out in the world how they think data is versus those of us who know sitting back and going, it’s not as clean as what you think it is.
[00:02:59] Angelique Russell: Definitely. [00:03:00] Yeah.
[00:03:00] Bill Russell: And I think a lot of the questions at that point were around interoperability and how can I get my data and that kind of stuff. And we were sitting there going, well we could probably get it to you, but I’m not sure it would do you much good because at least five years ago we were looking at it going, man, it’s in a lot of different silos and a lot of different places and it’s stored in different formats and I’m not even sure we can get to a proper definition of some of the terms that we were going to go ahead and we have all this data we’re going to give to you and you’re gonna look at it and say, I have this.
[00:03:29] And it’s like, [00:03:30] no, you don’t have that because this was the definition back five years ago. And the definition today is very different and that it was a ,that was a fun conversation. All right. So tell, tell us about yourself, current previous roles and your work in data science.
[00:03:48] Angelique Russell: Sure. Yeah. I actually had a career in an analytics and technical project management that precedes healthcare, but for the last decade, I’ve really [00:04:00] pivoted and focused, exclusively on healthcare. And I got in at the ground floor is like an EHR analyst. So I did a lot of builds in Allscripts. And since then I’ve learned Epic, things like clinical documentation, structured notes, flow sheets, order sets.
[00:04:18] And at that time in 2011, the focus in healthcare was really, we just need to modernize. We just need to have systems in place. But once we had those systems in place and we [00:04:30] started looking at what could we do with this data? I went back to my core skillset and started getting involved in predictive analytics.
[00:04:37] I was at City of Hope for a few years, which is a comprehensive cancer center where we formed a data team to look at how we could predict outcomes like sepsis and mortality. As you mentioned, I was with Providence Health System for a few years, and now I’m returning back to my data science consulting roots and[00:05:00] I’m pretty excited to be in this space right now in healthcare doing data science.
[00:05:06] Somehow I missed that I have a master’s in public health. I never know like where to fit that in, but, and a masters in public health from UC Berkeley.
[00:05:14] Bill Russell: When you got the master’s in public health, what were you thinking you were going to do?.
[00:05:18] Angelique Russell: What was I thinking? It’s such a fun question in a pandemic because it’s a time when people look at public health experts but what I was thinking was healthcare is a forever [00:05:30] focus for me.
[00:05:30] I’m always going to be in healthcare and I really wanted some expertise around our population. And when I began my MPH, population health was not yet a buzzword. But that’s what I was thinking. I was thinking, I, I don’t just want to know healthcare data. I want to know the whole picture of what makes a healthy person and what produces health outcomes and of that whole picture healthcare is only maybe 10 to 20% of a contributor to those outcomes.
[00:05:59] Bill Russell: [00:06:00] Actually, we’re going to do an interesting walk through your LinkedIn posts and just tell stories. I just, I think it’s fascinating. You have so many interesting things there but just going back to population health. 10 to 15 to 20% of the overall is the actual delivery of healthcare to overall outcomes in health. Who has the best data to really attack this problem today? I mean the most complete data sets really attack that problem.
[00:06:25] Angelique Russell: Yeah, that’s a great question. I’m not sure that it [00:06:30] exists. I think I can say conclusively. It probably isn’t your doctor. Your doctor is least likely to have that data health systems that you’re associated with maybe second best because they can at least get some of your social determinants of health information out of what they know about you. But I think there’s a real opportunity there to collaborate between different stakeholders because I don’t think that dream dataset that has [00:07:00] where you live and your social circumstances and your behavior combined with your health information from your healthcare provider combined with your genomics and information about the environment on top of that. I don’t think that that comprehensive data set has really been put together.
[00:07:19] Bill Russell: Plus financial and educational background as such a significant impact as well. So we try to get that through surveys but it’s still pretty incomplete [00:07:30] when we get that information, I would imagine. Plus it’s always changing. That’s the other challenge of that data set as well. All right. You are one of the people every time I see a post by you on LinkedIn, I have to read it. There’s a handful of people out there but you’re a good storyteller and data scientists are good storytellers.
[00:07:48] They have to take the data and they have to come up with stories. What is this data telling us? And a lot of times we look at the data and we say this is what it’s telling us. And then the data scientists look at us and go, no, [00:08:00] you can’t, you can’t make that, can’t make that job. We’ll start with early on because people are always asking me how do you become a data scientist? How do you get into this? And you had a post here of your early job story of how you were down to very little cash. And you sort of looked at it and said, I’m cash strapped, what should I do? And you decided to take a trip to Disneyland. That sounds like a very early on in your career kind of thing. How old were you when you were making that decision?
[00:08:29] Angelique Russell: [00:08:30] 16. Yeah. Yeah. I moved to Southern California when I was 17 but that decision was, just before my 17th birthday. And yeah, it was very much just the opportunity driven. I’m from a small town up north and when I saw how many jobs there were in Southern California, I got so excited and knew that that’s where I needed to be in order to be inducted.
[00:08:53] Bill Russell: What’s interesting because I talked to people a lot about, I moved around a lot in my career. You actually haven’t [00:09:00] moved around that much. You went to Southern California and you stayed and stayed in Southern California, which is an area that has a lot of jobs.
[00:09:06] But one of the things I talk about a lot, when people are asking me about careers, the first question I ask is, are you willing to move? And recently I had two different conversations with two people pretty far in their career. And one of them, I, I served that question and one of them said, yeah, I’ll move for an opportunity.
[00:09:23] In which case I can now talk about the full breadth of what’s available in healthcare. And one person said, no, I want to live in this town. I [00:09:30] go, okay, well you have the local health systems. You have anyone, that’ll let you work remote and you have people that will let you travel for your job. I mean, so it really does, it does limit you. And that’s one of the big points that you make here is so you feel like there’s no opportunities where you’re at. Maybe it’s time to get in your car and go to Disneyland. I don’t.
[00:09:50] Angelique Russell: Yeah, yeah. And I think things have changed a lot since, since I moved here 20 some odd years ago. Housing prices have gone up [00:10:00] a lot and the cost of living is such that I don’t think I wouldn’t necessarily recommend Southern California for people trying to get into healthcare. But definitely if you don’t find opportunity where you’re at relocating can be a huge advantage to getting your career going.
[00:10:16] Bill Russell: It’s interesting because when I left orange county St. Joe’s was headquartered in Southern California, and now it’s headquartered out of Providence, up in Seattle and a lot of people were like where do you go from there if you’re in healthcare? I’m like, well [00:10:30] there, there are some Memorial care still there. And so is Hoag is still there, I guess, according to the news articles, breaking off from Providence again. But at the end of the day, the largest health system in Orange County is Kaiser, which is out of Oakland and Providence which is out of Seattle.
[00:10:46] That’s an example of you think you’re moving into a place to do healthcare when in reality the jobs are being hired out of somewhere else.
[00:10:54] Angelique Russell: Yeah, absolutely.
[00:10:55] Bill Russell: All right. I want to talk some data science concepts with you. Since I have you on the line, you wrote [00:11:00] an article you said model drift, concept trips, historic time bias when working with out there data to train predictive models, it’s always prudent to have an extra hold of recent data to make sure the accuracy is the same across time. Help us to understand those three concepts and what you’re talking about about holding some data back.
[00:11:18] Angelique Russell: Sure. Yeah. Healthcare, so when you’re doing predictive analytics in healthcare, there are two signals that you’re picking up on. There’s an individual is like a biometrics, [00:11:30] right? Like you might think of your vital signs as revealing that you’re going downhill or declining. And your lab values can be that way, but there are also a number, there’s an overlap there. So there’s your vital signs. And then there’s your lab values. Your lab values actually overlap with the treatment domain, right? Because a physician making a treatment decision is going to order lab values to monitor the effects of that treatment and also to confirm the justification for that treatment.[00:12:00]
[00:12:00] And once you get into the treatment domain, this is very subjective data based on how you are being treated. You will have a different data set. So if we’re talking about sepsis and it was pre 2011, I think it was there were others drugs that were on the market that were pulled from the market. Xigris I think, just fell out of my brain, but there was a big change in treatment in 2011. [00:12:30] So treatment decisions that might predict in your algorithm would no longer predict after that point in time. And that’s just, that’s one big decision, but treatment guidelines and order sets, which are how a health system or a hospital standardizes the treatment that patients receive these change all the time.
[00:12:53] And these changes can result in, and drift in your model. If you’re detecting [00:13:00]decisions, treatments, labs that are no longer available after a certain point in time, your model might be really predictive when those indicators are available. And then drift, there is also another example. I think I gave in that post of historic bias related to just general scientific knowledge.
[00:13:19] And we could see some of this in the COVID pandemic. So early in the pandemic, we believed we were having a very bad flu [00:13:30] year. And the very bad flu year myths continued long after we knew that we were in a pandemic because we didn’t have testing available. If you recall, there was a real strict criteria for who could be tested for COVID.
[00:13:44] So even among hospitalized patients we were never really quite certain who had COVID and who had a bad case of influenza in the beginning until the patterns emerged and became just very obvious to the care teams. So there was an inflection point [00:14:00] after that we kind of knew just looking at a CT who has COVID and who doesn’t have COVID with severe pneumonia, but there was before that point, we didn’t know. So the data was frequently mislabeled. We had cases that looked like COVID, but were labeled as influenza. Influenza is not always tested. So there isn’t always a confirmatory test to rely on and that mislabeled data can send all kinds of [00:14:30] wonky signals if what you’re trying to do is for example detect COVID.
[00:14:35] Bill Russell: That’s interesting. Gosh, these are, these are challenges. I mean, you say order sets changing all the time and whatnot. Do you have to capture those order set changes as part of your model?
[00:14:45] Angelique Russell: I think it’s important to be aware of them so that you’re not using signals that are not going to be like reliably available. But I think early in, in how you’re choosing to design your model I think[00:15:00] really limiting to things that are less likely to change is important. So there’s an idea in healthcare data science that we can apply deep learning. And basically in an unsupervised way, just take a big, vast database and say, we’re going to let the algorithms find the signals. And the risk there is that the signals you’re choosing are not going to be consistent over time. They might be related to things that are in flux. And you won’t know that they’re in flux because you let the algorithm find the [00:15:30] signal and you don’t really know what the signal is and that kind of black box approach just, I don’t think it works at all in healthcare. And certainly can create problems.
[00:15:43] Bill Russell: Well, we’re going to talk about Watson here in a minute, cause you had a post about Watson. You talked about holding out some recent data. So I guess what you do is you compare the data against itself, along the model, and that’s one of the ways she determined if there’s been a [00:16:00] change in it, or some kind of drift or some kind of challenge that the data itself has, will the data will reveal the problems of the data itself it soundslike what you’re saying.
[00:16:13] Angelique Russell: Yeah. So traditionally when you’re training a model what you’re doing is you’re teaching and algorithm how to detect a pattern. And the typical way of doing that is to is to split your data [00:16:30] into training and tests. So you have one big set, usually 80 to 90% of your data, and you’re going to train on that and then you hold out 20%. And then you’re going to test on that and confirm. That it has the same approximate performance on your test dataset. So what I’m suggesting on top of that is also just take some recent data. So if you’re looking at five years of data and you’re saying, well, across five years, this is the accuracy we have in healthcare.
[00:16:57] It’s pretty common for five [00:17:00] years of data to not be purely consistent with the last year of data. So making sure you have a holdout that’s more recent than your entire set is important.
[00:17:11] Bill Russell: How has analytics, just straight up analytics, different from data science at a healthcare provider?
[00:17:17] Angelique Russell: Yeah. I, I noticed recently that that the terminology is blending a lot. A lot of roles that I would have considered to be like just a purely analyst role or being [00:17:30]labeled as data science roles. So it might be a little less separated today than it still is in my mind, but I think of analytics as being descriptive and it can be very advanced.
[00:17:42] You can use advanced statistics in your descriptive analytics. But what you’re trying to do is describe the current state. And I think of data science as being most focused on how can we. Use the the [00:18:00] vast data assets we have to anticipate what’s going to happen next. So less oriented toward, we really need to describe what’s happening and more oriented toward predictive analytics and machine learning models.
[00:18:14] Bill Russell: I once heard it described that analytics answers questions, and data science explores what questions we should be asking. And I’m like, I think I get it. But, you just sorta look at the data and then the data sort of reveals, Hey, we should be looking at this. [00:18:30] Vioxx is a problem kind of thing.
[00:18:32] Angelique Russell: Yeah. Yeah. I have some fun stories about things like that. I remember one machine learning model we were looking at we kept seeing a signal related to patient fall events. So patients falling over in the hospital and being injured as a major liability thing. Most that they frequently want to intervene upon. And we were able to detect a signal related to a drug class that didn’t make sense to us.
[00:18:59] But when we told the [00:19:00] nurses about it, the nurses were like, oh yeah, that’s definitely from Adavan. And it was just related to a common workflow we had where patients who were receiving chemotherapy were being given Adavan as an antiemetic to prevent severe nausea. And also to help them relax, but then they weren’t being reclassed as high risk for fall events.
[00:19:21] And it was actually kind of a known issue. They knew about it, but having the data to support it and being able to show once we had the data, we were able [00:19:30] to do some statistics and show an odds ratio and really show the dose relationship as well to physicians to help them come to a consensus on reducing the doses and intervening earlier on that. And that’s an example of the overlap between like descriptive analytics and data science. That was a data science project. It yielded some really valuable descriptive analytics and that was actually what was needed to affect change. [00:20:00]
[00:20:00] Bill Russell: All right. So you wrote a very interesting article. Four reasons why sepsis predictive models fail and specifically the most talked about one was ethics challenge. Their algorithm only detected 70% of sepsis cases missed by clinicians. And you had four reasons. Was this something that was like a project that was kicked off, or just something that you you were reading these articles saying, hey I want to look at this and see why this happened?
[00:20:26] Angelique Russell: That was my immediate reaction to that [00:20:30] story.
[00:20:30] Bill Russell: You read the story and you’re like, I know exactly why this is happening.
[00:20:34] Angelique Russell: Yes, that is, that is actually how that happened. And I think anyone watching this, who’s also done a sepsis prediction models had the same similar nod along because it’s if you’ve tried it, pedicting sepsis, you probably know very predictable, but in the actual workflows not very accurate and potentially even not useful.
[00:20:59] Bill Russell: All [00:21:00] right. So let’s walk through the four things. So the first was lack of timely automated data in the EHR, including vital signs.
[00:21:08] Angelique Russell: Yeah, so important. So in a modern hospital, in an ED and in an ICU, and usually they’re, they have a unit in between, they may call telemetry or sub ICU. You can rely on pulse oximetry and vital signs to automatically go into the EHR system anywhere between every 30 minutes to every two [00:21:30] hours. Fairly well automated, really great nurse to patient staffing ratios.
[00:21:35] So even when a nurse action is required to make the data available, it’s still very timely. But where there’s often a need to detect sepsis in a hospital is not where you have a patient constantly being monitored by a nurse. It’s where you’re in a Med surge floor recovering from a [00:22:00]procedure and suddenly you have a fever that’s when you would want to know that this patient is going septic, they may need to have an intervention.
[00:22:08] They may end up in the ICU and those floors do not consistently have. Automated devices for entering even basic vital signs. And the workflow is often not even what you would expect it to be nurses taking vital signs, use napkins, paper towels. And [00:22:30] this, these artifacts travel back to the nurses station where she’s going to input them three or four hours later.
[00:22:39] And that lack of timeliness, it, it just ruins any potential for accurate and reliable algorithms because the algorithms have to depend on the most recent time they have available, which can be the night before.
[00:22:53] Bill Russell: And I’m not sure. So I’m coming from the CIO standpoint. I’m not sure why that is today.
[00:22:57] I mean, we have these tools. We have capsule. [00:23:00] We have the ability to capture all that data and pull it in. There’s part of me that’s like, why haven’t we done that? That’s not a fair question for you, but it doesn’t make sense to me why we haven’t.
[00:23:07] Angelique Russell: Yeah. Well, as a CIO, you probably needed to take some of your budget and give it to your CNO because the CNO traditionally, and this can vary from hospitals, but traditionally the CNO has a budget that includes medical devices.
[00:23:22] So smart beds, smart pumps, capsule devices or Hill-Rom vital sign devices. Those [00:23:30] often come out of the CNO budget. So as money has been pouring into hit for modernization and EHR implementation, it isn’t necessarily pouring into the CNOs budget for those automations that would be needed to really make the clinical decision support tools we wish we had.
[00:23:48] Bill Russell: Interesting. Thanks for mentioning Hill-Rom who’s a sponsor of the show. So always, always good when you mentioned someone. Number two, upcoding an uncomfortable truth and source of label bias, [00:24:00] help us to understand that one.
[00:24:01] Angelique Russell: Yeah. So I just came from a non-profit health system. Actually, all of my health providers I’ve worked for have been non-profit and I want to be really clear about that because this isn’t the kind of profit driven concern that a lot of people mentioned in the nonprofit space it is the responsibility of the nonprofit organization to recover as much revenue as [00:24:30] possible for each encounter in order to cover the cost of care. Like they’re not trying to make huge profits or to give out big bonuses. They’re really just trying to cover the cost of care. And there are many many encounter types like stroke that routinely don’t, what is reimbursed, doesn’t even cover the cost of care. So this is the driver behind upcoding. Before I say what it is, we’re just all trying to cover the cost of care. So there are many [00:25:00] systems in place to recover the cost of care. They include computer aided coding. So there’s software that scans over it.
[00:25:07] And identifies key terms and individual coders. So humans who review notes or review charts, and they create corrections to medical records that doctors then have to sign off on for the purpose of making sure the bill is accurate. And in the case of sepsis, how this often works is you have [00:25:30] someone who maybe he has a fever or they have a hypotension, low blood pressure, and a physician may order a sepsis order set and their intention is to rule out sepsis. So they’re going to take the Lac de and run some blood cultures, but there are billing rules that if you if there was suspected sepsis and all of the criteria was met then it is sepsis.
[00:25:58] And so they’ll add that [00:26:00] code to the chart, even if the clinicians at the time of when they were treating the patient, didn’t consider sepsis to be the probable cause. And I it’s so hard for outsiders to kind of understand this concept because we think of things like sepsis as being totally like objective. Like there must be a definition for sepsis.
[00:26:22] Bill Russell: It’s like broken knee or a broken leg. Like alright there’s a definition.
[00:26:28] Angelique Russell: Yeah. How could you not know? [00:26:30] Yeah. Yeah. But there are a number of medical conditions that can produce a systemic inflammatory response and systemic inflammatory response is what you look for in sepsis and there’s often cases where you have sepsis but your blood cultures are negative. So we don’t rely on cultures. And there has been so much change in how we define sepsis medically over the last 15 years that even in the medical community, there is not a consensus on the [00:27:00] definition for sepsis.
[00:27:01] Bill Russell: Do you have a medical background or is this all from your view, from the data side that you’ve had to learn all of these terms?
[00:27:09] Angelique Russell: Yeah. I’ve had to learn all of these from my view, from the data side, for sure. And I think it was helpful to go through a graduate program, like my MPH program to learn how to review literature and research and incorporate some of that into my knowledge. But no, I don’t have a clinical background.
[00:27:28] Bill Russell: But did you pick it up from, [00:27:30] from research or do you just pick it up a lot from conversations or is it sort of a split?
[00:27:34] Angelique Russell: Both. When I was at City of Hope, I was very fortunate that my little data science team was actually inside an informatics team. So I worked elbow to elbow with nurse informaticists and though they did have a clinical background in critical care and in med surge and ER nursing. And I worked for our CMIO who was a medical doctor. So that just the [00:28:00] experience of being surrounded by clinicians and really making sure we had a nurse and physician leadership on every project we worked on was very very helpful in my own personal development of understanding the science in that.
[00:28:18] Bill Russell: I get this question a lot. Do you think it’s easier to go from a medical background and learn data science or data science and learn the things in medical that you need to learn
[00:28:28] Angelique Russell: I’m going to go [00:28:30] medical to data science on that one. I know some really fantastic doctors and nurses who have really embraced data science and predictive analytics. And I think you can always bring on a machine learning engineer when you reach a wall in your your data science skillset. That things could be better if you just had someone with that right match of skills. But at the end of the day, like what determined the success of a data [00:29:00] science project in healthcare is usually how much clinical leadership and involvement that it had.
[00:29:07] Bill Russell: Interesting. All right. So let’s get back to the sepsis model. Number three, sepsis models may not generalize to other patient populations. This gets back to definitions again. So help us understand that.
[00:29:21] Angelique Russell: Yeah. So it does relate to things like upcoding and definitions, because certainly if you have two institutions have two different definitions of what is [00:29:30] sepsis then you have some labeling bias and in one data set, it’s going to be labeled another way and another we’ll meet a different definition.
[00:29:38] And so that, that might not generalize. But the only paper I contributed related to sepsis, which came out of it, My work at city of hope with Dr. Dodd wall, who’s an infectious disease doctor. We were very sure that our bone marrow transplant oncology population was very different when they developed [00:30:00] sepsis compared to a community population.
[00:30:04] And there were some conversations with epic where in the beginning they were less sure. They were like, no, we have big data, so much data we think that our model is going to be predict. But we were able to demonstrate through our own model development and comparing it with Epic’s that there really is a significant difference there.
[00:30:25] If the underlying pattern is physiologically different in your patient [00:30:30] population, then a model based on how things usually occurs is just not going to apply. And in the case of bone marrow transplant, those are immunosuppressed patients. So it just didn’t generalize well, but I can think of other scenarios where you could have difficulty generalizing because you could be in a part of the country with a lot of retirees, Bill, and you might have a higher than average age in your population [00:31:00] and you should really consider the possibility that your population is not going to present the same exact way as a younger population in a nationwide dataset. And you can there are ways to validate to confirm this.
[00:31:15] You can t ake your validation data sets and you can stratify them to answer how accurate is this across different demographics and how accurate is this in my own patient population. And do I have vulnerable [00:31:30] patients such as pediatrics, elderly, or immunocompromised that are distinctly different from a general population and how accurate it is in that population.
[00:31:39] Bill Russell: It’s interesting that dataset I’m scanning your article as we’re having this conversation. So the the Epic data set relies on claims data a fair amount. So that claims data, that gets back to your first point, which is the telemetry data is going to be the most current, most accurate and most beneficial for predicting any [00:32:00] or creating any sepsis model.
[00:32:02] The claims data is kind of, it’s a huge dataset, but because of definitions and a lot of things you’ve talked about that’s, I mean, it’s good to have that data set, but it’s not necessarily the best work creating this kind of model.
[00:32:16] Angelique Russell: It’s definitely not the best for labels. If we were going to label sepsis there, there are other options. I think when it comes to labeling a sepsis dataset, the optimal way to do it is to try to rely on some [00:32:30] objective rules, which are in your day. So you can look for example, SIRS criteria or Mews criteria and lack D or a blood culture or some kind of rule that your clinicians have signed off on, and then use that to label sepsis versus not sepsis.
[00:32:48] But you still have to resolve all those ambiguous cases because there’s a lot of ambiguity in real life and you have to figure out how to, how to do that.
[00:32:56] Bill Russell: So where do we go from here? I mean 7% [00:33:00] of sepsis cases missed by clinicians is all it’s detecting at this point. We want to do better. So we’re where do we go from here? How do we create a better model?
[00:33:08] Angelique Russell: Well, you just cited one of my favorite statistics from the findings related to the epic sepsis model, which is that epic is only detecting 7% of missed sepsis cases. And that really brings us back to what is it that we are trying to predict or detect. And are we even [00:33:30] predicting or detecting? Right? Because prediction is before the diagnostic criteria is met and detection is somewhat after that point. In the case of sepsis where we have preventable mortality is usually we missed it. We dropped the ball and and that’s what we want to prevent. We don’t want to miss a sepsis case or delay intervention or care in a sepsis case.
[00:33:56] And so the cases we want to be detecting the [00:34:00] most, that’s where the Epic model unfortunately has the worst performance. So I think we have to go back to the drawing board and ask, well, what could we be detecting and how could we be in labeling our data set. I’d love to see a big data set out of hyperspace Epics database that includes a separate label for missed sepsis. Like let’s just see how we can optimize the prediction for missed asepsis and see if we can find those drivers. [00:34:30] I tend to think my hypothesis would be that a lot of those missed sepsis cases, we relate in collecting and processing vital signs so they weren’t seen by the right people.
[00:34:41] I think that’s likely to be a pretty big cause in which case the solution is again, those devices we were talking about a minute ago. It’s not even necessarily a predictive model, it’s have the right technology in place for the clinicians. And then once you have that really understanding how is [00:35:00] missed asepsis happening and how can we predict that specifically.
[00:35:04] Bill Russell: Interesting. I’m going to skip to a different story now. So you wrote a little bit about Watson. I think this has been written about a lot in terms of just their biggest mistake was they came in loud and proud. Like, Hey, we won jeopardy. Now we’re going to cure cancer.
[00:35:22] So they came in a little loud and proud, which was a huge mistake. They didn’t understand the data that they were sort of wading into. And[00:35:30] they weren’t able to really do the things that they thought they were going to do. So part of me wants to ask you the question to really succinctly talk about what went wrong with Watson, but also what’s the future of machine learning, AI, NLP. What’s the future of these machine-driven technologies with regard to healthcare data?
[00:35:49] Angelique Russell: Sure. Well, I’ll answer the last question first. I think the future is an understanding how we can augment human decisions. I [00:36:00] think every time I hear about someone’s breakthrough algorithm, that’s about to make it to bedside and has the most potential, usually what we’re talking about is a tool that gives the physician insight that they didn’t previously have that’s machine aided insight. Like here’s some prognostic information and here’s some trends and this is how the AI algorithm is interpreting them. Do you agree? And how do you want to base your treatment decision on this information?
[00:36:29] I think [00:36:30] that’s where the future is. What Watson tried to do was kind of build some kind of a quasi recommender systems where they used medical literature and treatment notes and progress notes that detailing treatment to kind of tell physicians what treatment should be, or like in the case of oncology, they would recommend drug regimens based on historical patterns, but that [00:37:00] doesn’t, the value add there is very little because the past isn’t necessarily optimal. So if we’re using the past patterns to predict future action or trying to automate off past patterns, we have to be realistic and honest about the fact that not every treatment decision is optimal in the current state.
[00:37:21] That means our historic data contains a lot of treatment decisions, a lot of diagnoses a lot of bias in what tests were [00:37:30] ordered and what information is available. And if we only rely on past patterns, then that those treatment decisions and bias and and suboptimal care is going to be what we predict and recommend going forward, which no one wants that.
[00:37:45] Bill Russell: I sat across from the doctor one time and I said help me to understand, are you guys just guessing. He said educated guesses but yes. I mean, what we’re doing is we’re taking the data that you present us. We’re taking our [00:38:00] knowledge, that our learning, our experience, the journals that we’ve read, we’ve taken all of that and we were saying, we believe you have this.
[00:38:08] And we, we prescribe a plan of action or medication. And then you come back and you tell me it worked, or it didn’t work. In some cases, it’s just not an exact science. And sometimes we, prescribe medicine, people go use it, they come back and it didn’t work. And we go, all right, try this. And then that works. It’s like, well, why don’t you prescribe that the first time? Well, there’s a lot of reasons why [00:38:30] they wouldn’t prescribe that the first time.
[00:38:32] Angelique Russell: Yeah, I really love this book. It’s this is Michael Lewis’s The Undoing Project. It has a whole section about medical decision-making and about algorithms and medical decision-making.
[00:38:43] And it’s so neat. And when I say algorithms in medicine algorithms existed prior to machine learning, so there was a whole, they already were there and they were based on research that was done in the last 75 years that found that [00:39:00] experts, even clinical experts couldn’t outperform just a simple decision-making algorithm that would have like a checklist.
[00:39:10] Like if, if the tumor looks like this, if it’s yay big, if the margins look like that, check, check, check, then the diagnosis is this. So medical experts could create really good workflows like that, which are called algorithms in medical decision-making. But they couldn’t outperform their own tools, just using their own [00:39:30] judgment alone.
[00:39:32] And that revolutionized medicine. And it resulted in a whole bunch of like systematic ways to do diagnoses that we have today. But I think there’s a lot of potential to go further than that. Now that we have things like machine learning, we just have to figure out how to use our data.
[00:39:47] Bill Russell: I would assume this is why imaging is one of those areas where it has excelled. Because you have an image and they can actually go through that algorithm pretty well. Now there’s still some things that are missed, but for the [00:40:00] most part, you’re seeing the reads of computers are getting pretty close to the radiologists and whatnot. I don’t want to take all those hate emails that are going to come at me right now, but
[00:40:09] Angelique Russell: There’s a lot of caveats to that.
[00:40:12] Bill Russell: There are a ton of caveats. But one of the things I talked to doctors about is the nature of work is changing. It is machine assisting. And if that’s not to minimize the medical degree.
[00:40:23] That’s not to minimize the value of the patient interaction or actually seeing the patient and taking [00:40:30] that knowledge in or even trumping the computer algorithm from time to time, just because you look at it and you go, no, that’s not, that’s not right. There is that interaction that is going to the nature of how we practice medicine.
[00:40:44] It’s already changing. I mean, it’s to take a look at what happened through the pandemic. And I’m not sure when physicians ask me, it’s like w what’s it gonna look like? I’m like, it’s going to keep changing. It’s going to keep morphing. And still 30 years from asthma look very different than it looks today. But I think five [00:41:00] years from now, it’s going to look pretty different than it looks today.
[00:41:02] Angelique Russell: Absolutely. I think when I talk to physicians, I think physicians want the kind of changes that I can see coming in terms of having more insight into the data especially in the time of COVID.
[00:41:22] I mean, I sat on some calls with ICU physicians that were painful, just to be on hearing the [00:41:30] frustration from clinicians who were unable to determine who they needed to admit and put on a ventilator and who would be okay with oxygen. Or in an emergency room who could go home and not having that answer. Not being able to rely on their own medical judgment and there wasn’t any literature out yet. That was really difficult. But in medicine today, even when we’re not in a pandemic, there are still a lot of decisions where we just, we’re nowhere near optimal [00:42:00] and there’s so much potential to use our data, to get to optimal. And. And pharmaceutical research is that the concept of like numbers needed to treat.
[00:42:12] Like how many patients need to get lipid tore in order for one heart attack to be prevented, it’s like a hundred patients need to get it for one person to benefit. Now that we have these large data assets, what if we could reduce that down to 50 people or 10 people like the cost, the safety, [00:42:30]the outcomes are so different when we’re able to target treatments and we can do a lot of things that we haven’t been able to do up until this point with predictive analytics.
[00:42:40] Bill Russell: This has been a fantastic conversation. I’m going to close with this only because it’s fun. And I like the human interest stuff. So your daughter made a father’s day card. Is that accurate?
[00:42:51] Angelique Russell: Yeah. Poster.
[00:42:54] Bill Russell: A poster. Dad you’re as brave as Bilbo, as strong as Thorin, as clever as Gollum, [00:43:00] as generous as Elron, as wise as Gandolf and I would face a dragon for you. And it has these graphic images and those kinds of things. And this is what I love about your writing.
[00:43:10] You’re like, here’s the five lessons we learned from this. I assume you remember this post. It’s it’s about a month old, I guess. So five lessons. The first lesson is the right tool is the one that you can master and get the job done. So your daughter was probably figuring out, okay, how can I make this thing and she decided to use Microsoft paint. [00:43:30] Wow. That had to be hard. Isn’t that hard? Microsoft paint.
[00:43:34] Angelique Russell: It’s it’s so the wrong tool, which I got a kick out of, and also my husband always laughs at me for using Paint for anything, but she, but I wasn’t going to teach an eight year old Photoshop so.
[00:43:49] Bill Russell: And that’s the first thing you use the right tool. The second was commercial off the shelf can often accomplish what you want at a lower cost and you shouldn’t pursue a custom solution [00:44:00] without ruling out custom off the shelf. We couldn’t sign any proper Hobbit theme poster, but you chose some clip art from an Etsy store as well.
[00:44:09] That’s interesting. Man, I hope your eight year old learned that lesson. That’s a lesson every CIO needs to learn in technology.
[00:44:19] Angelique Russell: Yeah she didn’t go get a degree in graphic design in order to make her a father’s day present. She just went to Etsy and I think it cost us $8.
[00:44:29] Bill Russell: $8. [00:44:30] Well, that’s a really nice poster by the way. Let’s see. Number three, if you’re trying to do something beyond your skillsket try pair programming designing with someone more skilled, like your mom. It’s interesting. This pair programming thing was really has come into Vogue. I mean two are better than one, they’re able to fill in the gaps for each other, look at each other’s code, help each other to learn things and whatnot. Is that true in data science? I would assume it is.
[00:44:58] Angelique Russell: Absolutely. Yeah. There are all [00:45:00] kinds of engineering principles and software engineering principles around the right person being able to get something done in an hour that would take the wrong person a hundred hours or even a thousand hours. So having the right person there is hugely valuable, but when it comes to data science, usually the right person is actually the right team. We talked about needing to have that clinical knowledge and also needed to have that machine learning knowledge. Sometimes you can just have two people with those expertise working together [00:45:30] and kind of following along with each other as they work and you can get quite a lot done.
[00:45:34] Bill Russell: I think about that when I’m reading some job descriptions and I’m looking at it. I think there’s five of those people in the world, or you can just get two people or even three people at far less costs and they could do what that one person who is in high demand can actually do.
[00:45:52] Your number four outsourcing is cheaper than investing in your own infrastructure. Many thanks to Staples for printing the poster for only [00:46:00] $16. As a CIO, I would have to say proper outsourcing because I’ve had to insource two or three times in my career from improperly outsource. But yeah, Staples makes a lot of sense.
[00:46:11] You don’t want to go out and have to buy one of those massive printers for your house just to print this thing. And I love your last point. If you believe you can make something awesome, you probably can. And for an eight year old that’s a great lesson. Angelique, thanks. Thanks for your time. I love your insights. I love your writing. I hope you’re going to continue to do that in your in your new role. [00:46:30] How can people folow you?
[00:46:32] Angelique Russell: Yeah, you can follow me on LinkedIn. I also accept quite a number of connections. I know different people have different opinions on how to connect on LinkedIn, but I’m always open to, to chat and connect with people who are deep in healthcare, data science, and trying to make a difference.
[00:46:47] Bill Russell: Yeah, I’m curious what your philosophy is. I get like, I don’t know, five or six marketing every week. Somebody is like, hey, you only have 9,000 followers. And I’m like, yeah, because I say no to like half of the ones that come in cause it’s like, [00:47:00] I can grow your number of connections, I can, and I’m like, I just don’t, I don’t invite those people into my network usually.
[00:47:06] Angelique Russell: Sure, sure. But I’m always looking for genuine connections.
[00:47:10] Bill Russell: Yeah and I appreciate it. You’re now in the consulting world again. So if people have questions, they can reach out to you through LinkedIn and maybe contract your services in the future. We’ll see. See what happens. Thanks.
[00:47:22] Thanks again for your time. Really appreciate it.
[00:47:23] What a great discussion. If you know someone that might benefit from our channel, from these kinds of discussions, please forward them a [00:47:30] note, perhaps your team, your staff. I know if I were a CIO today, I would have every one of my team members listening to this show. It’s conference level value every week. They can subscribe on our website thisweekhealth.com or they can go wherever you listen to podcasts, Apple, Google, Overcast, which is what I use, Spotify, Stitcher. You name it. We’re out there. They can find us. Go ahead. Subscribe today. Send a note to someone and have them subscribe as well. We want to thank our channel sponsors who are investing in our mission to develop the next [00:48:00] generation of health IT leaders. Those are VMware, Hill-Rom, StarBridge Advisers, Aruba and McAfee. Thanks for listening. That’s all for now.