Joe Petro CTO of Nuance on Making the Keyboard Disappear and Other Magic

This Week in Health IT Joe Petro with Nuance
Share on linkedin
Share on twitter
Share on facebook
Share on email
The combination of Voice, ML, AI, and NLP have the ability to make the computer disappear from the exam room. Nuance demonstrated it and Joe Petro explains it in this discussion form the HIMSS floor.

Episode Videos

Bill Russell:                   00:11                Welcome to this week in health it where we discuss the news information and immerging thought with leaders from across the health care industry. This is Bill Russell. Recovering healthcare CIO and creator of this week in health it, a set of podcasts and videos dedicated to training the next generation of health it leaders. This podcast is brought to you by health lyrics, helping you build agile, efficient and effective health. It let’s talk visit HealthLyricscom to schedule your free consultation. We were recording a series of discussions with industry influencers at the Chime Himss 2019 conference. Here’s another of these great conversations. Hope you enjoy.

Bill Russell:                   00:47                Great. Uh, here we are from above the himss for, we’ve actually elevated our game. Gone one step up. Uh, we’re at the, uh, nuance booth and uh, we’re here with Joe Petro who is the CTO for nuance and a, as I keep saying, I love getting the CTOs in the room so that we can have a in depth conversations and as, uh, as we’ve heard from our audience, they want to hear more about voice voices is one of those things that I think it was an emerging technology a couple of years ago and now it’s okay, where else can we use it? I mean, we’ve been doing notes and dictation for awhile now they’re saying, hey, where else can we do it? So what are you guys showcasing here? So talk a little bit about, um, you know, what is, what’s the big announcements for nuance at this show?

Joe Petro:                    01:35                Sure, downstairs we have what’s called an experience room. And um, the, uh, the, the name of the product or the solution that we’ve created is called ambient clinical intelligence. And fundamentally it’s, it’s almost a combination of everything that the company has been doing literally over the last 20 years. So, uh, it’s a combination of, um, a hardware device, which basically listens to the patient and the doctor conversation it. Uh, and as it’s actually listening. It turns that into a, into a transcript and it diarize as the speech, which means it splits it up. So it, it separates out what the patient is saying versus what the physician is saying. And then it starts to derive meaning. And this is where the intelligence part comes in, provides the physician with feedback as, um, as the conversations in real time at the point of care.

Joe Petro:                    02:27                So, uh, and, and we’re extracting facts and evidence and we’re creating documentation so it automatically is generating the documentation and it allows the physician to stay engaged with the patient without turning their back and go into the computer. So it’s a super exciting stuff.

Bill Russell:                   02:41                So what are we talking about? We were talking about, uh, you know, voice recognition. We’re talking about really machine learning AI, we’re talking aboutNLP, natural leraning processing. Yep. Yeah. What other things am I missing? I mean, you’re really hitting all the, all the buzzwords and actually doing

Joe Petro:                    02:57                it hits, it hits all of it. And that’s the amazing thing. We believe that this really is the future destination that the company, in terms of all the different products we’ve created over the last 15 or 20 years, it has been pointed at in a natural way. And, um, the vision that we actually create by bringing folks through that experience actually it’s just speaks to just literally everything that we brought together.

Bill Russell:                   03:22                So is this starting from the conversations because our doctors always say, my patients hate this. Hey, how you doing? How are things going? Yeah. Um, so now we’re going to be able to put this in the, in the clinic. They’re just going to be able to have conversations. Is it, or the notes going in the Ehr?

Joe Petro:                    03:38                Yeah, so it basically enhances and augments the experience inside of the electronic medical record. One of the things that we say for our product dragon medical one that’s uh, that’s uh, uh, cloud based speech products. Um, about, uh, 25% of the physicians in the u s are on it now and about 50% of the physicians will be on it over the course of the next couple of years. Um, so it’s a very prevalent product, but we like to say is we turn the chair around, right. And that’s one of the values that we actually bring to the physician patient experience just with regular speech.

Joe Petro:                    04:10                This even takes it a step further. So there is no chair. All right. So we’re, we’re basically listening to what the conversation is transpiring during the conversation with deriving meaning from that and in real time and it’s all happening at the point of care. And so the idea is once that once that note is created, physician will review the note and then that will automatically get journaled into the electronic medical record. And so it’ll relieve the physician from a lot of the typical burden that they actually feel through the documentation process. Uh, and it’s exciting. We’re, we’re starting out with a set of specialties that there’s an Ortho demonstration that we’re actually doing downstairs where it’s a condition of the patient was experiencing, they’re experiencing in their knee. And then we’ll expand through a number of different specialties like ear, nose and throat, ophthalmology, et cetera.

Joe Petro:                    04:59                There’s like 18 different specialties and we’ll eventually migrate our way up to a general medicine type of a scenario.

Bill Russell:                   05:06                So is that just because you have to get the, the language down for those different specialties?

Joe Petro:                    05:10                Yeah, it’s a good point. It’s like, it’s not so much the language, it’s, it’s kind of like when you think about it, it’s the narrowness of the problem, right? So by narrowing it to a very specific domain like Ortho, it’s very prescriptive in terms of the dialogue that the patient generally, uh, the physician generally has with the patient and what they’ve document, right? So it’s kind of the simplest form of a patient, physician and counter. And so by starting out with a simple form and narrowing it, it, it kind of in an artificial way imposes this condition where the accuracies and all that stuff go up. So where, so the technology is more capable of solving the problem.

Joe Petro:                    05:47                And if you think about how speech evolved, we started with something called interactive voice response. It was just the menu system navigation by in the early days when speech wasn’t very accurately, it was like 75% accurate. By narrowing it to a really constrained vocabulary. The accuracy went way up, went up to like 95% created businesses around that, continue to exploit the technology more and more until the general accurate accuracy kind of tipped up over 90% and then it turned into long form dictation where the physician could actually just rail for 15 minutes and then do editing.

Bill Russell:                   06:18                It’s interesting, this is an example of that sort of exponential curve because back in the day, you know you had dragon naturally speaking and you were trying to navigate and eventually you threw it away cause you’re like, ah, it’s just quicker to click on the mouse. But we’ve gotten to a point now where you incremental, incremental, incremental and now machine learning AI and you just see it go through the roof. Where else’s voice is going to be, are we going to see being used in the or for checking the record?

Joe Petro:                    06:43                I think, um, I think the way this is going to progress as part of ambient clinical intelligence is something we call virtual agents. And this is, it’s a similar to the type of stuff that you actually use on your phone. When you ask what your calendar looks like. You asked for the weather information retrieval use cases, we believe that’s going to be the pathway between like where we are in terms of just general speech in general, kind of clinical decision support to that kind of next phase where you’re actually engaging with the system and you’re skipping most clicks, right? So let’s say you want to pull up an abnormal lab values on a patient. Just being able to say, let’s look at their abnormal lab values and have it instantly pop that saves the physician x number of clicks and they might do that 30 or 40 times a day.

Joe Petro:                    07:30                Stack that up with a bunch of different, we call them, show me use cases like show me the growth chart, show me where the patient sits on the growth chart. Um, show me the uh, the patient’s last procedure or show me their patient’s imaging study. It just short circuits the distance between the physician and the information and they can make it a very interactive experience. Because these, uh, these treatment rooms and so forth, the kind of the treatment room 2.0, they all have large screen TVs in them and they’re all kind of high fidelity experiences. And so the ability to kind of leverage that screen, uh, leverage the electronic medical record with all the information that has in it. And then using voice to kind of control that experience. We believe there’s going to be a natural migration from where we are through virtual agents to ambient clinical intelligence.

Bill Russell:                   08:16                So you’re, I mean, you’re hitting both the consumer experience and the clinician experience. Is there anything that you’re really focused in on the, uh, uh, consumer experience specifically?

Joe Petro:                    08:27                Not this business, not today, but, um, if you look at nuance, we’ve got a large footprint in enterprise as well as auto. Okay. So we’re in like 500 million cars where that voice experience speaker diarization what we call a gaze detection. So we’ve got technology in a car nowadays where you can look at something, you can look at a store and you can say what are the hours and it knows that you’re actually looking at the store, if it figures it out relative to the gps relative to where the gaze is actually going and it will actually retrieve the information off of the, off of the Internet. That’s completely a consumer experience. And we’re like in essentially every major car brand that you can imagine.

Joe Petro:                    09:10                Uh, and then on the, uh, on the enterprise side, we’ve been using virtual agents on the enterprise side forever, basically. And it’s all about engaging with the enterprise. So if you call up like, uh, uh, any major airline, any major bank, and you’ve got an agent and intelligent agent, electronic agents asking you questions to get you to the right information, that’s generally us as well. And we’ve, we’ve taken that technology and the, all the battle hardening associated with it and we’ve migrated it to the, to the healthcare use case. And this is what has allowed us to really accelerate because we had to develop that from scratch, we call that conversational AI

Bill Russell:                   09:44                and it would be interested in conversational Ai on top of a call center where I call it into a health system and interact with my medical record, my father in law is great, because he’s 87 years old, he now lives with us and I’d take him to the urgent care center. He just just moved here and he’s getting a new set of doctors and uh, he said, well, what meds are you on? Oh yeah, he pulls out this long list. Well that’s, that’s how a lot of uh, older people carry around their, their, uh, their med list. And there are times where, um, you know, we need to verify something or whatever from his health system. It would be nice to do that. And not all these health systems are connected, right? So you have to give back. Uh, we have some international, uh, listeners and, uh, you know, this conversation is in English. So, I assume a lot of stuff we’re talking about is an English. What other languages are you?

Joe Petro:                    10:40                We have, we have a large variety of, uh, of languages. So you know, our business from healthcare, sports,

Bill Russell:                   10:45                so when you’re doing orthopedics, you’re doing it in multiple languages at the same time.

Joe Petro:                    10:49                We can do it in multiple languages at the same, the same time. This, of course will be released in where, wherever there, that English is the language of medicine, right? So it’ll can be like, uh, easily brought from the United States into places like the UK, even some Asian places where English is the language of medicine. But we’ve got solutions in just about every language that you could possibly imagine. We’ve got great business and you know, France, France, Germany, the Nordics, you know, Australia, obviously the UK. It’s a major expansion opportunity for us. And so we’re really serious about the business. Um, and our, I won’t say that all the solutions are language agnostic, but they’re agnostic enough so that if we get serious about a market where, you know, we can go there.

Bill Russell:                   11:33                Well, uh, biggest obstacle at this point. It’s just time.

Joe Petro:                    11:38                it’s time and you know, it’s time and money. You don’t know how quickly some of these problems are going to yield. Right. And um, I appreciate like what happened in the early speech days because when speech was 75% accurate, our company continue to throw money at the problem until it yielded. And there were a lot of people, I’m sure from the outside that we’re looking at this like when, when you’re running at a 25% error rate, that means like one in four words is wrong. Right? And, but what they saw was, they saw the opportunity to narrow the problem. And so there’s a lesson that we should take from that, that by kind of changing the definition of the problem and moving the technology to some place where it works, it creates a commercial opportunity so that you can keep the business going as you’ve kind of grind through that, that kind of, that, that resistance from a tech point of view.

Joe Petro:                    12:23                So we’ll see how fast the problem is. You know, the problems yield. We’ve got some human assistance and human augmentation where in the virtual scribing, for example, we’re using the virtual scribing to collect data and train the models because that helps us create the truth sets. Um, so it’s gonna it’s, it’s interest is going to take some time. I’m going to take a bunch of money

Bill Russell:                   12:42                cause the inner rates in medicine or have to be administrator

Joe Petro:                    12:46                and this is the pressure that technology actually puts on the physician. And uh, this is just kind of a sign of the times. I mean, the physician is responsible for, you know, delivering the quality of care at a high level of accuracy. So we can’t be putting stuff in front of them that, um, that creates even more problems from that point of view. So we’re very thoughtful about as we roll out solutions into prime time, but we have to be, um, you know, very kind of engaged with what the physician is going through and make sure that we’re not disrupting that experience anymore than you’d have to.

Bill Russell:                   13:16                Well as a CIO for a health system. It was interesting because, because we use your product throughout, um, throughout our clinical setting in, um, there were people that trained it. There’s people who didn’t. So there’s a huge, we got a team that was just dedicated to training them how to use it because we knew that the benefit was so high because the satisfaction rating of the physicians that were using it was really high and, but there was a whole bunch of people that used it. Saw one word wrong and said, I don’t trust it, I can’t use it, you know, um, but we knew it worked and we knew it worked effectively. So we utilize those clinicians to train other clinicians and get them up speed.

Joe Petro:                    13:51                One thing that’s changed relative to that, by the way, because everything’s multitenant and it’s in the cloud now we can go into, we can go into one of our clients and say, this is what your physician population looks like. This is how they’re using the product. Some of them are using macros, some of them are using advanced commands. Um, this is what the correction rates are and we can sit down and have a really informed dialogue with them relative to what are your very best physicians doing and how can we take that best practice and actually apply it to the rest of the physician community. And that’s super helpful because one of the questions I always ask when I’m meeting with a client is, you know, what’s your adoption rates? So they might say 75% so that, okay, so there’s a category that we need to revisit, like one of the 25% doing why isn’t the tech there? But then within the 75%, how many of them are using it efficiently, high productivity, it’s an opportunity there as well. So we use this to kind of continually improve and make the client successful.

Bill Russell:                   14:45                That’s one of the first big data projects we did is we collected all the, the uh, interactions with the Ehr and we identified the physicians that were struggling with the Ehr just based on the raw data that was coming back to us. And, uh, these are the advantage of the cloud, advantage of big data, advantage of machine learning, identify these patterns. This is fantastic. Thank you very much for your time. I really appreciate it. Keep up the good work. Thank you.

Bill Russell:                   15:11                I hope you enjoyed this conversation. This shows is a production of this week in health it for more great content. You can check out our [email protected] or the youtube channel at Thanks for listening. That’s all for now.


Recent Episodes