Joe Petro, Nuance CTO on Ambient Clinical Intelligence Progress vs the Hype


Bill Russell / Joe Petro

About this guest...

Share Now...

Share on linkedin
Share on twitter
Share on facebook
Share on email

Show Sponsor(s)

March 9, 2020: Welcome to another episode of This Week in Health IT for another industry influencer interview. Today on the show we are joined once again by Joe Petro, the CTO for Nuance, for an incredible conversation. In this episode, Joe shares an update on where they’re at in terms of Nuance’s ambient clinical intelligence (ACI), where that is heading next, and what progress they have made since last year. We also dive into the role of accuracy and how it transfers to ACI from technology such as Dragon Medical One. Finally, we talk about the Microsoft agreement that they did and how this partnership has formed over the last few years. So stay tuned for incredible insights from Joe on the latest at Nuance, and what to expect from their presentation
at the HIMSS Conference.

Key Points From This Episode:

  • What Joe and his team from Nuance will be showcasing at HIMSS this year.
  • What they have learned over the last year as people interact with pilot technology.
  • The top takeaways from physicians using Dragon Medical One.
  • Accuracy rates achieved with Dragon Medical One: are they transferrable?
  • More detail about Nuance’s announcement with Microsoft.
  • How the partnership between Nuance and Microsoft developed.
  • The acceleration of ACI in the next year: where will expansion be focused?
  • Understanding Nuance’s conservative approach towards data usage and storage.

Joe Petro, Nuance CTO on Ambient Clinical Intelligence Progress vs the Hype

Episode 190: Transcript – March 9, 2020

This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

[0:00:04.8] BR: Welcome to This Week In Health events where we amplify great thinking with interviews from the floor. My name is Bill Russell, healthcare CIO coach and creator of This Week in Health IT, a set of podcast videos and collaboration events, dedicated to developing the next generation of health leaders. 

If you’re not familiar with our show, every Tuesday, we do Tuesday News Day, we take a look at the news that is going to impact health IT and every Friday, we do an interview with an industry influencer to look at best practices, look at how people are thinking about the industry. A couple times of the year, we forgo that normal pattern to cover events and no week is busier or packed than the HIMSS Chime Week that happens ever spring. Last year we dropped about 13 episodes during the HIMSS Chime Week, this year we’re going to drop many more.

My goal, for this show is that you would learn more from listening to this interviews than you would if you actually went to the show. What I do is I vet a lot of different organizations/ companies’ interview potential and I’m looking for those things which are particularly of interest to health IT based on the trends that are going on based on the work that’s going on within health IT departments across the country.

If you’re wondering, I’ve already vetted hundreds and hundreds of these requests for interviews and so, one of the things that happens is, there’s just not enough time at the show and I wanted to kick this off a little early because I wasn’t able to schedule a time with Joe Petro, the CTO for Nuance, to have a conversation and we had such a great conversation last year at the show.

I wanted to do an update where they’re at in terms of Nuance’s clinical ambient intelligence and where that is heading and what progress they have made since last year. Also, want to talk a little bit about the Microsoft agreement that they did and a handful of other things and he was kind enough and their team was kind enough to make it available prior to the show.

While not from the showroom floor, here’s our first HIMSS interview for HIMSS 2020. Hope you enjoy.


[0:02:01.9] BR: Today we’re joined by Joe Petro, CTO for Nuance in out of Boston. Joe, this is your second time at the show, welcome back. 

[0:02:10.0] JP: Thanks. Thanks for the invite, I appreciate it.

[0:02:12.3] BR: Well, I’m looking forward to this conversation. Last year was really exciting, we met at HIMSS voice, you guys were showing off the ambient clinical intelligence and listening in your booth and I thought it was one of the most exciting things that was going on at the show. Give us an idea of – let’s start with what will you be showcasing this year at HIMSS?

[0:02:32.5] JP: Yeah, sure. You know, at this point, we’re still being a little bit vague on it for probably competitive reasons. But, you know, with a high level, we’re basically showing the next generation of that demonstration that you actually saw last year. We’re coming out with a generally available version of the product here in the next month or so. That’s what we’ll be showcasing.

It’s going to be the super high level, it’s going to be a lot less structured, a lot more real world, the audience is going to participate a little bit. So it’s exciting.

[0:03:01.6] BR: Voice, clinical, in the patient room seems to be sort of the hottest thing going. Talk about what you’ve learned over the past year. You’ve put it out there in pilots and what are you learning? What are you finding as people start to interact with the technology?

[0:03:17.2] JP: Yeah, we’ve got a number of clients that I would call early adopters at this point. I think what we’re basically learning is that this is spot-on in terms of hitting the center of gravity of the pain point, you know, that physicians are feeling, what this whole thing, what ACI is really targeted at it, is attempting to remove the screen between the patient and the physician and you know, for a long time now, we’re Dragon Medical One, we’ve always said we’d turn the chair around as this famous crayon drawing that’s been in GEMA, which a little girl actually drew of her sister who is very sick and she drew a picture of her sister kind of sitting on the exam table and her mom and her other sibling sitting next to her but with the physician turned around, sitting in a chair, typing on a computer.

That’s really what this is targeted at, we’re trying to turn the chair, continuing to turn the chair around, kind of extending on the Dragon Medical One, you know, proposition of relieving physician burden, trying to bring back some of that joy of care to the physician and also from a patient point of view, you know, improve the experience for ourselves as well.

So that’s fundamentally what it’s targeted at and it seems to be spot on and it’s been overwhelmingly positive, the feedback that we’ve gotten from the physicians that are on the system.

[0:04:27.4] BR: Are you learning anything in regards to Dragon One Medical is microphone, pretty close to the physician, generally in a quiet area but when you’re doing this in sort of a room setting, you’re probably learning, I would assume, a ton of things in terms of positioning the microphones, picking up the video, picking up visual cues and those kind of things. Are you learning anything in those areas?

[0:04:52.1] JP: Yeah there’s a number of kind of mundane things that it’s just important to kind of work through it, you know? As we kind of learn the ropes. Of course, it’s a noisy environment so we’re pushing the state of the art in terms of what we call signal enhancement and noise filtering and we leveraged technology that we’ve brought over from our other divisions. 

Some people don’t realize it but, you know, Nuance has been in the automotive space for a long time. If you’ve got a car and you’re talking to it, the odds are you’re probably talking to Nuance, we’re in 65% of the vehicles out there and that technology has allowed us to really advance the state of the art in terms of signal enhancement and noise filtering because of road noises as well wind noise, as well as kind off inter party noise, kids in the background kind of yelling and so forth and you’re trying to control your car. 

We always expected that that was going to be a challenge because it’s a noisy environment, it’s a multiparty environment, you know? You have a doctor, you could have a number of nurses, you could have a number of family members. Those kind of stuff that we actually expected. There’s some more mundane things like how we actually have the patient opt into the experience is important. You know, what happens is this is fully disclosed to the patient so that the patient understands what’s going on and we understand what’s going on and we understand how this is actually going to help their experience. 

So how you do that, what forms you put in front of them, what signatures you get from them and how you actually explain it, there’s kind of an art form to that of course. Less largely on our clients, not so much on Nuance, but that’s an important step because with all crazy stuff that’ s going out there in terms of privacy and so forth. We want to be very clean on that so we learned a lot there and our opt in rates are still very high, which is awesome because that’s what we wanted to be because we do really, truly believe this is going to be good for everybody. 

There’s other little things like – there were times during the episode of care where you don’t want the machine listening, it might be a very intimate conversation that’s going on between the patient and the physician and the patient might say something like, you know, “Can we shut this down for a minute? I need to talk to you about something,” and then how, you know, you kind of maintain the context to that when you come back in, simple stuff like how do you make sure that all of these various snippets of audio and so forth are kind of stitched together.

Seems like a simple mundane problem but, you know, doing that in a way that maintains the continuity of the dialog so that we can make sense of it, it’s little things like that, but there haven’t been a lot of very big discoveries. This is a hard problem, we know it’s hard. We kind of walked into it knowing it’s hard and the problem has been actually nicely yielding over the course of the last year.

[0:07:19.9] BR: I’m hearing the industry, I mean, people are ecstatic about Dragon Medical One. The people who are using it, cloud based solution, it’s their – the physicians are figuring out, it really has moved pretty far along, pretty rapidly, compared to what I was looking at and maybe six or seven years ago and I’m hearing really good things. Your accuracy rates on that, at least I think they’re hovering in the high 90s, 96, seven, eight percent. Are you able to get those same kind of accuracy rates in the ambient clinical environments?

[0:07:53.9] JP: Yeah, so let me just explain — the answer is no, not yet. This is part of the problem yielding to us but let me kind of explain how we’re creeping up on this. So I joined Nuance about 12 years ago or so and when I got here, the out of the box accuracy on Dragon was okay, it was like 88 or 89%. But the fact is that, you know, 88, 89%, that’s still one in 10 words, you know? That you need to correct.

So even though it would be a good grade on a test maybe if you’re in college, not such a good grade if you’re streaming speech, right? There is no scale here like there is in college. Over the years, we made that yield that what we call 15% relative error rate per year, we basically chipped away at the problem and now as you say, it’s extraordinarily high. Basically, we reached a level of what we call human agreeance.

It’s fundamentally means that if we have a human being listening to the conversation and we have the speech engine listening for the conversation, the speech engine and the human are going to agree in terms of what they heard. ACI is somewhat of a different beast. Number one, it’s conversational in nature. So it’s sloppy the way humans are sloppy, in terms of the way we speak back and forth to each other in conversation.

Versus, if you look at Dragon Medical One, Dragon Medical One is a very controlled, formalized form of clinical documentation delivery. Physicians tend to speak in complete sentences, complete thoughts, You know, they’re trying to do like a soap note, they’re trying to do progress note. They know and they’re kind of pre-contemplating and premeditating what they say. 

So, you know, “Patient presents with complaints of severe headache, patient is on 150 milligrams of XYZ, patient has history of headache.” Basically a very kind of formalized prescriptive type of a speech. But when we’re talking back and forth, if you’re the physician and I’m the patient, it’s sloppy. We’re talking about the kids, we’re talking about the kid’s games, we’re talking about the score of the last football game, we’re talking about some ache in my knee, it’s very sloppy, it’s back and forth, there’s a lot of oh’s and ah’s.

By its very nature, conversational speech has a lower level of accuracy than formal speech, that’s one step kind of down in terms of accuracy level. Then, we have to diarize the speech so you have to take what you said and what I said and separate it so that you can track both of those kind of threads and that is what you apply to speech to text to.

You apply to the patient thread, you apply it to the physician thread or multiple physicians of and possible family members. There’s another accuracy rate that applies to that. And then, once you turn it into text, you start to harvest it for information so that you can summarize it. Because that’s what ambient clinical intelligence does, it summarizes the conversation into clinical documentation that gets embodied in the EMR. It’s another level of accuracy there. 

So the challenge with ACI is you might be multiplying several high accuracies together but there’s this inertia or this pull towards the lower level of accuracy. The question is, what level of accuracy do we need inside of ACI so that the physician accepts the result and they just want to edit it? Okay? It’s easier for this thing to listen and then for the physician to look at the summarized result and actually edit it, will they be willing to do that? That’s a tipping point and we discovered this tipping point in the old days with transcription because you know we went on this journey where transcriptionists were involved. They would do the typing for physicians and even with transcriptionists, we knew we had to get to a certain level of accuracy. 

The magic number with the transcriptionist with kind of a relatively lower paid person, lower paid knowledge worker than a physician, happened to be 85%. Once you got to 85% it was a lot faster to edit it than it was to type it even though it was 15 word to 100 and what we learned in that process is we learned that formatting the document was actually far more important than the speech accuracy and then as speech started getting higher and higher and higher then we flipped it over to the physician. 

And when we started, when we shift that trigger point of 88, 89, 90 physicians started wanting to edit. We don’t know yet because we don’t have any experience. We have a bunch of opinions because we are highly experienced in this field, we don’t really know yet where that trigger point is and that is what we are basically working on. But the problem is yielding nicely from a technical point of view. Accuracies are continuing to go up. We are not hitting any blocking points at this point in time. 

So it’s really, really exciting and we’re looking forward to going through the same type of journeys. 

[0:12:12.2] BR: I mean I appreciate you going into that level of detail. As you know, the listeners of the show are primarily health IT professionals and that is a level of detail they really want to understand because the hype around this is amazing and you guys are tackling the hardest area to tackle and so people are saying, “Hey, you know, we could put Amazon Echo in the patient room and we get this.” Yeah, but you are just asking for a blanket. We’re asking for a nurse to come up. 

That’s a completely different level of accuracy that’s required and risk tolerance quite frankly that’s going on. So you made an announcement with Microsoft earlier this year. Was that about tapping into AI capabilities or what is that all about? 

[0:12:53.8] JP: Yeah it’s a super exciting opportunity for us. So just to tell you a little bit about the history, Microsoft approached us two and a half years ago at HIMSS. A guy named Peter Lee who runs their healthcare business had approached us and they came to me and they were talking about different ways that we could work together and to be perfectly honest whenever kind of a big platform player approaches you, you are not exactly sure why and you’re a little bit skeptical. 

And so we had a healthy well with skepticism in the early days but what ended up happening just to kind of fast forward through a multi-year journey, just started to realize that Peter’s team and Microsoft legitimately were trying to get to the business of improving the situation in health care and really improving the situation with the physician in particular. We talk about all different kinds of ways that we could potentially work together. 

You know, we’ve got something called the AI marketplace, which is a diagnostic in radiology so we do some image processing, we’ve gotten an AI and image algorithms for the detection of certain disease conditions and so forth. So we looked at a whole bunch of stuff but we kept coming back to the ACI problem because they had a project called EmpowerMD, which was similar to ACI. A little bit different but [inaudible] that was right. 

We started to realize they’re thinking about the problem the same way we are and we realized that if we could create a partnership that was a legitimate partnership, it wasn’t just some kind of like “Azure licensing deal” that this could be really good for us because this could help us accelerate our ACI efforts but it could also get Microsoft into the market as well the way they desired and so this kind of spiritual alignment between the two companies and the somewhat friendships we developed with Peter’s team, that’s the thing that kind of field it. 

And so what the partnership is all about is about bringing the net horse power of Microsoft and the way we think about Microsoft as an enterprise class R&D company and solutions company bringing the full horse power from them in terms of AI, their research, all the various things they have in the market as well as privately, marrying that up with the Nuance experiences and history in terms of kind of a documentation expertise, our partnerships with the EMR’s and so forth and attempting to accelerate the ACI problem. 

So as you might imagine, this is a very complex and difficult deal to contemplate and think about for two companies like ours. It builds on our territories but we got through all of that. It has support all the way at the very top. CEO level at Microsoft as well as the CEO level on our side. So it was hard to do but we think it’s very much going to be worth it and it will help accelerate us. 

[0:15:23.7] BR: Yeah that’s it. I think that is the most exciting thing because the expectation on the voice and conversational technologies is that you’re going to alleviate the clinician burnout and the clinician problem. I checked my 87-year-old father in law into a hospital last night and there was literally [inaudible] and there is literally a person dedicated to the keyboard. They were sitting in front of Epic, they were doing all of their stuff, and then nurses and doctors came in and out and after a while I just said to them like, “Hey, if you don’t mind me asking what’s your level?” and she is a nurse practitioner but she really hardly ever left the keyboard because it was required. I mean somebody had to put all of this information in and there was no other way to get the information in. 

So the acceleration of this is really important and last year we talked about you’re doing this in orthopedics but in order to accelerate this, you almost need new vocabularies to move forward. Are we going to be able to accelerate this over the next 12 to 18 months or is this going to be more of a five to six year journey kind of thing? 

[0:16:30.9] JP: Yeah, this is where I could easily follow the life cycle so I won’t do that. You know, the reality is this problem is going to yield overtime. I think it is going to yield in pieces, right? So the way we are doing is we’ve already got over 1.5 million cases of care in our AI training library and that spans five different specialties. So we talked about ortho last year, we are also attacking — I’m just reading off a list here — podiatry, dermatology, ears, nose and throat as well as ophthalmology. 

Those will all come out the early part of this year. So we’ll have support for those specialties inside the ACI foot rent and as I said, please come to the booth when you’re at HIMSS. It’s an open invitation to anybody that’s interested and they can take a look at actually what we’re doing there and then over late 2019 and 2020 we are attacking a remainder of the 18 specialties and they’ll be coming out of the clip of one to two per month over that period of time. 

So I think that the data collection piece of it for us at least because we know how to do this. We have been doing this for a long time, it’s the easier part of it. The vocabulary part of it is also the easier part. The hard part about this is you need to be an expert in each one of these specialties. Like so for example, I just went and did one of these executive physicals a short time ago and I went through several different care settings. So I dealt with a dermatologist, I dealt with an ophthalmologist, I dealt with a general care person, cardiologist, etcetera and if you look at their notes and you look at the way they behave in the treatment room or the exam room everyone is different. 

Like the ophthalmologist hardly said anything to me, right? So they were just doing measurements. They have the thing on your face and they’re making the adjustments, they are working in your eyes and so forth. He really didn’t say anything until the very end and he barely summarized it. I mean he didn’t summarize the measurements for me. So he barely verbalized and so the ACI experience on the ophthalmology setting and what we capture is going to be different than say the dermatology setting. When I was in dermatology, this physician actually had a scribe. A scribe walked in and she sat behind the curtain at the back of the room and the ophthalmologist came up and examined me. 

And everything he pointed to on my body and on my skin he described it in dermatological terms but he also summarized it for me in layman’s terms so that I understood what was going on and whether I should be worried or not. Each one of those care settings was different. So the fines here or the specmanship or the art form to this is going to be each one of these specialties we encounter trying to figure out what has to be in that summarized document, what has to go from the summarized document through the electronic medical record? And so in that respect it’s brand new territory I mean this is where the subject matter expertise and the specialties come and this is how we are chipping our way at it. 

[0:19:10.4] BR: The last question is, we focus a lot of storing the information in the EHR, are we starting to look at storing the information elsewhere and making it available for research, looking at training and these kinds of things in other technologies, data technologies? 

[0:19:23.8] JP: Yeah there are companies that are looking at that. We tend to operate in very, very narrow band around the usage of data. We take a very conservative approach. We basically labelled ourselves as stewards and custodians of the data and we have a very clear conversation with our clients that were borrowing the data for a short period of time so that we can train our models so that we can deliver a very specific service back to you. 

We are not going to derive value from the data, we are not going to try to resell the data, we are not going to try to build products that you’re unaware of. So we have something in our point tracks called the data usage clause that really, really limits our scope. With that said there is a lot going on out there, some of it a little spooky as consumers just in terms of the way our data is used and our health data is used. It is not making me very comfortable. 

And I think there are companies that are starting to get into the business of potentially having secondary monetization of health care data and so that is an interesting space. I think if a company tries to crack it, it will probably end up being a partnership with one of these big IBMs or whatever trying to come up with a way to anonymize the data and then create kind of a honey pot kind of a resource for AI training. That is not the business that we’re in and we can potentially try to stay away from that. 

The other thing is that the EMR for us is our central. It makes our world go round and so we have a very open relationship with the EMR. We try to push everything back to them. 

[0:20:43.7] BR: Joe, thanks for your time. I’l look forward to stopping by the booth at HIMSS. I will likely air this part of the HIMSS show so that people will get a feel for what you guys are going to show even though you were kind of cryptic with us early on. But I understand the marketing aspect of that. 

[0:20:58.0] JP: I’ve got people standing over here off camera by the way that will attack me if I share too much, so I’ve got to be careful. 

[0:21:03.3] BR: Yeah, I understand. Thanks again. I look forward to seeing you at HIMSS. 

[0:21:07.8] JP: All right, thanks man. We’ll see you soon. 


[0:21:11.3] BR: I really want to thank Joe and the team for New Ounce for making him available for this interview. A lot of great things going on in voice conversational technologies, AI and it is exciting to hear how focused they are on making that a reality. 

We want to thank our founding channel sponsors who make this content possible. Health Lyrics, Galen Healthcare, StarBridge Advisers, VMware and Pro-Talent Advisers. If you want to be a part of our mission to develop health leaders, go to for more information. If you want to reach me you can always shoot me an email, [email protected]. I love your feedback, feedback on the show. 

A couple of you have said ideas for future shows, people I should interview and those kinds of thing. Keep the feedback coming it is extremely helpful. This show is a production of This Week in Health IT. For more great content, you can check out the website at or the YouTube channel as well. 

Thanks for listening. That is all for now.