Newsday – Deep Dive into Sky Lakes Ransomware Event with Karl West
Episode 417: Show Notes
June 21, 2021: It’s Newsday with Karl West, Chief Information Security Officer, Healthcare at Sirius Healthcare Solutions and former CISO for Intermountain Healthcare. With an extensive history in healthcare cybersecurity, he and Bill complete a deep dive on Sky Lakes Medical phishing incident and subsequent ransomware attack. How can security teams prevent these emails from ever reaching their organization’s inbox? How can training combat the heightened sophistication of ransomware attackers? What can your health system do to stay updated on the latest phishing campaigns and attacks? Segmentation can go a long way in preventing damage in a ransomware attack, but 80% of health systems may not have had the resources to prioritize this. Does your health system have a plan and prioritization for if you’re breached? Create tiered response and keep it brief before it is needed.
welcome to this. We can help it. It’s news day. And today we’re going to go through the sky lakes ransomware event in detail with former CSO for Intermountain Karl west. My name is bill Russell. I’m a former CIO for a 16 hospital system and creative this week in health. It a channel dedicated to keeping health it staff current.
And engaged, especially thanks to serious health care, our new state sponsor for today’s show and health lyrics and WWT, and for investing in our mission to develop the next generation of health leaders. If you want to be a part of our mission as well. You can become a show sponsor, send an email to [email protected] a quick note, check out our latest article on the changing role CIO.
This has gotten a ton of hits. It’s a great article. It highlights BJ more. Ed marks, William welders trusses, spring men, Craig Richard Ville amongst others. The role is changing. You want to hear how it’s changing from those people who are living it all right onto today’s show. Okay, today, it is a Newsday episode and we’re going to do something a little different.
We’re going to camp on one topic, one story, and that is a ransomware and the attack specifically on sky lakes, medical center. And today we have Carl West with us, with Sirius computer. How how are you doing Carl?
I am good. And you’ve picked an awesome topic, bill
really it’s relevant to these days.
This is a conversation everywhere I go. And so you’ve picked a, and certainly the precedent is having this conversation right now at in his world.
I was shocked to find this video, to be honest with you. I was shocked for a couple reasons. One is as a CIO. Deloitte was our internal auditor and they forbid me from talking about our security posture with the media.
They just flat out said no articles and whatnot. Do not talk to the media. Now, if you want it to share it in a smaller setting, a more controlled setting with just healthcare leaders by all means, go ahead and do that for the good of the industry, but do not allow it to get published in black and white.
And I remember way back when. After I was told that I saw that Derek Dworkin CIO at Cedars wrote an article for the wall street journal on preparedness for healthcare, for security. If you haven’t seen that, article eight is exceptional. It just shows the thinking that Darren has.
And when he got done, I saw him at a conference. I said, man, that was an exceptional article. He goes, I got in so much trouble for writing that article.
Do you know? And in fact, I think most healthcare’s have that posture bill. We don’t want to talk about, we don’t want to broadcast what we’re doing.
It puts a target on everyone’s organization. And so the article you’re referencing actually the video from skylights is it is so telling and behind the scenes bill, many of us as CSOs. We’re calling, talking about, we are calling and talking about what did just happen and we make notes of the details, but to see it so open in every aspect from the attack that’s rev vector, what happened minute by minute, if people have been seen that sky lakes medical center Presentation go out and take a look at that.
It is very telling it’s the kind of thing. Every CSO needs to be looking at understanding what happened, one of their top five or six recommendations, whether they telling us we should be doing and what do we need to be doing? Very informative.
Here’s how we’re going to structure this. You and I are going to go through it and we’ve both watched the video.
I’ve outlined it here. So we’re going to start by. The events that happen. So identifying that you’ve been ransom because there was a time period where things were happening and they were like maybe this is just normal slowness of the network or systems and that kind of stuff, which happens which is part of the process.
We’ve got to determine what it is. So we’re going to identify again, we’re going to do a section where you and I talk about working through it, right? So now, it’s ransomware, what are the steps they took and we’ll go back and forth on that a little bit. And then they do share lessons learned.
And I want to talk about their lessons learned, but I also. One to tap your brain on, what are your lessons that you learned? And I’ll share some of the ones that I learned.
Does that make sense? I guess we’d go into this for our listeners. I would tell you what you’re going to hear is if you’re looking at maturity scale, it was ad hoc.
What occurred is ad hoc and that’s what happens in most cases. So the very first thing I would tell you. As you listened to this and you think, why didn’t they, why didn’t they, you need to have documented processes, some playbooks so that you know exactly what you want your caregiver to do.
Exactly. What’s your essays, your deviates, your network engineers should be doing. And as you listen to this I’m just so pleased at sky lakes was so candid and we’ll hear ad hoc.
Yeah. It’s it’s a cause it’s very easy to sit there and go, oh, why didn’t they, why didn’t they? Why didn’t they?
Okay, so let’s put this in context. Skyline medical center is located in Southern Oregon. Okay. They’re there, they serve a radius of about 75 miles. In other words, 75, in some cases it’s a hundred miles or 140 miles to the next hospital. All right. So this is not in downtown LA. This is pretty remote.
It’s. It’s a community connect site, which means they probably don’t have a significant it budget. I would guess their staff for it is less than 25 is my guess. Just based on size. I didn’t do a lot of research on this. I don’t have any insider information. They have about a hundred thousand patients. So 25 might even be a
I think that’s way too high. I think if they had two or three, I’d be surprised, dedicated to cyber. It’s a small
shop. Yeah. I was talking to 25 total in it because their EHR is being handled by a Santee and others. So it’s a small organization, right? Small organization, small budget, small allocation for cybersecurity.
So I, I would say to people give to cut them some slack. Now we’re going to point them. Yeah. And bill,
this is healthcare in America, so you don’t have to cut them any slack. This is 80% of healthcare in America today.
That’s a little scary, but we’ll come back to that because we’re going to look at both things.
I want to take some time and look at the position that satay was put in as the host system, because we have large health systems that have community connect sites and other types of sites where they’re sharing systems with smaller health systems in their community. That they could be opened up to attack through those sites.
So I want to look at that the host system, but I also really want to look at, the events that went on at this house system and see if we can get some wisdom out of it. So October let’s see, October 26th, you have phone calls, stating systems were slow. The computers were slow. Earlier that day about noon on October 26th.
And the email came in to one of the employees at the medical center. There was an email that had information about bonuses. She clicked on it. We’re going to talk about the email in a minute, but she had met with HR previously and assumed the email was about that meeting within about 30 minutes.
The system went out to a malicious site that delivered the payload to the computer that contained the ransomware code. Her computer froze and she couldn’t control her mouse when she regained access to her mouth, she closed the window and didn’t think much about it, nor would any of us. Your system slows down a window, slows down.
You close that window. You go back and Hey, everything’s working well about an hour later, PowerShell command cobalt strike was enacted. That’s the words that were on the slide. I don’t know if that’s correct or not. I’m not sure. That in depth. In my understanding of the cobalt strike was enacted about 10:50 PM.
The times don’t really match up here. I’d want a better timeline, but I’m just relaying what was communicated in the presentation. The code was a ransomware binary executable. It started encrypting everything it could going across the network. And the CIO was notified at about 3:30 AM. And what they term it would determine was that around 11:00 PM, the system started slowing down.
All CPU usage was being sapped and the files were being locked by ransomware. All right. So Carl let’s walk through this a little bit. That’s a pretty, I would assume that’s a normal or typical ransomware attack. But that presupposes that there is a normal attack is there a normal attack?
is very common. It’s a, the kind of thing I hear every day. And those symptoms, those things that are happening are things around footage, processes, structure needs to be developed. And the CIO is very funny, candid in his video. When he says, how would you feel. When this happens and you get this call we know how we’d all feel.
Just like we’ve just been punched in the gut and we’re all terrified. And I just felt for him as he’s representing here’s, what’s just happened and it’s been happening. So recently I just got a note. I think two and a half weeks ago from the FBI of another 250 healthcare’s who have just been hit.
And this is back in October. Now we’re talking now in may, that folks are being hit again. So let’s,
let’s dig into that a little bit. So I’m going to do it by timeframe. So at noon, she clicks on the email. 10:50 PM is when they’re saying the PowerShell command. Pits is that a window that software or even a monitoring service today would have been able to identify it or prevent it in some way?
I think a number of things that could help our listeners first email education, phishing education. She had something happened later. Other things occur. We need to have our users educated. So they’re alert if we can have a fish hook, a some type of an alert in the mail system, so that if you’re worried, if you’re concerned, you’ve been trained enough through fishing education, Click and this goes right to cyber security and they’re taking a look analyzing the email prior to email coming in.
Many organizations have put into place sandbox technology and what it does. It examines every piece of mail coming into an organization. It’s a best practice. It’s a thing I strongly recommend. There are many providers of this type of technology. It would have detonated and attempted to open. And would have removed this should have removed it from the email system.
But even if you miss, if you’re doing regular fish campaigns, if you’re educating training your employees, when you see, when you click, be careful, don’t click links, don’t give out credentials. Those are all things that should have been a part of. The process is the end. And I’m not saying anything that sky lakes and their CEO CIO didn’t point out.
He knows that he recognized it after the fact, I think we just have got to get these kinds of products and these kinds of solutions funded.
Yeah. So it’s, multi-layer, I started at the point that the email got clicks, but there that email probably shouldn’t have gotten to that person. So there should have been tools that identified it before it
there. And really build the CSO, his or her role should be, I’ve got to get all of this stuff out of the system because no matter how much I educate, no matter how much I train. These people are good. And this looked like that’s what I caught.
The lady said, I just sat in an HR training and this looked just like what we were told. So someone knew someone was doing reconnaissance inside their organization, knew about a bonus plan, made something, looked like that. We’ve got to get education about what is real, what isn’t we’ve got to have. People are aware of and trained, and we’ve got to have tools to eliminate everything that we can.
And in fact, it goes beyond just fishing because we miss it. Many of the fish attacks because we get numb, we get hit every day by so much. Spam the stuff that used to come to our doors and we would see it and throw it right in the ma and the junk we need as CSOs. We need to be eliminating spam, not only doing the sandbox that I mentioned, but also help your users.
Don’t let them be confused. There’s so much coming in. They’re weary. And then they accidentally click or in this granting best of intentions to this poor lady, it looked like the meeting she had just been in. And that’s how fish attacks are getting through.
You’re kinder than me. I’m going to go. You are a lot kinder than me.
I could just tell by your disposition. I wouldn’t be a little more critical cause he has a picture of the email and it’s an obvious, it’s a. It’s obvious
in a minute and just say, this is why it’s multi-layer that email should easily be picked up by software. Number two, the education would say, you say, look, we should be well beyond flicking on PBS from people we don’t know. At this point or any of those other things. Now there’s a little sophistication in the email but not much.
But then, the sandbox technology and other things. So there’s a lot of things we can do around this. So let’s talk about the email a little bit. The title is anal bonus report PDF. Now the person who was the person who met with HR just said, I’ve met with HR. And, but it wasn’t really a meeting around her bonus or anything to do with payroll, but this, no, this is the kind of email that just blanketed one out.
In fact, we’re going to hear later this same email. Is the one that took down St. Lawrence in New York, the same exact email. So what is predicating on is we, hackers will sit there and go, you know what, we’re gonna send this out to a thousand people. I bet you, at least five of them met with HR this week.
And we’ll say, Hey here’s a note from HR here’s information on your bonus. And I’m looking at this thing. The links to the files are, just Google docs. That would be a red flag for starters, especially if you’re not a Google docs kind of place, if you’re a Microsoft place, you’re probably not using a lot of Google docs for internal emails as my guests.
The, it has things like, bonus reports like here has PDF click here, expanded preview, click here. If the document and it has a document name and that’s a link as well. And then it has a, if it doesn’t open automatically click here, another link. And then it says published by Google drive dash report abuse.
And that’s a little sophisticated, but, I just had a conversation. I just did a a show with the people from proof point. And they were talking to me about how sophisticated they’re getting now. They’re watching your social media and other things, and they are now coming to new saying, Hey, you got an email Hey, welcome back from vacation.
We missed you while you were gone. Love to catch up with you on the project that’s going on before we do that, can you review the document and let me know if the budget is on track for the new tower, right? So it did enough. If they did enough research to know your organization’s doing a new tower, they did enough research on you to know you, you were just on vacation.
And that kind of email was just looks like a run of the mill. I’m sending it to you saying, Hey, what’s the status. That’s a lot more sophisticated and probably gets through the net of the. The initial the software that would be identifying that kind of email.
Yeah. We need to be running fish campaigns in our organizations and many people don’t do favors to their organizations cause they make the fish attack look very bad, very easy to spot and the things that are happening.
Are much more aligned with what’s going on now inside our organizations, because these bad actors know what happens. So they’re making it look just like a regular review of your your information as an employee or. Your annual review process. And so I think we have to just train, teach, look at the Earls.
You said that you spotted it right off. I looked at it and thought, why would you have clicked this? And oftentimes someone will ask me, will you look at this? And I just say first hover on that link. What does that link look like it’s going to HR? Does that look like it came from HR? And if we can do those kinds of things and train that, and then say question, if you have any questions forward that over to your cyber team, have them take a look at it for you.
I think the other thing that needs to be done here is in this case, you’d point out and the CIO points out from skylights. This was the rye grass, and we’re coming out of Russia. Lots of intelligence, lots of threat information around that. So in cyber, in our network teams, we’ve got to be harvesting, mining that information, the bad actors do.
That’s how they know what we’re doing. We need to know what they’re doing, which means we need to have a program to block known bad sites. So going further down the chronology of what occurs. Now we’ve got to go to a site, click on it, and we’ve got to receive information from a bad site. Those are things we need to be doing in our hygiene block, known bad sites.
Keep up on those kinds of things. The information comes from the HII SAC there’s information available from HHS from the FBI. You can go out and subscribe. FBI will send you threat intelligence, Homeland security. We’ll send. And if we keep up on that, we can block many of these things before they can get started.
Phishing is still the number one way. They’re getting it. Isn’t it.
It is. And if you take a look bill at 4 0 5 D out of Washington, HHS in a partnership with the provider organizations put out a list, what are the top risks? Email is a number one on their list. And it’s these phishing attacks and having appropriate email protection has got to be on everyone’s radar.
Carl, what’s going to replace email so that we don’t have this problem, or is it not going to matter? Is there like a slack equivalent of this.
I’m starting to see now on my phone bill almost as much spam coming to my phone, independent of the mail system, coming to the channels that I use, my actual text messaging system.
And they’re all the same kinds of. Things that have a link for me to collect, to go out and take a look at. And some of them are selling. Some of them are advertising. Some of them are just malicious. It is going to spread to those other channels. It is spreading.
Yeah. And I don’t want to hear people. It might sound like I’m being critical of the person who clicked on the email, but at the end of the day, we’re all at the level of our education, right?
So if that person hadn’t gone through cyber education and those kinds of things, she would not have been aware that this exists, which is what you’re talking about. And you’re making people aware of during those campaigns internal campaigns, so that people can practice, they can learn those kinds of things.
And I’m not saying that I’m not a target of phishing campaigns or could succumb to them switching campaign. I’m sure I could. And I’ve talked to many CEOs who have as part of those campaigns, they do internally. Eventually they’ll click on one of them and go, ah, I can’t believe I did that.
But it’s hard. You’re moving fast throughout the day. And sometimes you’re trying to get through a hundred emails and you just, you just happen to click on it as a matter of habit,
yeah. And you hit some important kinds of points that and you’re talking about detection.
Can I, as a user detect, this is anomalous, this is wrong. And detection. If you look at my responsibility as a CSO, an organization’s responsibility, we’ve got to educate those users. But we’ve also got to have detection systems. In addition to the sandbox that I mentioned I listened to the CIO who said they had an endpoint detection system.
It wasn’t fully deployed. It was in an initial phase of rollout was what I understood. And I think. That becomes so important. If you look at this risk that you identified and it’s number one on HHS central CRS, a threat risks for healthcare, we’ve got to have a detection system. One half of a one part of it is me as a user.
What can I do as a nurse, as a physician, as a caregiver that I can spot this, but what can the system do to help me and really having that end point that server-based protection. All around the environment and you’ve got to measure, monitor, make sure you’re not in the initial phases. You’re not half-baked with the cake that needs to be complete.
That is, that has got to be done and that’s going to help us. So the detection, even if my caregiver missed that, like she did, I needed to have that end point fully rolled out so that it could have protected everything that happened after it got through.
So let me keep going through the, the identifying it whatnot.
It was three days later, actually October 30th, the FBI put out their warning, that 400 health systems were going to be targeted. And three of them had already come to it. That was sky lakes, university of Vermont and St. Lawrence health system. That’s not completely succumbed to it. Now. That’s not to say that others didn’t have infiltrations and issues, but they may have.
Had the right software in place to limit its impact or or to detect it early and those kinds of things. But those three L systems had significant events. And as I mentioned earlier, St. Lawrence was the same exact email. Do we have a way that we’re sharing that information like with other health systems so that they can be prepared?
Yeah I think a couple of ways people can know that first joining the, a part of HII SAC. That’s where this information comes from. They run a threat intelligence center. That threat intelligence center is connected to another center. That’s run by the federal government. All this information flows through those environments.
We can also encourage all of our listeners. Go to FBI or to Homeland security, web pages, sign up, ask for their threat intelligence. They will send it to you. In fact, it flooded my mailbox, but it was important. Every day I get 50, 60 pieces of email about what is happening coming from FBI from Homeland security.
This H AIESEC feed that I talked about. It will tell you. All this information that the press is sending out, but if you’re relying on Google, you’re a day late, maybe already hit. As in the case, you mentioned October 26 there hip four days later, information comes out. It’s for them a little bit. Yeah, a little too late.
The I knew some I talked to some CEOs who were on that call with the FBI. I think it was posted actually these, the systems being hacked. And one of the complaints they had was, the FBI was kept the party line, which is we can’t comment on a current investigation. We can’t comment on the current investigation now.
It was pretty close to when it happened. And I understand that response. But they’re there. Is there a way to get that kind of information because you really want it on the same day. This is the email that’s floating around. Is, are these groups that you’re talking about that current?
I think the recommendations they give are very consistent.
They’re very, I’ve sat through. I have secret clearance with the FBI have sat through regular briefings over the past many years. They are consistent. They’re predictable. They’re going to tell you, watch get the signature. Value is not new. What happened October 28 is not new right out. We have been watching monitoring in organizations for three or four years.
If you already know this signature. Update your files, update your perimeters update to our endpoints to make sure of the most current, I don’t have to wait for sky lakes to send me the information, say, here’s the specific signature. There are other recommendations that always come out they’ll they always, and you’ll see at the end of this video, they give you their seven or eight.
This was critical. This was high. These were things we needed to do. Always in that list of things. What we’ve already began to talk about in addition, certainly multifactor and NEC network segmentation, which will, I know we’ll come to in our conversation, but those are critical kinds of things. And the fish education that we’ve talked about, the sandbox technology, they’re always consistent recommendations.
And if you look at sky lakes and say, what did they do? Those are the things. So what should I be learning from this? Don’t wait for the threat intelligence to come from from Homeland security or HII sec, do those things now get your MFA in place. Get your segmentation in place. Email, fish education, going on, get your detection built up block known bad sites.
Do that right now. Don’t wait for another strain of right from the
Russians. All right, so I’m going to get us through this. So identifying it. Here’s our first section. And the last thing I would say is something you already mentioned. Philbin was trending up in the community. So people are preoccupied and they already had a network ticket that the network was running slow.
So there’s a bias towards that being the problem initially. And as you said, they had endpoint security that was not fully deployed and not fully configured. So that’s where we were at. I’m going to guess that if I asked you this question, you’re going to say very common, but how common is that situation across healthcare where a security software is not fully deployed or fully configured?
Very common when I said 80%, really 80% of the health care in America is what we consider small. And that’s what we’re looking at here. And then there’s the medium organizations. And then there’s those very large. I think if you looked at the large. They are not going to fall into this, but the large represents three to 5%, 10% of our, all of our healthcare infrastructure.
So this is very common in 80% of what’s going on in healthcare.
All right. So let’s start working through it. They identify that it’s a. That is a ransomware attack. And I keep wanting to say malware, is it a type of malware or is it is
a type of, it is a type of malware and it is financially motivated with different types of instruments that are delivered.
There may be a financial, there may be an IP. There may be. They may be looking to steal data and collect the data and sell it. In this case, it was all about money. And these, it was quite clear if you read the, if you read the stories that have happened recently, these folks just want money and they don’t care that you’re a healthcare they’re just about getting
Yep. I agree. So initially, they’re working with their cyber insurance company who says, look, you need to. Bring in some third parties and they bring in a Cisco Talos to bring in you. The Telus was, and they split them up. So telos determine the root cause because you need to determine the root cause.
And the other one was recover. The other company was recover offline systems. All right. So they split that up. The next are he has these five things that they did that in, and I’m not sure they’re exactly in order, but we’ll just go with it. So the first steps were segmented, the network shut off all systems.
I would think at this point, when you say segment the network item, plug all the routers, sub switches and everything. I’m not sure I’m configuring anything. I It’s just shut it down, make sure it can’t propagate any more. And I think that’s what he’s communicating here, that once you identify that this is happening, It’s too late.
You have to do everything you can to cut off access. Is that the right first step?
Correct. And if you have a good plan, you don’t have to do that. If you’re segmented, then you can say, ah, this is only in segment a or B or C shut off that segment. But again, for most organizations and this is a significant concern to HHS.
They recognize that healthcare runs flat networks. Everyone can go anywhere. And so that why they described this as that’s what they had to do, they did not have segmentation, no way to detect to to separate what’s going on, but a good strategy and a plan would include on the back end. I know where my data is.
I know where the virus is, the malware. Can I just segment that one piece, some organizations that I have talked to have been able to successfully do that, which is why if we see that 400 were hit it’s wide, probably we didn’t see all of those stories like skylight somewhere. They were able to catch it segment, do some cyber hygiene to prevent.
The rapid spread of this virus through the organization. If you don’t have that is what’s been occurred. And I think one other piece you talk about how he talks. He as the CIO, I was talking about this, he talked about different processes. I would stress to our listeners three processes.
You must understand, you must know the difference when you get hit detection, response and recovery. They are not the same. They’re not even close to the same. What causes. This rapid re-infection think of COVID. We think we’ve got a handle on it. And then it comes back at a resurgence. The problem that occurs is people go from detection before they’ve completely detected it and they start their response and then they recover systems.
Then those systems get infected and we have a resurgence understand the difference between detection, response, and recovery. Don’t move until, the difference until, you’ve completely identified in a good shop. You should be measuring how long to detect and know that you have the threat vector completely understood how long to respond, how long to recover.
If you measure those three things. You’ll know how you’re improving in your capability and maturity. And that’s what we want to be doing. If we’re building healthy, good resilient infrastructure, like president Biden is stressing. Start measuring. How long to detect and know that we’ve got that right. How long to respond, which means I’m down, I’m hurt.
How long does it take me to fix that? And then recover, get the EMR back up gifty O 365, the imaging
backup. It’s interesting because you as you would imagine, once you shut off the network and all the systems, their communication was really shot, they had cell phones. And they had Cisco WebEx teams.
And so that was the two forms of communication that they had. So they said we got to get our nurses and clinicians communicating again. So that Vocera, they brought that back up as quickly as they could. And it was immediately infected again. So that’s that the point you’re making
Yeah, exactly. Bill. When I read, when I listened to that, I thought, oh gosh, he didn’t get, they didn’t get detection complete. And it’s a little bit like what we have just witnessed. It’s very. Analogous to what’s just been going on for us in COVID. We think we understand it. We start down a path of opening up and then we have a resurgence and we’ve got to understand, do we have this contained?
We know exactly where it came from and we have containment. Now you can move to this respond and recovery, and each are different phases.
All right. I want to camp on this one for a minute. Next step was B he contacts, Leah Santa’s health system CIO. All right. So keep this in mind that at noon, the previous day they were infected and he contacts the Assate C I O at 7:27 AM.
The question that’s not really answered. He said we cut off connectivity. I don’t want to know when they cut off connectivity, because my guess is. Given the sophistication of their response, why not? They really didn’t cut off connectivity until the CIO for a satay said you’ve been infected.
We’re cutting off connectivity. That’s my guess is what happened. And I guess my question to you is if you’re the host EHR systems, CIO, Is it appropriate to cut off access? You’re cutting off access to their EHR, essentially
with every organization that I don’t want to mention names of organizations because they become targets, but the playbook should call for both sides.
If I have a partnership with epic, with Cerner, with Agfa, I should have a process that says we cut. You’ve got, we both cut. We both have, and we know exactly how to do that. And we know the consequences of it. In many instances, I have seen organizations proactively cut because they see something happening through a third party partner and they have a playbook that tells them, cut the network.
And what happens is that organizations will be scanning. They’ll see something happening. They will sever both organizations have to have that cyber hygiene in place so that they isolate and segment their systems and networks. And yesterday your question that that , who’s their EMR provider, their third-party partner should have processed.
They should have been detecting and probably. They were scared, stiff when they called back the next day, they were probably asking this question, do you know the source of the threat? Can you tell us that? And if you can’t, we’re all segmented, we’re all separated. That’s what you must do until they can identify.
And literally it would be best practicing your playbook. Never reestablish that link and tell the CSO comes back and tells you. I know if they tell you, oh, we’re bringing back up systems, but they can’t answer question one. If you don’t want to reconnect. You weren’t part of their poor cyber hygiene.
Yeah. That’s you went exactly where I’m going next, which is later on in the presentation. Sunday comes back to sky lakes and says, look and it was disconnected at that point. It’s seven 30 or when, or earlier that it was disconnected, they came back and said, look, in order to reconnect.
Here’s what we want. We want the steps that you’re going to take. We want the build documentation because essentially you’re relaunching, it’s almost 30 days of downtime. You’re relaunching the EHR and all the connectivity and all the systems that you’re going to connect them. They made them sign, didn’t make them sign.
They asked them to sign a memorandum of understanding and the memorandum of understanding asked for four main things. According to this presentation, third-party clean bill of health. So third-party to come in and audit them. Which makes sense. Cause they probably didn’t have that sophisticated audit capability internally annual risk assessments with pen testing.
So penetration testing. That’s interesting. Annual may not be enough, but it might be, what they can afford kind of thing. Incident notification more timely incident notification. My guess is as a satay was looking at this, they’re like, Hey, we were exposed to something for the better part of 12 to potentially.
19 hours. So we want incident notification and more timely response. And then they have D is interesting NIST, V2, frameworks, security, posture, and culture. So they’re asking them to implement one of the security, frameworks and chapters to be in this version two. Is, are there other things you would add to the memorandum of understanding, or is this a good start?
That is a
great start. And I think as I look through that, I thought they’ve done a good job. They brought in a good, they did some great things. They brought in good partners. They separated processes. I think I think in a list of things I would want to know what is the signature? What’s the strain of this virus is endpoint protection complete.
Did you do some scans? Is there no Riach anywhere else in the environment, in an offline, in a segmented once you’ve completely eliminated than that piece of the process is good. Now you can move on.
All right. So we camped on that for a while. If you’re a CIO for a health system today that has some community connect partners or other EHR partners, then you’re serving them.
It’s if you don’t have this kind of language in your contracts, w would you go back today and make sure that language is cleaned up?
Definitely. And I think I’d start first on my processes internally, and I would, the processes internally have to involve. Your ability and knowledge of how to segment, if something occurs last year, bill 2020 HHS reported that 25% of breach occurred to third party partners, which means it happened to someone else, but because of co the day.
We know about this spread and because of cyber hygiene, things are spreading from third-party to the primary health care. Make sure you know about your BAA, make sure you have language around our ability to sever your ability, your responsibility, to protect me by severing and probably in your process.
You need to make sure you have phone numbers. I’ve seen many people at seven 30 in the morning. Like we heard the CIO saying. Where is the phone number for the CIO of a Santa? I can’t remember where that was. It’s not on my speed. Dial my computer’s down. You need to have a playbook where, where is this Santee?
Where is epic? Where is Cerner? Where is my key providers? And we’ve got to be able to quickly call them notify were infected or expect the notification process from them.
All right. Let me take you through the rest of what they did here. So they established command center. That’s a pretty obvious and standard procedure in something like this.
They had to go to huddles communication obviously was very stilted. So not only was there a command center around this, but command center around caring for the patients and whatnot, making sure that all those processes were in place, they had to start prioritizing servers and bringing them back up.
He talks about the fact that they had a lot of different lists of, what was the most important thing to bring back up and whatnot? They finally determined that the place to start with cancer. But before I get there, I want to talk about some other things that what they did is they created a, an environment where they had what they called the dirty Zeeland.
So that’s the infected, the land. They had the staging B land, where they could bring systems back up, see how they acted for a period of time. I would imagine. And if they remain clean and whatnot, then they could then move them into the clean V land. So they created that segmentation so that they could start to bring systems up and online.
I assume that’s pretty much best practice there.
Yeah. In spite of all the things they did bad, there are many things they did well, and that’s a pattern I would recommend our listeners listen to this. Review it, put it into your processes and playbooks. If you get to this point and cross our fingers, we don’t get there.
That is best practice right there. And I think some things you can do in advance, they talk about they’re doing this and at the same time, they’re prioritizing service, a service B application Y and Z. That should be a process you go through now in advanced. It’s called the criticality matrix. You build a matrix and say, I need, what do we all need?
The EMR that’s number one. We all know that. And so put that as number one, but before you finish notice what services are required to have the EMR up. You have layer one physical layer, two network layer, three is application. What’s it. That lowest layer that the EMR. Must have you can’t bring the EMR up and tell your hardware is up.
Your network is up. Your database is up. So a criticality matrix tells me what are the critical applications and what do they depend on in order to be up? Interfaces need to be up. I need to have hardware. I need to have network build the criticality matrix now, so that in the middle of the crisis, you know exactly number one is, and here’s the services required.
Uh, Number two is here’s the services required and that will help you in this process.
Absolutely. We brought a guy in from financial services who helped us with that. He called it something else, but essentially what we did is we identified. Our criticality, we identified tier one, two and three apps and tier one, two and three systems.
And he was adamant that we keep the documentation simple. He goes, cause when you’re flying around, you don’t want to pull out the binder. That’s this thick and people go, ah, you, you need to pull out the piece of paper that says. Here are the here’s the list of applications or two pieces of paper says, here’s the list of applications in the order.
And like you said, he came up with multiple lists right here on the fly. That’s something we can do ahead of time. Very, not easily. It does take some work. It took us a while to really think through that because. You don’t realize how intercut healthcare everything’s interconnected. It’s crazy.
Yeah. A real quick recommendation for our listeners. As you look at this exercise, you can make it simple. What is life critical? It must be up in three hours, two hours. Put a bucket around that. What could I do without, for 12 hours, 24 hours? What could I do without for a week? And what could I go?
30 days? That’s if you’re dealing with the big organization that does business continuity and disaster recovery, they’ll ask you to put things into buckets like that. So as you built your criticality matrix, just start to think. What’s life critical. What do I have to have? I’ve got patients in hospitals.
We’re going to go to divert mode. Patients are going to be at, in life critical situations. What is that category? What’s next day? What’s couple of days a week. What’s
the this is one of the more interesting slash so they had to rebuild 2,500 PCs, essentially any PC that was connected to the network at the time of the infection, they decided they were going to rebuild.
And they had to replace about 680 legacy PCs as well in the process. I don’t know about you, but I went through a significant process while at St. Joe’s to get us down to six images. And it was Herculean effort to get us to six images. You’re smiling cause it’s not, it was hard to get to that.
2,500 PCs in this kind of organization. How many images do you think they have.
The inner mountain, we had 13 and it was. It was like pulling teeth from a bear to get to 13. It was very hard. And if you don’t have that, what it means if they don’t have, and don’t know how many they have, it means they probably have many images that are customized for everyone.
And so if you can get that’s another critical kind of a thing. Build that as we come down to, what should you do in a future model? You need to have workstation classification, which means. Tell me what it is. Is this a blue, a red, a green what’s the classification of this workstation and that rebuild that they did because they didn’t have scanning and ability to do.
It’s probably best practice. If you have internal scanners ability to do could have prevented some of that. But those were things probably just at the last minute flying, like they were, they just said. Best thing to do start claim. Yep.
And it probably makes sense. A third-party systems, Pyxis, diagnostic imaging, CTS, MRIs, x-rays and other smart pumps.
So you have all these devices online which and even monitors, they talk about the fact that they’ve disconnected at all from the network. So the monitor still work. They collect all the information locally. But at some point you’re going to bring them back onto the network and you have to verify and certify those devices.
What’s best practices around those devices or does it depend on the device?
Yeah. You hit a kind of, a bit of a sore spot for me in the middle of fighting the fire, which is. And I just was so sensitive as I watched this, I thought, oh man, they were in the middle of the biggest battle to fight, but there’s an obligation or responsibility to be contacting the third-party partners.
Probably you need a partnership with someone in communications because your team’s not going to have time while they’re trying to scan and determine, and detect and respond and recover. So you need a process that says contact the partners. So that they can protect themselves, give them the signature, let them know what it looked like, what it is, and make sure that they’re protected.
And yeah, that, as I heard that I that would have been a huge it is a huge help, but in the middle of the fight of the fire that they’re dealing with, it, isn’t something they take a look at.
Yeah. They I’m just going to keep moving through this. The if people want, deeper answers, they can contact you.
You bet. They go on and say, they, we did their their password policy. And I think this was dictated by the vendors. To be honest with you, they came in and said, what’s your password policy now that’s not good enough. And they made him hardened that across the board.
They bring up the cancer system. And this is a probably, I don’t know, maybe 20 days in 25 to 20, about 20 days in, they bring up the cancer center, but he also talks about these side roads. Like they didn’t expect there to be a snow storm and the snow storms coming and they said, Hey wait.
Our heating systems and some of these things are connected up to the computers. If we don’t know how to turn this stuff on without this. And that’s a good point here. He also talks about, they had clinicians that had only worked in the EHR world. They had not worked in the paper world, so they weren’t they weren’t used to paper processes.
They weren’t, they don’t know how to turn on these systems that are all digitally controlled. And those kinds of things are going to come up as you’re. As you’re laser focused on the clinical systems, those kinds of sidetrack things are going to come up and they get bumbled up, through the command center.
I think it’s a great lesson learned for our listeners bill, as you go through the processes of thinking ahead, always in it. And in cybersecurity, we will think of the technology kinds of things, but the business processes have to be thought through, which means if you’re putting together a good business continuity plan, you need.
You need home care, you need hospice. You need pharmacy sitting in the room because for them to survive the CML, the CNL for them to survive an outage. They have to know what are their processes that are not it processes. And if we learned anything from what has just occurred in our gas pipeline, shut down in our meat packaging, shut down.
That just occurred again, some Russian attackers. What we need to learn and understand is. The business process, not the it process. It is part of enabling the business, look at the whole business process and say, what would we do if, and what’s going to occur in the absence of that kind of a plan. The Cecil is going to be under immense pressure, just restore everything.
And he, or she is going to be fighting to detect. And the business is saying, I have to have this up. I can’t do ophthalmology without therefore you must just bring me up first. If you have a thought out plan where you say, what is your business continuity? What did it look like before we had technology?
What can you do in a non. The technology-based process, have those plans ready, have them build out. It’s not an it thing. It usually has led in good sophisticated organizations either by the risk officer or by the chief medical officer, but it and cyber have to sit there and be a part of that conversation.
Okay, Carl. So here. So we’re going to transition to the last section, which is what did they learn? And they have on the first slide, big takeaways that you have to have the tools of the old ways to make it work. In other words, paper, toner, paper, processes, those kind of things you have to because they had, in his words, they had good processes to be down for a day.
They had bad processes to be down for 30 days. And I think that’s true across the board. I can’t tell you. That we had a good process to be down for 30 days at St. Joe’s that would have been, to have the data center down to have the primary clinical systems down for 30 days. I’m not sure many health systems do have that kind of thing in place.
Let me go through the other. So there are providers. Who only knew the EMR not paper, which is what we talked about. And they said there’s a massive backlog of page paper post outage. So there’s they had to restore the data. They had to plan on the data that was being generated as they were down that had to be reentered and all that stuff.
So those were three of his key findings. You talked about key findings around preventative prevention. He said you need to have good backups security operations center seven by 24, 365 education as first line of defense playbook for extended outages and plan for a rapid deployment of systems. And those are pretty good and strategic recommendations and priority.
Yes, implemented multi-factor authentication as high continuous monitoring as high medium was centralized log repository. Medium was incident response team on retainer and low was incident response plans and playbook. And security awareness program. So those were the major findings. I’m sorry to go through those real quick, but as is usually the case, when you and I talk, we have so much to talk about what, w what other learnings, or what do you take from from take away from this incident?
I’d prioritize a little differently. I would tell people, spend an invest time on your incident detection, response management monitoring that is going to, you cannot prevent this from happening. And the CIO concludes that what you can do is reduce the impact. If you can detect and respond and recover quickly, spend time and invest in network segmentation.
They didn’t talk about patch and patch management. What makes us system susceptible as they forgot to get their vaccine. And this is not a political and it’s not moral. Just get the vaccine and vaccine your systems. It’s called patch management. You need to be doing that. I think that’s so critical building the BCDR plan identity.
Wasn’t talked about heavily here, but I would tell you I’d invest time, spend time understanding the identity. Where should this identity be used? Is it okay for four bills in case this lady, Susie or whatever her name can Susie’s identity be used in Russia? If we had identity management, we would have said, that’s anomalous.
Why is Susie’s identity being used in Russia? Catch that even if you missed everything else, get an identity management strategy, the multi-factor strategy. Those are things I would encourage our users to take a look at.
Yeah, and I would only add to that, buying the buying the right security software is not costly.
You have to get it installed and configured correctly. And by the way, I would check those configurations on a pretty regular basis because you just have to, things are changing and whatnot. I do want to touch on one comment. I hear over and over again, and it just sits the wrong way with me.
And that is it’s not if, but when. So everybody’s sentence. It’s not if we get hacked, but when, and when I hear that there’s something that’s defeatist about it. And there’s something that says, look, I’ll agree with this. It’s not if you’re going to be attacked, but when you’re going to be attacked and it’s also not if you’re going to be infiltrated, but when you’re going to be infiltrated, But it shouldn’t be, it’s not if they’re going to shut down the EHR, but when they’re going to shut down the VHR, there’s different levels of where that statement is true. Yeah. You’re going to be attacked. Yes. They’re going to get in and yes, they might even get to one or two key systems, but yeah, it should be so hard. For them to get to the EHR and shut it down the PAC system and completely annihilate that they completely annihilated their packs.
So they had to bring up a different pack system altogether. When I I’m curious when you hear that, it’s not if, but when, it doesn’t sit well with me, but I’m curious what your thoughts are on it. I like
what you’re saying, and I tell you, that’s why bill, when they said incident response would be a low, I put it at a high or a critical, because if you can detect, respond and recover quickly, the damage is minimal organizations.
An organization I used to work at would see seven to 10, right? Like attacks per week. And the difference was, if you can detect that in minutes, respond and recover in minutes, they’re not going to affect, they’re not going to take down, which is what you’re saying. Don’t say it’ll never happen. That’s a fabulous kind of a view it’s going to happen, but we can minimize the effects.
If we have capabilities in our systems. To protect, detect, respond, recover, get that right. And you can sleep well at night and you can say it will happen seven times a week, eight times, 10 times. We can route that out before it has any impact. We took the cyber hygiene and we did that that necessary step.
Carl. I want to thank you for your time. I know this is a busy time for you and I really appreciate you going through this, taking the time to watch the video and then going through with me. I really appreciate
it, bill. Thanks you for educating. This needs to be out there. We need to invest more money.
That’s one of our issues as well for us to improve. President Biden is pushing. We’ve got to invest those healthcare and you educating is so helpful. Thank you, bill.
Appreciate it. Take care. Okay, thank
What a great conversation. I really appreciate Carl coming on the show. If you want more on this topic, specifically, the sky lakes ransomware event, go ahead and hit today in health.
It’s the daily show that I do. I did three episodes on this last week and we go into more detail, actually not we, I go into more detail. It’s a show where I just take one news story and break it apart. So three, three days, Tuesday, Wednesday, Thursday of last week. And I think it’s one of the most important topics that’s facing healthcare right now.
All right. That’s all for today. If you know someone that might benefit from our channel, please forward them a note, let them know that they can subscribe on our website this week, health.com or wherever you listen to podcasts, apple, Google, overcast, Spotify, Stitcher, you get the picture. We are everywhere.
We went. I want to thank our channel sponsors who are investing in our mission to develop the next generation of health leaders. They are VMware Hill-Rom Sturbridge advisors, McAfee and Aruba networks. Thanks for listening. That’s all. .