Robert Rice, COO of Health Lyrics joins us to discuss Cloud Infrastructure for Healthcare. We explore 8 years of experience migrating to the cloud with lessons learned and best practices.
Bill Russell: 00:04 Welcome to this week in health it where we discussed the news information emerging thought leaders from across the healthcare industry. This is episode number 29. Today we do a deep dive this, uh, this one is in the cloud infrastructure and migration with our sponsor health lyrics. Find out how to leverage the cloud to new levels of efficiency and productivity. Visit healthlyrics.com to schedule your free consultation. My name is Bill Russell, recovering healthcare cio, right, or an advisor with the previously mentioned health lyrics. Uh, before I get to our guests and updated our listener, Dr. uh, I recorded a brief segment yesterday was Shawna Smith, the executive director for hope builders. And uh, I’m going to play that right now for you as we’re coming to the close of our campaign and support and hope builders and organization that gives disadvantaged youth the job training and life skills that they need in order to succeed. I wanted to share a little bit more about the organization with everyone today. I’m joined by Shawna Smith, the executive director of hope builders. A good morning Shawna, and thanks for joining me today.
Shawna Smith: 00:59 Hi, Bill. It’s really a pleasure to be here. Thank you so much.
Bill Russell: 01:02 Well, I’m. I’m really excited to have you. Here are our great community. As I shared with you, have listeners, has responded and raised close to $2,000 for hope builders over the last couple of weeks. We still have a couple more weeks in our campaign or our sponsor has agreed to give a thousand dollars for every hundred new podcast listeners or youtube subscribers and I just thought it was this great opportunity to connect with you and give the audience a little bit more idea of what you guys do. So I’m just going to throw it to you with a tell us a little bit about hope builders. Where’d you come from? Who Do you serve and what? What are you doing in the community that we live in?
Shawna Smith: 01:39 Well, I think I’ll start with the where we came from question because it’s a really, I think, compelling one and unique one. Actually, you know, almost 25 years ago, a group of four Catholic nuns, the sisters of Saint Joseph of orange were living in a really tough neighborhood in city of Santa Ana and Orange County, California, not always a place thought of as a place of high need, but there are definitely pockets of that and that’s especially true in the early nineties. Violence and youth violence were really prevalent at that time and the sisters are really caught in the middle of that and realized that for many young people there weren’t a lot of options outside of the neighborhood. And so they were compelled into action. And as is our tradition, they engage the community and 23 years later, we’re serving close to 600 young people a year providing workforce training, really seeing jobs as a great vehicle for helping young people who are really disenfranchised, move out of poverty and build lives of self sufficiency for themselves. So that’s really what we focus on.
Bill Russell: 02:44 You find these, these, these youth that are struggling and you give them a second chance. So how do you identify them and, and how do you, um, you know, and what is your program do for them,
Shawna Smith: 02:55 you know, so we really try to seek out those young people who aren’t in school. They’re in work, we’re particularly focused on young adults so that 18 to 25, 26 year old who’s struggling and stuck maybe not in work or in a minimum wage job, going nowhere, you know, and, and, and where we are in southern California, like many parts of the country, it’s very high cost of living. So we want to ensure these young people get into the workforce and we’re particularly focused on four industries instruction, healthcare it and business. Uh, we, we provide training and then we partner with local employers to provide them employment opportunities which really kind of help enrich and grow their businesses. So it’s a win win for both the youth and our local employer community.
Bill Russell: 03:40 I want to thank you again for the opportunity. I want to thank you for the work that you’re doing and then it’s such a great opportunity for us to support and I really appreciate our listeners who have subscribed to the podcast and our sponsor for, uh, for uh, giving that money. We have another four weeks of the, um, of the, uh, a fundraiser going so I would just encourage people to share with their friends and uh, let’s see if we can drive the number higher and thanks again for, for coming on, Shawna. I really appreciate it.
Shawna Smith: 04:08 Well, thanks to you and your listeners for your generosity and the opportunity to share about our Good work. Really appreciate it.
Bill Russell: 04:14 I appreciate Shauna coming and joining me and, uh, and I also really appreciate our listeners that have spread the word about the show and we really appreciate it. As I said, we’re a, we’ve raised $2,000 but we hope to raise more in the next couple of weeks and we appreciate everything you guys do with regard to that. So, uh, today’s guest is a Robert Rice who is the chief, everything, chief architect chief technology or chief chief operating officer for health lyrics. Rob and I have worked together for a while. A good morning, rob and Welcome to the show.
Robert Rice: 04:48 Good morning. Thank you.
Bill Russell: 04:50 So you have a, you have an interesting background. You’ve worked in a lot of, a lot of different industries besides healthcare. Uh, what are some of the other industries you’ve, uh, you’ve worked in?
Robert Rice: 05:04 I started in finance and then prior to healthcare I worked for the entertainment industry. So The Walt Disney Company, for example,
Bill Russell: 05:11 Walt Disney Company. So you worked for Disney. What’s, what’s it like to work for Disney? Is it, is it as magical as we would think?
Robert Rice: 05:18 A lot more intense than magical. Yeah.
Bill Russell: 05:21 It’s intense for the people that make it magical for the people who are visiting, I guess what’s, what’s the, what’s the craziest it project you had ever had to pull off at Disney? Just out of curiosity.
Robert Rice: 05:35 There’s one intensity over the other it’s size. So for example, in identity project at a normal large institute might have 100,000 identity accounts. At Disney It was about 2 million identity accounts, just sheer volume alone caused you to rethink and relook at the way you’re doing things.
Bill Russell: 05:53 Was there anything about the characteristics in terms of downtime? I would, I would assume Disney had like a zero tolerance for downtime kind of situation.
Robert Rice: 06:04 Yeah. We worked when most people were off, so we had to make sure that we didn’t impact that and it was kind of mantra that we only have one time to get right for the user experience. Family experience We kind of measured twice cut once.
Bill Russell: 06:22 Yeah. So that was a good, good training for you to, uh, to come into healthcare. So then you came into healthcare. You actually join me at, at St Joe’s and Orange County and um, and that’s really going to be the. Gist, we’re going to talk a lot about that time there. Uh, and then when the providence merger happened, you and I went off and a full disclosure, we are a partners in health lyrics and we’ve been doing cloud migrations now really since about 2010, uh, not only with a St Joe’s but with other health systems. And that’s the gist of this. So this is a deep dive episode, which means we take one topic and really explored, we’ve, we’ve done this before with ai and now we’re going to do this around cloud computing, but more specifically we’re going to focus in on cloud infrastructure. We can talk about cloud data models and some other things, but this one’s going to be about cloud infrastructure.
Bill Russell: 07:15 Uh, I’ll do the setup and then we can move deeper into the process. Um, and just, you know, as I said, I, I, you know, Rob and I have a lot of the same shared experience with this. A plus. Some of my talking more on this episode is just going to be just sympathy for rob because his plane got into lax after midnight last night. So, um, we really appreciate, I really appreciate you coming on the show. Maybe not at the top of your game right now. I guess. So. So let’s, uh, let’s start off with a quote. This is pretty recent. This is happening this morning. So, uh, Dale Sanders, president of technology with health catalyst, uh, had this to say on linkedin this morning at roughly 35 minutes ago, if and when, I ever returned to the cio role in healthcare I’ll race as fast as possible to move my data centers to the public cloud.
Bill Russell: 08:07 There’s no way I can provide the same capabilities on any dimension with my data center. Even the data centers like viawest can come close to matching azure, aws, et. Um, and I think it’s interesting. I think you’re seeing a lot of the John Halamka had on the show also have similar comments about, uh, the, the capabilities and things that are available in the cloud. So, um, so let me set this up. So let me put a little definition around it. Uh, I came in as cio in 2010. I looked at the organization, looked at rob and said, you know, we’re going to, here are the characteristics of cloud that we’re looking for. And this was back in 2010. Not a perfect definition, but it sets the groundwork. We said it has five characteristics. First, a self service provisioning from anywhere. So a web based provisioning really from a, from your phone, from Alexa, from self service provisioning, from anywhere, uh, that was number one.
Bill Russell: 09:02 Number two, rapid elasticity, which meant you could scale up and scale down. Now scaling up was not that big of a deal back then, but uh, but, but it was somewhat, sometimes you had to add a crack units and air conditioning and power to your data center, in which case it was millions of dollars. Uh, other times you had to add servers, took a lot of time. But one of the things that we were looking at it when I came in in 2010 was the ability to scale down the org. We kept asking the Organization for tens of millions of dollars for new data centers and whatnot. And they were saying, okay, hey, we’re, we’re actually selling off portions of our business. Can we have some of that money back and The answer is no. So cloud gives you the ability for self service provisioning and rapid elasticity. So scale up, scale down a meter service, pay for what you use.
Bill Russell: 09:47 So, um, one of the benefits of the cloud is if you have a thousand users today, you pay for a thousand users. If you have 800, you’re paying for 800, but even more than that, the, uh, the services are much more granular. So, uh, you’re only paying for the storage you need and the compute you need, and now it goes all the way down to a, you know, much, you know, Ai and other things as well. Uh, the fourth thing was programmatic access to resources. This is where we get into the devops. This is where a lot of your efficiency comes in. And then the fifth thing was granular accounting. I’m to the point where aws now you can do compute by the minute. So, uh, for those of you are wondering what you would use that for, sometimes you have a processing that’s done at night. You could scale up 100 servers, use them for 10 minutes, do all your processing scale and back down, and only pay for using those servers for 10 minutes.
Bill Russell: 10:39 So, so granular accounting, so self service provisioning, rapid elasticity, metered service, programmatic access to resources and granular accounting, and uh, like I said that, that, not a perfect definition, but it’s sort of set up what we were looking to accomplish. We were looking to get agility, efficiency and automation within our health system. So we thought the cloud was going to bring. Us really a two important things first in 2010 when I came in, we, we interviewed a lot of the executives and said, where’s healthcare going? And the most common response was we don’t really know. We know that there’s a lot of change coming, hop health, value based care, a mergers and acquisitions, new competition. Silicon Valley was coming in. Um, and uh, you know, one of the things is when you don’t know what’s next in your industry, you need to create something that is, that you’re able to make quick moves, you’re able to adapt very quickly.
Bill Russell: 11:33 And so cloud gives you that ability. Uh, the second thing we knew is that healthcare is moving outside the walls of the traditional health system. So we used to have data centers which served our 23,000 staff, uh, but that wasn’t going to be the case moving forward. We were going to start putting devices all over the place. We also had clinics, critical access facilities, long term acute care facilities, more and new types of partnerships so that not only were we looking at 23,000 end points, we were now looking at literally millions of endpoints, uh, accessing resources from our data center and the cloud lend itself pretty well for that. So, uh, one last piece of background, rob, before we get going. So I’m just to give people a little idea of where we started our journey. Um, you know, our, our scenario is pretty challenging.
Bill Russell: 12:19 We had a year, some of the statistics of 50 percent of our data center equipment was end of life when we started our project. Sixty percent of the workstations were also end of life, uh, 1800 instances of 900 distinct applications, which is not the worst we’ve seen in the industry. I know if you, I’ve seen organizations, when we say 1800 instances, they just go, oh, well that’s, that’s nothing. We have 10,000 of you just sort of shake your head like that’s a almost impossible to manage 10,000 instances of distinct applications. But we had 1800, 900 a distinct applications. Uh, we had $50, million in data center upgrades that needed to be done and I’m the data center or I mean this is the real thing here. The data center had just gone down eight times in six weeks prior to my arrival as cio. Uh, it’s actually one of the things that precipitated my coming in as cio.
Bill Russell: 13:13 So, uh, and the other thing I will give people just because it gives a picture of our relationship, I wish the unreasonable cio, you are the vp of infrastructure and operations and I looked at you and said, hey, you know, get our data centers to the cloud. And Oh, by the way, here’s one last thing. You can only have two racks per hospital and they have to be self contained racks and not need data centers. And I remember actually I told the team that before you, before I hired you and they looked at me and said, that’s not possible. And then when I brought you in I said, hey, here’s the requirement, I thought you were going to tell me the same thing, and you looked at me and said, oh yeah, piece of cake. Yeah, we could do that. So we’ll, uh, we’ll shed some light on why that was a piece of cake and actually how you exceeded that expectation, which was pretty interesting. All right, are you ready? Great. So we’ve set it up. It’s obviously, it’s a picture perfect environment. Most of the stuff’s end of life, data centers going down. You come in as a VP of infrastructure and operations and uh, we have this cloud strategy. So you take the cloud strategy. But one of the things that we talked a lot about is we’re not moving to the cloud for cloud’s sake. We’re looking to really derive benefits for healthcare organization. Um, let’s start out there. So what are some of the benefits that a healthcare organization will get from moving to the cloud?
Robert Rice: 14:40 First and foremost, it gets you out of the business. You’re not an IP business, you shouldn’t be there and we don’t do it. Well, we don’t attract the of talent that we need because honestly I know so that little talent wants agreement once a Disney and Amazon on their resume.
Bill Russell: 15:00 In fairness, we were in southern California, so we were competing with literally with google with startups with Disney and others. So it was, it was a tough labor market.
Robert Rice: 15:09 Right, right, exactly. But the beauty is that application is also not very complex in the grand scheme of things so that lends itself easily to get yourself out. Once you get over that emotional hurdle is, you know, I need to own the servers.
Bill Russell: 15:26 We did have people that wondered if we move to the cloud, how are we going to walk into the data center? How did you address some of those, some of those concerns and, and I mean the benefits far outweigh that, you know, being able to walk into the data center.
Robert Rice: 15:42 Yeah. Well first and foremost, it’s understanding improving. Do we know what we need to know and what do we don’t know? A kind of funny speak that the basic is for is what is our operation, what are we actually managing and what do we think we know versus all that shadow stuff that we know that doesn’t make sense. Once you start getting your hands around that or I’ll say it becomes an easier picture to paint that we probably don’t manage enough of it already. So it’s easier to start kind of boxing up and shipping out.
Bill Russell: 16:16 Yeah. Between you and David Baker Baker, who we had on the show earlier between you and David, you guys kept reminding me that it’s not compute storage and uh, those things that people buy, they buy what they want from it is uptime, uh, ease of use access. Um, they’re, they’re buying services and, and so much of our, a challenge up front was to get it to think different. You could get the culture to think different of what’s the best way to deliver service to the organization, not what’s the best way to get, you know, storage compute and all that stuff to the, to the business.
Bill Russell: 16:58 It was a, it was interesting cultural change. So we, we, we saw a financial benefits, you know, once you get to Dev ops and everything’s programmatic. We went from 35 down to five, uptime went up, the, uh, the, uh, overall costs were more tied to the level of the business. Uh, and then I think the other thing that people are surprised to hear is that, um, our, our availability actually goes up in the cloud and we’re going to talk about that a little bit later, but, uh, you know, you have disaster recovery capabilities that aren’t available in traditional models, uh, that you were able to leverage on a regional basis and on a, on a system wide basis, which were a, were pretty interesting. So, um, alright, so let’s, let’s just, we’ll step into the process here. So, uh, and again, this is the process that you gave me way back when.
Bill Russell: 17:56 Um, there’s really three movements to get us, get us to the cloud. One is to establish business case, second to architect a plan, and then the third to migrate and operate. Then under each one of those, there’s a, there’s a handful of steps and movements and we’re going to touch on those. So the first thing under established a business case, you have a readiness assessment and I think the thing that people will be interested to hear is not every health system is ready to go to the cloud. So why is that? Why do you think what makes one health system more ready than another to move to the cloud?
Robert Rice: 18:33 I Don’t think we’ve ever talked about this. At least the simplicity of it. I think the context is organic growth versus measured growth when that most health systems that I’ve seen have grown organically, like I’m gonna buy this system, that system or that system and just keep adding onto the bandaid ball, but as opposed to logical and methodical growth, you know, we know we’re going to grow by a factor of x, every year and we’re gonna plan for it and I think that’s kind of really at the epicenter of why what I’ve seen coming from other industries we’ve done that going into healthcare. I’ve yet to see that logically thought through.
Bill Russell: 19:11 Yeah. I think the other thing we noticed is that some health systems still don’t have a handle on the basics, the basics of blocking, tackling. So when we go in there, like for example, um, you know, when I came in as cio, I asked, okay, I would like a list of all the organizations that we’re connected to and what data we’re sharing. Um, that took almost six months to a year for me to get that list. So organizations that have trouble with the basics of, hey, can we have access to this data, this information, do you know where your contracts are? Those organizations need to, not that it’s an insurmountable hurdle because we got, we got through it and um, but you know, being able to have access to that data even though it’s, you know, a lot of times we can just take raw data and process it. Um, just knowing where it is and having access is one of the things. So what are some other things you found in terms of organizations being ready?
Robert Rice: 20:15 Process, maturity. is key So standardization, and this is not process for process sake, is standard repeatable processes that a large portion of your staff follow and in a lot of cases that doesn’t exist and that’s the baseline for building blocks of automation and kind of build from that and build on what you said, asset management is Wonky, but do I have standardized procedures for patch and reboots, for change or things like that. So if it’s measured and applied consistently across a larger part of the organization you’re much more mature and ready to those leaps, then you have to set those building blocks in place first,
Bill Russell: 20:54 So if they’ve don Itell and those kinds of things, you’re, you’re going to be in good shape, I would assume. Right? Alright. So the next step is to establish the point of truth. And uh, you know, I wish I found this interesting because, uh, you know, for me, you came in and started dumping all this uh, information on me and it didn’t match with what the organization had told me. Um, you know, prior to you doing this deep dive into our information. And, uh, and to be honest with you, I was, I was a little defensive because I’m looking at, you know, we had just done a Microsoft agreement based on information that Microsoft has given us, so they’d come in and done an audit and said, hey, here’s all the stuff you have, here’s what you should license. So that’s what we licensed. And then you came in, did this deep dive and came in and said, hey, you over licensed by about a million and half dollars. And I got defensive and I’m like, how can you say that, you know, more than Microsoft, who has all these tools on my environment, but give us an idea of what it means to establish the point of truth and how you get to, um, really that, that starting point for the cloud migration,
Robert Rice: 22:04 A lot of your, we’ll say your source logical systems have the data of how you actually use it from a licensing perspective, or consumption perspective. So active directory, for example, every user that’s gonna log in and every account that’s going to do it, but not every account is active. So when you start looking at it like how fast and frequently do we actually become or kind of terminate my accountant and that’s one of the problems that we had is, you know, we had approximately 50,000 something accounts but 15000 have been fallow for multiple years, meaning that we don’t use them at all, so we have licensing for the wrong amount, the same thing applies in most other areas, right? So we never do a good job of rationalizing down that we can buy we can acquire, we could build, but what’s our lifecycle management process? And that’s kind of one of those easy sacred cows to go after first and foremost. Say how much have we been deep conditioning over the years, which is almost something?
Bill Russell: 23:05 I think I found it interesting that you, you go after source data, You literally are just saying, Hey, I just want the raw data out of active directory. I want a, I want to look at the physical contracts. You, you, you really this process, you go back and one of the things you sort of drove home to me is that every report I’m seeing has a bias or a purpose, a purpose or a bias, right? It has a bias to make something look a certain way or it has a purpose that it was designed for and I might be using that report for a different purpose. Um, so talk about why you go back to the source data and, and how and why that doesn’t take forever. I would think if you go to the source data, it would just take forever to get done.
Robert Rice: 23:47 Well I mean and you’re Right? And that was part of our journey together. Generic performance was asset reports are just that. They’re generic, they don’t apply to the way we actually use it, they applied to the way the tool is done. So when you look at the problem from a tool perspective, you’re solving a problem with a hammer when you actually need to build a house, when you go to the law of the data, you’re looking at actually how you’re using it and which is really the key, which takes a little bit more time upfront, but it moves a lot faster once you lay out all the cards. So when you’re solving a, putting together a puzzle, for example, you typically look at the edges rather than starting in the center. You know, and that’s kind of the approach, what you have to kinda look at it. How am I actually using it and what am I actually doing with it first, which allows us to move faster once you get the edges of the puzzle figured out.
Bill Russell: 24:31 Yup. And then we have this point of truth and then you establish the, the business case itself, which closes out this first step. And in the business case you have things like financial models. You have usage, you have skills, you have a, you know, I’m, I’m, I’m actually shortening this up for, for purposes of the show, but, uh, there’s a lot of elements of this business case. Uh, one of the things that the combination of you and I really works so as people can tell, you’re very analytical and I like to tell stories. And so, uh, I really strongly encourage people to tell the whole story of how health care transitions in this process. And then you come back with just all the supporting data in terms of, uh, how, how agility is going to play out in the organization. How, uh, there are cost savings even though people need to be aware of the fact it’s, it’s operational dollars versus capital dollars and in a lot of cases and some organizations struggle with that transition. So we now move into architect and plan which is the second step that you have here. And the first part of that I think is probably one of the more fascinating which is decomposed the applications and the services to core elements um what are you actually doing there?
Robert Rice: 25:51 You’re just looking for the commonality across your application portfolio, right? Every application has the same components, a wall where the data exists, how does it process what the client side look like? So all we’re doing is we’re taking that and establishing a commonality across the largest portion of the application portfolio that we can start standardizing and really focusing on because you’re not going to solve the problem individually. If you have, you know, hundreds of thousands of applications, you’re going to look for the least common denominator across the group which allows you to do that. So a database, for example, most applications have a database and an on premise legacy world. You have your production database plus your backup, maybe a a high availability one, so now, but from a core conceptual it’s all the same thing. So that allows us to start reducing down and focusing on solving the database problem rather than the thousand application problem.
Bill Russell: 26:46 Yup. It’s interesting because I think, I forget the number of layers, but it was a, you know, it’s everything you would think about. You have the fiscal hardware, the networking and the virtualization layer. I’m going to get some of these wrongs storage layer a os and whatnot. And when you got done that, one of the, one of the big findings I found was you came over and said, hey, we have a. It was a couple hundred images. I forget the number. Do you remember the number of images? Server images we had?
Robert Rice: 27:16 It was a little over 200 now.
Bill Russell: 27:20 Yeah. So it’s. So that’s one of the things when you look at it through a different Lens, you’re able to see this, wow, we have a couple hundred images. Well, that’s very hard for an organization to maintain that level and then when you broke it down even further you, I think what your team came back and said is immediately we can get this down into the teens because of, you know, we just do some work around the images and eventually I think the goal was to get it down to six. Is that, is that pretty close?
Robert Rice: 27:49 Yeah, so about 200 down to five common images and based on that you know you could automate you can start leveraging efficiencies because now I’m not spending time chasing after 200 different images across 40,00 types than just chasing down 5 images which is the exact same.
Bill Russell: 28:06 And it was really interesting. You had, before you came in, we were chasing the CMDB, a nirvana where we were going to have a cmdb which would always tell us what was going on in the servers and, and you just, you sort of minimize that. You’re just sort of like, hey, you know what, if we get down to five images then you know, it’s, it’s the CMDB becomes not irrelevant but becomes first of all, a lot easier to manage. But second of all, I mean it’s one of five images. You don’t have to figure out all the things that you would normally have to dive deep into a CMDB to figure out. So, um, it was, I found that to be interesting. So the next thing, and this really highlights the mistake that I made prior to you coming in is the skills Matrix and training plan.
Bill Russell: 28:48 Um, so I skip this step and one of the things people need to know is that the biggest obstacle they’re going to get to moving to the cloud is not from the organization. The organization’s going to say, Hey, I use the cloud at home and I got 25 Gig of storage for my email and I use dropbox or box at home and I, you know, I have all these capabilities and they understand the cloud gives them better services. So they’re not, they’re not pushing back to you when you say, Hey, we’re moving to the cloud, but it does. And one of the things that I skipped was the skills matrix part and because I did, people didn’t see where they fit in the organization when all this was done. And uh, and so you, you made a step back, you created, um, you know, a skills Matrix for where we’re at today and what skills they would need to develop to get to where they were going. Uh, so talk about the process of not only developing the skills matrix but also having the conversations with the employees and the staff around moving to the cloud.
Robert Rice: 29:48 Well first and foremost, right, that the skillset shifts or elevates up in the cloud because again, you’re not emphasizing the petty engineering skills. You don’t need a lot of really core storage depth of knowledge because all you’re doing is splitting up gigabytes and terabytes of data at a time here. So as you’re building up the skills matrix, what we started realizing was we never really defined what success was when you look at it from a scale perspective. So we emphasized planning, design, reporting, those types of skills that you actually really need more so because you need to map out more templates or more creative skills matrix, you rated all the skills that we’re gonna need in different types of roles on a one to five basis. And we added in more soft skills and those design skills and those reporting skills that are often neglected that we need to emphasize. So we kind of reset what success was for each individual team and group to say right now your engineering heavy but, we need to get to is this level to be able to emphasize Itell service, things like that because that’s what really kind of helped us and not everybody wants to do that. Or it takes that leap to do that as well. So we gave him a different.
Bill Russell: 31:00 Yeah. And that was the fascinating thing to me. I mean, you sat down with a significant staff and started mapping that out and some of them just self opted out. You looked at him and said, hey, here’s where it needs to get to. And they said, you know, I’m fine, I’m fine working with Kobalt. And you’re like, okay, that’s fine because we’re going to have this legacy environment for at least a year, maybe a year and a half while we do the migration and we need somebody to take care of it. So you know. But at the end of the year and a half, we’re not going to need Kobalt programmer. I’m using that as an example to Kobalt programmers. So you’ve essentially self opted out over the next 18 months. But, you know, but people appreciate it knowing that, of just, okay, so what you’re saying is my job going to go away in 18 months, but you need me for the next 18 months to get there. And they, they were really appreciative. Um, and I, I would just strongly encourage people to do it seems so obvious to CIOs, but for some reason I skipped it. A, you have to show people where they’re going to fit in the future state. Otherwise they, uh, they will fight, fight the process. So, um, yeah,
Robert Rice: 32:05 You also need to identify those soft skills and those kind of intangibles in a sense too. In a Dev ops world, there’s no step by step guide. So how do you identify your innovative thinkers, your testers, your tinkerers, because part of the process is learning self learning and if you need a, you know, cbt or a training module to that, you’re probably not the right fit for that role. And so some of people we found came out of the most unlikely places when we started doing that.
Bill Russell: 32:31 Yeah, it always surprised me where you were, you, you were moving people around. And I’m like, yesterday that person was doing that and now you have been doing, are you kidding me? And you’re like, no, let me tell you how these things match up. This is, you know, this is the new world. And um, I think the other thing was we just had a conversation with a client with a potential client. Actually I should be honest there. So with the potential client and they were saying, are you saying in the Dev ops world, I’m going to have to hire all these programmers because that’s going to be really hard for me and I, I think you. Oh well why don’t you, why don’t you talk about that a little bit? Because I think that’s a myth. Common misconception,
Robert Rice: 33:14 Yeah they think the inverse is true right? Like once you have dev ops you gonna loose them Because they are going to the bigger names, they’re building their references and their resumes based on that. So you have to create more than the grooming and training on, just ike we talked about before how do you invest in the people that need the time to learn and train and develop their skills to do because you can’t compete with those big guns in the industry that you kind of give them a growth path so that you have the ability to really kind of build that skillset.
Bill Russell: 33:42 And then the other thing we’re finding is that the tools are becoming more configuration based and less, you know, hardcore coding based, right? So it’s, it really is going into an interface and clicking on a couple things and then boom, you’ve just, you’ve just stood up a whole lot. Not that you don’t need a couple of those hardcore programmers who really understand how the underlying framework works, but a lot of the, uh, tools or are moving higher up in the, in the, uh, uh, in really the, into the presentation layer. So, uh, so people who just understand how that entire stack works can make this stuff work.
Robert Rice: 34:22 Yeah. when starting we used to, and we’re talking like, mid 2005ish you had to have core development skills in order to do this, but now like 50 to 75 percent of the work is configuration based, there’s still an element of development in Dev ops now it’s kind of funneling higher and higher.
Bill Russell: 34:43 Yep. Absolutely. So the next thing, and uh, you know, probably the sexiest thing here is establish a digital architecture, security models and aligned work loads with the appropriate a deployment models. Um, and what, what we talked about a lot here is that each cloud will look a little bit different from client to client based on what they’re trying to accomplish. But you do have five primary environments that you consider today. Um, can you, can you walk us through the five environments again? I mean you gave us. So there’s a, there’s a legacy of environment, uh, and you know, what list I’m talking about.
Robert Rice: 35:18 Yeah. So the legacy environment is acknowledging the reality that not all your systems are going to lift out on that. So you have to maintain it.
Bill Russell: 35:31 So why is that? Is that just, is that just because because of the way they’re written and the way they function today, they are just not going to go to a cloud environment,
Robert Rice: 35:45 yea, so there are some devices that still require direct physical connection. So in the health industry, biomed devices, you’re just not going to get out of them anytime soon. Then there are larger ones that are critical systems, like your EMR, that your vendor supports different processing models because they’re monolithic in nature, but so you may as well just acknowledge them, put them in the corner and then come back to them in a couple of years time and then focus on the other, your scattered.
Bill Russell: 36:12 So the first one we have is a legacy environment. Walk us through the rest.
Robert Rice: 36:17 Yeah. So the next level up is you have applications that are still server based, but they’re virtualized. A virtualized server can exist anywhere. It doesn’t have to exist in your data center for the most part. So those are the next target to move out into infrastructure service so you’ll actually begin hosting your aws instances, but they’re still server based so your building server instances for that. And then you take the next step. Beyond that, now you’re focusing higher up, which is more workflow based. So you’re going after a softwares and service platforms, which everybody has right now, so whether you’re using salesforce or dynamics or other software based platform, you’re focused on workload or you’re not focused on anything below the workflow for the most part. And that also kind of lends itself to platform as a service emitegating those investments into sharepoint online, into salesforce, into other crm platforms. Now you have more platforms that I could kind of build different workloads, different processes, different solutions in a similar platform instance rather than different softwares. And then ultimately the utopic vision when people get to that level is native development called native development models where you’re designing micro services, web services, and the like. But that’s typically the end state of a journey.
Bill Russell: 37:44 So that sort of distinction. So every health system will have those five components, legacy or a private cloud infrastructure as a service, software as a service platform, and then cloud native. So what’s the distinctive is how much of each of those you’re going to use. So some some health systems might have 20 percent legacy, uh, you know, 30 percent infrastructure as a service and so forth and so on. But the goal really is whatever, whatever that, that core layer of legacy, we’re shrinking that over the next couple of years. We’re moving to infrastructure as a service, even that layer, we’re moving to software platform and cloud native. And the reason we’re doing that as the levels of efficiency, agility, cost savings, and access to innovation that organizations are gonna have I go into the cloud, are just significantly more then a traditional traditional models. All right? So we create these really cool diagrams. We map out the entire, uh, digital architecture, the security model, and we aligned the workloads to the various layers of, of cloud infrastructure that we’re going to use. Now we’re ready to migrate and operates. And uh, this is the easy part, right? So we, uh, we just have to migrate 1800 applications, 1800 instances of application’s pretty straightforward. Um, and uh, I’m being a little facetious here. So here’s where we, here’s where you start this process. You start with a negotiating agreements and Kpis with the cloud vendors. What are some of the things that healthcare cio or healthcare organization should be thinking about when they’re negotiating these cloud based contracts?
Robert Rice: 39:29 I guess first and foremost to our earlier conversation on Knowing your environment, knowing your instances when you know how many virtual servers or instances I’m going to need to maintain because they just can’t get rid of them, because they’re still server based. You could lock in certain pricing models with your vendors, so when you’re locking in reserve pricing for I’ll use integer for example, let’s say I know I’m gonna maintain $5000 the ends, you’re gonna get a very different price point than if you say, you know, I’m going to spin up instances willy nilly and it’s a bachelor of magnitude to. So knowing what people reserved pricing forecast will be and say, I know I’m going to release three years. Allows you to negotiate very differently than just kind of winging it and going into it. So again, back to that very first step, which is knowing your world and what you’re going to maintain. It’s going to be key for you. And then from there it’s kind of peeling back to your usage case.
Bill Russell: 40:20 Yeah. And some of the things that we’ve talked about, you, you can’t outsource security because you have a background in security, so you can’t out source security. So you need to negotiate that and uh, so understand the HIPPA requirements, understand any PCI requirements. You have a and you have to be able to provide the reports and, and pass through all the way, all the way down so you can’t outsource it. You’re still responsible. You still have to be able to supply that. Um, the other thing, the other Gotcha that think people don’t think about is, um, especially with cloud applications, like, uh, where, you know, you’re moving vast amounts of data into these cloud environments. Um, make sure you can get it out. Um, you know, when you’re negotiating these contracts, uh, not only make sure you can get it out technically, make sure you can get it out financially.
Bill Russell: 41:10 Uh, I know that some of these clouds, it’s interesting, they will, they’ll literally send trucks out to pick up your data, bring it to their data center and pump it in. And some of those, some of the services almost free. But to get it out, they hit you with bandwidth costs and all sorts of other stuff and it costs a fortune to get it back out. So be careful of those kinds of things. And you know, this is a, we only have a couple more minutes, but those are, those are just some of the things off the top of my head that I would, uh, I would say just keep those things, keep those things in mind. So this is the part where I,
Robert Rice: 41:47 The other thing to emphasixe too is what type of data, so data that I need fast response on is when it costs very different than data. I probably only need once in a long periods. So your pacs image, research, imaging for example.
Bill Russell: 42:02 Yeah. And when we’re looking at, when we’re looking at the architecture, we’re looking at those bandwidth requirements and we’re looking at those access requirements because the, the, the more access you need in the cloud, you can pay for very high availability, very high access, very low bandwidth or latency, low latency, low latency servers. But if you don’t have to, you really shouldn’t. So, uh, so really knowing your workloads and aligning correctly to the, to what you’re buying is really important. So here’s where I want to spend a bunch of time because this is the, you spent so much time here and this is where the big payoff is. So, um, you’ve, you’ve done all this, you’ve done this deep dive, you’ve done all this planning and now we’re getting ready to migrate workloads and uh, you know, at first migrating the workload goes a little slow, but then it really picks up pace because you’re just doing the same thing over and over again. So give us an idea of some of the things you think about going into a migrating to the cloud and some of the processes that you, uh, you take a look at.
Robert Rice: 43:05 Well, first and foremost, if we’re doing a lift and shift migrations, the easiest way to do it that we’ve always found is to do a disaster recovery, fail over. So we create the disaster instance in the cloud, which allows us to create a full production of snapshot everything we need to go the scale, build the competence, build the expertise for it, and then go through a few iterations of fail over without impacting the business. So in those cycles, now we’ve built everything that we need to at a low price point, at a low impact to the production environment and got the business to buy in and sign off on it and build competence and that’s kind of the probably the primary way that we start the process because it allows us to kick the tires and allows us to build confidence and build the models we’re gonna need for those kind of migrations.
Bill Russell: 43:52 Yeah, you really encouraged people to start with the easy stuff first because your team is learning a new skill so you, you, you, you don’t want to start with, hey, let’s start with pax. You really want to start with something that’s a pretty, pretty small, pretty self contained. Not a complex architecture and your teams just getting that, building up those skills and then as you, as you sort of moved down the road, then you start doing some of the more challenging, more difficult, uh, migrations. But I want to go back to something you said. So you start by standing up a Dr Environment or a greenfield environment that you can fail over to that, that really is a, one of the, one of the secrets that you brought to the, to the process that kept our, uh, kept our downtime pretty minimal in this migration process. I mean, because when you can fail over from one to the other and it just sort of switched those users over, now you’re not looking at extended downtime’s a and plus in the process you’re building out dr capabilities for each of these applications as you move forward. Um, so does that work for all the applications or just a certain set of those applications?
Robert Rice: 45:13 In a large portion of them it does, I would Qualify that to say a large portion of them is transparent, but the user experience is barely on the radar for them. There are some that struggle that. But architecturally you identify them way before.
Bill Russell: 45:26 And so I think the other thing was, you and David both drove home to me was a, this migration process is more of a people and culture a challenge than a technical challenge. Um, obviously we’re, we’re bringing our it staff along, we’re training them as we go. Uh, but also you guys built up a pretty solid team with Rachel and the rest of the team at Rachel who we’ve hired at health lyrics now, um, who, who put together great processes for engaging the end user community. So they, uh, engage them early in the process, walk them through it, talked about the benefits. Um, so that team became really good after, you know, after you do a couple hundred applications, they, they actually ended up with an extensive database of everyone in the organization. They knew who the key decision makers were. They were able to engage them at every point within that and they became a trusted source for that. So, uh, is that, I mean, do you still see that as a pretty critical team to have in place for this migration?
Robert Rice: 46:29 Oh yeah absolutely, it’s not worth doing if You don’t understand your business, and if you’re not talking with them right? Because they’re going to define their tolerances They’re going to define their appetites, they’re going to tell you, don’t bother me these days, but I’m okay these days, which gives you your operating model to build your competence and there are so many other problems that you find that exist outside, normal it that you win a lot of good will just by solving easier problems to building that relationship is absolutely critical.
Bill Russell: 46:58 Some of the not necessarily seen benefits for us were, I think we got rid of 10 percent of the applications just by talking to people just by going out there and saying, hey, we’re going to be migrating this application. You’re listed as the owner. And they looked at you like, uh, you know, we don’t use that application anymore. Great. Um, and so we, we didn’t know we were doing a, uh, an application rationalization projects at apparat project, but we ended up, um, you know, archiving a whole bunch of data and a whole bunch of applications just in the process. Because as you pointed out earlier, a lot of healthcare just grew over time of we added this application, add this application, we acquired this company, we’d never really cleaned all this stuff up and this is a great opportunity, uh, to do that. So we get 24 months later, here are some of the things, you know, we, we, we never did spend that $15 million dollar on data center, uh, improvements because we didn’t need to, um, we went from a server provisioning from and people find this hard to believe, but really from a two to three weeks to provision a new server with the processes that we had to almost two to five minutes to provision a new server.
Bill Russell: 48:14 And sometimes our clients find that hard to believe. But in, uh, uh, to be honest with you, with a credit card right now, I could go online and provision 100 servers in the next 10 minutes on aws and I don’t know, I don’t even have the requisite stuff lined up yet. So, um, that’s the kind of stuff that’s available in the, in the cloud. Um, and then the other thing is we, we got access to a whole bunch of new capabilities. Uh, you created a, a, a, the ability to fail over regionally, uh, and, and some other things, but I want to get back to the one thing because you private cloud, you went into hyper converged route, which I found interesting and I came to you and said, hey, no more than two racks in every hospital. And I know when, when I say that to the team, they, they sort of were incredulous and I think some, some of our listeners from health systems are saying, well, you know, we, there’s no way our health system, uh, in that hospital could get down to that, that number. Um, but talk to a lot of, talk a little bit about what hyperconvergence was able to do a at the hospital level, for us
Robert Rice: 49:26 So all hyperconvergence is really condensed, compute and memory power, right? So most applications that exist nowadays are nowhere near the level that you can provide in basically what I could fit under a desk right now. So when we started getting into the applications, I think a, you know, these are my toys element to the local staff too. So being able to come in and say we’re not going to take your toys out of there, but we’re going to do is just put them in a new case. But that new case now is a fraction of the size, right. So this is kind of a, just the convergence of data and compute. So we found almost immediately that in about six years, which is roughly six to 12 inches of space, could hold hundreds of servers of what you currently have. So getting to that point it was extremely easy and it also kind of want to a lot of different competencies to by saying you’re not moving it out. And once it was in the hyper converged with our ability to migrate out and to conquer the Vr stuff was almost push button easy.
Bill Russell: 50:24 Yeah, absolutely. And then you can go to third parties and be able to migrate, migrate in as needed. And uh, so yeah, you were able to get down to, and a lot of our hospitals you able to get down to a half rack. I think the largest was maybe two racks. I didn’t know you always argued with me on that, but um, but essentially self-contained racks, you didn’t have to have a specialty room. You can move that stuff around. I think that has some Dr. Dr. benefits for it. Uh, the other thing is people, people always ask me, you know, if you’re moving this stuff into the cloud, what happens in the case of disaster? So if your connection to the cloud goes, what a,
Bill Russell: 51:05 what, you know, what do you do in that case? And so one of the more ingenious things I think that your, you and your team came up with was a almost reversed the, our process, which is a, it was our primary was in the cloud, but we were actually a trickling data back down into the local data center so that if, if the, if something went down, I’m on the network. Let’s say there was an earthquake in southern California network went down, those hospitals would still be able to run a certain core systems because we have trickled the data back. So it was almost a reverse a Dr kind of thing. Um, hey rob, I want to thank you for, you know, I know you got in late last night. I want to thank you for coming on the show. Um, and uh, no, it’s, it’s, it’s always fun to, uh, to talk through this stuff with you and to be a part of, uh, helping now other health systems, uh, to go in this direction. So thanks again. Um, let’s see. All right, so let’s go. So you can follow, uh, you can follow the show on [email protected] andyoucancheckoutourwebsiteatthisweekandhealthit.com. Watch the videos on the youtube channel this week in health it.com/video. Uh, please come back every Friday for more news, information and commentary from industry influencers. We have a great lineup of people over the next four, four or five weeks and we’re starting to set our fall schedule as well. So you’ll want to check us out every Friday. That’s all for now.
Robert Rice, COO of Health Lyrics joins us to discuss Cloud Infrastructure for Healthcare. We explore 8 years of experience migrating to the cloud with lessons learned and best practices.