Brac­ing for a crash

While IT out­ages are rare, providers need a plan

Modern Healthcare - - INFORMATION TECHNOLOGY - Joseph Conn

Many bil­lions of dol­lars have been spent in the past decade on new or up­graded health­care in­for­ma­tion tech­nol­ogy sys­tems, but the money has bought im­proved func­tion­al­ity, not in­fal­li­bil­ity, ac­cord­ing to health IT ex­perts.

Sys­tem re­li­a­bil­ity has im­proved sig­nif­i­cantly in the past decade, says Dr. Wil­liam Bria, a veteran physi­cian in­for­mati­cist and chief med­i­cal in­for­ma­tion of­fi­cer for the Tampa, Fla.-based Shriners Hos­pi­tals for Chil­dren. And while full­blown, blar­ing head­line com­puter crashes are now rare oc­cur­rences, when it comes to down­time, “ev­ery­body has some,” Bria says.

“The eu­phemism is planned ver­sus un­planned and the mea­sure­ment is po­lit­i­cally sen­si­tive,” Bria says. “Very few times it’s a hard­ware is­sue. It’s most of­ten a soft­ware is­sue. The worst times, of course, are when things have been very sta­ble for long pe­ri­ods of time and peo­ple let their guards down.”

Ernie Hood, se­nior re­search di­rec­tor for the Ad­vi­sory Board Co. and a for­mer chief in­for­ma­tion of­fi­cer with the Seat­tle-based Group Health Co­op­er­a­tive, re­calls one of those times.

At Group Health, Hood says, the or­ga­ni­za­tion ini­tially mea­sured IT sys­tem per­for­mance as a per­cent­age of up­time. But as per­for­mance im­proved to 99.5% up­time, “We started to mea­sure min­utes of down­time. Even a planned down­time was such a rare event.” Group Health launched its sys­tem wide elec­tronic health record in 2003 “and we had about a to­tal of about 30 min­utes of down­time from 2003 un­til mid­way 2005, and then, we had an out­age that lasted a cou­ple of hours.”

“It was one of those glitches,” Hood re­calls. “We iden­ti­fied and fixed it fairly quickly.”

Still, he says, “Ev­ery­one had to drop to pa­per.” And while the hospi­tal was pre­pared with pa­per forms at hand, there were is­sues about catch­ing up with the reg­u­lar work­flow, up­dat­ing the elec­tronic record and over­time. “It does cre­ate a lot of havoc,” he says. “It was such a sig­nif­i­cant event, I was asked to make a pre­sen­ta­tion to the board ex­actly what hap­pened and why it was never go­ing to hap­pen again.”

Un­til that time, the board “rarely ever had any dis­cus­sion about IT,” Hood says. But af­ter the out­age, board in­ter­est “es­ca­lated so they got into dis­as­ter pre­pared­ness and con­ti­nu­ity plan­ning, which was a good thing. That’s an area where health­care or­ga­ni­za­tions tend to un­der­spend. Busi­ness con­ti­nu­ity is re­ally a busi­ness re­spon­si­bil­ity, not an IT re­spon­si­bil­ity.”

And once crash-pre­pared­ness plan­ning be­gins, Hood says, “It’s like pulling the clas­sic thread on a sweater, and you might think this is a rel­a­tively small is­sue and when you start pulling on it, then it starts to broaden—what if we have an earth­quake or a pan­demic? — and then it be­comes an or­ga­ni­za­tion wide plan­ning process.”

And that soon leads to plan­ning on an even broader scope.

“There is this trend in health­care to go to­ward com­mu­nity,” Hood says. “You look at things like ac­count­able care or­ga­ni­za­tions or com­mu­nity net­works. The old days of a stand-alone prac­tice or hospi­tal that doesn’t or­ga­nize with pe­riph­eral ser­vices are go­ing by the way­side.”

Dr. Howard Landa is chief med­i­cal in­for­ma­tion of­fi­cer at Alameda County Med­i­cal Cen­ter in Oak­land, Calif. But in 2007, he helped or­ga­nize a “down­time sum­mit” con­fer­ence at Kaiser Per­ma­nente, where he has served as CMIO of its Hawaii di­vi­sion; he also has lec­tured on down­time at the Health­care In­for­ma­tion and Man­age­ment Sys­tems So­ci­ety’s an­nual con­fer­ence.

“Ba­si­cally, what we came up with was a se­ries of pro­ce­dures to deal with ei­ther a planned or un­planned down­time,” Landa says. There needs to be a com­mu­ni­ca­tion plan to keep ev­ery­one in­formed that there will be down­time—if it’s planned—and what the pro­ce­dures are, and what’s go­ing on if it’s un­planned.

The down­time plan needs to pro­vide “as much ac­cess as you can” to “his­toric” in­for­ma­tion in past records. It needs to deal with opera- tions, cre­at­ing new records dur­ing the down­time. We cre­ated down­time forms, a sub­set of the doc­u­men­ta­tion in the EHR,” he says. “Those forms were scanned af­ter the down­time was over.”

Fi­nally, a plan must pro­vide for the re­cov­ery of the sys­tem and how, and in what form, pa­tient in­for­ma­tion from the down­time is reen­tered into the main elec­tronic record sys­tem.

“The re­cov­ery is the thing a lot of peo­ple don’t do well,” Landa says.

One ques­tion plan­ners re­ly­ing on pa­per dur­ing a down­time need to ask is, what’s the value of struc­tured data com­pared with the ef­fort it would re­quire to re-en­ter it?

It may be OK to scan a pa­tient’s vi­tal signs and a physi­cian’s notes from pa­per copies, Landa says. “Al­ler­gies, cer­tain chemo­ther­apy drugs, cer­tain an­tibi­otics, those things are im­por­tant enough you’d want to re-en­ter.”

To­day, “most of the EHRS have ways of do­ing this, reg­u­larly stor­ing copies of all or­ders as fre­quently as once an hour on a sep­a­rate “down­time ma­chine” con­nected to a hard­wired printer. “The ma­chine has a big sign that says, ‘Don’t turn me off.’ ”

With the pro­lif­er­a­tion of mo­bile in­for­ma­tion tech­nol­ogy, se­cu­rity-re­lated threats to sys­tem de­pend­abil­ity come into play, ac­cord­ing to Gary Barnes, CIO for Med­i­cal Cen­ter Health Sys­tem in Odessa, Texas.

And with the gov­ern­ment’s EHR in­cen­tive pay­ment sys­tems re­quir­ing providers to use cer­ti­fied EHRS, “We’ve prob­a­bly ex­pe­ri­enced more down­time due to ven­dors up­grad­ing our soft­ware. It’s just eat­ing our lunch. We’re go­ing to be down for six to eight hours to put a mean­ing­ful-use patch in.”

Stephen Stew­ard, CIO of 25-bed Henry County Health Cen­ter, a crit­i­cal-ac­cess hospi­tal in Mount Pleas­ant, Iowa, rec­om­mends start­ing to plan for down­time from the worstcase sce­nario: “What would we do if the build­ing were gone?”

“Even if you’re not in busi­ness, you have to be able to re­trieve your records. Pa­tient records come first,” he says, but “al­most as good as the health records are your em­ploy­ment records. You still have to pay peo­ple.”

Mov­ing up from a to­tal dis­as­ter, “You pick the low­est hang­ing fruit first and at­tack the things that are go­ing to be your most likely fail­ure points and try to move on from there.”

“An­other point of vul­ner­a­bil­ity to­day that didn’t ex­ist 20 years ago was be­ing at­tacked from the out­side,” Ste­wart says.

Ear­lier this month, the Fed­eral Emer­gency Man­age­ment Agency re­ported that the num­ber of cy­ber­at­tacks on fed­eral agen­cies in­creased more than sev­en­fold from 5,503 in fis­cal 2006, to 41,776 in fis­cal 2010. FEMA ranked the U.S. against 31 “core ca­pa­bil­i­ties” for na­tional pre-

pared­ness across “the full range of haz­ards at all lev­els of gov­ern­ment and across all seg­ments of so­ci­ety,” in­clud­ing “count­less threats posed by those who wish to bring harm to Amer­ica” as well as “many nat­u­ral and tech­no­log­i­cal haz­ards that face the na­tion’s com­mu­ni­ties.”

The good news for the health­care com­mu­nity: public health and med­i­cal ser­vices topped the pre­pared­ness list. The bad news? Cy­ber­se­cu­rity de­fense ranked dead last. Health­care IT is not im­mune.

Re­ports on more than 50,000 breaches of med­i­cal records have been sub­mit­ted to HHS’ Of­fice for Civil Rights since the fall of 2009 when a re­port­ing man­date and an on­line re­port­ing mech­a­nism were cre­ated un­der the Amer­i­can Re­cov­ery and Rein­vest­ment Act.

The vast ma­jor­ity of these breach re­ports in­volve records af­fect­ing fewer than 500 in­di­vid­u­als, which “are in the PDF for­mat and are not in data­base for­mat,” said HHS spokesman Bill Hall. In­for­ma­tion about these lesser breaches “is not recorded or con­verted into a data­base” within the Civil Rights Of­fice, and it is un­der no obli­ga­tion “to cre­ate or main­tain” such a data­base, Hall says. Copies of these records have not been made public, thus, how many of those lesser breaches were caused by hack­ers is un­known.

But of the 421 largest breaches, which the Civil Rights Of­fice does an­a­lyze and re­port pub­licly, nearly 6% were linked to hack­ers and led to the ex­po­sure of records af­fect­ing more than 550,000 peo­ple.

In March, hack­ers “be­lieved to be op­er­at­ing out of East­ern Europe” pen­e­trated a state-run com­puter sys­tem in Utah and ex­posed the records of 780,000 Med­i­caid re­cip­i­ents.

So far, with the health­care in­dus­try, hack­ers have fo­cused their at­ten­tion on crim­i­nal, not de­struc­tive, pur­suits, ac­cord­ing to Mac Mcmillan, CEO of Cyn­er­gis­tek, an Austin, Tex­as­based se­cu­rity con­sult­ing firm.

“We haven’t seen any ac­tiv­ity for the pur­pose of dam­ag­ing or de­stroy­ing sys­tems” in the health­care in­dus­try,” Mcmillan says. Hack­ing in health­care has “mostly been re­lated to the theft of data.”

That’s not to say the health­care in­dus­try will re­main un­tar­geted by de­stroy­ers, he says.

“If they wanted to, if the hacker com­mu­nity re­ally de­cided it wanted to take a shot at health­care, knock­ing down net­works, they’d have a fairly good chance at suc­cess, par­tic­u­larly with the smaller or­ga­ni­za­tions.”

On Fri­day the 13th in July 2006, IT staffers at the UPMC sys­tem in Pitts­burgh wit­nessed their own pri­vate hor­ror movie come to life. An en­tire Pitts­burgh neigh­bor­hood—where UPMC’S data cen­ter was lo­cated—lost power, re­calls Chris Car­mody, the sys­tem’s vice pres- ident of in­for­ma­tion ser­vices.

“The power gen­er­a­tor worked for about four hours and then failed and we couldn’t get it run­ning again. They brought in this mo­bile unit and that worked for maybe 45 min­utes, and then the fuel line fil­ter clogged be­cause it had been sit­ting at some con­struc­tion site and had dirt in it.”

Mean­while, the data cen­ter flick­ered off and on for 36 hours.

“That height­ened the aware­ness of how sen- sitive and how crit­i­cal it was for us to plan how we can de­fend against and pre­pare for and min­i­mize the im­pact to our clin­i­cal users,” Car­mody says.

Risk man­age­ment be­came in­te­grated into ev­ery as­pect of UPMC op­er­a­tions, from bud­get­ing to the im­ple­men­ta­tion of a sin­gle soft­ware ap­pli­ca­tion, Car­mody says.

UPMC is now on its 58th con­sec­u­tive month of con­tin­u­ous up­time.

PHOTO IL­LUS­TRA­TION/ERIC SEMELROTH/GETTY IMAGES

Newspapers in English

Newspapers from USA

© PressReader. All rights reserved.