Bracing for a crash
While IT outages are rare, providers need a plan
Many billions of dollars have been spent in the past decade on new or upgraded healthcare information technology systems, but the money has bought improved functionality, not infallibility, according to health IT experts.
System reliability has improved significantly in the past decade, says Dr. William Bria, a veteran physician informaticist and chief medical information officer for the Tampa, Fla.-based Shriners Hospitals for Children. And while fullblown, blaring headline computer crashes are now rare occurrences, when it comes to downtime, “everybody has some,” Bria says.
“The euphemism is planned versus unplanned and the measurement is politically sensitive,” Bria says. “Very few times it’s a hardware issue. It’s most often a software issue. The worst times, of course, are when things have been very stable for long periods of time and people let their guards down.”
Ernie Hood, senior research director for the Advisory Board Co. and a former chief information officer with the Seattle-based Group Health Cooperative, recalls one of those times.
At Group Health, Hood says, the organization initially measured IT system performance as a percentage of uptime. But as performance improved to 99.5% uptime, “We started to measure minutes of downtime. Even a planned downtime was such a rare event.” Group Health launched its system wide electronic health record in 2003 “and we had about a total of about 30 minutes of downtime from 2003 until midway 2005, and then, we had an outage that lasted a couple of hours.”
“It was one of those glitches,” Hood recalls. “We identified and fixed it fairly quickly.”
Still, he says, “Everyone had to drop to paper.” And while the hospital was prepared with paper forms at hand, there were issues about catching up with the regular workflow, updating the electronic record and overtime. “It does create a lot of havoc,” he says. “It was such a significant event, I was asked to make a presentation to the board exactly what happened and why it was never going to happen again.”
Until that time, the board “rarely ever had any discussion about IT,” Hood says. But after the outage, board interest “escalated so they got into disaster preparedness and continuity planning, which was a good thing. That’s an area where healthcare organizations tend to underspend. Business continuity is really a business responsibility, not an IT responsibility.”
And once crash-preparedness planning begins, Hood says, “It’s like pulling the classic thread on a sweater, and you might think this is a relatively small issue and when you start pulling on it, then it starts to broaden—what if we have an earthquake or a pandemic? — and then it becomes an organization wide planning process.”
And that soon leads to planning on an even broader scope.
“There is this trend in healthcare to go toward community,” Hood says. “You look at things like accountable care organizations or community networks. The old days of a stand-alone practice or hospital that doesn’t organize with peripheral services are going by the wayside.”
Dr. Howard Landa is chief medical information officer at Alameda County Medical Center in Oakland, Calif. But in 2007, he helped organize a “downtime summit” conference at Kaiser Permanente, where he has served as CMIO of its Hawaii division; he also has lectured on downtime at the Healthcare Information and Management Systems Society’s annual conference.
“Basically, what we came up with was a series of procedures to deal with either a planned or unplanned downtime,” Landa says. There needs to be a communication plan to keep everyone informed that there will be downtime—if it’s planned—and what the procedures are, and what’s going on if it’s unplanned.
The downtime plan needs to provide “as much access as you can” to “historic” information in past records. It needs to deal with opera- tions, creating new records during the downtime. We created downtime forms, a subset of the documentation in the EHR,” he says. “Those forms were scanned after the downtime was over.”
Finally, a plan must provide for the recovery of the system and how, and in what form, patient information from the downtime is reentered into the main electronic record system.
“The recovery is the thing a lot of people don’t do well,” Landa says.
One question planners relying on paper during a downtime need to ask is, what’s the value of structured data compared with the effort it would require to re-enter it?
It may be OK to scan a patient’s vital signs and a physician’s notes from paper copies, Landa says. “Allergies, certain chemotherapy drugs, certain antibiotics, those things are important enough you’d want to re-enter.”
Today, “most of the EHRS have ways of doing this, regularly storing copies of all orders as frequently as once an hour on a separate “downtime machine” connected to a hardwired printer. “The machine has a big sign that says, ‘Don’t turn me off.’ ”
With the proliferation of mobile information technology, security-related threats to system dependability come into play, according to Gary Barnes, CIO for Medical Center Health System in Odessa, Texas.
And with the government’s EHR incentive payment systems requiring providers to use certified EHRS, “We’ve probably experienced more downtime due to vendors upgrading our software. It’s just eating our lunch. We’re going to be down for six to eight hours to put a meaningful-use patch in.”
Stephen Steward, CIO of 25-bed Henry County Health Center, a critical-access hospital in Mount Pleasant, Iowa, recommends starting to plan for downtime from the worstcase scenario: “What would we do if the building were gone?”
“Even if you’re not in business, you have to be able to retrieve your records. Patient records come first,” he says, but “almost as good as the health records are your employment records. You still have to pay people.”
Moving up from a total disaster, “You pick the lowest hanging fruit first and attack the things that are going to be your most likely failure points and try to move on from there.”
“Another point of vulnerability today that didn’t exist 20 years ago was being attacked from the outside,” Stewart says.
Earlier this month, the Federal Emergency Management Agency reported that the number of cyberattacks on federal agencies increased more than sevenfold from 5,503 in fiscal 2006, to 41,776 in fiscal 2010. FEMA ranked the U.S. against 31 “core capabilities” for national pre-
paredness across “the full range of hazards at all levels of government and across all segments of society,” including “countless threats posed by those who wish to bring harm to America” as well as “many natural and technological hazards that face the nation’s communities.”
The good news for the healthcare community: public health and medical services topped the preparedness list. The bad news? Cybersecurity defense ranked dead last. Healthcare IT is not immune.
Reports on more than 50,000 breaches of medical records have been submitted to HHS’ Office for Civil Rights since the fall of 2009 when a reporting mandate and an online reporting mechanism were created under the American Recovery and Reinvestment Act.
The vast majority of these breach reports involve records affecting fewer than 500 individuals, which “are in the PDF format and are not in database format,” said HHS spokesman Bill Hall. Information about these lesser breaches “is not recorded or converted into a database” within the Civil Rights Office, and it is under no obligation “to create or maintain” such a database, Hall says. Copies of these records have not been made public, thus, how many of those lesser breaches were caused by hackers is unknown.
But of the 421 largest breaches, which the Civil Rights Office does analyze and report publicly, nearly 6% were linked to hackers and led to the exposure of records affecting more than 550,000 people.
In March, hackers “believed to be operating out of Eastern Europe” penetrated a state-run computer system in Utah and exposed the records of 780,000 Medicaid recipients.
So far, with the healthcare industry, hackers have focused their attention on criminal, not destructive, pursuits, according to Mac Mcmillan, CEO of Cynergistek, an Austin, Texasbased security consulting firm.
“We haven’t seen any activity for the purpose of damaging or destroying systems” in the healthcare industry,” Mcmillan says. Hacking in healthcare has “mostly been related to the theft of data.”
That’s not to say the healthcare industry will remain untargeted by destroyers, he says.
“If they wanted to, if the hacker community really decided it wanted to take a shot at healthcare, knocking down networks, they’d have a fairly good chance at success, particularly with the smaller organizations.”
On Friday the 13th in July 2006, IT staffers at the UPMC system in Pittsburgh witnessed their own private horror movie come to life. An entire Pittsburgh neighborhood—where UPMC’S data center was located—lost power, recalls Chris Carmody, the system’s vice pres- ident of information services.
“The power generator worked for about four hours and then failed and we couldn’t get it running again. They brought in this mobile unit and that worked for maybe 45 minutes, and then the fuel line filter clogged because it had been sitting at some construction site and had dirt in it.”
Meanwhile, the data center flickered off and on for 36 hours.
“That heightened the awareness of how sen- sitive and how critical it was for us to plan how we can defend against and prepare for and minimize the impact to our clinical users,” Carmody says.
Risk management became integrated into every aspect of UPMC operations, from budgeting to the implementation of a single software application, Carmody says.
UPMC is now on its 58th consecutive month of continuous uptime.