The business value of protecting your assets

Regulatory compliance impacts all companies, in all industries. However, mandates specific to healthcare organizations are some of the most stringent, requiring the preservation of patient records, images and other files – often for a patient’s lifetime. The loss of such data has serious implications on healthcare organizations – the risks of non-compliance coupled with the risks associated with providing poor quality patient care.

There have been numerous studies done on the number of businesses prepared for a disaster, including a business continuity plan. The long standing position on preparedness for natural disasters has been “it can’t happen to me.” However, this attitude started to change with 9/11 and Hurricane Katrina. Organizations realized that disastrous events can impact a business no matter where they are located, and often regardless of the magnitude. While emergency preparedness is imperative for saving lives in a catastrophe, it can be just as important for an organization to maintain a comprehensive and well-maintained business continuity or disaster recovery strategy in order to mitigate this risk.

The Southeast Louisiana Veterans Health Care System was just one of the many local hospitals to suffer extensive damage when Hurricane Katrina struck the Louisiana-Mississippi border in 2005, causing the failure of the New Orleans levee system. The subsequent flooding resulted in one of the largest natural disasters to ever hit the United States and in the flooding of one of the Southeast Louisiana Veterans Health Care System’s 354-bed acute care facilities. This closure significantly impaired the hospital’s ability to provide healthcare services to the more than 220,000 veterans living in the New Orleans area.

However, the Southeast Louisiana Veterans Health Care System had a disaster recovery plan in place they believed could successfully weather the storm. Unfortunately, like many hospitals and businesses in the area, the wrath of Katrina would cause pain and suffering for months and even years.

After the hurricane hit, the hospital’s Health Systems team sprang into action. The goal of Kenneth Allen, health systems specialist at the Southeast Louisiana Veterans Heath Care System was to recover as much data as possible. Unsure what he might find, Allen and his team physically went to the site to begin the lengthy clean-up recovery process.

Once onsite, the recovery team discovered that water had entered the basement of the building knocking out all utilities – electrical, plumbing, and communications – thus straining the hospital’s emergency and non emergency systems. After further inspection, Allen and his team realized that the hospital’s medical-records storage area was also compromised.

The hospital’s medical-records storage area included a Storage Area Network (SAN) with two clustered servers, five image Gateway servers, a Plasmon archive library and a RAID system, a computer data storage system that can divide and replicate data among multiple hard disk drives. This equipment was housed on the second floor of the main hospital building and was the main SAN that stored all patient records.

After the system safeguarding the computer network exhausted its battery power, all electronic medical patient records became unusable. The backup power had only lasted a few hours and with the entire area’s communication and transportation in disarray, the medical center’s IT staff could neither access nor drive to the hospital to perform a safe shut down of the system.

As a result of the flooding and the power outages, all medical records from the hospital were destroyed or severely damaged. The only recoverable data found was stored on UDOtm media from Plasmon stored in the empty hull of the New Orleans hospital. While the media was not immediately exposed to flood waters, the high humidity from stagnant waters two floors below caused significant concern over the reliability of the storage disks containing the sensitive patient data.

Adding to the problem was the fact that the RAID disk drives in the SAN and the servers had been physically removed in an effort to prevent the theft of confidential patient records. In the chaos, the hospital pulled the disks from the drives and put them into unmarked boxes that no one was ever able to find. As a result, all of the RAID data was lost.

Had the RAID drives remained in place, and if power and network connectivity had been quickly restored, then this tragedy would not have been so bad. However, with the loss of the RAID drives, the lack of a complete series of backup tapes, and the uncertainty of reliable utilities, ‘normal’ disaster-recovery plans would have been futile.

“The media had been exposed to extremely high temperatures and humidity from brackish standing water, they were covered in debris and dust for more than a month,” said Allen. “We were able to recover all of the patient images from the Plasmon archive libraries. We had just started to Plasmon UDO for an archive solution when the hurricane hit.”

Allen and his team retrieved media from the Plasmon library for recovery efforts. The team brought the 1,100 disks, which represented approximately 6.2 terabytes of data, to its facility in North Little Rock, Arkansas for recovery. There they built a new SAN exclusively for the purpose of recovering patient data.

“All told the recovery ended up being a year long process that began with trying to determine the technical status of the hospital,” said Allen. “We had to determine what information was retrievable among the absolute chaos of the first week after Katrina.”

A year after the tragedy, the Southeast Louisiana Veterans Health Care System had restored 99.5% of its pre-Katrina patient data. After this experience, the Southeast Louisiana Veterans Health Care System chose to install Plasmon UDO Archive Appliances because of its Katrina tested recoverability as well as because of its capacity, scalability and price.

“Our disaster-recovery plan for the hospital was backup tape and RAID,” said Allen. “The backup tapes were never found in the debris and the RAID system was lost. When people start thinking about disaster-recovery plans, they need to determine the levels of disaster and then develop the appropriate solution. Since this disaster, we have continued a migration to Plasmon’s UDO Archive Appliance solution to meet our policies and regulations for record retention.”

Although the New Orleans hospital building itself is no longer available for patient care, Veterans continue to receive some diagnostic services from a myriad of outpatient clinics in the surrounding area. Clinicians working in these outpatient clinics have been able to access restored patient data since February of 2006 through the VAs VistA Remote Image View process. There are plans to build a new hospital in the New Orleans area, and when it is opened for patient use, patient records, past, present and future will await the hospital staff to continue to fulfill the VA’s pledge to “Serve Those Who Have Served.”