When a data centre data recovery becomes disaster recovery

When a customer loses most of their data centre to water damage, it is nothing to take lightly. We received an emergency call just like this recently. An enterprise level company had a burst pipe which flooded and physically damaged portions of their data centre. We immediately went to work.

The situation

The customer sent us 10 cabinets with a total of 1,400 drives. These were a mix of several storage and software manufacturers (SAS, SATA and FC drives from EMC, IBM and Oracle). The shipping logistics alone were a challenge. In order to save time, the customer sent entire cabinets to us intact so we could start working right away.

The work done

Each enclosure in the cabinet was inspected and inventoried. Ontrack found visible water damage on the first set of cabinets. This water damage was present on the drives, shelves, power supplies for the storage processors and the head units for the equipment.

Each drive was then removed from the original enclosure and inventoried (asset tagged). Once tagged, the media was transferred to the Engineering Services Team for triage. At this time each piece of media was inspected and a decision made of whether to image in engineering services (no damage or contamination) or to transfer the media to the clean room for additional work (contamination, visible damage, non-functional drives).

Any drives found to be damaged or contaminated were transferred to our cleanroom and repaired using our proprietary tools and techniques.

All media was imaged (sector by sector copy) and reports were provided to the customer detailing the results of the imaging process.

During the imaging process, we needed to have the capacity to host all of their data. Some quick, but extensive, adjustments were made to our data centre and we were able to oblige.

What was the result?

After the data was recovered, repaired, and restored; quite the feat with 1,491 drives, the systems were reassembled and physically and electronically verified to be in working condition. Done…correct? Wrong! Once the equipment was returned to the customer in working order, it was time for them to rebuild their data centre. Those of you who work in enterprise storage know you cannot just set the racks up and plug them into the wall. We proceeded to work with the customer to restore all of the disks from the point of failure so that all of the data was present. When you have older storage systems working with newer equipment, multiple manufacturers and software programmes , there are bound to be a few failures before everything works seamlessly together. Our engineers supported our customer by working with them and the OEMs to troubleshoot and consult on problems and procure the proper equipment until everything was in working order.

The summary of this job was as follows:

• Over 15 Ontrack Employees from engineering, logistics, sales and technical support in constant contact with the customer the entire process.
• 1,491 hard drives imaged
• 1.2 petabytes of data processed and stored until the customer’s data centre was fully operational
• Customer was fully functional in 2.5 weeks. Amazing when you think about the amount of time it takes when you build a data centre from scratch without having to recover the data first.