Only Available at Ontrack: IBM Storwize Data Recovery

The Client

The Situation

A client recently experienced a remote ransomware attack that resulted in Ontrack engineers being presented with one of their most extraordinary data recovery efforts to date: restoring 120 damaged HDDs within an IBM SVC Storwize v7000 system…with no backup to rely on. 

 

The Solution

After noting the critical nature of the project, Ontrack’s data recovery experts proceeded with a comprehensive process to potentially restore the deleted data:

  1. Consultation
    The Ontrack team was able to join the client’s team and scope the data loss event and the storage systems impacted. Based on the scope set forth, Ontrack was able to determine a project plan, set timing expectations and determine costs for data recovery.

     

  2. Diagnosis
    Data recovery engineers used Ontrack’s proprietary tools to analyse the disks, determine the likely array configuration as well as detect indications of Windows storage space and VMware storage virtual machines.

    3.  

  3. R&D Simulation and Software Programming
    After the initial diagnosis, engineers analysed a minimal hardware setup of the IBM SVC Storwize v7000 as a means of detecting the layout of on-disk structures used to map Raid Arrays, including managed disks, SVC pools, virtual disks, and physical disks (=LUN).


    Ontrack then began to work closely with the client’s IT department to get the hardware running on a new setup of the IBM SVC Storwize v7000 system.

    Simulations were performed to see if the client’s environment could be recreated on the live hardware and if any structure could be found to possibly reconstruct the deleted data. All findings regarding the simulated structures were compared to the structures on the original hard drives.

    A positive prediction was formed based on the comparison of the structures, and Ontrack was able to move forward with the creation and modification of proprietary tools to extract functional storage systems and proceed with successful SAN system data recovery.


  4. Data Recovery

    5. With an enormous challenge ahead of them, Ontrack’s data recovery experts performed extensive research on IBM’s proprietary software which resulted in engineers modifying their recovery tools to allow for the virtual rebuild of the DRAID that was in use on the IBM system.

    Figuring out the distribution patterns for DRAID proved to be the most intricate part of the recovery process, given that all of the data sitting on the DRAID6 MDisk was combined with a number of other MDisks and dynamically allocated multiple levels of both VDisks and Dynamic Disks.

    Once the array was virtually rebuilt, the Ontrack team was able to virtually rebuild the volumes, transforming them into 1,152 devices in order to display the overall layout of available data contained within to generate reports for the client and complete the IBM Storwize data recovery.

     

The Resolution

When the client initially introduced the issue, there was little hope for full (if any) recovery given the complex nature of the IBM Storwize data storage system. However, thanks to the diligence of our engineers, an unprecedented Ontrack data recovery solution for all IBM Storwize systems is now solely available via Ontrack.

Ontrack Performs Emergency Raid 5 Data Recovery

The Client

The Situation

A multinational client had fallen victim to the disappearance of crucial financial data and office files, with no backup data plan in place. After realizing the urgency of the data loss situation, they immediately contacted Ontrack’s expert engineers to help with recovery.

The Solution

Given the importance of the lost data, the client expedited all media over the weekend via courier to an Ontrack engineer who happened to be on call. Per client request, Ontrack provided project updates every 15 minutes as a data recovery plan was being prepared.

It was quickly discovered that the client’s hard drives used the Raid 5 system, and in order to have a successful data recovery, engineers would have to virtually rebuild the drives to guarantee perfect quality of the data found on their VMFS VMware ESX 5.1 volumes.

The Resolution

Through a fantastic show of teamwork and diligence, Ontrack’s data recovery engineers were successfully able to bring all drives back to nearly 100% of their data volume within 6 days after rebuilding the system; a testament to the company’s unmatched expertise and dedication to solving even the most complex data challenges. The client expressed enthusiastic gratitude for such quick data recovery and was excited to have their company back up and running in just under a week with the help of Ontrack’s emergency recovery service. 

Damaged Mac Laptop? Ontrack to the Rescue.

The Client

The Situation

A customer contacted Ontrack’s experts in need of help with Mac data recovery for a laptop that had been dropped and accidentally run over by a truck, causing its battery to combust. 

The Solution

Ontrack’s recovery experts surveyed the Mac device damage and implemented a 3-step recovery process for data extraction which included:

  • Decontamination – A thorough process in which all contaminants that adversely affected the damaged Mac laptop’s operability were removed.
  • Micro-Soldering – A small soldering tool was used to replace a pool of damaged capacitors and resistors which were located near the burned battery.
  • Diagnostics – The MacBook was placed in a special diagnostic mode where both Apple-provided OEM tools and Ontrack’s own proprietary tools were used to mount the internal storage device as a volume onto lab machines and process the copy out.

Once the process was complete, Ontrack was able to successfully extract and verify all data.

The Resolution

Some accidents aren’t easy to account for. Ontrack’s recovery experts can be trusted to provide satisfactory service when it comes to Mac recovery, as well as data recovery for similar devices, whether damage is due to an occasional drop and break or a truck that comes out of nowhere.

A Deep Dive for Dell/EMC Isilon Data

The Client

The Situation

A client inadvertently ran a command that deleted critical files on a Dell/EMC Isilon storage array containing 270 disks totaling 2 petabytes (PB). 

The Solution

After having the drives in question flown in, Ontrack’s recovery engineers began an in-depth evaluation and determined that the JIT* development team would be needed to assist in data recovery from this version of Isilon. Within eight weeks of working nights, weekends, and holidays to develop a proper solution, an initial set of data was delivered to the client. 

While the client was ecstatic to receive the recovered files, they also requested that subsequent tasks be done to prove that no stone was left unturned in their recovery efforts, per regulatory requirements. Ontrack’s JIT development team complied by combing through the 270 drives in search of files that matched regulatory requirements and implementing a process that would search, copy, and deduplicate specific files that were found across each disk.

After months of conducting a thorough secondary search, more than 300 million files and 13 terabytes (TB) of PDF and JPG files were produced. 

The Resolution

This project exemplifies the ability of Ontrack’s team of recovery experts to go above and beyond to meet the client’s needs. Our team of engineers is not only well-equipped to restore files lost from Dell/EMC Isilon storage, but they are also prepared to recover your data according to regulatory standards in place.

 

*JIT stands for "Just In Time". It means that our development team can make changes on our tools on the fly to accommodate custom problems or changes to third party file systems and others. It's a development capability that sets Ontrack apart from other data recovery companies that just buy fixed data recovery tools off the shelf.

 

Ransomware Recovery – Veeam Agent for Windows

The Client

The Situation

A health care customer was affected by a ransomware attack that not only targeted their server data, but also “Veeam Agent for Windows” backups located on an external HDD. Their IT / managed services provider agreement did not include regular off-site backups, so this was the only copy of the data that existed.

The Solution

 

The customer was able to send the affected HDD to Ontrack, where an image of the drive was taken to preserve the original state of the customer media.

Ontrack engineers assessed the damage to the affected Veeam backup files and identified that partial recovery would be possible as the files had not been fully encrypted, meaning there was a chance that some data could be recovered from within the files. However, it was determined that the version of Veeam used was newer than Ontrack could support with current tools and required development assistance.

With a global engineering presence, as well as internal development teams that maintain and improve our proprietary tools, Ontrack was able to research, develop and implement support for the new version quickly. In fact, much of the time-intensive research required had already been completed for similar jobs seen in our European offices. This allowed Ontrack developers to quickly and efficiently modify tools to the level required to be able to support this restore scenario. Rather than building out a fully-fledged tool, Ontrack engineers were able to use the improved version of the tools to complete searches for required structures to allow them to manually rebuild internal components critical to the recovery of data from within the file.

The Resolution

Once repairs to the files had been completed, engineers were able to use their remaining Veeam tool set to complete an extraction of data from within the repaired files. The recoverable data consisted of many flat file data types that had been completely lost to the customer during the ransomware attack.

Ransomware attacks server – backup tapes erased

The Client

The Situation

A ransomware attack of a company server encrypted the Microsoft Dynamics 365 data and demanded payment. Recent backups of the server were stored on multiple LTO-6 backup tapes, which had been erased by the malware. 

The Solution

After assessing the extent of the ransomware attack, Ontrack representatives identified the company’s backup tapes as the best option for data recovery—even though the malware had erased them. 23 LTO-6 backup tapes from the backup library were sent to the Ontrack office in Böblingen, Germany. Working in conjunction with the R&D department in the United Kingdom, Ontrack developed a custom solution to recover the data from the erased backup tapes.

The Resolution

Ontrack was able to restore 46TB of data from 18 of the LTO-6 tapes. Due to the type of attack on the tapes, Ontrack had to repair the logical damage, shipping the data and tapes separately back to the customer.

Ransomware VBK Recoveries on Tape - Server & NAS Systems

The Client

The Situation

The attacked volume was originally also used to back up data to LTO8 tapes at regular intervals. Most of these backup tapes were also in the tape library at the time of the incident and were quickly formatted by the attackers. However, the customer was able to save an original unformatted tape with a fairly old backup date, which was then completely restored to the now empty Windows volume with a total of 6 TB. Only then was Ontrack commissioned to examine data recovery options. The HP server DL380 with the 55 3TB hard disks were transported to Ontrack in Böblingen Germany.

The Solution

During the diagnosis, a large number of the searched VEEAM vbk files were successfully found on the Windows volume with Ontrack Tools and 27 records were extracted according to a priority list. The restore of the LTO8 tape partially overwrote some of the data sets and damaged the backup files. 

The Resolution

A large part of the data could still be repaired and extracted in several steps.

Later on, 19 significantly older LTO8 quick formatted tape backups were successfully recovered too.The attack also affected numerous European sub offices of the customer. Here were predominantly QNAP NAS systems in use which had stored virtual VMs under VMware, including backup VMs that were partially deleted or internally reformatted with another file system. Ontrack was also able to successfully restore complete backup data in 90% of the seven cases ordered.

Lost Data - What's next?

The Client

ViewLift

The Situation

Meet Doug, Director of Client Services from ViewLift, a digital content distribution technology company located in the NoHo neighborhood of New York City. Doug experienced both a computer crash and an operating system that was wiped clean, and found himself asking those exact questions.

The Solution

After some initial research and numerous referrals from computer retail and repair businesses, including a local Apple retail store that said it no longer performed data recovery work, Doug found answers to all his questions at the uBreakiFix store in Greenwich Village.

The uBreakiFix team introduced him to Ontrack, the global leader in data recovery. The in-store team talked Doug through his data recovery options and explained the process.

“I really didn’t have a preconceived notion of what uBreakiFix actually did. I thought that they were only a cell phone repair store, but they do much more,” said Doug. “The experience was great, the prices were fair and it was great to know that when I walked in the store I was greeted by the staff in an friendly manner—and in New York that is not always the case.”

The Resolution

No one ever plans for data loss, but when it does happen you need to know who to trust to provide a fair price and phenomenal customer service. Doug is confident that if he does experience a data loss of any kind, he knows the Greenwich Village uBreakiFix store will be there to help.

Accidental Deletion of Virtual Machines Results in 15TB Lost.

The Client

The Situation

An accidental deletion at a large wireless provider causes a massive loss of email databases.

The wireless carrier stored all of their Microsoft® Exchange databases spread across 24 separate 2TB LUNs on an EMC® VNX 5400 using VMware® virtual machines. It was also set up so each database had a mirror copy on a different LUN. All of the virtual machines were accidentally deleted resulting in the loss of email for the entire company.

The Solution

The client originally contacted VMware for support.

When VMware’s support team realized the extent of the data loss, they immediately contacted Ontrack for assistance. An Ontrack® Data Recovery™ Engineer assessed the situation and determined that a remote data recovery would be the fastest and most cost-effective option for the customer. The engineer fully explained the process and the client agreed and connected the LUNs to Ontrack’s proprietary Remote Data Recovery system for 24 hour emergency service. Ontrack assembled a team of three data recovery engineers and two developers in order to provide the fastest possible recovery.

The Resolution

The Ontrack developers quickly created the tools needed to improve the success of the recovery. After only a few hours, the first virtual machine was rebuilt allowing for the extraction of the Exchange databases to be returned to the customer. The team continued to rebuild all of the critical virtual machines until the client’s email was back in production. At the end of the project, a total of 15TB of data was recovered with minimal downtime for the client.

New tools made recovery from highly specialized EMC® Isilon® big data storage possible.

The Client

The Situation

The pharmaceutical company lost almost 4 million files of highly critical research and development data when moving files within the Isilon storage system.

The “lost files” were mostly raw data gathered from chemical analysis in ongoing laboratory research work. The firm, together with EMC support, was able to recover 90 percent of the data using a standard rebuild process. To recover the desperately needed remaining files, Ontrack was contacted by the customer for help. Ontrack worked with the EMC support in order to get detailed information on the situation.

The Isilon IQ 6000x, designed for big data, consisted of four nodes with 15 terabytes of data storage in total with 500 gigabyte hard drives installed.

It was set up as a single volume where all of the data was striped across the disks. Unlike a traditional RAID system, Isilon systems are built on the concept of a huge data lake where all data is stored and managed inside one data pool. To manage the system and this “data lake” a unique file system was created by Isilon called OneFS. In this case one of the four nodes inside the system experienced a kernel panic and several disks showed multiple errors. EMC was able to gather most of the files with a rebuild before a consistency check showed that several disks had errors. To recover the missing data, the file system had to be analyzed by the Ontrack team to find out how the data was laid out in the whole storage system.

The Solution

To find out how the data originally was distributed over the disks and determine the data mapping, Ontrack s own R&D department developed a brand new set of tools unique to the OneFS.

With these tools, an existing OneFS volume can be analyzed in depth and missing or faulty data file structures inside the “big” data lake can be discovered even more quickly. With the now gathered information on how the data structure of the Isilon system was setup, the engineers were finally able to rebuild and recover the missing data.

The Resolution

In the end, Ontrack engineers were able to recover almost all of the missing files with only a couple of bad files due to damaged encryption.

The client was delighted that Ontrack handled such a complex recovery so quickly. None of the confidential data was compromised and the solution provided by Ontrack was cost-effective and completed within a very short timeframe. With the newly developed tools for this case, Ontrack is currently the only data recovery service provider with the ability to recover data from Isilon storage systems.

Ontrack is assisted by NetApp’s technology to solve a ransomware infection.

The Client

The Situation

A single user’s laptop at a large pharmaceutical company was infected with CryptoLocker ransomware.

This malware encrypts the user’s files and withholds the encryption key until you pay the ransom amount. The laptop was connected to the corporate network which allowed the malware to infect a CIFS volume which was set up as a file share on a NetApp FAS. The malware was able to infiltrate the file share and encrypt the majority of the files. The IT team was not notified of the infection until after the backup retention period had expired, meaning that the backup contained only encrypted data. The total impact resulted in inaccessible data on:

■ 46 drives

■ 1 aggregate

■ 1 volume infected on a RAID-DP

To perform the recovery, the aggregate needed to be taken offline, which affected 17 volumes in total.

The Solution

The customer brought their 46 drives into our New Jersey lab for evaluation and Ontrack engineers got to work on a solution.

The engineering team from Ontrack:

■ Virtually rebuilt the RAID groups which were strewn across 10 different shelves

■ Virtually rebuilt the aggregate

■ Virtually rebuilt the critical volume

An additional challenge on this recovery was that the aggregate was in use for two weeks after the incident occurred which resulted in some data being overwritten.

The Resolution

Ontrack was able to virtually rebuild the volume containing the CIFS share and encrypted data.

Leveraging NetApp’s proprietary OS (OnTap) and file system (WAFL), Ontrack engineers used multiple consistency points to “walk back” in time to find and merge unencrypted copies of the critical data to return to the customer. This type of recovery is only possible on storage like NetApp’s FAS because of the way the data is stored on the volume.

Ontrack Successfully Recovers Data from iPhone 5.

The Client

The Situation

Customer Testimonial

“I was taking video of my [then] 7 month-old in the bathtub, and of course, ended up dropping my phone in the tub! It was completely soaked. I tried all the usual tricks like putting it in rice, etc., but nothing worked. So we contacted a couple of data recovery companies for quotes, and ended up sending it to a company other than Ontrack. We were told initially by the other data recovery company that there was an 85-90% chance of recovering the data. We paid a deposit to have it sent to this other company, who we later found out sent the phone to Canada to have it diagnosed. After all that, they came back and said the phone was unrecoverable. We ended up losing the deposit, and having them send the phone back to us. We decided to send it to Ontrack for a second opinion. We wanted to work with a well-known company that absolutely knew what they were doing if we were going to take another chance. We had Ontrack evaluate the phone and they were able to recover all the photos! We were so happy. There were over 2,500 memories that we never would have been able to recreate. We couldn’t have gotten them back without Ontrack.” - Stacy Holm

Case Details

■ iPhone 5 recovery needed due to water damage to the PCB.

■ Ontrack’s expert engineers were able to perform repairs to the PCB utilizing the company’s proprietary data recovery methods.

■ 17GB of data was recovered from the phone which consisted of approximately 2,500 files.

The Solution

With the implementation of newer, more secure technology, it is more difficult to recover data from the iPhone 5.

Ontrack had to make extensive repairs to the device. Even after the physical restoration was complete, there were still logical failures within the device. Despite these logical faults, Ontrack was able to recover all of the requested data from the phone. The entire recovery only took two days and Ontrack succeeded where the competition had failed.

The Resolution

Partnering with EMC Creates New Tools for Recovery of 10 TB of Data.

The Client

The Situation

A global insurance company was storing PST data on an EMC Isilon OneFS cluster. The IT staff completed a replication task which accidentally deleted 10 TB of PST data. Unfortunately, the insurance company did not have a current backup of the PST data.

The data lost was vital to company operations, so they contacted EMC for assistance who in turn recommended Ontrack due to our increased capabilities recovering from EMC Isilon system.

The Solution

All of the drives affected were brought to Ontrack’s London office for imaging. During the initial investigation of the drives, our team of engineers found that the Isilon OneFS version used on this volume was different from those we had seen in the past.

In order to update our current data recovery solution for this new version of Isilon OneFS, our team worked with EMC engineering to understand the changes between versions. We used this information to create updated data recovery tools and ultimately find the deleted data.

The Resolution

Due to the Copy-On-Write system used by Isilon OneFS, the engineering team at Ontrack was able to find many copies of the deleted file metadata in the unallocated portions of the volume. They extracted 120,000 business critical files which were accidentally deleted and returned them back to the customer.

Double disk failure of RAID 5, and all patient data is gone.

The Client

The Situation

An Italian manufacturer of medical devices experienced the simultaneous failure of two hard drives in a RAID5 system, losing highly sensitive files that were urgently needed.

Without access to the missing data, the company faced potential lawsuits from their clients. Knowing the recovery of the data on the RAID 5 could not be successfully completed without an expert, the client reached out to Ontrack for assistance.

Two hard drives in an EMC NX4 SAN system based on a RAID 5 hard disk array failed simultaneously, leading to an inconsistency of the system files.

The company decided to fix this situation by installing two new hard drives into the system. Their IT department knew how to setup the RAID system, but while preparing the data restoration they noticed that the backup of several SAN drives was missing. Even in cases such as this, it is still possible to restore the system’s latest configuration status, but only if the four most important configuration files are accessible. Unfortunately, the company had set up the system internally and never saved or stored the respective configuration files. They also had not used the manufacturer’s support services, so even EMC couldn’t help. Important patient data, such as MRI scans that was stored for hospitals, became inaccessible.

The Solution

The client sent all the hard drives to Ontrack to mechanically repair the drives, image the data, and then create a 1:1 copy of the content on Ontrack’s servers.

There were a total of 43 LUN’s on the 48 hard drives. Luckily, the client was able to name the two LUNs that contained the important patient data so they could be accurately searched for fragments of both LUNs, as well as file-links within the system’s tables. The search was conducted using Ontrack’s proprietary tools combined with manual search by our data recovery engineers.

The Resolution

► More than three million DICOM files, image files of MRI scans, could be rescued from the two critical LUNs.

The first LUN contained 1.46 million files and the second LUN 1.78 million files. Finding and reassembling the fragments was not easy. Fourteen days of highly complex data recovery experience had to be invested in each of the two volumes. The effort paid off for the medical equipment producer as the critical data was recovered allowing patient treatments to move forward.

Hospital databases rescued from ransomware.

The Client

The Situation

A ransomware attack with the ‘Locky’ virus had severe effects for a large German hospital.

Many servers at the hospital were paralyzed by the virus, limiting operations. Uninfected servers became affected during the panic when their power supplies were disconnected while they were still in operation. In highly complex virtualized storage systems, an improper power shutdown can result in unexpected issues. This was the case for a Dell EqualLogic PS6500ES storage array with a total of 148 professional grade 100-gigabyte hard drives. After the hospital’s IT staff and Dell’s technical support were unable to solve the problem, the specialists at Ontrack were called in to help. All of the drives were delivered to the data recovery laboratory in Germany where they were assessed.

The Dell EqualLogic PS6500ES system typically contains multiple hard drives arranged on 16 or 48 hard drive shelves and are connected together to form RAID 5 or RAID 50 systems (sub-arrays). These sub-arrays in turn are connected to ’members,’ with one or more members belonging to a logical unit (a group). LUNs are created and stored in the group, then fragmented and distributed over all members and sub-arrays. They are ‘tracked’ by a map, which in turn distributes itself to the members or to the various subarrays when it gets proportionally large. In this case our specialists discovered of those seven shelves with 148 hard drives, three shelves with 80 hard drives contained the LUN with the Oracle databases needed. However, many of the links (mappings) of the data fragments (which were distributed over all hard disks) were either corrupted or no longer available, so arranging the fragments proved to be a very difficult task. The mapping of an EqualLogic PS system is also encoded in a specific logic, so the links here aren’t easy to locate either.

The Solution

To map the links, specialist engineers from other Ontrack offices developed new software tools to specifically solve the logic and corruption problems regarding the RAID and the LUN mapping.

With the help of the new tools, the engineers were able to recreate the RAID 5 and RAID 50 systems as well as display the LUN. Within this LUN a virtual hard disk (a VMDK file) was located, in which an NTFS file system with two Oracle databases were hidden. Two file layers had to be identified and recovered within the LUN before these databases could be finally exported.

The Resolution

The team of data recovery engineers from several Ontrack offices were finally able to successfully extract and recover the required databases and send the data by courier to the client.

The hospital was very pleased with the mediation support from Dell to Ontrack and the fact that they finally had all their important data available again. In addition, the tools developed for this project can be used again in upcoming data recovery cases of Dell EqualLogic PS Array systems, significantly reducing future data recovery times.

Ontrack Provides Database and Backup Restores After a Flood.

The Client

The Situation

Dell Equallogic™, Storage Area Network, VMware ESX base and RAID 10 backup server.

A flash flood in Baden-Wurttemberg, Germany in Spring 2016 permeated the walls of a server room in a hobby and art supplies store, severely affecting the IT system.

Vast amounts of water flooded into the server room affecting two Dell devices: an EqualLogic SAN with 96 hard drives and a RAID 10 backup server with 12 hard drives.

The storage capacity of the SAN hard drives was between 300 and 700 gigabytes and the backup server contained 24 terabytes of data. About 30 terabytes of data were lost when the SAN volume and the iSCSI connections for the SAN in a VMware ESX Server were damaged. Due to the importance of the data and that most of it contained critical customer information, an emergency recovery was arranged with Ontrack. Several SAN virtualized LUNs running in a VMware environment were prioritized as critical information, with one of them storing a particularly important Oracle database that needed to be recovered as soon as possible. The client also needed to recover two additional LUNs with important data, as well as all the data on the backup server.

The Solution

After working with the client to assess the data loss and prioritize the data that needed recovering, it was determined that the hard drives in both systems needed to be processed simultaneously. All the hard drives were picked up by Ontrack and delivered to the data recovery lab in Boblingen, Germany.

Upon arrival, the hard drives were initially processed in a cleanroom environment to safely remove dirt and inspect the full extent of the damage. The data on each hard drive and its server location were also documented at this time.

Fortunately, the drives in the SAN had no mechanical problems and could be read properly. However, some of the drives from the backup server were faulty and these had to be processed further in the cleanroom in order to extract copies of the data.

The SAN data recovery was very complex, as the water damage interrupted the connection to the VMware ESX Server and the power supply while in operation. This meant that the mapping links to the EqualLogic SAN and the LUNs in the VMFS datastores (as well as the Oracle database and other files) were heavily corrupted.

To perform the recovery, Ontrack’s proprietary data recovery software tools were required to reconstruct the system and the file structures in order to get to the actual files. However, reassembling the Dell backup server was relatively easy since it hardly experienced any data corruption.

The Resolution

The engineers from Ontrack’s Boblingen data recovery lab succeeded in reconstructing the main points of both affected devices so that the data could be accessed again, including the critical Oracle database and the full backup.

Overall, the customer was very satisfied with the work of the recovery experts from Ontrack. The recovery efforts were also mediated by the Dell Support Team so the data could be restored into a new storage system, as the information required was urgent and needed as soon as the recovery was complete.

Missing Dell® EqualLogic™ LUNs Recovered via Remote Data Recovery.

The Client

The Situation

A large municipal event center in the US lost data on a Dell® EqualLogic™ iSCSI SAN configured with in a RAID 50 running VMware® ESXi™ 5.5.

VMware snapshots filled up the datastore causing the system to crash. The customer attempted to delete one of the snapshots, but after four hours of processing without success, they had to give up. Working with VMware support, they were able to get the VMware ESXi 5.5 host to boot, but were missing critical data from six of the iSCSI LUNs. This system was unique because it was using the EqualLogic LUNs as raw device mappings (RDMs) attached to the guest instead of the traditional virtual disks (VMDKs) on VMFS datastores.

The Solution

The event center called Ontrack at noon on a Saturday for emergency service.

Highly-trained data recovery engineers connected remotely to the EqualLogic LUNs using their proprietary remote data recovery (RDR) solution and started assessing the damage. During the evaluation, the engineers were able to locate the snapshots containing the missing data and virtually apply them to RDMs. Once the snapshots had been applied, the Ontrack engineering team was able to access the underlying NTFS volume, virtually repair the NTFS corruption, and extract the data.

The Resolution

Within 12 hours Ontrack was able to reunite the customer with the lost data which totaled over 250,000 files (-250GB of data).

“I was most impressed with the customer service I received from Ontrack throughout the data recovery process, the speed at which all the data was restored and the fact that during the entire restore process we were able to have our live environment up and running.”

Ontrack recovers over 230 million files from Commvault database.

The Client

The Situation

A large North American based manufacturer of building materials experienced a serious data loss due to a corrupt database file on their Commvault Media Server.

The client’s data storage solution consisted of the Commvault server as well as a Media Agent under the Commcell® management system. A Commvault Simpana® 9 was used for backup and archiving. The company lost access to media files crucial to the operation of their business.

The client attempted to open one of its virtual tape library backups only to find the MediaAgent database file, the gateway to the files on the media server, was corrupted and showed zero bytes in size.

The virtual tapes and the files stored inside the media server were still intact, but were not able to be accessed. The client contacted Commvault support and they attempted to retrieve the missing data from the media agent volume residing on the deployed Dell® MD1200 storage using the Commvault Media Explorer Tool. Their efforts proved to be unsuccessful. More than 3500 virtual tapes and 25 tape-sets were lost resulting in over 230 million inaccessible files. Fortunately, the client was able to copy the entire volume from the Dell server to external disks by using the Commvault Simpana 9 and sent them to Ontrack to recover the data. Unfortunately, the metadata containing the shortcuts leading from the database towards the files was not transferred with the media agent volume; therefore the files still could not be reached. Intensive investigation was necessary in order to discover the data mapping.

The Solution

Ontrack engineers were able to rebuild the original structure of the virtual array with the information contained on the external hard disks.

To reconstruct the data mapping used by the Commvault system, Ontracks internal software development team researched how Commvault distributes files, over the whole storage system and the built-in disks. After determining the custom mapping used by Commvault and decoding the virtual array, another challenge arose when the engineers discovered the virtual tape backup sets were also de-duplicated. The Ontrack software developers and data recovery engineers were able to create custom tools to

read the files from the system, rebuild the missing catalog and mapping information, and gain access to the file data. The engineers were then able to restore the files upon extraction and remove the de-duplication.

The Resolution

In the end, Ontrack engineers were able to recover the missing tape libraries as well as tape-sets with the 230+ million files included in just a couple of weeks.

The client was amazed at the success and speed of the recovery. With the insights gained into the Commvault system and the newly developed tools, the data recovery specialists at Ontrack are able to retrieve data even faster from all Commvault-based systems.

German service partner turns to Ontrack to recover business and personal data from external RAID drives.

The Client

The Situation

A marketing freelancer based in Switzerland entrusted all of their business and personal data to two RAID drives attached to their Apple Macintosh and asked their IT service partner for help when the data disappeared. The partner called on Ontrack to provide specialist engineering skills and know-how to fix the problem.

The client used an Apple Macintosh desktop to work on multiple design projects for clients, backing them up to two external RAID drives.

Both drives were attached directly to the Apple Mac, but there was no mechanism in place to perform automatic backups. One day, the client discovered that the drives had failed due to an unidentified issue, believing all their data had been lost in the process. The client was under pressure to deliver design and photography jobs to their customers on time and to budget. The system failure meant that the client’s entire business was at risk.

The Solution

The client approached their IT service provider for help to recover the lost data and they in turn sent the job to Ontrack.

The engineering team at Ontrack evaluated the two external devices and made the surprising discovery that the second external USB device consisted of two internal hard drives. The client, like so many small and home businesses, had taken advantage of the availability of low-cost external devices to extend the storage capability of their Apple Mac when they ran out of space. Having installed the first one terabyte drive, they implemented a special Span Set to attach the two terabyte drive - with a hidden internal stripe set inside. No emphasis had been placed on how the external storage should be configured, how often manual backups should be made or whether a larger, dedicated storage drive with automated backups would have been preferable to a more complex RAID array setup than it appeared to be in the first run. It wasn’t immediately clear to the engineers at Ontrack what had gone wrong with the client’s set-up, but since

there was no physical damage to the drives the problem was most likely caused by a power failure or an issue with the cables used to connect the drives to the computer. In any event, the complexity involved in the way in which the system was backing up data meant that while the set-up had appeared to work well for the client for a certain length of time, it was always at risk of sudden failure and data loss. While Ontrack’s service partner was able to help with an initial survey of the problem drives, it did not have the specialist engineering skills and resources to rebuild the file structures and retrieve the data - and could have potentially made the situation worse.

The Resolution

The engineers at Ontrack were able to rebuild and restore all of the information lost by the client: a total of 423,064 files and almost two terabytes of data.

The recovery was 100% successful and the drives were reconfigured correctly so that the same problem would not happen again. Some of the most common reasons for failure of backups to supply lost data are:

■ The external hard drives used by the majority of companies are only connected on an occasional basis, hence backup is not automated and instead performed on demand

■ The computer was not switched on during the scheduled backup nor configured to perform at a different time

■ The backup software failed

■ The backup ran out of destination space

■ The backup profile did not cover all of the device requiring backup

■ File was lost before the scheduled backup

Bonus Section

Having seen so many cases of critical data loss, Ontrack recommends the following tips to ensure backup success:

■ Take the time to invest in a backup solution and set up an automated backup schedule

■ Ensure backups are running regularly in accordance with the determined schedule

■ Check backup reports for error indications or failure

■ Test backups on a regular basis to ensure data has been accurately captured and files are intact

When things do go wrong, calling a trusted data recovery provider to identify and assess your data recovery options can increase the likelihood of successfully recovering your data.

24 terabytes of data recovered from RAID 6 array with newly developed toolset.

The Client

The Situation

A large UK Government organization had to learn the hard way that even RAID 6 arrays, known for their reliability, are not 100 percent impervious to hardware failure.

Unfortunately, the system failed to rebuild the data after two hard disk drives failed resulting in the loss of access to 24 terabytes of highly critical data. The organization approached the experts at Ontrack for help.

The client was using an Infortrend® EonStor RAID 6 array to run a range of business applications.

They experienced failures on two 2TB SATA drives in the system and replaced both of the failed drives. Even though it was a RAID 6, when the second drive failed, it also caused the array to fail. After the replacement drives were installed, the system failed to rebuild, which meant the business critical data was not accessible. The engineers at Ontrack virtually rebuilt the RAID 6 array with the two missing disks in order to recover the missing data. Due to the manufacturer-unique algorithm in a RAID 6 array, a rebuild of the secondary parity stripe from this specific system had not been completed before.

The Solution

Due to the rebuild failure, missing data from two failed drives had not been replicated onto the new drives when the new drives were added.

Being the client was using a RAID 6 array, the missing information could be rebuilt from the existing data on the other drives. The challenge of a RAID 6 recovery is locating the data to be restored; each RAID controller uses different algorithms and a concept called parity to create a RAID 6 configuration. To locate and access the missing data, Ontrack engineers developed a solution to support the Infortrend controller type. The engineering team utilized the specialized toolset to recover and rebuild all 24 terabytes of missing data from the RAID array.

Ontrack assembled a team of three data recovery engineers and two developers in order to provide the fastest possible recovery.

The Resolution

The Ontrack developers quickly created the tools needed to improve the success of the recovery. After only a few hours, the first virtual machine was rebuilt allowing for the extraction of the Exchange databases to be returned to the customer. The team continued to rebuild all of the critical virtual machines until the client’s email was back in production. At the end of the project, a total of 15TB of data was recovered with minimal downtime for the client.