Only Available at Ontrack: IBM Storwize Data Recovery

The Client

The Situation

A client recently experienced a remote ransomware attack that resulted in Ontrack engineers being presented with one of their most extraordinary data recovery efforts to date: restoring 120 damaged HDDs within an IBM SVC Storwize v7000 system…with no backup to rely on. 

The Solution

After noting the critical nature of the project, Ontrack’s data recovery experts proceeded with a comprehensive process to potentially restore the deleted data:

  1. Consultation
    The Ontrack team was able to join the client’s team and scope the data loss event and the storage systems impacted. Based on the scope set forth, Ontrack was able to determine a project plan, set timing expectations and determine costs for data recovery.

     

  2. Diagnosis
    Data recovery engineers used Ontrack’s proprietary tools to analyze the disks, determine the likely array configuration as well as detect indications of Windows storage space and VMware storage virtual machines.

  3. R&D Simulation and Software Programming
    After the initial diagnosis, engineers analyzed a minimal hardware setup of the IBM SVC Storwize v7000 as a means of detecting the layout of on-disk structures used to map Raid Arrays, including managed disks, SVC pools, virtual disks, and physical disks (=LUN).


    Ontrack then began to work closely with the client’s IT department to get the hardware running on a new setup of the IBM SVC Storwize v7000 system.

    Simulations were performed to see if the client’s environment could be recreated on the live hardware and if any structure could be found to possibly reconstruct the deleted data. All findings regarding the simulated structures were compared to the structures on the original hard drives.

    A positive prediction was formed based on the comparison of the structures, and Ontrack was able to move forward with the creation and modification of proprietary tools to extract functional storage systems and proceed with successful SAN system data recovery.

  4. Data Recovery

With an enormous challenge ahead of them, Ontrack’s data recovery experts performed extensive research on IBM’s proprietary software which resulted in engineers modifying their recovery tools to allow for the virtual rebuild of the DRAID that was in use on the IBM system.

Figuring out the distribution patterns for DRAID proved to be the most intricate part of the recovery process, given that all of the data sitting on the DRAID6 MDisk was combined with a number of other MDisks and dynamically allocated multiple levels of both VDisks and Dynamic Disks.

Once the array was virtually rebuilt, the Ontrack team was able to virtually rebuild the volumes, transforming them into 1,152 devices in order to display the overall layout of available data contained within to generate reports for the client and complete the IBM Storwize data recovery.

The Resolution

When the client initially introduced the issue, there was little hope for full (if any) recovery given the complex nature of the IBM Storwize data storage system. However, thanks to the diligence of our engineers, an unprecedented Ontrack data recovery solution for all IBM Storwize systems is now solely available via Ontrack.

Ontrack Performs Emergency Raid 5 Data Recovery

The Client

The Situation

A multinational client had fallen victim to the disappearance of crucial financial data and office files, with no backup data plan in place. After realizing the urgency of the data loss situation, they immediately contacted  Ontrack’s expert engineers to help with recovery. 

The Solution

Given the importance of the lost data, the client expedited all media over the weekend via courier to an Ontrack engineer who happened to be on call. Per client request, Ontrack provided project updates every 15 minutes as a data recovery plan was being prepared.

It was quickly discovered that the client’s hard drives used  the Raid 5 system, and in order to have a successful data recovery, engineers would have to virtually rebuild the drives to guarantee perfect quality of the data found on their  VMFS VMware ESX 5.1 volumes.

The Resolution

Through a fantastic show of teamwork and diligence, Ontrack’s data recovery engineers were successfully able to bring all drives back to nearly 100% of their data volume within 6 days after rebuilding the system; a testament to the company’s unmatched expertise and dedication to solving even the most complex data challenges. The client expressed enthusiastic gratitude for such quick data recovery and was excited to have their company back up and running in just under a week with the help of  Ontrack’s emergency recovery service

Damaged Mac Laptop? Ontrack to the Rescue.

The Client

The Situation

A customer contacted Ontrack’s experts in need of help with Mac data recovery for a laptop that had been dropped and accidentally run over by a truck, causing its battery to combust. 

The Solution

Ontrack’s recovery experts surveyed the Mac device damage and implemented a 3-step recovery process for data extraction which included:

  • Decontamination – A thorough process in which all contaminants that adversely affected the damaged Mac laptop’s operability were removed.
  • Micro-Soldering – A small soldering tool was used to replace a pool of damaged capacitors and resistors which were located near the burned battery.
  • Diagnostics – The MacBook was placed in a special diagnostic mode where both Apple-provided OEM tools and Ontrack’s own proprietary tools were used to mount the internal storage device as a volume onto lab machines and process the copy out.

Once the process was complete, Ontrack was able to successfully extract and verify all data.

The Resolution

Some accidents aren’t easy to account for. Ontrack’s recovery experts can be trusted to provide satisfactory service when it comes to Mac recovery, as well as data recovery for similar devices, whether damage is due to an occasional drop and break or a truck that comes out of nowhere.

A Deep Dive for Dell/EMC Isilon Data

The Client

The Situation

A client inadvertently ran a command that deleted critical files on a Dell/EMC Isilon storage array containing 270 disks totaling 2 petabytes (PB). 

The Solution

After having the drives in question flown in, Ontrack’s recovery engineers began an in-depth evaluation and determined that the JIT development team would be needed to assist in data recovery from this version of Isilon. Within eight weeks of working nights, weekends, and holidays to develop a proper solution, an initial set of data was delivered to the client.

While the client was ecstatic to receive the recovered files, they also requested that subsequent tasks be done to prove that no stone was left unturned in their recovery efforts, per regulatory requirements. Ontrack’s JIT development team complied by combing through the 270 drives in search of files that matched regulatory requirements and implementing a process that would search, copy, and deduplicate specific files that were found across each disk.

After months of conducting a thorough secondary search, more than 300 million files and 13 terabytes (TB) of PDF and JPG files were produced. 

The Resolution

This project exemplifies the ability of Ontrack’s team of recovery experts to go above and beyond to meet the client’s needs. Our team of engineers is not only well-equipped to restore files lost from Dell/EMC Isilon storage, but they are also prepared to help your company provide the proper proof of data recovery according to regulatory standards.

Ransomware Recovery – Veeam Agent for Windows

The Client

The Situation

A health care customer was affected by a ransomware attack that not only targeted their server data, but also “Veeam Agent for Windows” backups located on an external HDD. Their IT / managed services provider agreement did not include regular off-site backups, so this was the only copy of the data that existed.

The Solution

The customer was able to send the affected HDD to Ontrack, where an image of the drive was taken to preserve the original state of the customer media.

Ontrack engineers assessed the damage to the affected Veeam backup files and identified that partial recovery would be possible as the files had not been fully encrypted, meaning there was a chance that some data could be recovered from within the files. However, it was determined that the version of Veeam used was newer than Ontrack could support with current tools and required development assistance.

With a global engineering presence, as well as internal development teams that maintain and improve our proprietary tools, Ontrack was able to research, develop and implement support for the new version quickly. In fact, much of the time-intensive research required had already been completed for similar jobs seen in our European offices. This allowed Ontrack developers to quickly and efficiently modify tools to the level required to be able to support this restore scenario. Rather than building out a fully-fledged tool, Ontrack engineers were able to use the improved version of the tools to complete searches for required structures to allow them to manually rebuild internal components critical to the recovery of data from within the file.

The Resolution

Once repairs to the files had been completed, engineers were able to use their remaining Veeam tool set to complete an extraction of data from within the repaired files. The recoverable data consisted of many flat file data types that had been completely lost to the customer during the ransomware attack.

IBM server with a RAID 5 comprised of 5 SCSI hard drives failed

The Client

The Situation

According to the manager of the resort, the server contained every single record of their operations since the day that they opened – property management, reservations, accounting, reporting and revenue, payroll – everything. After allowing a local IT provider to work on the server for 8 hours, they referred the job to Ontrack Data Recovery. 
By that stage, the resort's server had been out of operation for a full day and they were starting to run out of time. There wasn’t enough time to wait for couriers so the hard drives were put on a plane with a staff member and flown
to the nearest Ontrack location. 

The Solution

They arrived in the Ontrack office and clean room facility at approximately 3pm in the afternoon, and within two hours the client was given confirmation that the data was recoverable. “From speaking to the client, I knew that his business would be in serious trouble if data was not restored quickly” explains Adrian Briscoe. “Due to our ‘follow the sun’ support capabilities, our local engineers were able to image the hard drives and then send the images to teams in Europe and the US where they  pieced the RAID back together. The critical data was then uploaded to the FTP and made available for the client to download.”

The Resolution

The recovery was 100% successful and every single file that was on the server was able to be recovered. When asked about the Ontrack service, the client said “communication was great I would say, there was response and we didn’t have to sit around the resort waiting for communication or having to chase updates ourselves – we were informed about what was happening every step of the way.” They continued on to state that Ontrack scored “10 out of 10 – communication, speed, response, recovery. I would definitely recommend their services to other businesses.”

Ontrack supports an IT service provider to ensure its end-customer can access legacy backup tapes

The Client

An IT service provider had to guarantee access to the legacy backup tapes of a new end-customer from the insurance industry.

The Situation

The end-customer needed to have access to data on a large number of 3592 and 3592/JA tapes for a period of five years, in order to comply with data retention and governance regulations. The backup tapes had been created using Tivoli Storage Manager, however the end customer did not want to incur the costs of maintaining this environment for a five year period for infrequent backup tape restore requests.

The Solution

Working with the IT service provider, Ontrack came up with a cost effective and efficient solution to allow continued direct access to the end-customer’s backup tapes over the retention period.

The first step was to conduct a Proof of Concept (PoC) where the end-customer sent 5 tapes to Ontrack. The purpose of the PoC was to ensure full support for the client’s tape and backup software combination and confirm the scope of the project and timelines.

Following the PoC, Ontrack extracted the Tivoli Storage Manager catalogue into a standard database format. This allowed the IT service provider to identify the location of folders or specific files and send the relevant tapes for restore to Ontrack when required. The data from the tape restore could then be delivered back to the end-customer on an encrypted USB drive or via secure FTP for low volume or urgent restore requests.

The data restores and maintenance of the required infrastructure are covered by a multi-year tape service agreement allowing for a pre-defined number of tape restores, with incremental restores available upon request.

The Resolution

The IT service provider was able to offer a bespoke tape processing service to their end-customer which meant that they no longer had to sustain the significant costs of maintaining a Tivoli Storage Manager environment while still having the ability to extract data from the tapes as and when required.

The multi-year tape service agreement for restoring backup tapes on demand meant that the services were tailored to the client’s needs and ensured greater predictability in expenditure planning and budgeting. The service agreement also allows requests for, and delivery of, tape restores to be processed quickly and efficiently and avoid administrative delays. Under the service agreement Ontrack is able to guarantee the tape extraction capabilities over the full period of the contract.

Ontrack computer forensics team assists pharmaceutical company with onsite tape cataloging and restoration.

The Client

A pharmaceutical company carrying out an internal investigation on product liability.

The Situation

The client encountered a product liability issue which led to an internal investigation involving product managers as well as other management staff. The internal legal department, together with the internal investigators, needed email files to be restored from a total of 65 LTO 3 and 4 tapes to an external storage device. A catalog was also required with searchable listings of items on the recovered tapes.

The Solution

Working with the pharmaceutical company, Ontrack developed a solution which allowed the client to maintain the security requirements of the internal investigation while providing the evidenciary data required.

A team of two computer forensic specialists from Ontrack were sent with their mobile hardware devices required for carrying out the project. This included seven high-end computers, a tape reader and other storage reading devices.

The team inspected all the tapes in order of save sets. They also verified the labels (based on creation dates) and performed a tape restore of the backup tapes using proprietary data recovery software in order to partially read damaged save sets and tapes. Our specialists were able to restore most of the Exchange database files and export several .pst files that were significant to the case.

The Resolution

All the restored data was extracted on to a new storage device, which had been deployed on-site for the purpose of the investigation.

Additionally, Ontrack computer forensic specialists provided the client with a catalog and searchable listing of items that were stored on the recovered tapes. This allowed the client to search for not only emails regarding this specific case, but also others that were within the backup timeframe covered by the 64 tapes.

Ransomware attacks server – backup tapes erased

The Client

The Situation

A ransomware attack of a company server encrypted the Microsoft Dynamics 365 data and demanded payment. Recent backups of the server were stored on multiple LTO-6 backup tapes, which had been erased by the malware. 

The Solution

After assessing the extent of the ransomware attack, Ontrack representatives identified the company’s backup tapes as the best option for data recovery—even though the malware had erased them. 23 LTO-6 backup tapes from the backup library were sent to the Ontrack office in Böblingen, Germany. Working in conjunction with the R&D department in the United Kingdom, Ontrack developed a custom solution to recover the data from the erased backup tapes.

The Resolution

Ontrack was able to restore 46TB of data from 18 of the LTO-6 tapes. Due to the type of attack on the tapes, Ontrack had to repair the logical damage, shipping the data and tapes separately back to the customer.

Ransomware VBK Recoveries on Tape - Server & NAS Systems

The Client

The Situation

The attacked volume was originally also used to back up data to LTO8 tapes at regular intervals. Most of these backup tapes were also in the tape library at the time of the incident and were quickly formatted by the attackers. However, the customer was able to save an original unformatted tape with a fairly old backup date, which was then completely restored to the now empty Windows volume with a total of 6 TB. Only then was Ontrack commissioned to examine data recovery options. The HP server DL380 with the 55 3TB hard disks were transported to Ontrack in Böblingen Germany.

The Solution

During the diagnosis, a large number of the searched VEEAM vbk files were successfully found on the Windows volume with Ontrack Tools and 27 records were extracted according to a priority list. The restore of the LTO8 tape partially overwrote some of the data sets and damaged the backup files. 

The Resolution

A large part of the data could still be repaired and extracted in several steps.

Later on, 19 significantly older LTO8 quick formatted tape backups were successfully recovered too.The attack also affected numerous European sub offices of the customer. Here were predominantly QNAP NAS systems in use which had stored virtual VMs under VMware, including backup VMs that were partially deleted or internally reformatted with another file system. Ontrack was also able to successfully restore complete backup data in 90% of the seven cases ordered.

New tools made recovery from highly specialized EMC® Isilon® big data storage possible.

The Client

The Situation

The pharmaceutical company lost almost 4 million files of highly critical research and development data when moving files within the Isilon storage system.

The “lost files” were mostly raw data gathered from chemical analysis in ongoing laboratory research work. The firm, together with EMC support, was able to recover 90 percent of the data using a standard rebuild process. To recover the desperately needed remaining files, Ontrack was contacted by the customer for help. Ontrack worked with the EMC support in order to get detailed information on the situation.

The Isilon IQ 6000x, designed for big data, consisted of four nodes with 15 terabytes of data storage in total with 500 gigabyte hard drives installed.

It was set up as a single volume where all of the data was striped across the disks. Unlike a traditional RAID system, Isilon systems are built on the concept of a huge data lake where all data is stored and managed inside one data pool. To manage the system and this “data lake” a unique file system was created by Isilon called OneFS. In this case one of the four nodes inside the system experienced a kernel panic and several disks showed multiple errors. EMC was able to gather most of the files with a rebuild before a consistency check showed that several disks had errors. To recover the missing data, the file system had to be analyzed by the Ontrack team to find out how the data was laid out in the whole storage system.

The Solution

To find out how the data originally was distributed over the disks and determine the data mapping, Ontrack s own R&D department developed a brand new set of tools unique to the OneFS.

With these tools, an existing OneFS volume can be analyzed in depth and missing or faulty data file structures inside the “big” data lake can be discovered even more quickly. With the now gathered information on how the data structure of the Isilon system was setup, the engineers were finally able to rebuild and recover the missing data.

The Resolution

In the end, Ontrack engineers were able to recover almost all of the missing files with only a couple of bad files due to damaged encryption.

The client was delighted that Ontrack handled such a complex recovery so quickly. None of the confidential data was compromised and the solution provided by Ontrack was cost-effective and completed within a very short timeframe. With the newly developed tools for this case, Ontrack is currently the only data recovery service provider with the ability to recover data from Isilon storage systems.

Ontrack is assisted by NetApp’s technology to solve a ransomware infection.

The Client

The Situation

A single user’s laptop at a large pharmaceutical company was infected with CryptoLocker ransomware.

This malware encrypts the user’s files and withholds the encryption key until you pay the ransom amount. The laptop was connected to the corporate network which allowed the malware to infect a CIFS volume which was set up as a file share on a NetApp FAS. The malware was able to infiltrate the file share and encrypt the majority of the files. The IT team was not notified of the infection until after the backup retention period had expired, meaning that the backup contained only encrypted data. The total impact resulted in inaccessible data on:

■ 46 drives

■ 1 aggregate

■ 1 volume infected on a RAID-DP

To perform the recovery, the aggregate needed to be taken offline, which affected 17 volumes in total.

The Solution

The customer brought their 46 drives into our New Jersey lab for evaluation and Ontrack engineers got to work on a solution.

The engineering team from Ontrack:

■ Virtually rebuilt the RAID groups which were strewn across 10 different shelves

■ Virtually rebuilt the aggregate

■ Virtually rebuilt the critical volume

An additional challenge on this recovery was that the aggregate was in use for two weeks after the incident occurred which resulted in some data being overwritten.

The Resolution

Ontrack was able to virtually rebuild the volume containing the CIFS share and encrypted data.

Leveraging NetApp’s proprietary OS (OnTap) and file system (WAFL), Ontrack engineers used multiple consistency points to “walk back” in time to find and merge unencrypted copies of the critical data to return to the customer. This type of recovery is only possible on storage like NetApp’s FAS because of the way the data is stored on the volume.

Missing Dell® EqualLogic™ LUNs Recovered via Remote Data Recovery.

The Client

The Situation

A large municipal event center in the US lost data on a Dell® EqualLogic™ iSCSI SAN configured with in a RAID 50 running VMware® ESXi™ 5.5.

VMware snapshots filled up the datastore causing the system to crash. The customer attempted to delete one of the snapshots, but after four hours of processing without success, they had to give up. Working with VMware support, they were able to get the VMware ESXi 5.5 host to boot, but were missing critical data from six of the iSCSI LUNs. This system was unique because it was using the EqualLogic LUNs as raw device mappings (RDMs) attached to the guest instead of the traditional virtual disks (VMDKs) on VMFS datastores.

The Solution

The event center called Ontrack at noon on a Saturday for emergency service.

Highly-trained data recovery engineers connected remotely to the EqualLogic LUNs using their proprietary remote data recovery (RDR) solution and started assessing the damage. During the evaluation, the engineers were able to locate the snapshots containing the missing data and virtually apply them to RDMs. Once the snapshots had been applied, the Ontrack engineering team was able to access the underlying NTFS volume, virtually repair the NTFS corruption, and extract the data.

The Resolution

Within 12 hours Ontrack was able to reunite the customer with the lost data which totaled over 250,000 files (-250GB of data).

“I was most impressed with the customer service I received from Ontrack throughout the data recovery process, the speed at which all the data was restored and the fact that during the entire restore process we were able to have our live environment up and running.”

24 terabytes of data recovered from RAID 6 array with newly developed toolset.

The Client

The Situation

A large UK Government organization had to learn the hard way that even RAID 6 arrays, known for their reliability, are not 100 percent impervious to hardware failure.

Unfortunately, the system failed to rebuild the data after two hard disk drives failed resulting in the loss of access to 24 terabytes of highly critical data. The organization approached the experts at Ontrack for help.

The client was using an Infortrend® EonStor RAID 6 array to run a range of business applications.

They experienced failures on two 2TB SATA drives in the system and replaced both of the failed drives. Even though it was a RAID 6, when the second drive failed, it also caused the array to fail. After the replacement drives were installed, the system failed to rebuild, which meant the business critical data was not accessible. The engineers at Ontrack virtually rebuilt the RAID 6 array with the two missing disks in order to recover the missing data. Due to the manufacturer-unique algorithm in a RAID 6 array, a rebuild of the secondary parity stripe from this specific system had not been completed before.

The Solution

Due to the rebuild failure, missing data from two failed drives had not been replicated onto the new drives when the new drives were added.

Being the client was using a RAID 6 array, the missing information could be rebuilt from the existing data on the other drives. The challenge of a RAID 6 recovery is locating the data to be restored; each RAID controller uses different algorithms and a concept called parity to create a RAID 6 configuration. To locate and access the missing data, Ontrack engineers developed a solution to support the Infortrend controller type. The engineering team utilized the specialized toolset to recover and rebuild all 24 terabytes of missing data from the RAID array.

Ontrack assembled a team of three data recovery engineers and two developers in order to provide the fastest possible recovery.

The Resolution

The Ontrack developers quickly created the tools needed to improve the success of the recovery. After only a few hours, the first virtual machine was rebuilt allowing for the extraction of the Exchange databases to be returned to the customer. The team continued to rebuild all of the critical virtual machines until the client’s email was back in production. At the end of the project, a total of 15TB of data was recovered with minimal downtime for the client.

A leading IT services company needed to access legacy tapes in a short timeframe.

The Client

A leading global IT services company, who provides technology-enabled business solutions & services.

The Situation

As part of a larger bespoke IT project the client was facilitating for a third-party, the identification, restoration and compliant archiving of 2,500 individual Microsoft Exchange mailboxes was required.

The third-party client employed rigorous data backups throughout their global locations. However their new backup system was not compatible with the legacy system, and data restores required from the legacy system put severe pressure on the ability to maintain the backup schedule.

As a result, the third-party client required assistance to restore 6,000 mailboxes from 40 LT03 tapes, then identify, extract and process the required 2,500 mailboxes all within a very tight timeframe.

The Solution

After consultation with Ontrack tape services specialists, the project timescales, budget and critical success factors were identified, a project proposal and delivery schedule were formulated, and the tapes sent to Ontrack for analysis.

The Resolution

With extensive tape restoration and extraction experience, the engineers at Ontrack tape services successfully extracted and mounted the 40 EDBs contained on each of the tapes. Using the proprietary Verifile™ data reporting software, the client identified the required 2,500 mailboxes and Ontrack then employed their advanced ‘find & restore’ software technology to extract the required mailboxes. Ontrack delivered the required mailboxes in an encrypted and compatible PST format.