Ransomware attacks server – backup tapes erased
A ransomware attack of a company server encrypted the Microsoft Dynamics 365 data and demanded payment. Recent backups of the server were stored on multiple LTO-6 backup tapes, which had been erased by the malware.
After assessing the extent of the ransomware attack, Ontrack representatives identified the company’s backup tapes as the best option for data recovery—even though the malware had erased them. 23 LTO-6 backup tapes from the backup library were sent to the Ontrack office in Böblingen, Germany. Working in conjunction with the R&D department in the United Kingdom, Ontrack developed a custom solution to recover the data from the erased backup tapes.
Ontrack was able to restore 46TB of data from 18 of the LTO-6 tapes. Due to the type of attack on the tapes, Ontrack had to repair the logical damage, shipping the data and tapes separately back to the customer.
Ransomware VBK Recoveries on Tape - Server & NAS Systems
The attacked volume was originally also used to back up data to LTO8 tapes at regular intervals. Most of these backup tapes were also in the tape library at the time of the incident and were quickly formatted by the attackers. However, the customer was able to save an original unformatted tape with a fairly old backup date, which was then completely restored to the now empty Windows volume with a total of 6 TB. Only then was Ontrack commissioned to examine data recovery options. The HP server DL380 with the 55 3TB hard disks were transported to Ontrack in Böblingen Germany.
During the diagnosis, a large number of the searched VEEAM vbk files were successfully found on the Windows volume with Ontrack Tools and 27 records were extracted according to a priority list. The restore of the LTO8 tape partially overwrote some of the data sets and damaged the backup files.
A large part of the data could still be repaired and extracted in several steps.
Later on, 19 significantly older LTO8 quick formatted tape backups were successfully recovered too.The attack also affected numerous European sub offices of the customer. Here were predominantly QNAP NAS systems in use which had stored virtual VMs under VMware, including backup VMs that were partially deleted or internally reformatted with another file system. Ontrack was also able to successfully restore complete backup data in 90% of the seven cases ordered.
Accidental Deletion of Virtual Machines Results in 15TB Lost.
An accidental deletion at a large wireless provider causes a massive loss of email databases.
The wireless carrier stored all of their Microsoft® Exchange databases spread across 24 separate 2TB LUNs on an EMC® VNX 5400 using VMware® virtual machines. It was also set up so each database had a mirror copy on a different LUN. All of the virtual machines were accidentally deleted resulting in the loss of email for the entire company.
New tools made recovery from highly specialized EMC® Isilon® big data storage possible.
The pharmaceutical company lost almost 4 million files of highly critical research and development data when moving files within the Isilon storage system.
The “lost files” were mostly raw data gathered from chemical analysis in ongoing laboratory research work. The firm, together with EMC support, was able to recover 90 percent of the data using a standard rebuild process. To recover the desperately needed remaining files, Ontrack was contacted by the customer for help. Ontrack worked with the EMC support in order to get detailed information on the situation.
The Isilon IQ 6000x, designed for big data, consisted of four nodes with 15 terabytes of data storage in total with 500 gigabyte hard drives installed.
It was set up as a single volume where all of the data was striped across the disks. Unlike a traditional RAID system, Isilon systems are built on the concept of a huge data lake where all data is stored and managed inside one data pool. To manage the system and this “data lake” a unique file system was created by Isilon called OneFS. In this case one of the four nodes inside the system experienced a kernel panic and several disks showed multiple errors. EMC was able to gather most of the files with a rebuild before a consistency check showed that several disks had errors. To recover the missing data, the file system had to be analyzed by the Ontrack team to find out how the data was laid out in the whole storage system.
To find out how the data originally was distributed over the disks and determine the data mapping, Ontrack s own R&D department developed a brand new set of tools unique to the OneFS.
With these tools, an existing OneFS volume can be analyzed in depth and missing or faulty data file structures inside the “big” data lake can be discovered even more quickly. With the now gathered information on how the data structure of the Isilon system was setup, the engineers were finally able to rebuild and recover the missing data.
In the end, Ontrack engineers were able to recover almost all of the missing files with only a couple of bad files due to damaged encryption.
The client was delighted that Ontrack handled such a complex recovery so quickly. None of the confidential data was compromised and the solution provided by Ontrack was cost-effective and completed within a very short timeframe. With the newly developed tools for this case, Ontrack is currently the only data recovery service provider with the ability to recover data from Isilon storage systems.
Ontrack is assisted by NetApp’s technology to solve a ransomware infection.
A single user’s laptop at a large pharmaceutical company was infected with CryptoLocker ransomware.
This malware encrypts the user’s files and withholds the encryption key until you pay the ransom amount. The laptop was connected to the corporate network which allowed the malware to infect a CIFS volume which was set up as a file share on a NetApp FAS.
The malware was able to infiltrate the file share and encrypt the majority of the files. The IT team was not notified of the infection until after the backup retention period had expired, meaning that the backup contained only encrypted data. The total impact resulted in inaccessible data on:
- 46 drives
- 1 aggregate
- 1 volume infected on a RAID-DP
To perform the recovery, the aggregate needed to be taken offline, which affected 17 volumes in total.
The customer brought their 46 drives into our New Jersey lab for evaluation and Ontrack engineers got to work on a solution.
The engineering team from Ontrack:
- Virtually rebuilt the RAID groups which were strewn across 10 different shelves
- Virtually rebuilt the aggregate
- Virtually rebuilt the critical volume
An additional challenge on this recovery was that the aggregate was in use for two weeks after the incident occurred which resulted in some data being overwritten.
Ontrack was able to virtually rebuild the volume containing the CIFS share and encrypted data.
Leveraging NetApp’s proprietary OS (OnTap) and file system (WAFL), Ontrack engineers used multiple consistency points to “walk back” in time to find and merge unencrypted copies of the critical data to return to the customer. This type of recovery is only possible on storage like NetApp’s FAS because of the way the data is stored on the volume.
Partnering with EMC Creates New Tools for Recovery of 10 TB of Data.
Due to the Copy-On-Write system used by Isilon OneFS, the engineering team at Ontrack was able to find many copies of the deleted file metadata in the unallocated portions of the volume. They extracted 120,000 business critical files which were accidentally deleted and returned them back to the customer.
Double disk failure of RAID 5, and all patient data is gone.
Hospital databases rescued from ransomware.
Ontrack Provides Database and Backup Restores After a Flood.
Missing Dell® EqualLogic™ LUNs Recovered via Remote Data Recovery.
Ontrack recovers over 230 million files from Commvault database.
24 terabytes of data recovered from RAID 6 array with newly developed toolset.
Unfortunately, the system failed to rebuild the data after two hard disk drives failed resulting in the loss of access to 24 terabytes of highly critical data. The organization approached the experts at Ontrack for help.
The Ontrack developers quickly created the tools needed to improve the success of the recovery. After only a few hours, the first virtual machine was rebuilt allowing for the extraction of the Exchange databases to be returned to the customer. The team continued to rebuild all of the critical virtual machines until the client’s email was back in production. At the end of the project, a total of 15TB of data was recovered with minimal downtime for the client.
Accidental wipe command brings down critical production database server.
Four terabytes of data recovered from flood damaged HP EVA SAN
A flooded data center left a client’s servers and storage systems partially submerged in water.
At the center of the damage was a HP Storage Works EVA (Enterprise Virtual Array) 6000 containing business critical SQL database files as well as employee file shares. The EVA sustained substantial physical damage due to the flood water preventing access to the data. The severity of the damage from the flood was increased when an attempt was made to access the data by powering on the drives that were still wet. The customer contacted HP Support for help and they handed the project over to Ontrack.
The SAN consisted of 80 hard disk drives which were divided into 2 EVA disk groups; in total there were 18 virtual RAID volumes consisting of both VRAID1 and VRAID5.
A HP EVA system is fully virtualized and has a unique way to write data which adds to the complexity of any data recovery effort. It works with disk groups and virtual disks instead of normal RAID sets and logical drive volumes. The disk groups consist of physical drives organized in a proprietary manner. LUNs or Virtual disks (vDisks) in an EVA are then distributed over all of the installed HDDs.
Due to the physical damage, all of the drives were sent to one of Ontrack´s cleanroom facilities to be assessed.
Once the 80 drives were decontaminated and cleaned, 55 were found to be fully recoverable. 25 of the drives had severe water damage and were not recoverable. To regain access to the data on the damaged drives, the engineers needed to research how the EVA RAID and file system was structured. After the engineers were able to map the disk groups and determine how the vDisks were distributed, they had to rebuild the whole EVA system. To recover the data included in the vDisks, the R&D team and its software developers had to create completely new tools to extract the data. Once the development was complete, Ontrack engineers virtually assembled the disk groups and virtually rebuilt the vDisks which allowed access to the underlying file systems. The file systems were virtually repaired and the data extracted.
After extensive development, reengineering and recovery work the project successfully ended.
With the newly created tools the data recovery specialists were able to recover four terabytes of sensitive data including the critical SQL database files. In all, approximately 86 percent of the total data lost was recovered. With the HP EVA SAN data mapping knowledge gained and the integration of the newly developed tools, Ontrack is able to quickly recover data from all models of the HP Enterprise EVA storage systems.