Data Loss Experiences for Users of Virtual Platforms

Thursday, June 29, 2017 by Michael Nuncic

Users of virtual platforms are victims of data loss because of their own mistakes

Virtualization is still a major issue when it comes to a company’s IT infrastructure, leading to countless incidents of data loss. As the technology matures, the benefits remain the same:  lower hardware costs, lower energy consumption and less administrative work hours for maintaining the guest operating systems and applications. Market analysts from Gartner estimate that companies already reach an internal server virtualization rate of over 75%. Today, every main computer technology can be part of virtual platform.  This includes: storage, operating systems, network or applications.

Even though virtualization hypervisors have become better and better, there is still a good chance that a data loss will occur. Since 2008, when Kroll Ontrack first introduced data recovery for the two most widely used virtualized platforms - VMware and Microsoft Hyper-V, their data recovery specialists have seen a variety of data losses.

From the beginning, Kroll Ontrack virtualization experts gathered information and data on the reasons for data loss with a VMware or Hyper-V based system. What comes as surprise, is that after almost a decade of computer virtualization in companies, users make almost the same mistakes today in both VMware and Microsoft Hyper-V based virtual platforms as they did in the beginning.

Common reasons for data losses in VMware and Hyper-V

Kroll Ontrack used analysis and insights from its internal database of all worldwide data recovery labs and published statistics on the most common problems that lead to data loss in virtualized systems.

According to these figures, the five most common reasons for data loss in virtualized environments in 2016 were:

  • Deleted VMs (Virtual Machines) (40%)
  • Hardware failures (30 %)
  • Migration Failures (10%)
  • Snapshots gone wrong (10%)
  • Other (10%)

When you compare these figures with the ones from 2010, they differ only slightly. In 2010, hardware/RAID failures accounted for 40% of all data losses in virtual systems, while deleted virtual disks and/or snapshots totaled 36%. These figures clearly show that the most common reasons remained almost the same. What has become less of a factor over the years is formatting and reinstallation problems that lead to data loss. While this accounted for 10% of all cases in 2010, the figure has dropped and is now just one of the many reasons combined in the 10% of “other reasons”.

A good sign is that data loss due to the used virtualization platform itself became less a factor over the years. As an example: “VMFS Metadata Corruption” accounted for 13% and “virtual disk corruption” for 1% of all “virtual” data losses in 2010. In 2016, these reasons for data loss have fallen dramatically and are now part of the 10% “other reasons” section. This clearly shows that the virtualization platform software products have become better and more reliable other the years.

While virtual platforms became better, many users didn’t!

What has remained the same is that many failures and data losses are due to human error:

  • Deleted Virtual Machines
  • Migration Failures
  • Snapshots gone wrong

In most of these human error cases, the IT administrator simply made poor decisions. With better user interfaces and administration panels, deleting or migrating hundreds or even thousands of virtual machines or virtual disks is now achievable by just a simple click of a button. The main risk here is that sometimes the user is either too fast or they don’t really know what they are doing. The administrator then attempts to recover the missing data by using internal or 3rd party recovery tools.

How can data recovery experts help in virtualization related data losses?

For almost a decade, Kroll Ontrack engineers have developed highly specialized tools to recover data from both VMware virtual platforms, like VMware ESX, VMware Virtual SAN (VSAN), as well as Microsoft virtual platforms, including Hyper-V, Virtual Server and Virtual PC. For Hyper-V alone, Kroll Ontrack researchers have developed a unique Hyper-V tool suite for their data recovery specialists, which can help in almost all common data loss scenarios listed above.

Even though Kroll Ontrack has the all the necessary tools to recover data from virtual platforms and systems like VMware ESX and Microsoft Hyper-V, every case is different. Utilizing a data recovery specialist with deep knowledge is critical.

How are missing VMs recoverable after deletion?

First, the specialists try to find the original pointers to the missing virtual disk. If found, they virtually rebuild the pointers to the missing virtual disk and extract the data. Pointers can be missing becasue of corruption or blocks errors. Kroll Ontrack engineers can extract the data inside the virtual disk.  In the situation where pointers are missing, engineers will scan the entire data store for the blocks that make up the missing virtual disk.  They will then virtually put all identified blocks of data together in the right order and extract the data.

Hardware failure! OMG!

First, all individual drives belonging to the system are imaged in the clean room. If some drives are ruined, the experts will replace them and use special tools to virtually rebuild the RAID or storage system. Once the storage has been virtually rebuilt, the engineers will attempt to recover the missing data.

What if a migration goes wrong?

Typically when a migration goes wrong, the experts examine the internal structure of the virtualized source storage first and then rebuild the original virtual machine.

How is virtual data recoverable when snapshots have gone awry?

If the snapshots are missing or deleted, the specialists will try to find and virtually rebuild the original pointers to address the original files.  If they cannot recover the pointers, they will then recover the raw blocks that make up the snapshot and virtually rebuild them. When a snapshot is corrupted, engineers will use Kroll Ontrack’s special tools to merge the flat file with the corrupted snapshot. Afterwards, they virtually repair any file system damage at the guest level and then extract the data inside of the virtual disk and/or the snapshots to get to the desired original files in both cases.

Data recovery from virtual platforms and systems is too difficult and dangerous for non-professionals

As one can see, data recovery from virtual platforms and systems is definitely not an easy task. It involves a lot of knowledge, specialized tools, as well as time. In many cases, several technological challenges are combined in one single data recovery project due to the different data structure layers that have to be virtually “recreated” in order to regain access to the original files. Therefore, if a data loss occurs, it is wise for the user to stay calm and \not attempt a do-it-yourself data recovery. In many of these attempts, the missing data is still available on the hard disks but can be permanently deleted when trying DYI. Additionally, it is wise to power down the affected hardware, so no applications can run any scripts on the disk and overwrite the “missing” data. In any case, it is far better to contact a professional data recovery service provider like Kroll Ontrack. The experts can give sound advice and a professional estimation of the needed steps for a successful data recovery from a virtual platform or system.