Myth: Deleted VMware VMDK files cannot be recovered

With reference to the myth that deleted VMware® virtual disks are not recoverable I would like to discuss a case where Ontrack busted this myth wide open.

The Client

A VMware customer contacted VMware support after an incident at their premises. Due to the severity of the damage and uncertainty of what exactly had happened VMware support asked for their agreement for Ontrack to help assess the data loss situation and see if they could recover any of the critical volumes. Ontrack on many previous occasions had helped VMware’s customers’ to recover their lost data.

They were a London-based drinks import business with a turnover of approximately £10 million per annum.

The Situation

The employee in charge of the IT systems requested a pay rise. When his request was declined he left the company. Following his departure the company detected they had been hacked and the entire virtual infrastructure was destroyed and they were unable to access any of their systems. Suspicions immediately were placed on the callous ex-employee.

The System

The company ran a SAN with two 270GB VMFS (Virtual Machine File System) LUN (Logical Unit Number) and one 4TB NTFS LUN:

• The two VMFS volumes contained virtual machines running all the business systems, i.e. the email server, SQL server, file server, etc.
• The NTFS volume contained Veeam backups of all the virtual machines from the VMFS volumes.

Data lost

All the systems running in the VMware environment were lost. This included email servers, SQL servers running the business systems, file servers hosting documents from all the employees.

All the backups for the systems mentioned above which were stored in the NTFS partition, which were also lost.

Recovery attempts by the customer

The customer tried to bring the systems online through various methods. It was later found that in their attempts they reformatted the NTFS volume which contained the backup files, further damaging data.

The customer contacted Ontrack Legal Technologies computer forensics team to help investigate the breach, but expressed that their main priority was to get their business back up and running.

The forensics team then called in the Ontrack data recovery team in order to coordinate the data gathering in a way that allowed the recovery of the systems without compromising the validity of the data in any forensic investigations carried out afterwards.

The Solution - work carried out by Ontrack

The first step in this recovery was to create a copy of the dataset following computer forensics evidential preservation guidelines. For this Ontrack sent an engineer onsite with the equipment necessary to clone the systems.

Once the clone copy of the data was complete it was brought back to the data recovery lab to start on the data recovery work.

An analysis was carried out on the NTFS Veeam backup volume first since in theory, this volume would have all the required data in backup files.  However, the investigation carried out by the data recovery engineers determined that this volume had suffered further damage when an attempt to bring it online led to a reformat. This reformat laid enough new structures on the LUN to overwrite critical file records required for the Veeam backups database.

Following this discovery, work on the two VMFS LUNs  started right away. Recovering deleted virtual machines on VMFS volumes can be one of the most complex recovery scenarios Ontrack engineers have to deal with.  The addition of snapshots and thin provisioning can make things even more complicated.  However, since Ontrack conducted their first deleted virtual machine recovery a few years previous, the proprietary recovery software tools and techniques have been significantly improved through ongoing development.

The Outcome

Within 48 hours Ontrack engineers had recovered all critical deleted VMDKs (Virtual Machine Disk). These were provided to the customer who then imported them into a new environment and were able to get back up and running immediately.

The Lesson

Malicious damage is one of the most difficult disaster recovery scenarios to protect against and although the risk of it occurring are very low the catastrophic damage it can cause should not be overlooked.