The situation
Enterprise Storage System Symmetrix VMAX by EMC Corporation (this system was developed specifically to be used in virtual data centres).
The ISP contacted Ontrack on a Thursday - the content of a LUN with 50 GB was not displaying correctly after a system crash. For no apparent reason the Unix-based file system had used up both the inode references and the free space so the system no longer ran properly after being restarted. To try to solve the problem and regain access to all the information inside the LUN, they had tried several times to restart the system in rescue mode. The file system was mounted in read-only mode and the command to determine which file had depleted the resources in the system was not available. It appeared that human error had occurred during the operation of the complex system and the allocation of available space.
A total of about 30GB of log files from the affected LUN needed to be recovered as, according to the local legal requirements, telecommunications data must be stored for a period of 12 months from the date of transfer.
The solution
A technical questionnaire was immediately sent to the ISP which helped to gain detailed information about the affected storage system. The Ontrack Data Recovery team was able to provide the client with a concrete proposal for the procedure within three hours of receiving the questionnaire.
Late Friday night the ISP contacted Ontrack to enquire if the information could be available by Sunday. Given the urgency, it was agreed to escalate the service to an emergency level ensuring the recovery process was performed 24/7.
As a result of the ISPs internal corporate security policy remote data recovery was not possible, therefore an image of the affected LUN was requested. The hard disk with the image was delivered to Ontrack at 2am on Saturday.
The image of the LUN was approximately 52 GB in size and contained a Linux EXT3 file system under RedHat 5.9 x64.
Since this was a logical problem, the specialist recovery engineers carried out a detailed analysis of the complex data structure. The damage to the directory with the log files was found, which was preventing Linux from properly accessing the content. The metadata of the damaged logical structure was reconstructed and access to 100% of the files was promptly restored.
The resolution
The data recovery was successfully completed by 10am on Saturday. Less than 24 hours after the project was started and far sooner than agreed. Ontrack presented the ISP with the salvaged logs, so the legal obligation to retain the customer data was again fully guaranteed and no penalties had to be paid.
