The pharmaceutical company lost almost 4 million files of highly critical research and development data when moving files within the Isilon storage system.
The “lost files” were mostly raw data gathered from chemical analysis in ongoing laboratory research work. The firm, together with EMC support, was able to recover 90 percent of the data using a standard rebuild process. To recover the desperately needed remaining files, Ontrack was contacted by the customer for help. Ontrack worked with the EMC support in order to get detailed information on the situation.
The Isilon IQ 6000x, designed for big data, consisted of four nodes with 15 terabytes of data storage in total with 500 gigabyte hard drives installed.
It was set up as a single volume where all of the data was striped across the disks. Unlike a traditional RAID system, Isilon systems are built on the concept of a huge data lake where all data is stored and managed inside one data pool. To manage the system and this “data lake” a unique file system was created by Isilon called OneFS. In this case one of the four nodes inside the system experienced a kernel panic and several disks showed multiple errors. EMC was able to gather most of the files with a rebuild before a consistency check showed that several disks had errors. To recover the missing data, the file system had to be analyzed by the Ontrack team to find out how the data was laid out in the whole storage system.