Recover everything!
He’s no cryptkeeper, but Mayank Sharma knows a thing or two about bringing deleted files back to life.
He’s no cryptkeeper, but Mayank Sharma knows a thing or two about bringing deleted files back from the dead.
You don’t have to be inexperienced to accidentally lose data. From what we have seen here at LXF Towers, the more experienced users make the most severe mistakes. So while amateurs curse themselves for accidentally zapping the wrong files, the pros usually mess up their partition tables, wipe the MBR and even format the wrong partition (haha, never
If that makes you feel bad about data loss, then get this: there’s a good chance that your data loss is reversible. There are lots of tools that’ll help you out of a sticky situation and help carve data out of failing or dead disks. There are tools that will not only help restore deleted files but also entire partitions. Remember, however, that the success of the tools depends on a variety of factors. If your data loss is the result of a failing drive, it’s best to minimise further interactions, as a failing drive’s condition worsens over time. If however the data loss is because of a clumsy operator, the construction of the disk plays a big factor in the recovery of the file.
Traditional hard drives use rotating magnetic platters in which data is read and written by physically repositioning a read/ write head. In contrast, the relatively newer solid state drives (SSDS) have no moving parts. SSDS are silicon-based chips that use electrons for storage, pretty much like USB drives and flash memory cards. On a traditional drive, files remain in place even after being deleted: the underlying data will only be overwritten when the space it occupies is required.
SSDS also work on this principal. However, since they can only store data in write-ready blocks that are cleared of all data, these disks run the garbagecollection TRIM command to wipe data. In addition, an SSD will start the background garbage collection process and erase trimmed blocks automatically.
When you lose data due to a hardware failure, it’s always a good idea to image your disk instead of subjecting the dying disk to the rigours of data recovery. Although you don’t need to do this to recover data you’ve deleted accidentally, it’s still a good idea. Virtually all recovery tools can work with disk images just as they do with real disks.
Perhaps the most comprehensive file recovery opensource tool is Photorec. Don’t be misled by its name; besides the most common image formats, Photorec can also pick out files in various formats including ODT, PDF, 7ZIP, RPM, DEB and even virtual disks. The tool works on all sorts of disks, including hard disks and removable media such as USB disks. In addition to reading unbootable disks, Photorec will also recover files from partitions that have been formatted and reinstalled into.
Although Photorec is a command-line tool, it breaks the file recovery process into steps, much like a wizard. When you launch the tool it first asks you to select the disk and then point it to the partition that housed the lost file, and also specify its filesystem. It’ll then ask you if you want it to search only the free unallocated space or the entire partition. You’ll also have to point it to a folder where it should store the recovered files.
Depending on the size of the partition, Photorec can take quite a while to complete. By default it looks for files of all the formats it supports, but you can limit the filetypes to recover. Once Photorec is done, you’ll discover gazillions of weirdly named files of all different formats under one directory. Photorec names them as it finds them, leaving the sorting to you.
Just as with files, it doesn’t take much effort to corrupt a healthy disk. A wrong keypress in Fdisk or
Gparted can wipe the MBR, or banish a partition into oblivion. And, just as with files, the situation is salvageable – more so if you stop using the disk as soon as you realise your mistake. Testdisk is the best tool to fix partition tables and put non-bootable disks back into service again. Using Testdisk is quite similar to
Photorec. See the LXF152 tutorial (http://bit.ly/ lxf257recover) for a hands-on with both these utilities.
Expecto restorum!
The Magic Rescue command-line tool provides another approach to recovering corrupted partition tables or data lost by accidental deletion. However, the tool does not rely on filesystem allocation tables. Instead, it works by reading a file’s ‘magic bytes’ or ‘magic pattern’, which is the unique signature that designates each file type. This signature is often located within the very first few bytes of a file.
Magic Rescue uses its collection of recipes to identify the magic bytes in all deleted files of a particular type and then transfers the deleted files to a directory for you to sort them. Magic Rescue is available in official repositories and upon installation dumps the recipes in the /usr/share/magicrescue/recipes/ directory.
Before you go hunting for deleted files, first create a target folder to store the data you unearth. It’s best if this location isn’t on the filesystem from which Magic Rescue is trying to extract files. The following command will hunt for PNG files from the /dev/sda1 partition and store it under the /media/recovery directory: sudo magicrescue -r png -d /media/recovery/ /dev/ sda1
Running Magic Rescue can take several minutes, depending on the size of the disk/partition it has to search and the number of file types it has to hunt. Once it’s done, the target folder will be full of awkwardly named files. To bring some order to the mess, you can use the magicsort tool to arrange the files into folders.
The command magicsort /media/ recovery will create and file the contents of the recovery folder into directories categorised by their magic number.
Dig deeper
Digital forensics is a branch of forensic science that involves recovering data from digital devices. The above-mentioned data recovery tools are good for recovering files that have been marked for deletion but lie untouched in the area they occupy on the disk.
Foremost, on the other hand, is what is known as a file carver: it digs out untraceable files by trying to identify existing data structures and recover occupied sectors by referencing the metadata, headers, footers and – like Magic Rescue – the magic number of the file. The tool was originally written by agents of the U.S. Air Force Office of Special Investigations and is included in the official repositories of major Linux distros.
As we mentioned earlier, make sure that before you run Foremost you create a recovery folder on a separate partition, or better still on an external disk. Now assuming the lost files are on sda1, you can recover files with: sudo foremost -i /dev/sda1 -o /media/recovery/ foremost To run Foremost on an image, just replace the device name with the name of the image, such as: sudo foremost -i sda1.image -o /media/recovery/ foremost
When it’s done, head to the /media/recovery/ foremost directory, which will contain various folders for different file formats, such as JPG, PNG, PDF, MP4 and dozens of others. In addition to various multimedia files, Foremost also recovers some binary and document formats without manual rework. It also recognises some types of archives. The software generates an audit file called audit.txt containing information on the data-reconstruction process.
As you can see, Foremost is designed to reconstruct numerous data formats out of the box. To recover only specific file types use the -t option, such as: sudo foremost -t png -i /dev/sda1 -o /media/ recovery/foremost
As with the other recovery tools, the recovery process will not be able to recover the original name of the file, so once Foremost has done its job, you’ll have to manually go through all the folders to check and rename the files. Depending on the type of files you’ve asked it to restore, you’ll probably end up with hundreds of them, and sorting through them is a task in itself. If there’s a particular file you’re after, you can save yourself a lot of time if you know the approximate size of the file.
Precision carving
While you can’t deny the effectiveness of Foremost, the tool is painstakingly slow. Scalpel is another file carver that’s based on foremost, but promises to be more efficient. Scalpel too is available in the repositories of most distros. Before you can use Scalpel, however, you’ll have to edit its configuration file (usually placed under the /etc/scalpel/ directory) in order to uncomment the descriptors for the type of file you want to recover. The developers ask you to avoid the temptation to uncomment the entire file since that will create an unnecessary overhead by digging up a whole lot of files, much like Foremost. Again, it’s advisable to store the recovered files on a separate partition/disk. Invoking Scalpel on a disk is a lot simpler:
PREVENTION IS BETTER THAN… “While you can’t use any of these tools as an excuse for not taking backups, one of them will surely come to your aid to help you recover data.”
sudo scalpel /dev/sda1 -o /media/recovery/scalpel Instead of the disk you can also point it to an image file: sudo scalpel sda1.image -o /media/recovery/scalpel
Scalpel will then perform two passes of sda1. It’ll process data in chunks of 10MB, first searching for file headers, and then for their corresponding footers. Any recovered files are placed in an output directory you specify, along with a log of Scalpel’s progress.
Down the rabbit hole
The tools we’ve seen till now are excellent and will even dig up files that were deleted quite a while back. But they won’t be of much help if the headers or footers of the files have been overwritten. You’ll also not have any success with these tools if the target drive is encrypted. In this case, none of the tools we’ve covered till now will be able to scan the contents of the disks to identify the data structures of the files, making recovery impossible.
But don’t despair. The Sleuth Kit (TSK) is an advanced recovery tool that can detect encrypted and password-protected files. It can also identify Bitlocker and Veracrypt volumes, which makes it useful for recovering files from encrypted silos. TSK uses code from the file system analysis tools of
The Coroner’s Toolkit (TCT) with additional support for FAT and NTFS file systems. As its name suggests, TSK is made up of a bunch of tools. However, the developers recommend using the command-line tools from the unified graphical interface, called Autopsy. Autopsy and
TSK are available in the official repos of several distros. Even if they aren’t in yours, the process to install them is well-documented on the project’s website.
Slice and dice
Autopsy is a browser-based app and by default runs on http://localhost:9999/autopsy. The interface is intuitive enough. The app is designed for forensic analysis and should be used as such, even if all you need it for is to recover deleted files on your own machine. You’ll first have to create a New Case, followed by details about the host being investigated.
Once that’s out of the way, use the Add Image button to associate an image of the disk or the partition you want analysed. Of the three import methods, it’s always a good idea to use Copy, which doesn’t touch the original image for the recovery process, and instead works on a copy (or a copy of a copy, if you imaged the original disk). Once the image has been associated the tool gives you various options to prod and poke it. Use the Image Details to get various details about the image that helps forensically identify its contents,
Autopsy presents various analysis modes to scan the image. The first is the File Analysis mode, which resembles a file manager and enables you to examine the contents of the filesystem within the image. For every file and directory, the interface will show you when it was last written, accessed and modified, along with several other metadata including its size. You can get more details about a file by clicking its corresponding metadata entry. Forensic investigators use this to view the hexadecimal entry for a file to verify whether its extension was changed.
Unlike investigators, however, we have a very limited objective, which is to recover deleted files. To that end, files that have been deleted are listed in red. The All Deleted Files button in the left-hand column under the File Analysis view displays all deleted files. You can click and view the contents of a file and once you’ve found the one you’re looking for, use the Export button to extract it from the image on to your disk.
There you have it. While you can’t use any of these tools as an excuse for not taking backups (look, I said
I was sorry – Ed), one of them will surely come to your aid to help you recover data that you thought was gone for good.