ADVANCED PROJECT: DUPLICATES DAY

To be honest, this one isn’t all that advanced for techies, but it can feel that way for users when you ask them to go hunting around their systems for duplicate files. Windows doesn’t make this easy, especially when documents are spread across a variety of storage platforms and security settings.

You may question whether this is a worthwhile use of time and energy. Many people say exactly that. Many people, that is, who haven’t seen the benefits of deduplication, as found on the most upmarket NAS and storage server gear. These monsters – a single full-height rack packed with disks qualifies as small – can identify data that’s duplicated across multiple files and users, and refer hundreds of thousands of files all to a single block copy. The effect on storage requirements can be breathtaking: I’ve seen the time taken to back up an entire Exchange Server VM drop from several hours to a matter of minutes.

You’re not going to match those benefits by clearing out a load of individual duplicates, but you can still make a significant difference to your storage needs, which will in turn have a knock-on effect on the speed and efficiency of your backups.

The best approach is to standardise on a few lightweight utilities. The first task is to analyse where your storage is being eaten up, and to identify where you most need to free up some space. Then, you can use one of numerous free tools to search and destroy any space-wasting duplicates you may be harbouring. I split up these operations by size; I start by searching for duplicate files over 500MB and then work my way down. This way, you can achieve big space savings very quickly, and it’s up to you whether you continue to inspect the smaller files.

Shrink your storage needs and your backup load