How To:Use pigz to speed up compression

From The Open Source Backup Wiki (Amanda, MySQL Backup, BackupPC)
Jump to navigationJump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
NOTE: Some users on the mailing list have experienced data-corruption problems while using pigz. Test your restores before relying on this solution!

pigz, which stands for parallel implementation of gzip, is a fully functional drop-in replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data. Using it our new 16 thread (dual Xeon 5520 quad core multithreaded cpus) capable backup server has speed up our backups dramatically.

How I use it is by defining the following dump type:

 define dumptype scomp-user-tar {
     user-tar
     compress server custom
     server_custom_compress "/usr/bin/pigz"
 }

You could also run the compression on the clients:

 define dumptype comp-user-tar {
     user-tar
     compress client custom
     client_custom_compress "/usr/bin/pigz"
 }

but be aware that using pigz is going to put a lot more load on the machine running the compression. In our case, we want to affect our disk servers as little as possible while they are getting backed up. Finally, you will need to make sure that pigz is installed on all machines that need it.