Hardware compression

From The Open Source Backup Wiki (Amanda, MySQL Backup, BackupPC)
Revision as of 22:13, 20 August 2009 by Nathanst (talk | contribs) (Added explanation of tape drive v.s. linux kernel driver hardware-compression settings)
Jump to navigationJump to search

Compression in hardware, i.e. in the tapedrive itself, is not done by writing the bits in a higher density, or by using more tracks, or using both sides of the tape. Hardware compression is achieved with an algorithm, built into the firmware of the tapedrive, which removes redundancy from the byte stream, resulting in less bits to write to the tape medium. Manufacturers often claim to achieve a 2:1 or even 2.6:1 reduction on "normal data", hence the advertising of e.g. 36/72 GB tape capacity. Some of those algorithms, when given uncompressable data, actually result in an expansion by 5-15%. The net amount of tape capacity for such uncompressable data is lowered as well: e.g. the expected 36 GB capacity is now around 30-32 GB for such drives. Some tapedrives have a much better compression firmware, which does not have that negative impact (e.g. LTO tape drives).

It is usually recommended to disable hardware compression with Amanda, though it may have less of a negative impact with certain modern tape formats (e.g. LTO).

There are multiple methods to appropriately set (i.e. turn off) hardware compression. They may vary between OSes somewhat, not all tape drives support this setting, and tape drives may support a few methods at a protocol level for controlling hardware compression, methods which your OS's mt(1) tool or similar may or may not handle - e.g. the "device configuration" scsi-2 mode page 0x10 (common), the "data compression" scsi-3 mode page 0x0F (sometimes) or a particular special "density" setting (rare nowadays, thankfully). If your tape drive still has hardware-level DIP switches to control settings, it can be more pleasant to use them.

Resurfacing HW compression

Note that some tape drives re-enable HW compression if they read HW compressed data, as explained e.g. here. Therefore it may be necessary to properly overwrite existing tapes to ensure HW compression stays off.

Please note that compression information is not stored on the tapes ident header block until the tape has been written to.

Procedure for turning off compression and labelling tapes

  • Label the tape
  • Rewind the tape
  • Read the label to a file using dd command
  • Turn off tape compression using mt(1) command. See next section.
  • Re-write the label block and write more /dev/zero blocks to flush its buffers.

OS specific notes

Linux

Under Linux, there are actually two separate tape-drive-compression settings that can be configured, and on top of that there are two separate versions of the mt utility, which each have a different way of altering those tape-drive-compression settings.

Tape drive v.s. Linux kernel driver compression settings
Tape drive setting

Most SCSI tape drives support the Data Compression Characteristics mode page (page 15 or 0x0F). The host computer can read and set the hardware compression settings using the SCSI MODE SENSE and MODE SELECT commands, respectively. When this setting is written, the drive immediately changes to the specified setting, and reading this setting shows the physical drive's current status.

Linux kernel drivers

The Linux kernel st driver keeps track of a "default" setting for each tape-drive device that it is controlling. If that default is set, the driver will set the tape drive to match the setting (e.g. turn on compression if the default is set to "on") whenever the system first tries to write to a tape in that drive.

This "default" setting is controlled using the MT_ST_DEF_COMPRESSION subcommand of the MTSETDRVBUFFER ioctl on the device file (e.g. /dev/st0).

An additional feature of the kernel driver is the support for multiple "modes" within a single physical tape drive. Each "mode" has a separate device file; for example /dev/st0, /dev/st0a, /dev/st0l, and /dev/st0m each refer to the same tape drive via different modes. The kernel keeps track of separate "default compression" settings for each mode that is defined.

The st kernel driver also supports an MTCOMPRESSION ioctl, which will immediately set the compression status of the drive (using the SCSI mode page 15 commands internally to do so).

For more information, see the README.st or st.txt files in the Documentation directory within the kernel source tree. (For example, here is a link to the Linux 2.6.30 version of st.txt.)

Versions of the mt command

There are actually two different versions of the mt command in use on Linux, GNU mt and mt-st. For most operations they are interchangable, but for a few operations -- including controlling tape-drive hardware compression settings -- the two programs behave quite differently.

You can find out which version you are running by giving the command

# mt --version

In some distributions (e.g. Debian), it's possible to install both versions at the same time; you then invoke the command as either mt-gnu or mt-st as desired.

GNU mt
mt-st
Setting compression settings
  • Using stinit command: stinit(8) command initializes the SCSI tape drives at the system startup by sending driver ioctl commands. Use "comp" field in /etc/stinit.def configuration file to configure hardware compression for each type of tape.
  • Use (a recent version of) mt(1) command to turn off hardware compression at boot time. For example:
# mt -f /dev/st0 compression 0
  • Use (a recent version of) mt(1) command to set default compression - e.g.
# mt -f /dev/nst0 defcompression 0

(some tape drives don't seem to support this)

Checking compression settings

FreeBSD

  • Use mt(1) command to turn off hardware compression at boot time. For example:
# mt -f /dev/nsa0 comp off

NetBSD

  • Use mt(1) command to turn off hardware compression at boot time. For example:
# mt -f /dev/nrst0 compress 0

OpenBSD

  • At time of writing (OpenBSD 3.9), OpenBSD's mt(1) doesn't allow manipulation of hardware compression settings. However, one can use the swiss-army-knife scsi(8) command to change the compression setting (and more besides - be careful...) in the mode pages. This should probably be considered undocumented and subject to change, as the st(4) man page describes as "reserved" the device bit pattern enrst0 used here to put the st kernel driver into "control mode".
# EDITOR=vi scsi -f /dev/enrst0 -m 16 -e -P0 

( -m 16 selects the "Device Configuration" mode page. Change the value of "Select Data Compression Algorithm" to 0 in the text editor launched, save and exit. Using -P3 _might_ save the setting, equivalent to linux mt "defcompression" above, but some tape drives don't seem to support it. )

Solaris

Set ST_MODE_SEL_COMP in st.conf for your device and then use the correct tape device variant (do not use "c" and "u" device)

AIX

(untested, based on public man pages - someone aixy please verify)

# chdev -l rmt0 -a compress=no