Hardware compression

Compression in hardware, i.e. in the tapedrive itself, is not done by writing the bits in a higher density, or by using more tracks, or using both sides of the tape. Hardware compression is achieved with an algorithm, built into the firmware of the tapedrive, which removes redundancy from the byte stream, resulting in less bits to write to the tape medium. Manufacturers often claim to achieve a 2:1 or even 2.6:1 reduction on "normal data", hence the advertising of e.g. 36/72 GB tape capacity. Some of those algorithms, when given uncompressable data, actually result in an expansion by 5-15%. The net amount of tape capacity for such uncompressable data is lowered as well: e.g. the expected 36 GB capacity is now around 30-32 GB for such drives. Some tapedrives have a much better compression firmware, which does not have that negative impact (e.g. LTO tape drives).

It is usually recommended to disable hardware compression with Amanda, though it may have less of a negative impact with certain modern tape formats (e.g. LTO).

There are multiple methods to appropriately set (i.e. turn off) hardware compression. They may vary between OSes somewhat, not all tape drives support this setting, and tape drives may support a few methods at a protocol level for controlling hardware compression, methods which your OS's mt(1) tool or similar may or may not handle - e.g. the "device configuration" scsi-2 mode page 0x10 (common), the "data compression" scsi-3 mode page 0x0F (sometimes) or a particular special "density" setting (rare nowadays, thankfully). If your tape drive still has hardware-level DIP switches to control settings, it can be more pleasant to use them.

Resurfacing HW compression

Note that some tape drives re-enable HW compression if they read HW compressed data, as explained e.g. here. Therefore it may be necessary to properly overwrite existing tapes to ensure HW compression stays off.

Please note that compression information is not stored on the tapes ident header block until the tape has been written to.

Procedure for turning off compression and labelling tapes

• Label the tape
• Rewind the tape
• Read the label to a file using dd command
• Turn off tape compression using mt(1) command. See next section.
• Re-write the label block and write more /dev/zero blocks to flush its buffers.

OS specific notes

Linux

Under Linux, there are actually two separate tape-drive-compression settings that can be configured, and on top of that there are two separate versions of the mt utility, which each have a different way of altering those tape-drive-compression settings.

Tape drive v.s. Linux kernel driver compression settings

Tape drive setting

Most SCSI tape drives support the Data Compression Characteristics mode page (page 15 or 0x0F). The host computer can read and set the hardware compression settings using the SCSI MODE SENSE and MODE SELECT commands, respectively. When this setting is written, the drive immediately changes to the specified setting, and reading this setting shows the physical drive's current status.

Linux kernel drivers

The Linux kernel st driver keeps track of a "default" setting for each tape-drive device that it is controlling. If that default is set, the driver will set the tape drive to match the setting (e.g. turn on compression if the default is set to "on") whenever the system first tries to write to a tape in that drive.

This "default" setting is controlled using the MT_ST_DEF_COMPRESSION subcommand of the MTSETDRVBUFFER ioctl on the device file (e.g. /dev/st0).

An additional feature of the kernel driver is the support for multiple "modes" within a single physical tape drive. Each "mode" has a separate device file; for example /dev/st0, /dev/st0a, /dev/st0l, and /dev/st0m each refer to the same tape drive via different modes. The kernel keeps track of separate "default compression" settings for each mode that is defined.

The st kernel driver also supports an MTCOMPRESSION ioctl, which will immediately set the compression status of the drive (using the SCSI mode page 15 commands internally to do so).

On the other hand, the st driver does not provide any ioctls for reading the current setting of either the mode page 15 compression setting or the driver's "default compression" setting.

For more information, see the README.st or st.txt files in the Documentation directory within the kernel source tree. (For example, here is a link to the Linux 2.6.30 version of st.txt.)

Versions of the mt command

There are actually two different versions of the mt command in use on Linux, GNU mt and mt-st. For most operations they are interchangable, but for a few operations -- including controlling tape-drive hardware compression settings -- the two programs behave quite differently.

You can find out which version you are running by giving the command

# mt --version

In some distributions (e.g. Debian), it's possible to install both versions at the same time; you then invoke the command as either mt-gnu or mt-st as desired.

GNU mt

GNU mt is part of the GNU cpio package. However, the upstream version does not include support for various functions that are available from SCSI tape drives, so many Linux distributions include their own patches to support these functions.

In particular, the patches add support for a datcompression command which can be used to control the tape drive's compression setting. This command directly access the SCSI Data Compression Characteristics mode page as discussed above, so any changes made using this command take effect immediately (but the st driver's "default compression" setting is not changed).

mt-st

mt-st is version of "mt" patched to support various functions provided by the Linux st device driver. (The project does not seem to have a home page, but source code for the package is made available in the mt-st-*.tar.gz file found in http://ftp.ibiblio.org/pub/linux/system/backup/ .)

Since it uses the st driver functions, mt-st is able to set both the current SCSI mode page compression settings and the kernel driver's "default compression" setting -- but it is not able to read back the current values of either setting for display to the user.

The mt-st package also includes the stinit command, which can be used to initialize various kernel-driver settings for the tape devices, including the configuration of the "mode" devices (e.g. both /dev/st0 and /dev/st0a). (This command also uses the st kernel driver functionality, and can initialize that driver's "default compression" setting, etc.)

Setting and checking the compression settings

• Using stinit command: stinit(8) command initializes the SCSI tape drives at the system startup by sending driver ioctl commands. Use "comp" field in /etc/stinit.def configuration file to configure hardware compression for each type of tape.

• Use (a recent version of) mt(1) command to turn off hardware compression at boot time. For example:

# mt -f /dev/st0 compression 0

• Use (a recent version of) mt(1) command to set default compression - e.g.

# mt -f /dev/nst0 defcompression 0

(some tape drives don't seem to support this)

FreeBSD

• Use mt(1) command to turn off hardware compression at boot time. For example:

# mt -f /dev/nsa0 comp off

NetBSD

• Use mt(1) command to turn off hardware compression at boot time. For example:

# mt -f /dev/nrst0 compress 0

OpenBSD

• At time of writing (OpenBSD 3.9), OpenBSD's mt(1) doesn't allow manipulation of hardware compression settings. However, one can use the swiss-army-knife scsi(8) command to change the compression setting (and more besides - be careful...) in the mode pages. This should probably be considered undocumented and subject to change, as the st(4) man page describes as "reserved" the device bit pattern enrst0 used here to put the st kernel driver into "control mode".

# EDITOR=vi scsi -f /dev/enrst0 -m 16 -e -P0 

( -m 16 selects the "Device Configuration" mode page. Change the value of "Select Data Compression Algorithm" to 0 in the text editor launched, save and exit. Using -P3 _might_ save the setting, equivalent to linux mt "defcompression" above, but some tape drives don't seem to support it. )

Solaris

Set ST_MODE_SEL_COMP in st.conf for your device and then use the correct tape device variant (do not use "c" and "u" device)

AIX

(untested, based on public man pages - someone aixy please verify)

# chdev -l rmt0 -a compress=no