Splitsize too small

From The Open Source Backup Wiki (Amanda, MySQL Backup, BackupPC)
Jump to navigationJump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

This article is a part of the Troubleshooting collection.

Problem

Amcheck gives warnings about tape_splitsize or part_size:

WARNING: knuth /usr: tape_splitsize of 10240 KB < 0.1% of tape length.
 This may create > 1000 parts, severely degrading backup/restore performance.
 To remedy, increase tape_splitsize or disable splitting by setting it to 0
 See http://wiki.zmanda.com/index.php/Splitsize_too_small for more information.

or about fallback_splitsize or part_cache_max_size:

WARNING: euclid /var/lib: fallback_splitsize of 10240 KB < 0.1% of tape length.
 This may create > 1000 parts, severely degrading backup/restore performance.
 To remedy, create/set a split_diskbuffer or increase fallback_splitsize
 See http://wiki.zmanda.com/index.php/Splitsize_too_small for more information.

and sends you to this wiki page.

Solution

This is usually due to a misconfiguration - sometimes a misconfiguration that has been in place for many years! The misconfiguration can cause Amanda to produce too many parts for a dump. This has unfortunate consequences:

  • Writes to tape are slowed, because the tape drive must stop to flush its buffers and write a filemark thousands of times per tape. This is less significant with vtapes.
  • Amanda uses on-volume space inefficiently - filemarks and headers take up valuable space, even on vtapes
  • Recoveries using amfetchdump or amrecover are slow, because they must open and read from thousands of parts
  • Recoveries using amrestore are extremely painful, since it is difficult to concatenate thousands of files together with shell tools

Splitting Config

Amanda-3.2 and later

The primary splitting parameter is the tapetype parameter part_size. Set this to something greater than 0.1% of your tape length.

If Amanda may need to "rewind" a part that has failed during a write to tape, then the part cache comes into play. This is a rare circumstance, requiring:

  • the data is directly from the dumper, not via holding disk
  • Amanda is configured to use more than one tape (runtapes > 1)
  • the underlying device cannot detect end-of-medium in advance (LEOM support)
  • the part cache is only consulted if the device actually reaches EOM during the dump

If part caching is required, then Amanda will reduce the part size to part_cache_max_size. Especially when caching in RAM, setting this value to something smaller than part_size is useful to limit the amount of space required for the part cache. The part_cache_type "none" can be used to avoid caching even in cases where it would otherwise be required. In this case, if Amanda reaches end-of-medium and all of the above conditions are met, then entire DLE fails - but Amanda will retry it up to three times, so your data is still safe.

Amanda-3.1 and earlier

The three splitting parameters are tape_splitsize, split_diskbuffer, and fallback_splitsize.

When Amanda is writing to tape from holding-disk, it always creates parts of size tape_splitsize, and does not require any other buffers. However, Amanda can decide to skip holding-disk without much warning. For a dump direct to tape (also known as PORT-WRITE), the rules for the splitsize are a little more complicated:

  • if split_diskbuffer is set, the dirctory exists, and it has at least tape_splitsize bytes free, then the parts are of size tape_splitsize
  • if split_diskbuffer is not set, or if the directory does not exist or does not have enough free space, then
    • if fallback_splitsize is set, the parts are of size fallback_splitsize; otherwise
    • the parts are of size 10M (the default value of fallback_splitsize).

The common misconfiguration is to set tape_splitsize but not split_diskbuffer or fallback_splitsize, with the consequence that when a dump goes straight to tape, it is written in 10M parts. Unless you're watching your reports carefully, you won't notice this until you try to do a recovery - and possible not even then!

Fix

For the first warning, the recommended splitsize is something between 1% and 5% of your tape size, so you should adjust your part_size or tape_splitsize accordingly.

For the second warning, most likely you have not changed the default fallback_splitsize and have not set split_diskbuffer. If you have enough RAM to hold an entire part, set the fallback_splitsize to the desired part size. Otherwise, set split_diskbuffer to a directory with sufficient space for your tape_splitsize. If you are using part_cache_max_size, and you do not have space (either on disk or in memory) for at least 0.1% of your tape length, then you should consider acquiring more space or disabling caching altogether (part_cache_type "none").