Splitsize too small: Difference between revisions
(→Splitting Config: update for 3.2) |
(→Amanda-3.2 and later: more info on fallback) |
||
(One intermediate revision by the same user not shown) | |||
Line 31: | Line 31: | ||
The primary splitting parameter is the tapetype parameter ''part_size''. Set this to something greater than 0.1% of your tape length. | The primary splitting parameter is the tapetype parameter ''part_size''. Set this to something greater than 0.1% of your tape length. | ||
If part | If Amanda may need to "rewind" a part that has failed during a write to tape, then the part cache comes into play. This is a rare circumstance, requiring: | ||
* the data is directly from the dumper, not via holding disk | |||
* Amanda is configured to use more than one tape (''runtapes'' > 1) | |||
* the underlying device cannot detect end-of-medium in advance (LEOM support) | |||
* the part cache is only consulted if the device actually reaches EOM during the dump | |||
If part caching is required, then Amanda will reduce the part size to ''part_cache_max_size''. Especially when caching in RAM, setting this value to something smaller than ''part_size'' is useful to limit the amount of space required for the part cache. The ''part_cache_type'' "none" can be used to avoid caching even in cases where it would otherwise be required. In this case, if Amanda reaches end-of-medium and all of the above conditions are met, then entire DLE fails - but Amanda will retry it up to three times, so your data is still safe. | |||
=== Amanda-3.1 and earlier === | === Amanda-3.1 and earlier === | ||
Line 45: | Line 51: | ||
== Fix == | == Fix == | ||
For the first warning | For the first warning, the recommended splitsize is something between 1% and 5% of your tape size, so you should adjust your ''part_size'' or ''tape_splitsize'' accordingly. | ||
For the second warning, most likely you have not changed the default ''fallback_splitsize'' and have not set ''split_diskbuffer''. If you have enough RAM to hold an entire part, set the ''fallback_splitsize'' to the desired part size. Otherwise, set ''split_diskbuffer'' to a directory with sufficient space for your ''tape_splitsize''. | For the second warning, most likely you have not changed the default ''fallback_splitsize'' and have not set ''split_diskbuffer''. If you have enough RAM to hold an entire part, set the ''fallback_splitsize'' to the desired part size. Otherwise, set ''split_diskbuffer'' to a directory with sufficient space for your ''tape_splitsize''. If you are using ''part_cache_max_size'', and you do not have space (either on disk or in memory) for at least 0.1% of your tape length, then you should consider acquiring more space or disabling caching altogether (''part_cache_type'' "none"). |
Latest revision as of 17:35, 6 July 2010
This article is a part of the Troubleshooting collection.
Problem
Amcheck gives warnings about tape_splitsize or part_size:
WARNING: knuth /usr: tape_splitsize of 10240 KB < 0.1% of tape length. This may create > 1000 parts, severely degrading backup/restore performance. To remedy, increase tape_splitsize or disable splitting by setting it to 0 See http://wiki.zmanda.com/index.php/Splitsize_too_small for more information.
or about fallback_splitsize or part_cache_max_size:
WARNING: euclid /var/lib: fallback_splitsize of 10240 KB < 0.1% of tape length. This may create > 1000 parts, severely degrading backup/restore performance. To remedy, create/set a split_diskbuffer or increase fallback_splitsize See http://wiki.zmanda.com/index.php/Splitsize_too_small for more information.
and sends you to this wiki page.
Solution
This is usually due to a misconfiguration - sometimes a misconfiguration that has been in place for many years! The misconfiguration can cause Amanda to produce too many parts for a dump. This has unfortunate consequences:
- Writes to tape are slowed, because the tape drive must stop to flush its buffers and write a filemark thousands of times per tape. This is less significant with vtapes.
- Amanda uses on-volume space inefficiently - filemarks and headers take up valuable space, even on vtapes
- Recoveries using amfetchdump or amrecover are slow, because they must open and read from thousands of parts
- Recoveries using amrestore are extremely painful, since it is difficult to concatenate thousands of files together with shell tools
Splitting Config
Amanda-3.2 and later
The primary splitting parameter is the tapetype parameter part_size. Set this to something greater than 0.1% of your tape length.
If Amanda may need to "rewind" a part that has failed during a write to tape, then the part cache comes into play. This is a rare circumstance, requiring:
- the data is directly from the dumper, not via holding disk
- Amanda is configured to use more than one tape (runtapes > 1)
- the underlying device cannot detect end-of-medium in advance (LEOM support)
- the part cache is only consulted if the device actually reaches EOM during the dump
If part caching is required, then Amanda will reduce the part size to part_cache_max_size. Especially when caching in RAM, setting this value to something smaller than part_size is useful to limit the amount of space required for the part cache. The part_cache_type "none" can be used to avoid caching even in cases where it would otherwise be required. In this case, if Amanda reaches end-of-medium and all of the above conditions are met, then entire DLE fails - but Amanda will retry it up to three times, so your data is still safe.
Amanda-3.1 and earlier
The three splitting parameters are tape_splitsize, split_diskbuffer, and fallback_splitsize.
When Amanda is writing to tape from holding-disk, it always creates parts of size tape_splitsize, and does not require any other buffers. However, Amanda can decide to skip holding-disk without much warning. For a dump direct to tape (also known as PORT-WRITE), the rules for the splitsize are a little more complicated:
- if split_diskbuffer is set, the dirctory exists, and it has at least tape_splitsize bytes free, then the parts are of size tape_splitsize
- if split_diskbuffer is not set, or if the directory does not exist or does not have enough free space, then
- if fallback_splitsize is set, the parts are of size fallback_splitsize; otherwise
- the parts are of size 10M (the default value of fallback_splitsize).
The common misconfiguration is to set tape_splitsize but not split_diskbuffer or fallback_splitsize, with the consequence that when a dump goes straight to tape, it is written in 10M parts. Unless you're watching your reports carefully, you won't notice this until you try to do a recovery - and possible not even then!
Fix
For the first warning, the recommended splitsize is something between 1% and 5% of your tape size, so you should adjust your part_size or tape_splitsize accordingly.
For the second warning, most likely you have not changed the default fallback_splitsize and have not set split_diskbuffer. If you have enough RAM to hold an entire part, set the fallback_splitsize to the desired part size. Otherwise, set split_diskbuffer to a directory with sufficient space for your tape_splitsize. If you are using part_cache_max_size, and you do not have space (either on disk or in memory) for at least 0.1% of your tape length, then you should consider acquiring more space or disabling caching altogether (part_cache_type "none").