Tar level-1 dumps are the same size as level-0 dumps: Difference between revisions

From wiki.zmanda.com
Jump to navigation Jump to search
(writeup of the situation as I understand it)
 
(GNU tar now has a "--no-check-device" option)
Line 17: Line 17:
There are a few long-term solutions to this problem that may occur, but are not currently planned:
There are a few long-term solutions to this problem that may occur, but are not currently planned:
* the kernel developers can assign permanent numbers for LVM and the other dynamic but common devices in the [http://www.lanana.org/docs/device-list LANANA]
* the kernel developers can assign permanent numbers for LVM and the other dynamic but common devices in the [http://www.lanana.org/docs/device-list LANANA]
* GNU tar could have an option to ignore device numbers, although there are legitimate arguments for including them in the indices


You can also stabilize the device numbers yourself on Linux, at least for the device mapper, assuming you're using modules.  Just add
However, you can also stabilize the device numbers yourself on Linux, at least for the device mapper, assuming you're using modules.  Just add
  options dm-mod major=238
  options dm-mod major=238
to {{file|/etc/modprobe.conf}}.
to {{file|/etc/modprobe.conf}}.


Finally, if your device numbers have changed, and you need to manually update the index files, you can do that too.  Tar contains a Perl script which can "edit" index files, swapping one device number for another.  See http://www.gnu.org/software/tar/utils/tar-snapshot-edit.html (the utility is not posted, as of this writing, but should be there soon; consult the amanda mailing lists if you need a copy immediately).
If you have a one-time situation where the device number changes (e.g. upgrading to a new version of the OS kernel causes the number to change), you can manually update the index files.  Tar contains a Perl script which can "edit" index files, swapping one device number for another.  If it is not included in your version of tar, you can also download it from http://www.gnu.org/software/tar/utils/tar-snapshot-edit.html or
http://git.savannah.gnu.org/gitweb/?p=tar.git;a=blob;f=scripts/tar-snapshot-edit;hb=HEAD .
 
On the other hand, the device numbers change frequently in your environment, you may need to use the --no-check-devices option to GNU tar, available in version 1.20 and later.  For more information, see:
* http://www.gnu.org/software/tar/#releases
* http://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html#device-numbers
 
In Amanda 2.6.1, the "amgtar" Application API script already includes a property that controls whether this option is passed to GNU Tar: http://wiki.zmanda.com/man/amgtar.8.html
 
(If you are running an earlier version of Amanda but have a recent-enough version of GNU Tar, you may be able to for usage of this option by arranging for the TAR_OPTIONS environment variable to be set on the Amanda client when the dumps are run, or by following the "wrapper script" instructions found in
http://wiki.zmanda.com/index.php/Backup_client#pre_and_post-_backup_scripts_.28wrappers.29
)


= Links =
= Links =
* http://marc.info/?t=117591139700003&r=1&w=2
* amanda-users list thread with subject "testing a patched tar question.":
* http://www.mail-archive.com/[email protected]/msg01265.html
** http://marc.info/?t=117591139700003&r=1&w=2
** http://tech.groups.yahoo.com/group/amanda-users/message/62562
** http://www.mail-archive.com/[email protected]/msg37675.html
 
* bug-tar list thread with subject "Changing 'dev' in snapshot files?":
** http://marc.info/?l=tar-bug&m=117624216431722&w=2
** http://www.mail-archive.com/[email protected]/msg01265.html
 
* bug-tar list thread with subject "incremental of nfs directory do full every times."
** http://marc.info/?l=tar-bug&m=120660815326563&w=2
** http://www.mail-archive.com/[email protected]/msg01860.html

Revision as of 22:27, 14 May 2009

Problem

After a hardware change, level-1 dumps are level-0 sized, because GNU tar sees every file as changed. For a highly leveraged Amanda installation, where the total data being backed up is much larger than available tape storage, this can cause dumps to fail badly, as Amanda cannot fit dumps for all DLEs on tape.

Explanation

Amanda uses index files not only to find files on tapes, but to allow GNU tar to calculate what has changed on a partition from day to day. When tar runs, it takes an index file describing the previous dump as input, and produces a new index file describing the new run.

Among other data about files that are backed up, GNU tar's index files contain the device number of the device on which the file is located. For example, if the /data partition is on /dev/sda2, then the device number might be 8,2; you can see the device numbers with ls, operating directly on the underlying device: 

# df /data
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2               988244    698452    239592  75% /data
# ls -al /dev/sda2
brw-rw---- 1 root disk 8, 2 May 23 00:27 /dev/sda2

This particular problem occurs when the device number for a particular piece of hardware changes. Some device numbers, notably those for device mapper (used by the Logical Volume Manager on Linux), are assigned dynamically at boot time, and changing a machine's hardware configuration (by e.g., adding a new USB CD drive) can affect those numbers. The result is that, on the next run, tar sees different device numbers for every file, and interprets that to mean that the file has changed.

Solutions

There are a few long-term solutions to this problem that may occur, but are not currently planned:

  • the kernel developers can assign permanent numbers for LVM and the other dynamic but common devices in the LANANA

However, you can also stabilize the device numbers yourself on Linux, at least for the device mapper, assuming you're using modules. Just add 

options dm-mod major=238

to /etc/modprobe.conf.

If you have a one-time situation where the device number changes (e.g. upgrading to a new version of the OS kernel causes the number to change), you can manually update the index files. Tar contains a Perl script which can "edit" index files, swapping one device number for another. If it is not included in your version of tar, you can also download it from http://www.gnu.org/software/tar/utils/tar-snapshot-edit.html or http://git.savannah.gnu.org/gitweb/?p=tar.git;a=blob;f=scripts/tar-snapshot-edit;hb=HEAD .

On the other hand, the device numbers change frequently in your environment, you may need to use the --no-check-devices option to GNU tar, available in version 1.20 and later. For more information, see:

In Amanda 2.6.1, the "amgtar" Application API script already includes a property that controls whether this option is passed to GNU Tar: http://wiki.zmanda.com/man/amgtar.8.html

(If you are running an earlier version of Amanda but have a recent-enough version of GNU Tar, you may be able to for usage of this option by arranging for the TAR_OPTIONS environment variable to be set on the Amanda client when the dumps are run, or by following the "wrapper script" instructions found in http://wiki.zmanda.com/index.php/Backup_client#pre_and_post-_backup_scripts_.28wrappers.29 )

Links

Retrieved from ‘http://wiki.zmanda.com/index.php?title=Tar_level-1_dumps_are_the_same_size_as_level-0_dumps&oldid=6908’