Backup server (old): Difference between revisions

From wiki.zmanda.com
Jump to navigation Jump to search
(deprecate)
 
(39 intermediate revisions by 9 users not shown)
Line 1: Line 1:
{{Deprecated|See [[Getting Started with Amanda]] instead}}
== Server configuration file - amanda.conf==


Please see {{man|5|amanda.conf}} for information on amanda.conf parameters.


==amanda.conf==
* dumptypes
* dumptypes
* dumpcycle
* [[dumpcycle]]
* [[tapecycle]]
* [[runspercycle]]
* estimate timeouts
* estimate timeouts
* [[Tapetype definitions]]
* [[Tapetype definitions]]


==disklist==
==Client configuration file - disklist==
==tapelist==
The [[disklist]] file on the Amanda server describes the list of disk list entries ([[DLE]]s, usually directories or filesystem devices) to be backed up by the Amanda configuration.  A number of parameters can be specified for each DLE either directly in the disklist or in a dumptype definition in amanda.conf.  For more information on the disklist, see {{man|8|amanda}}.
==Exclude lists==
There are three ways of excluding data in an AMANDA backup:
* Exclude an individual item explicitly in the dumptype
* Utilize an "Exclude List"
* Do not include the data in the disklist  


The only way to exclude files or directories with AMANDA is to use GNU-tar as the dump program (others?). The file system dump programs provided with unix systems (e.g. dump, ufsdump) get data at a raw drive level and generally do not allow exclusion of specific files or directories.
==[[Exclude and include lists]]==


The GNU version of tar, (GNU-tar or gtar), reads its data at a file system, (or higher), level and does include the option to exclude specific files and/or directories. It should be mentioned here that tar will change the access times on files. Tar has the ability to preserve the access times however, doing so effectively disables incremental backups since resetting the access time alters the inode change time, which in turn causes the file to look like it needs to be archived again.
==POSIX filenames in configuration files==


For the purpose of this document an AMANDA backup configuration named "exclude-test" will be used. The machine that contains the tape drive which receives data to be archived will be referred to as "SERVER". The machine that data is being archived from will be referred to as "CLIENT". These two systems are usually different machines but are not required to be, and may be the same machine. Parts of this setup are on the server and some are on the client.
Amanda 2.5.1 supports the POSIX path naming standard for all filenames in dumptype configuration in {{man|5|amanda.conf}} and disklist configuration file.


: When AMANDA attempts to exclude a file or directory it does so relative to the area being archived. For example if  /var  is in your disklist and you want to exclude /var/log/somefile, then your exclude file would contain ./log/somefile.
This feature makes configuration of Amanda clients running Windows and Amanda clients that use international character sets possible.
You may use one exclude file in multiple dumptypes without any restriction.


===Before We Begin===
Any filename referenced, including include file names, may contain any character except a NULL ('\0') character. Actual path interpretation is system dependent. POSIX file name rules are also allowed when specifying configuration file tags.


* The first step that should be taken is to verify that backups are currently working. Connect to SERVER and run amcheck as your AMANDA user, to verify that there are no errors in the current setup.
For example we use spaces in a dumptype name in the examples below.


$ amcheck -cl CLIENT
===Specifying Special characters===
Pathnames with special characters must be enclosed in double quotes ("). Some unprintable charactes are given special escape sequences.


Output should look something like below for success:
{| style="text-align:left;" border="1"
|+'''Escape sequences'''
! align=center width="25%" | Sequence
! width="30" | Character
|-
|align=center width="25%" | \n || Newline
|-
|align=center width="25%" | \r || Carrage Return
|-
|align=center width="25%" | \t || Tab
|-
|align=center width="25%" | \f || Formfeed
|-
|align=center width="25%" | \" || Double Quote
|}


        AMANDA Tape Server Host Check
        -----------------------------
        /path/to/holding-disk: 4771300 KB disk space available, that's plenty. 
        AMANDA Backup Client Hosts Check
        --------------------------------
        Client check: 1 host checked in 0.084 seconds, 0 problems found.


* Next make sure that GNU-tar is the dump program currently in use. The easiest way to tell if your dumptype is using gnutar is to run the following:
'''Example: Spaces in a dumptype name, exclude list, include list in amanda.conf'''
  define dumptype "user-tar (a1)" {
    root-tar
    comment "User partition dumped with tar and a funny dumptype name"
    priority medium
    exclude "diskfile b*"
    include list "/tmp/include list with spaces in file name"
  }


$ amadmin exclude-test disklist CLIENT
'''Example: Disklist file entries with spaces in backup directory and dumptype names'''
  localhost "/tmp/disk name with spaces" user-tar
  localhost "/tmp/disk name with\nnewline" "user-tar (a1)"


Among all the output is the "program" value currently in use. This value is also specified with the "program" option in the dumptype. If the dumptype has the line "program GNUTAR" your setup should be ready to exclude data.
'''Example: Using file names with spaces in Amrecover sub-commands'''
 
  setdisk "/tmp/disk name with\nnewline"
If GNU-tar is not in use add the line "program GNUTAR" to the dumptype, and then run amcheck again to verify that backups should work. The capitalization of GNUTAR is required.
  add "diskfile *"
 
  delete "diskfile a1"
The dumptype should look something like:
 
define dumptype exclude-test {
      comment "test dumptype for documentation"
      priority high
      program "GNUTAR"
}
 
===Choosing an exclude mechanism===
 
If the need is to exclude only one file or directory then the easiest way to accomplish this is to exclude an individual item explicitly in the dumptype. If the need is to exclude multiple files or directories then use an Exclude List.
 
===Exclude Mechanisms===
====Exclude an individual item explicitly in the dumptype====
 
The easiest way to exclude a file or directory is to specify it with the "exclude" option in the dumptype. This option accepts an argument of the file or directory to be excluded. AMANDA allows only one exclude option in any dumptype at a time.
 
UPDATE: Recent AMANDA-releases bring the option "exclude append" which enables the administrator to define more than
one exclusion-pattern within one dumptype without using a exclude-list. Please look at the [[amanda.conf]] manpage for details.
 
Any path specified to be excluded must be encapsulated with quotes. Continuing with our example from above /var/log/somefile and using the same dumptype as above, the dumptype would now look like:
 
define dumptype exclude-test {
      comment "test dumptype for documentation"
      priority high
      program "GNUTAR"
      exclude "./log/somefile"
}
 
Next run [[amcheck]] again to verify that there are no problems with the revised AMANDA configuration. If the data is not being excluded as expected please see the Troubleshooting section below. This completes the setup of excluding an individual item in the dumptype.
 
====Utilize an Exclude List====
 
An exclude list is a file that resides on the CLIENT machine and contains paths to be excluded, one per line. This file can be in any location on the CLIENT so long as the same path is specified in the dumptype. Some find /usr/local/etc/amanda an appropriate location, but it is up to you. I personally like to have a subdirectory for exclude files but it is up to you where you place this file.
 
The exclude file may also be placed in the area being archived. This is an easy way to have a different exclusion file for each disklist entry without needing separate dumptype definitions. To use this technique, enter a path relative to the area being archived as the exclude file below instead of an absolute path.
 
Connect to CLIENT and create the exclude directory as root. For example:
 
$ mkdir -p /usr/local/etc/amanda/exclude
$ cd /usr/local/etc/amanda/exclude
 
Next create the exclude list for AMANDA to use. You can name the exclude file anything you wish it to be. Create a file, and in this file place all paths to files and directories that are to be excluded. Keeping with the /var example, assume that /var/log/XFree86.0.log, and /var/log/maillog need to be excluded. Remember that all paths are relative. The exclude list would look like:
 
./log/XFree86.0.log
./log/maillog
 
Make sure that permissions are restricted on this file. Run the following as root, where exclude-filename is the name of the file you just created. For example:
 
$ chmod 644 /usr/local/etc/amanda/exclude/exclude-filename
 
This concludes the necessary configuration on the client.
 
Connect to SERVER and cd to the exclude-test AMANDA configuration directory. Edit the AMANDA configuration file e.g. amanda.conf. Add an entry similar to the following line, to the dumptype for the client in question, where the exclude-filename is the file that was created on CLIENT in the step above including the quotes. For example:
 
exclude list "/usr/local/etc/amanda/exclude/exclude-filename"
 
The new dumptype should look something like:
 
define dumptype exclude-test{
  comment "test dumptype for documentation"
  priority high
  program "GNUTAR"
  exclude list "/usr/local/etc/amanda/exclude/exclude-filename"
}
 
Save the file. Run amcheck again to verify that there are no problems with the revised AMANDA configuration. If amcheck succeeds then run amdump to verify the data is being excluded correctly. If the data is not being excluded as expected please see the Troubleshooting section below. This completes the setup of an exclude list.
 
====Do not include the data in the disklist====
 
AMANDA uses disklist entries to define which directories or partitions should be archived. This allows us to exclude data by just not placing the data in question in the disklist. Assume that there is a disk mounted on /example. The directory /example has five subdirectories "a", "b", "c", "d", and "e". The directories "a", "b", and "c" need to be archived, while "d" and "e" should not. This can be accomplished by not specifying "d" and "e" in the disklist. Using the same dumptype and host in the above examples the disklist would contain:
 
CLIENT /examples/a exclude-test
CLIENT /examples/b exclude-test
CLIENT /examples/c exclude-test
 
Run amcheck to verify that AMANDA is working correctly. If the data is not being excluded as expected please see the Troubleshooting section below. This completes the setup of using a disklist to exclude data.
Expression
 
Quiz: what is the difference between the following entries in an exclude list?
 
./foo
./foo/
./foo/*
 
case 1 : directory ./foo won't be in the backup image (that's what you want) case 2 : matches nothing (don't use it) case 3 : directory ./foo will be in the backup image but nothing below it.
 
====Wildcard Expansion====
 
AMANDA has the ability to use wildcard expansion while excluding data as implemented by tar(1). The only places that wildcard expansion is allowed is in the "exclude" option in the dumptype, or in the exclude list. Some simple examples:
 
;<nowiki>./*.log</nowiki> : Exclude any file or directory that ends in ".log" e.g. ppp.log, XFree86.0.log
 
;<nowiki>*/*log*</nowiki> : Exclude any file or directory with the string "log" e.g. logfile, maillog, syslog, ppp.log, XFree86.0.log
 
;<nowiki>./*cron*.gz</nowiki> : Exclude any file or directory that starts with string "cron" and ends in ".gz" e.g. cron.1.gz, cron.2.gz, log/cron.1.gz
 
;<nowiki>./log.?</nowiki> : The question mark can be used to specify a single character. e.g. log.1, log.2, etc
 
 
===Troubleshooting===
 
If you find that you are having trouble getting the exclude patterns to match correctly, check out this really cool script written by John R. Jackson.
 
ftp://gandalf.cc.purdue.edu/pub/amanda/gtartest-exclude
 
This script allows you to test your patterns before placing them in an exclude list or in the dumptype. Instructions on how to run the script are included in the script.
 
====Broken gnutar?====
 
There are versions of GNU-tar that do not correctly exclude data. Version 1.12 (plus the AMANDA patches from http://www.amanda.org) are known to work correctly, as does version 1.13.19 (and later). Anything else is questionable.
 
UPDATE: Using GNU-tar 1.13.25 is recommended.
 
Connect to CLIENT and run a ps command and look to see exactly how the tar command is running. Look in the output for the --exclude or --exclude-from options in the running tar process. For example:
 
$ ps ax | grep tar
?        R      0:37 /bin/tar --create --directory /var
--listed-incremental /var/lib/amanda/gnutar-lists/CLIENTvar_0.new
--sparse --one-file-system --ignore-failed-read --totals --file
/dev/null --exclude-from=/usr/local/etc/amanda/exclude-test/exclude.var
.
 
In the above output notice the string "--exclude-from=". The string following the "=" is the exclude file currently in use. If the string was "--exclude" then the string following the "=" is the file or directory that is currently set to be excluded.


== Device configuration==
== Device configuration==
Line 185: Line 75:
The speed is currently unused.
The speed is currently unused.


AMANDA provides the [[amtapetype]] utility to calculate the size of a tape, to generate a "tapetype" entry for your amanda.conf.
AMANDA provides the {{man|8|amtapetype}} utility to calculate the size of a tape, to generate a "tapetype" entry for your amanda.conf.


Specifying the appropriate tape device, but beware that it may take many hours to run (it fills the tape twice ...). Make sure you do not use hardware compression, even if you plan to use hardware compression in the future. amtapetype writes random data to tape, and random data will expand instead of compressing, therefore you'll get an estimate that's smaller than expected.
Specifying the appropriate tape device, but beware that it may take many hours to run (it fills the tape twice ...). Make sure you do not use hardware compression, even if you plan to use hardware compression in the future. amtapetype writes random data to tape, and random data will expand instead of compressing, therefore you'll get an estimate that's smaller than expected.
Line 193: Line 83:
===Changers===
===Changers===


====Introduction====
This part has been moved to a separate page.
 
This document outlines the tape changer support design of AMANDA 2.2 and beyond. It defines a small interface for changer
software to follow so that AMANDA can remain device-independent but still support the widest range of tape software and
hardware possible. The interface is a bit simplistic and has only had complications added when there is a body of field
experience.
 
====Specifying a tape changer in amanda.conf====
 
All device-specifics are hidden by a glue program that the rest of AMANDA calls to interact with the tape changer.
 
The name of this changer program is given by the "tpchanger" variable in the file amanda.conf. Example entry:
 
tpchanger "chg-multi"   # use multi-unit tape changer emulator
The tapedev parameter is ignored if a tpchanger program is specified, unless the changer program itself reads tapedev from amanda.conf. The chg-multi changer doesn't, as it reads all its configuration arguments from its own configuration file, specified as changerfile.
 
If the tpchanger program does not begin with a '/', AMANDA expects it to reside in libexecdir, and possibly have the version suffix appended depending on how AMANDA was configured.
 
Two other amanda.conf parameters are available for changer use, however their definition is left entirely up to the changer script itself. They are changerfile and changerdev. Typically changerfile will point to the configuration file specific to the changer, or be a base name of several related changer files. The changerdev variable may point to the /dev entry used to access the changer device.
 
See the documentation with the specific changer you're interested in to see exactly how these variables are used (if at all).
 
====Device-specific tapechanger script====
 
The tape changer script/program is always from the directory with amanda.conf. It is never passed the configuration name it is running on behalf of, but since amgetconf works without a name from the current directory, that should be sufficient.
 
The script/program must support the following commands:
 
; <tpchanger> -slot <slot-specifier> : If changer is loaded, unloads the current slot (if different than "slot-specifier") and puts that tape away, then loads the requested slot. See the next section for the list of valid slot requests.
 
Outputs to stdout the slot name and name of the device file to access the tape on success, or a slot name and error text.
 
Returns 0 on success.
 
Returns 1 on positioning error (eg at bottom of gravity stacker or slot empty).
 
Returns 2 any other fatal error. The slot name may be invalid, but must be present. Error message goes to stdout in place of device name.
 
Examples:
 
% chg-multi -slot 0
0 /dev/nrst8 # exitcode returned is 0
 
% chg-multi -slot 1
1 slot 1 is empty # exitcode returned is 1
 
% chg-multi -slot bogus-slot
<none> no slot `bogus-slot' # exitcode returned is 2
 
; <tpchanger> -info : Outputs to stdout three or more fields:
 
* The current slot string (required)
* The number of slots (required)
* Flag indicating whether the changer can go backward (0 if it can't, 1 if it can). (required)
* Flag indicating whether the changer is searchable
* (optional). Shows whether the changer supports the -search and -label commands and is able to load a tape given only the AMANDA label string (0 or omitted if it can't, 1 if it can). (optional)
 
Examples:
% chg-multi -info
0 10 1 # exitcode returned is 0
 
% chg-zd-mtx -info
0 10 1 1
 
; <tpchanger> -reset : Resets the changer to known state and loads the first slot.
 
Output and error handling are the same as "<tpchanger> -slot first"
 
In the case of a gravity stacker that must be reset by hand, this could be run (via " amtape <conf> reset") to inform the software the stacker is positioned back at the top.
 
Examples:
 
% chg-multi -reset
0 /dev/nrst8 # exitcode returned is 0
 
% chg-multi -reset
0 slot 0 is empty # exitcode returned is 1
 
% chg-multi -reset
0 tape-changer not responding # exitcode returned is 2
 
; <tpchanger> -eject : Unloads the current slot (if loaded) and puts that tape away.
 
Output and error handling are the same as the -slot command. Note that a tape may or may not be loaded when this command completes, depending on the hardware.
 
Examples:
 
% chg-multi -eject
0 /dev/nrst8 # exitcode returned is 0
 
% chg-multi -eject
0 drive was not loaded # exitcode returned is 1
 
The tape changer program MAY support the following commands:
 
; <tpchanger> -search <labelstr> : Loads an AMANDA tape by name (labelstr).
 
Output and error handling are the same as the -slot command. taper, [[amcheck]] and [[amtape]] will use this command if the changer reports it is searchable.
 
Example:
 
% chg-zd-mtx -search DailySet005
5 /dev/nrst8 # exitcode returned is 0
 
; <tpchanger> -label <labelstr> : Associates the AMANDA label <labelstr> with the barcode of the currently loaded (in the tape drive) tape.
 
Outputs to stdout the current slot and tape device. amlabel will use this command if your changer is searchable to build up the barcode database.
 
Example:
 
% chg-zd-mtx -label DailySet006
6 /dev/nrst8 # exitcode returned is 0
 
; For all the commands :  An exit code of 0 implies that the operation was completely successful, and the output may be parsed by the AMANDA code as described above.
 
For non-zero exit codes, the first field is still the slot name, but the actual error messages are not fixed. They are just displayed and/or logged as-is by the calling AMANDA program.
 
An exit code of 1 implies the operation failed in a benign way, for example an empty slot or an attempt to go backwards in a gravity stacker. The calling AMANDA program will print the error message if appropriate and continue, perhaps requesting a different slot be loaded.
 
Any other exit code is considered fatal and will cause AMANDA to stop attempting to talk to the tape changer.
 
====Slot names and the "current" slot====
 
Some tape changers, such as carousels and gravity stackers, have a hardware notion of current position. Others have no current position when no tape is loaded: all tapes are in their slots and the changer arm is docked away from the slots.
 
Nevertheless, AMANDA requires tape-changer scripts to maintain the notion of a "current" position. This is for performance reasons: as tapes tend to be loaded into the rack in order, and AMANDA uses them in order, the next tape to use can be found much quicker if the position of the current one is remembered. As an example, the chg-multi script maintains the current position in a chg-multi.state file (or any other file specified in a `statefile' line in the changer configuration file).
 
AMANDA does not care how slots are available or how they are named. They could be numbered 0 to N-1, numbered 1 to N, or even designated by letter, A .. Z. The only requirement is that the names do not contain whitespace and that the names "current", "next", "prev", "first", "last" and "advance" retain their meaning as follows:
 
* current - The position of the last loaded tape, as described above
* next - The position after current, wrapping from the last slot to the first.
* prev - The position before current, wrapping from the first slot to the last.
* first - The first slot in the tape rack.
* last - The last slot in the tape rack.
* advance - The same as "next" except the next tape may not be loaded if the changer supports advancing to the next slot without putting that tape in the drive.
 
The current position must be updated even if there is a positioning error (such as "empty slot"). This allows amanda to step through the entire tape rack by issuing successive "slot next" positioning commands.
 
====Operator interface====
 
The [[amtape]] program is the main operator interface to AMANDA's tape changer support. The commands supported include:
 
; amtape <conf> slot <slot-specifier> : Load the tape from the specified slot into the drive
 
; amtape <conf> eject : Send an eject command to the tape-changer. Effect is changer specific.
 
; amtape <conf> reset : Send a reset command to the tape-changer. Effect is changer specific.
 
; amtape <conf> show : Go through the entire tape rack, showing the labels of all amanda tapes encountered.
 
; amtape <conf> label <label> : Find and load the tape with the specified label
 
; amtape <conf> taper : Perform taper's scan algorithm (see below), loading the tape which would be picked for the next amdump run.
 
; amtape <conf> clean : If a cleaning tape is defined by the changer, load it in the drive and put it away when done.
 
; amtape <conf> device : Output the current tape device name.
 
; amtape <conf> current : Display the contents of the current slot.
 
; amtape <conf> update : Scan the entire tape rack and update the barcode database.
 
See the [[amtape]](8) man page for more details.
 
In addition to amtape, [[amlabel]] has been modified to allow optionally specifying a slot:
 
; amlabel <conf> <label> [slot <slot-specifier>]
 
[[amcheck]] looks for the next tape in the rack the same way the taper does. If multiple tapes are used in one night, amcheck attempts to find all the needed tapes in turn if the tape-changer is random access. On a one-way gravity stacker, amcheck only finds the first tape, since finding the subsequent ones would put the first one out of reach of that night's amdump run.
 
[[amrestore]] and [[amrecover]] do not yet include any tape changer support directly, as amrestore knows nothing about the amanda.conf files or server-side databases. This is a deliberate decision to keep amrestore independent, so it can be run from any host with a tape drive, even if the original tape server host is down. To use amrestore in a tape-changer environment, use amtape to find the right tape, then run amrestore giving the resulting tape device name.
 
====How amdump interacts with the tape changer====
 
AMANDA does not require a particular tape label for a run. Any label that matches the labelstr regex and is determined to be "inactive" according to the tapelist database, may be written to. However, there is a preferred 'next' tape, the one that is next in the cycle implied by the current tapelist.
 
amdump uses two algorithms, depending on whether the tape changer can go backwards in the rack or not. If multiple tapes are needed in a single run, this algorithm is repeated in turn whenever a new tape is required.
 
; Normal tape changers (those that can go backwards) : With a full-access tape changer, amdump searches the entire rack for the preferred tape label. This tape will usually be found at the current or next position, but might be located anywhere. If the tape is found, it is used. If it is not found, the first tape encountered that matches the labelstr and is not active is picked.
 
; Gravity stackers (anything that can not go backwards) : To avoid going all the way to the bottom of the stacker only to find that the preferred tape isn't present and nothing can be done, AMANDA picks the first tape (starting at the current position) that matches the labelstr and is not active, regardless of whether it is the preferred tape.
 
====Builtin tape-changers====
=====chg-multi (formerly chg-generic)=====
 
This tape changer script supports several common configurations:
 
* Using multiple tape drives in a single host to emulate a tape changer. This can also be used with a single physical drive to write several tapes in an AMANDA run.
* Using a gravity stacker or a real changer configured to sequentially load the next tape when the current one is ejected. Also supports a changer which cycles to the first tape after loading the last one.
* Using a changer accessed through several "virtual" tape devices which determine which slot actually gets loaded in the tape drive.
 
The advantage of this changer script is that you do not need to get into the complexity of dealing with a real changer interface. All the action goes through the tape device interface with standard mt commands, which eases many portability issues. Many common tape jukeboxes can be configured in a sequential or cycle mode.
 
chg-multi ignores `tapedev' and `changerdev' because `changerfile' may specify several tape devices to be used. A sample configuration file may be found in example/chg-multi.conf.
 
=====chg-manual (formerly no-changer)=====
 
This is a poor man's tape changer that requires the backup operator to change tapes manually. It expects `tapedev' in amanda.conf to point to a valid tape device, and stores some status data in files whose names start with the `changerfile'. `changerdev' is ignored.
 
=====chg-mtx (formerly hp-changer)=====
 
An mtx-based tape changer script. `changerdev' must specify the tape device controlled by the mtx program, and `tapedev' must point to the no-rewind tape device to be used. More than likely, both `changerdev' and `tapedev' will reference the same device file. `changerfile' must specify a prefix for status files maintained by this script. It will maintain files named `changerfile'/changer-clean and `changerfile'/changer-access. You may have to edit the script to specify which slot contains a cleaning tape (cleanslot).
 
The mtx program must support commands such as `-s', `-l' and `-u'. If the one you've got requires `status', `load' and `unload', you should use chg-zd-mtx instead (see below).
 
=====chg-zd-mtx=====
 
Based on chg-mtx, but modified in order to support the Zubkoff/Dandelion version of mtx. Eric DOUTRELEAU <[email protected]>, who contributed this script, reported that it works on a Solaris/sparc box with a HP 1557A stacker.
 
In addition to the `changerfile'-clean and the `changerfile'-access files, it maintains a `changerfile'-slot file that indicates the currently loaded slot.
 
There are lots of comments at the start of the script describing how to set it up.
 
=====chg-scsi-chio (formerly seagate-changer, then chg-chio)=====
 
A C program that relies on scsi tape-changer interfaces. It may either use the tape changer interface specified in chio.h (Gerd Knor's SCSI media changer driver, a Linux kernel loadable module), or it may use built-in tape changer interfaces available on HPUX, Solaris 2.5, IRIX and possibly others, but only the chio and HPUX interfaces are currently implemented . `tapedev' specifies the tape device to be used; `changer_dev' is the device used to talk to the kernel module (for chio, usually /dev/ch0), and `changerfile' specifies a filename in which the current slot number will be stored.
 
Now there is another way, to get the chg-scsi a little bit more flexible. If you use only one digit in the `tapedev' parameter, the chg-scsi expects that changerfile points to a real configuration file, not only a counting file. In this configuration file you may specify that the tapedrive needs an eject command and an optional waittime, necessary after inserting the tape into the drive. You are also able to configure a range of slots which should be used by your configuration. The idea behind this is, that you don't want AMANDA to cycle all the tapes if AMANDA searches exactly one tape. If you have a library which supports more than one drive you can also specify which drive to use. For each configuration (there should be at least one) you have to specify a file, where AMANDA remembers which tape is actually in the drive. For future use there is also some stuff for cleaning the drives.
 
In amanda.conf:
 
tapedev "x"      with x between 0 and 9, selects the configuration to use
changerfile "filename"            specifies the changer configuration file
 
In the changer-config-file the following could be set:
 
number_configs x
# x between 0 and 9 number of configurations defined. This should be the first parameter in the config-file.
eject x
# x 0 or 1 1 means that the drives need an eject command, before the robot can handle the tape.
sleep x
# x between 0 and MAX_INT specifies the seconds to wait before the drive could be used after inserting a tape. 5 should be OK.
cleanmax x
# x some positive int How many cleanings does a cleaning tape survive
changerdev  <device>
# The device for the robot
And then there come some configuration sections, separated by the word `config` followed by the ordinal of that configuration (0 to 9). In each configuration section you should specify:
 
drivenum x x between 0 and the number of drives in the library
This one specifies the drive to use with this configuration
dev <device> The device for the tapedrive
startuse    x x between 0 and maximum slotnumber of your library
Starting here we may use the tapes
enduse x x between start and maximum slotnumber
This is the last tape we may use in this configuration. If we reach this one the next will be start..
statfile <filename> Here we remember the last used
slot for this configuration
cleancart x x between 0 and maximum slotnumber
In this slot we find the cleaning tape
cleanfile <filename>
Here we will remember how often we used the cleaning tape
usagecount <filename> This points to a file which is deleted after cleaning the drive
                                                e.g. the usagetime of the drive
Comments begin with an '#' until end of line. Allowed separators are TAB and SPACE.
 
=====chg-scsi (new interface, try to drive a robot with direct scsi commands)=====
 
The config and the syntax is the same as for chg-scsi-chio. New is the config type
 
emubarcode 1
 
With this option and the option labelfile chg-scsi will try to create an inventory. With this inventory it should be possible to use the search feature for loading tapes.
 
debuglevel x:y
 
This option will set the debug level and select for which part debug messages should be sent to the debug file. In case of problems you should set it to 9:0
 
havebarcode 1
 
This will force the program to read the barcodes, and don not try to figure out if there is an barcode reader available.
 
scsitapedev <devicename>
 
This device is used to control the tape, read status infos etc.
 
tapestatus <filename>
 
If this option is given on every eject/move the log pages of the tape device will be dumped in this file. There are 2 log pages were you can see how many read/write errors (corrected) are processed by the tape
 
labelfile <filename>
 
This file is used for the mapping from barcode labels to AMANDA volume labels. It is used if the changer has a barcode reader. To initialize run amtape show, this will add the mapping for the tapes in the magazine.
 
eject > 1
 
Use the mtio ioctl to eject the tape, use only if the standard (1) does not work, and send the debug output (/tmp/amanda/chg-scsi.debug) to [email protected]
 
changerident <ident>
 
With this it is possible to specify which internal driver to use for controlling/error handling of the robot
 
tapeident <ident>
 
Same as above but for the tape.
 
New command line option: -status [all|types|robot|sense|ModeSenseRobot|ModeSenseTape|fd]
 
<all> will show the result form all options. <types> will list the known driver types. <robot> will show the status of all elements (tape/robot/slots..) <sense> will show the result from a request sense <ModeSenseRobot> will show the sense page from the robot <ModeSenseTape> will show the sense page from the tape <fd> will show the devices which are open, and some info about it.
 
At the moment changer with tape and robot on the same SCSI id ( but on different luns) will run on the following platforms:
*HP_UX 10.20
*IRIX 6.x
*Solaris
*Linux
*AIX
*FreeBSD 3.0/4.0
 
Tape and robot on different IDs run native on
*Linux
*HP-UX 10.20
*Irix 6.x
*FreeBSD
 
'''Solaris'''
 
Tape and robot on different IDs with special modules run on: Solaris with sst kernel module, which is not any longer needed in solaris 2.8. See in the contrib/sst directory The configuration on solaris 2.8 with the sgen driver is done by creating the file /kernel/drv/sgen.conf
 
This file should contain at the beginning the following device-type-config-list="changer","sequential"
 
This will force the driver to attach only to the devices with type either changer (the robot) and sequential (the tape). Next you must tell the driver on which id it should check for devices (tape on id 5, robot on id 6 in this example),
 
name="sgen" class="scsi" target=5 lun=0; name="sgen" class="scsi" target=6 lun=0;
 
This will create the 2 device files /dev/scsi/sequential/c0t5d0 (scsitapedev option in chg-scsi.conf) /dev/scsi/changer/c0t6d0 (changer option in chg-scsi.conf)
 
So the complete sgen.conf looks like: device-type-config-list="changer","sequential name="sgen" class="scsi" target=5 lun=0; name="sgen" class="scsi" target=6 lun=0;
 
'''HP'''
 
You have to create the special device files for the pass throu interface. Check if the ctl driver is installed. Example:
 
# lsdev -C ctl
  Character    Block      Driver          Class
    203          -1        sctl            ctl
 
Next check on which bus your drives are connected. (ioscan) with the Character device num form the lsdev and the card instance from ioscan create the special file. Example:
 
mknod /dev/scsi/1 c 203 0x001000
                          ||||
                          ||| LUN of device
                          ||SCSI ID of the device
                          2 digit instance number from ioscan
 
'''FreeBSD 4.0'''
 
The syntax for the device files has changed. Now you have to tell chg-scsi the bus:target:lun combination. If you for example on your scsi bus 0 target 3 an robot the syntax is changerdev 0:3:0 To get this info you can use the camcontrol command, <camcontrl devlist> will give you a list of known devices. Don't specify dev and scsitapedev in your chg-scsi.conf !!, this will not work.
 
'''Linux'''
 
You need either sg (generic scsi) as module or it must be compiled into the kernel. If the sg driver doses not work try to use the ioctl interface. For that you have to undef the LINUX_CHG define in changer-src/scsi-linux.c Also you have to change the NORMAL_TIMEOUT in /usr/src/linux/drivers/scsi/scsi_ioctl.c from (10 * HZ) to (5 * 60 * HZ). On linux it does not run if you are using an aha1542 SCSI controller. The driver can not handle the extended request sense.
 
'''IRIX'''
 
You find the SCSI pass through interfaces for every device in /dev/scsi.
 
chg-scsi has been tested/run with the following devices: Exabyte 10h and eliant tape HP-Surestore 1200e and C1553A tape BreeceHill Q2.15 (EXB-120) and DLT7000 tape Powerstor L200 and DLT7000 ARCHIVE Python 28849-XXX TANDBERG TDS 1420 ADIC VLS DLT Library
 
It is now possible with a changer that has barcode reader to load tapes faster. Also amdump will find tapes faster. Every time a tape is labeled the information in the labelfile will be updated. To initialize the label mapping you can also do an amtape config show. This will put the mapping for all tapes in the magazine into the mapping file.
 
For all problems please contact [email protected]. Please include in your mail the debug file. (/tmp/amanda/chg-scsi.debug)
 
=====chg-chio=====
(new perl script that replaces the original chg-chio written in C) This script is based on the FreeBSD version of chio, a program to communicate with the jukebox. This script has for the moment only been test with FreeBSD and is likely not to work on any other system. Let me know if this is the case and send me the output of the chio program for your version of chio. It does not restrict the number of tapes, except that if there is only one tape in the juke, it is supposed to be in max_slot and not in slot 1. [This is the first version of the changer script and I would appreciate all comments on it, at [email protected]. It has been tested only with FreeBSD 2.2.5 and the accompanying chio program.]
 
=====chg-chs=====
(formerly chs-changer) A tape changer script very similar to chg-multi, that uses the `chs' program to change tapes. As in chg-multi, `tapedev' is ignored. `changerfile' names its configuration file, similar to chg-multi.conf. `changerdev' will be passed to CHS in a -f command-line switch, unless it is set to an empty string or "/dev/null" (watch out for default values!)
 
=====chg-rth=====
(formerly rth-changer) A perl5 script that controls an HPc1553 tape drive via a Peripheral Vision Inc. SCSI control subsystem that interprets commands sent on the SCSI bus. It expects `tapedev' to specify the tape device to be used. `changerfile' and `changerdev' are ignored.
 
=====chg-juke=====
A shell script that uses the Fermilab "juke" software (see http://www.fnal.gov/fermitools, the "juke" link) to control tape chagners. It supports mounting multiple tapes for RAIT tapedrive sets, both multiple jukeboxes, or one jukebox with multiple tape drives, or combinations. 'juke' must be configured to know tape drives by the same name AMANDA calls them. It uses 'changerfile' to track AMANDA's current tape state, 'tapedev' must be the tape drive (or RAIT set) name, and 'changerdev' is the juke software's name for the changer, or a csh-glob that expands to several jukebox names (i.e. "changer{a,b,c}").
 
=====chg-rait=====
A shell script that runs other changers in tandem, and returns a rait:{dev1,dev2,...} tape device based on the results of each other changer. So if you wanted to have 2 stackers striped with no parity, and you have chg-mtx support for your stackers, you would use the following changerfile:
 
  nchangers=3
  tpchanger_1="chg-mtx"
  changerdev_1="/dev/mtx1"
  changerfile_1="/some/file1"
  tapedev_1="/some/dev"
  tpchanger_2="chg-mtx"
  changerdev_2="/dev/mtx2"
  changerfile_2="/some/file2"
  tapedev_2="/some/dev"
  tpchanger_3="chg-null"
  changerdev_3="/dev/null"
  changerfile_3="/some/file3"
  tapedev_3="/some/dev"
 
The third uses the null changer. The tapedev_n entries are only needed if the changerfile in question uses them.
 
=====chg-disk=====
 
Clone of the chg-zd-mtx, but modified to be applied on local directories instead of tapes. This changer emulates a robotic that uses virtual tapes instead of real ones, where the virtual tapes are real directories on a hard disk.
 
The directory tree should be:
 
slot_root_dir -|
                |- info
                |- data -> slot1/
                |- slot1/
                |- slot2/
                |- ...
                |- slotn/
 
Where "slot_root_dir" is the tapedev "file:xxx" parameter and "n" the tapecycle parameter.
 
Please refer to How to use the AMANDA file-driver for details of usage.
 
=====chg-iomega=====


This changer script is designed for IOMEGA or JAZZ disks of various sizes as well as any other removable disk media. This is a PURELY MANUAL changer. It requests insertion of disk media via messages on /dev/tty. So it cannot be used via crontab. Make sure you comply with any of the following. - Add statements
See [[Changers]].
 
        tpchanger "chg-iomega"
        tapedev "file:<mount_point_of_removable_disk>"
        (e.g. tapedev "file:/mnt/iomega" )
        tapetype IOMEGA     
 
       
        define tapetype IOMEGA {
            comment "IOMega 250 MB floppys"
            length 250 mbytes
            filemark 100 kbytes
            speed 1 mbytes
        }
 
to your amanda.conf. - Add entry to /etc/fstab to specify mount point of removable disk and make this disk mountable by any user. - Format all disks, add a "data" sub directory and label all disks by using amlabel. - Be aware that as of version 2.4.4p1, AMANDA can't handle backups that are larger than the size of the removable disk media. So make sure /etc/amanda/<backup_set>/disklist specifies chunks smaller than the disk size.
 
=====chg-null=====
 
A trivial changer which loads/unloads on a null: device. Useful with chg-rait to throw away a parity stripe by puttin on a null jukebox, or for testing.


===RAIT===
===RAIT===
Currently it is only integrated with the chg-manual changer script


RAIT is an acronym for "Redundant Array of Inexpensive Tapes", where data is striped over several tape drives, with one drive writing an exclusive-or-sum of the others which can be used for error recovery. Any one of the data streams can be lost, and the data can still be recovered.
RAIT is an acronym for "Redundant Array of Inexpensive Tapes", where data is striped over several tape drives, with one drive writing an exclusive-or-sum of the others which can be used for error recovery. Any one of the data streams can be lost, and the data can still be recovered.
Line 638: Line 93:
This means that a 3-drive RAIT set will write 2 "data" streams and one "parity" stream, and give you twice the capacity, twice the throughput, and the square of the failure rate (i.e. a 1/100 failure rate becomes 1/10,000, since a double-tape failure is required to lose data).
This means that a 3-drive RAIT set will write 2 "data" streams and one "parity" stream, and give you twice the capacity, twice the throughput, and the square of the failure rate (i.e. a 1/100 failure rate becomes 1/10,000, since a double-tape failure is required to lose data).


Similarly, a 5-drive RAIT set will give you 4 times the capacity, 4 times the throughput (with sufficient bus bandwidth), and the square of the failure rate.
This means you can back up partitions as large as twice or four times your tape size with Amanda, with higher reliability and speed.
 
This means you can back up partitions as large as four times your tape size with AMANDA, with higher reliability and speed.
 
====Using a RAIT====
 
If you have several tape devices on your system [currently either 3 or 5 drive sets are supported] you tell AMANDA to use them as a RAIT by listing them as a single tape device using /bin/csh curly-brace-and-comma notation, as in:
 
tapedev = "rait:/dev/rmt/tps0d{4,5,6}n"
 
which means that /dev/rmt/tps0d4n, /dev/rmt/tps0d5n, and /dev/rmt/tps0d6n are to be treated as a RAIT set. You can now mount three tapes, and label them with amlabel, etc.
 
Also, you want to create a new tape-type entry, which lists an n-drive RAIT set, for this RAIT-set. So if you were using an entry like:
 
define tapetype EXB-8500 {
        comment "Exabyte EXB-8500 drive on decent machine"
        length 4200 mbytes
        filemark 48 kbytes
        speed 474 kbytes
}
 
You would want to make a new one like:
 
define tapetype EXB-8500x3 {
        comment "Exabyte EXB-8500 3 drive stripe on decent machine"
        length 8400 mbytes
        filemark 200 kbytes
        speed 948 kbytes
}
and change your tapetype entry to: tapetype EXB-8500x3 to tell AMANDA about the multiple drive set.
 
====Disaster Recovery====
 
To assist in disaster recovery (as well as changer scripts) the AMANDA package now also includes amdd, which is a simple dd(1) replacement which supports (only) the "if=xxx", "of=xxx", "bs=nnn[kMb]" "skip=nnn" and "count=nnn" options, but which can read and write RAIT tapesets.
 
Using [[amdd]] and your usual AMANDA unpack instructions will suffice for disaster recovery from RAIT tape-sets.


RAIT can also be used to mirror a backup to two drives, even when one of them is a virtual tape and the other is a real tape.


See [[Rait]] for a more complete description.


===File driver/Disk backups===
===File driver/Disk backups===
Stefan G. Weichinger, November - December, 2003 ; minor updates in April, 2005.
====Introduction====


Since release 2.4.3 AMANDA supports the usage of a output driver called "file". See man amanda, section OUTPUT DRIVERS, for more information on its implementation. As the name suggests, this driver uses files as virtual (or file) tapes. Once created and labeled, these file tapes can be selected and changed with the standard tape-changer-interface of the AMANDA server.
This driver uses files on disk as virtual tapes. Amanda can write to and read from virtual tapes, just like real tapes. A bunch of virtual tapes can even be manipulated with a changer.


Possible Uses
Possible Uses
* Test installations: You can easily explore the rich features of AMANDA on systems without tape drives.
* Inexpensive installations: Without buying a tape drive you can enjoy the benefits of AMANDA and backup to a bunch of harddisks. You can create CD/DVD-sized backups which you can burn onto optical disks later.
* disk-based installations: You can use the file-driver to backup onto a set of file tapes hosted on a bunch of hard-disks or a RAID-system. Combined with another AMANDA-configuration that dumps the file tapes to real tapes, you can provide reliable backup with faster tapeless recovery. This is called "disk-to-disk-to-tape"-backup by some people today.


====Setup====
* Test installations: You can easily explore the rich features of Amanda on systems without tape drives.
* Inexpensive installations: Without buying a tape drive you can enjoy the benefits of Amanda and backup to a bunch of internal or external harddisks connected with Firewire or USB. You can create CD/DVD-sized backups which you can burn onto optical disks later.
* Disk-based installations: You can use the file driver to backup onto a set of virtual tapes hosted on a bunch of hard-disks or a RAID-system. Combined with another Amanda configuration that dumps the virtual tapes to real tapes, you can provide reliable backup with faster tapeless recovery. This is called "disk-to-disk-to-tape" backup by some people today.


This guide assumes you have setup the basic AMANDA-services as described in Amanda Installation Notes
See [[File driver]] for a more complete description of virtual tapes, and their use.


The configuration in this HOWTO is called "daily". The file tapes are also called vtapes in this document, which stands for "virtual tapes".
==Server-side and Client-side encryption==


Please be sure to understand the differences between holding disks and file tapes. The two serve different purposes; holding disks allow for parallelism of multiple DLE's being backed up while file tapes are a replacement for physical tapes.
This section has been moved to a separate page.


Before beginning you will need to decide on (a) dedicated part(s) of your hard disk(s) for your file tape storage. While this space could be spread among several file systems and hard disks, I recommend to dedicate at least a specific partition, better a specific physical harddisk to the task of keeping your vtapes. The use of a dedicated disk will speed things up definitely.
See [[Encryption]].


The disk space you dedicate for your vtapes should NOT be backed up by AMANDA. Also, for performance reasons there should be NO holding disks on the same partition as the vtapes, preferably not even on the same physical drive.
==Custom Compression==
*compress client custom
**Specify client_custom_compress "PROG"
**PROG must not contain white space and it must accept -d for uncompress.


If you only have one harddisk, it will work out, too, but you will suffer low performance due to massive head-moving in your harddisk, resulting from copying data between the filesystems.
*compress server custom
**Specify server_custom_compress "PROG"
**PROG must not contain white space and it must accept -d for uncompress.


* Prepare the filesystem(s) used for the tapes. Decide on where to put your files, create the appropriate partition(s) and
*sample dumptype:
filesystem(s) and mount them. In our example we have the dedicated partition hdc1, mounted on /amandatapes for vtape storage.
   define dumptype custom-tar {
 
  global
$ mount
  program "GNUTAR"
[...]
  comment "root partitions dumped with custom compression"
/dev/hdc1 on /amandatapes type reiserfs (rw)
   compress server custom
[...]
   server_custom_compress "/usr/bin/my_gzip"
   priority low
Make sure there is space left. Determine the amount of space you will use.
  }
 
$ df -h /amandatapes
Filesystem      Size  Used  Avail  Use%   Mounted on
/dev/hdc1        20G    0G    20G    0%  /amandatapes
 
In our example we have 20GB diskspace left on /amandatapes.
 
* Determine length and number of tapes. After deciding on the number of vtapes you want to create, evenly allocate the
available space among them.
 
Look at the following rule of thumb: As many filesystems exhibit dramatically reduced performance when they are nearly full I have chosen to allocate only 90% of the available space. So we have:
 
      (Available Space * 0.9) >= tapelength * tapecycle
 
This is a very conservative approach to make sure you don´t suffer any performance drop due to a nearly-full-filesystem. As it is
uncommon for AMANDA to fill, or almost fill an entire tape you may also wish to use more space than that.  So you could determine
possible combinations of tapelength/tapecycle with the more general formula:
 
      Available Space >= tapelength * tapecycle
 
In our example we take the conservative approach:
:* 20 GB * 0.9 = 18 GB to use
and so we could create the following combinations:
:* 18 GB = 18 GB * 1
:* 18 GB = 9 GB * 2
:* 18 GB = 6 GB * 3
:* 18 GB = 3 GB * 6
:* 18 GB = ......... you get the picture.
 
Using only one tape is generally considered a bad idea when it comes to backup, so we should use at least 3 tapes (for testing purposes), better 6 or more tapes.
:* 18 GB = 3 GB * 6
so we get the value 3 GB for the tapelength if we want to use 6 tapes.
 
* Create a tapetype definition. Add a new tapetype definition similar to the following to your amanda.conf. I named my definition "HARD-DISK". Choose whatever name you consider appropriate.
 
define tapetype HARD-DISK {
      comment "Dump onto hard disk"
      length 3072 mbytes # specified in mbytes to get the exact size of 3GB
}
You don´t have to specify the parameter speed (as it is commonly listed in tapetype definitions and reported by the program amtapetype). AMANDA does not use this parameter right now.
 
There is also an optional parameter filemark, which indicates the amount of space "wasted" after each tape-listitem. Leave it blank and AMANDA uses the default of 1KB.
 
* Think about tapechangers.
 
As you will use a set of vtapes, you have to also use a kind of vtape-changer. There are several tape-changer-scripts included in the AMANDA. Read more about tape-changer-scripts in AMANDA Tape Changer Support. Right now there are two scripts that can be used with vtapes. These scripts take different approaches to the handling of tapes.
:*The script chg-multi handles many drives with a tape in each drive.
:*The script chg-disk handles a library with one drive and multiple tapes.
 
So with vtapes you could look at it this way: chg-multi simulates multiple tape drives with one tape in each drive. chg-disk simulates one tape-library with multiple tapes in. As chg-multi exists for a much longer time than chg-disk, it is still used in many AMANDA-vtape-installations.
 
Contrary to chg-multi, which is a generic changer-script that must be somewhat adjusted to the use of the file-driver, chg-disk offers exactly the behavior needed for handling vtapes. IMHO the approach is much more logical, so I recommend to use chg-disk in new AMANDA-vtape-installations.
   
      To use chg-disk you need to have at least amanda-2.4.4p1-20031202.
 
Choose the one that fits your way of vtape-handling and -maintenance.  In this HOWTO I only cover the use of chg-disk. Usage of
chg-multi is pretty similar and will maybe covered in a later version of this document.
 
* Set up your tape-config. In the general section of amanda.conf, you have to set the parameters tapecycle , tapetype , tpchanger , changerfile , tapedev , rawtapedev and changerdev.
 
$ vi /usr/local/etc/amanda/daily/amanda.conf
...
 
tapecycle 6
tapetype HARD-DISK
tpchanger "chg-disk"
changerfile "/usr/local/etc/amanda/daily/changer"
tapedev  "file:/amandatapes/daily"
This reflects the use of your defined tapetype.  The parameter tapecycle tells AMANDA how much tapes can be used, Set this value according to the number of tapes you want to use. The parameter tapetype , points to the tapetype definition you have created before.  The parameter tpchanger tells AMANDA to use the generic tape-changer-script to handle the vtapes. You can think of it as a virtual tape-changer-device.
 
The parameter changerfile is used to give chg-disk the "prefix" for the "%s-changer, %s-clean, %s-slot" files it needs. Use something like "changer" in your config-dir. Please note that this file does NOT have to exist, but it won't hurt anyway.  The parameter tapedev tells the chg-disk-script where the root-dir for your vtapes is.
 
In our example the vtape-files go to /amandatapes.  To separate multiple configurations, we decided to use subdirectories according to the configuration name "daily".
      The parameter changerdev is NOT needed with chg-disk as it is not parsed by chg-disk.
 
* Create the virtual tapes.
 
      Gene Heskett has committed a shell-script which creates and labels the vtapes in one step. Stefan G. Weichinger will
      generalize this script and contribute it, this script will just read your settings in amanda.conf and create the
      appropriate vtape-directories.
 
Now you have to create the tape-directories. chg-disk needs a directory structure like:
 
slot_root_dir -|
                |- info
                |- data -> slot1/
                |- slot1/
                |- slot2/
                |- ...
                |- slotn/
where 'slot_root_dir' is the tapedev 'file:xxx' parameter and 'n' is the tapecycle parameter.
 
So in our example we do:
$ mkdir /amandatapes/daily
for the 'slot_root_dir' and
    $ mkdir /amandatapes/daily/slot1
    $ mkdir /amandatapes/daily/slot2
      .... 
 
for the virtual slots that will later contain the vtapes. If you have many vtapes to create and their names follow a pattern you may be able to do them all with a single loop such as:
 
$ for n in 1 2 3 4 5 6 7 8 9 10 11 12
> do
>    mkdir /amandatapes/daily/slot${n}
> done
Create the info-file:
$ touch /amandatapes/daily/info
 
and link the first slot to the data-file (to "load" the vtape into the first slot):
$ ln -s /amandatapes/daily/slot1 /amandatapes/daily/data
 
Make sure the AMANDA-user has write-permissions on these directories:
$ chown -R amanda_user /amandatapes
$ chgrp -R amanda_group /amandatapes
$ chmod -R 750 /amandatapes
* Label the virtual tapes. As the virtual tapes are handled just like physical tapes by the AMANDA-Server they have to be
labeled before use.
 
$ amlabel daily daily1 slot 1
   ....
$ amlabel daily daily2 slot 2
   .... 
 
If you have many vtapes to label and their names follow a pattern you may be able to do them all with a single loop such as:
 
$ for n in 1 2 3 4 5 6 7 8 9 10 11 12
> do
>    amlabel daily daily${n} slot ${n}
> done
Label all your created tapes according to the "labelstr"-parameter in your amanda.conf. Consult the amlabel-man-page for details.
 
* Test your setup with amcheck. Run amcheck daily (or, more general, amcheck <config>) and look for anything AMANDA complains about.
 
A proper output looks like:
 
$ amcheck daily
Amanda Tape Server Host Check
--
Holding disk /amhold: 6924940 KB disk space available,
that's plenty
amcheck-server: slot 2: date 20031115 label daily02
(exact label match)
NOTE: skipping tape-writable test
Tape daily02 label ok
Server check took 0.377 seconds
 
Recheck your files if errors occur.
 
====Recovery====
 
Recovering files from vtapes is very similar to recovering files from a "real" tapechanger. Make sure you read the chapter Restore.
I will simply paste a amrecover-session here (provided by JC Simonetti, author of chg-disk):
 
# /usr/local/amanda/sbin/amrecover woo
AMRECOVER Version 2.4.4p3. Contacting server on backupserver.local ...
220 backupserver AMANDA index server (2.4.4p3) ready.
200 Access OK
Setting restore date to today (2004-10-08)
200 Working date set to 2004-10-08.
Scanning /BACKUP2/holding...
Scanning /BACKUP/holding...
200 Config set to woo.
200 Dump host set to backupserver.local.
Trying disk /tmp ...
$CWD '/tmp/RECOVER' is on disk '/tmp' mounted at '/tmp'.
200 Disk set to /tmp.
Invalid directory - /tmp/RECOVER
amrecover> sethost backupserver.local
200 Dump host set to backupserver.local.
amrecover> setdisk /
200 Disk set to /.
amrecover> cd /etc
/etc
amrecover> add passwd
Added /etc/passwd
amrecover> list
TAPE B3_14 LEVEL 0 DATE 2004-09-26
        /etc/passwd
amrecover> extract
 
Extracting files using tape drive file:/BACKUP2/slots/ on host
backupserver.local. The following tapes are needed: B3_14
 
Restoring files into directory /tmp/RECOVER
Continue [?/Y/n]? Y
 
Extracting files using tape drive file:/BACKUP2/slots/ on host
backupserver.local. Load tape B3_14 now
Continue [?/Y/n/s/t]? Y
. /etc/passwd
amrecover> quit
200 Good bye. 
 
Nothing spectacular? The trick is this: When AMANDA asks you
 
Load tape B3_14 now Continue [?/Y/n/s/t]? 
 
you have to run the following in a second terminal:
 
$ amtape woo slot 14
amtape: changed to slot 14 on file:/BACKUP2/slots/
 
This step is necessary to load the proper tape into your virtual changer. Let me express this in a more general way:
 
When amrecover prompts for the tape it needs to restore the files you requested, you have to "load" the tape it requests. The recommended way to do this is to use amtape. The options that make sense in this context are:
 
# amtape
Usage: amtape <conf> <command>
        Valid commands are:
                [...]
                slot <slot #>        load tape from slot <slot #>
                [...]
                label <label>        find and load labeled tape
                [...]
If you know which slot contains the requested tape (for example, if you have tape daily01 in slot 1, tape daily02 in slot 2, and so on) you may use the first option. If you just know the label of the tape you need, use the second option.
 
To continue the upper example:
 
amtape woo slot 14         # option 1 OR
amtape woo label B3_14 # option 2
 
amtape will return something like:
 
amtape: label B3_14 is now loaded. 
 
After this you can return to your amrecover-session and continue restoring your files.
 
Please be aware of the fact reported by JC Simonetti: " I have never never used the "settape" command of amrecover [with chg-disk] since there's some problems with it (tape not loaded correctly, or impossible to change from tape to tape when restoring data shared accross multiple tapes...) "
 
==Server side encryption==
 
This is based on a AMANDA-vtape-setup with the AMANDA-release 2.4.5. As this is still in the testing-stage, I have
coded the home-dir of the AMANDA-user into my scripts (/var/lib/amanda).
 
***This page needs updating***
 
===Additional packages needed===
* aespipe http://loop-aes.sourceforge.net/aespipe/aespipe-v2.3b.tar.bz2 and the bz2aespipe-wrapper that comes with it. It gets patched as described later.
* the wrapper-script /usr/local/libexec/amgtar, as listed down below,
* GNU-PG http://www.gnupg.org/(en)/download/index.html. This should be part of most current operating systems already.
 
===Setup===
 
* Configure and compile aespipe:
 
tar -xjf aespipe-v2.3b.tar.bz2
cd aespipe-v2.3b
./configure
make
make install
 
* Generate and store the gpg-key for the AMANDA-user:
 
# taken from the aespipe-README
head -c 2925 /dev/random | uuencode -m - | head -n 66 | tail -n 65 | \
gpg --symmetric -a > /var/lib/amanda/.gnupg/am_key.gpg
 
This will ask for a passphrase. Remember this passphrase as you will need it in the next step.
 
Store the passphrase inside the home-directory of the AMANDA-user and protect it with proper permissions:
 
echo my_secret_passphrase > ~amanda/.am_passphrase
chown amanda:disk ~amanda/.am_passphrase
chmod 700 ~amanda/.am_passphrase
 
We need this file because we don't want to have to enter the passphrase manually everytime we run amdump. We have to patch bz2aespipe to read the passphrase from a file. I have called that file ~amanda/.am_passphrase.
 
      Store the key and the passphrase in some other place as well, without these information you can't access any tapes that
      have been encrypted with it (this is exactly why we are doing all this, isn't it? ;) ).
 
* Create the wrapper for GNU-tar: /usr/local/libexec/amgtar
 
#!/bin/sh
#
# Original wrapper by Paul Bijnens
#
# crippled by Stefan G. Weichinger
# to enable gpg-encrypted dumps via aespipe
GTAR=/bin/tar
AM_AESPIPE=/usr/local/bin/amaespipe
AM_PASSPHRASE=/var/lib/amanda/.am_passphrase
LOG=/dev/null
LOG_ENABLED=1
if [ "$LOG_ENABLED" = "1" ]
then
LOG=/var/log/amanda/amgtar.debug
date >> $LOG
echo "$@" >> $LOG
fi
if [ "$3" = "/dev/null" ]
then
        echo "Estimate only" >> $LOG
        $GTAR "$@"
else
        echo "Real backup" >> $LOG
        $GTAR --use-compress-program="$AM_AESPIPE" "$@" 3< $AM_PASSPHRASE
fi
rc=$?
exit $rc
 
* Copy the wrapper-script bz2aespipe, which comes with the aespipe-tarball, to /usr/local/bin/amaespipe and edit it this way:
 
#! /bin/sh
# FILE FORMAT
# 10 bytes: constant string 'bz2aespipe'
# 10 bytes: itercountk digits
# 1 byte: '0' = AES128, '1' = AES192, '2' = AES256
# 1 byte: '0' = SHA256, '1' = SHA384, '2' = SHA512, '3' = RMD160
# 24 bytes: random seed string
# remaining bytes are bzip2 compressed and aespipe encrypted
# These definitions are only used when encrypting.
# Decryption will autodetect these definitions from archive.
ENCRYPTION=AES256
HASHFUNC=SHA256
ITERCOUNTK=100
WAITSECONDS=1
GPGKEY="/var/lib/amanda/.gnupg/am_key.gpg"
FDNUMBER=3
if test x$1 = x-d ; then
    # decrypt
    n=`head -c 10 - | tr -d -c 0-9a-zA-Z`
    if test x${n} != xbz2aespipe ; then
        echo "bz2aespipe: wrong magic - aborted" >/dev/tty
        exit 1
    fi
    itercountk=`head -c 10 - | tr -d -c 0-9`
    if test x${itercountk} = x ; then itercountk=0; fi
    n=`head -c 1 - | tr -d -c 0-9`
    encryption=AES128
    if test x${n} = x1 ; then encryption=AES192; fi
    if test x${n} = x2 ; then encryption=AES256; fi
    n=`head -c 1 - | tr -d -c 0-9`
    hashfunc=SHA256
    if test x${n} = x1 ; then hashfunc=SHA384; fi
    if test x${n} = x2 ; then hashfunc=SHA512; fi
    if test x${n} = x3 ; then hashfunc=RMD160; fi
    seedstr=`head -c 24 - | tr -d -c 0-9a-zA-Z+/`
    #aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${encryption} -H ${hashfunc} -S "${seedstr}" -C ${itercountk} -d | bzip2 -d -q
    aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${encryption} -H ${hashfunc} -S "${seedstr}" -C ${itercountk} -d
else
    # encrypt
    echo -n bz2aespipe
    echo ${ITERCOUNTK} | awk '{printf "%10u", $1;}'
    n=`echo ${ENCRYPTION} | tr -d -c 0-9`
    aesstr=0
    if test x${n} = x192 ; then aesstr=1; fi
    if test x${n} = x256 ; then aesstr=2; fi
    n=`echo ${HASHFUNC} | tr -d -c 0-9`
    hashstr=0
    if test x${n} = x384 ; then hashstr=1; fi
    if test x${n} = x512 ; then hashstr=2; fi
    if test x${n} = x160 ; then hashstr=3; fi
    seedstr=`head -c 18 /dev/urandom | uuencode -m - | head -n 2 | tail -n 1`
    echo -n ${aesstr}${hashstr}${seedstr}
    #bzip2 | aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -w ${WAITSECONDS}
    aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -w ${WAITSECONDS}
fi
exit 0
 
or apply this small patch (bz2aespipe.patch)
 
@@ -15,3 +15,5 @@
  ITERCOUNTK=100
-WAITSECONDS=10
+WAITSECONDS=1
+GPGKEY="/var/lib/amanda/.gnupg/am_key.gpg"
+FDNUMBER=3
@@ -36,3 +38,4 @@
      seedstr=`head -c 24 - | tr -d -c 0-9a-zA-Z+/`
-    aespipe -e ${encryption} -H ${hashfunc} -S "${seedstr}" -C ${itercountk} -d | bzip2 -d -q
+    #aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${encryption} -H ${hashfunc} -S "${seedstr}" -C ${itercountk} -d | bzip2 -d -q
+    aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${encryption} -H ${hashfunc} -S "${seedstr}" -C ${itercountk} -d
   else
  @@ -52,3 +55,4 @@
      echo -n ${aesstr}${hashstr}${seedstr}
-    bzip2 | aespipe -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -T -w ${WAITSECONDS}
+    #bzip2 | aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -w ${WAITSECONDS}
+    aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -w ${WAITSECONDS}
  fi
 
Patch changes:
* Decreased WAITSECONDS: No need to wait for 10 seconds to read the passphrase.
* Removed bzip2 from the pipes: AMANDA triggers GNU-zip-compression by itself, no need to do this twice (slows down things, blows up size).
* Added options -K and -p: This enables aespipe to use the generated gpg-key and tells it the number of the file-descriptor to read the passphrase from.
   
You may set various parameters inside bz2aespipe. You may also call bz2aespipe with various command-line-parameter to choose
the encryption-algorithm, hash-function etc. . For a start I have chosen to call bz2aespipe without command-line-options.
 
As described in How to use a wrapper you have to run configure again with the option --with-gnutar=/usr/local/libexec/amgtar, after that recompile and reinstall AMANDA. These steps are described in the mentioned document.
 
===Plans===


There are several wishes:
* I have tested custom compression using bzip2. Dumps works fine. Amrestore has a glitch on which
the image gets uncompressed correctly and written to a temp file but gets a broken-pipe error. I
am investigating the problem.


* Ability to switch encryption inside a dumptype. This HOWTO describes a method that enables/disables encryption for the whole installation. You might remove the amgtar-wrapper and simply link to plain GNU-tar again to disable encryption, but be aware that you also disable decryption with this step. You will hit problems when you then try to restore encrypted tapes.


* Ability to switch encryption-parameters inside a dumptype. Choice of algorithm, hash-functions etc. I don't know if it makes sense to put it into a dumptype or if it would be enough to configure it once inside amaespipe (I assume the latter).
==Tape hardware compression==


* All this leads to the need to code this into AMANDA itself: new dumptype-options and corresponding calls to GNU-tar etc. inside client-src/sendbackup-gnutar.c.
This section has been moved to a separate page.


==Compression==
See:  [[Hardware compression]].

Latest revision as of 23:48, 1 January 2011

Deprecated
See Getting Started with Amanda instead

Server configuration file - amanda.conf

Please see amanda.conf(5) for information on amanda.conf parameters.

Client configuration file - disklist

The disklist file on the Amanda server describes the list of disk list entries (DLEs, usually directories or filesystem devices) to be backed up by the Amanda configuration. A number of parameters can be specified for each DLE either directly in the disklist or in a dumptype definition in amanda.conf. For more information on the disklist, see amanda(8).

Exclude and include lists

POSIX filenames in configuration files

Amanda 2.5.1 supports the POSIX path naming standard for all filenames in dumptype configuration in amanda.conf(5) and disklist configuration file.

This feature makes configuration of Amanda clients running Windows and Amanda clients that use international character sets possible.

Any filename referenced, including include file names, may contain any character except a NULL ('\0') character. Actual path interpretation is system dependent. POSIX file name rules are also allowed when specifying configuration file tags.

For example we use spaces in a dumptype name in the examples below.

Specifying Special characters

Pathnames with special characters must be enclosed in double quotes ("). Some unprintable charactes are given special escape sequences.

Escape sequences
Sequence Character
\n Newline
\r Carrage Return
\t Tab
\f Formfeed
\" Double Quote


Example: Spaces in a dumptype name, exclude list, include list in amanda.conf 

 define dumptype "user-tar (a1)" {
    root-tar
    comment "User partition dumped with tar and a funny dumptype name"
    priority medium
    exclude "diskfile b*"
    include list "/tmp/include list with spaces in file name"
 }

Example: Disklist file entries with spaces in backup directory and dumptype names 

 localhost "/tmp/disk name with spaces" user-tar
 localhost "/tmp/disk name with\nnewline" "user-tar (a1)"

Example: Using file names with spaces in Amrecover sub-commands 

 setdisk "/tmp/disk name with\nnewline"
 add "diskfile *"
 delete "diskfile a1"

Device configuration

Tapetypes

Tapetype definitions are specified in amanda.conf configuration file. The tapetype definition provides AMANDA how much it is supposed to be able to store in a tape (length), how much space is wasted at the end of a dump image with the EOF mark (filemark) and how fast the tape unit is (speed).

The most important parameter is length, since AMANDA may decide to delay a backup if length is too small, but, if it is too large, AMANDA may end up leaving dumps in the holding disk or having to abort some dump.

Filemark is important if you have many disks, particularly with small incremental backups. The space wasted by so many filemarks may add up and considerably modify the available tape space.

The speed is currently unused.

AMANDA provides the amtapetype(8) utility to calculate the size of a tape, to generate a "tapetype" entry for your amanda.conf.

Specifying the appropriate tape device, but beware that it may take many hours to run (it fills the tape twice ...). Make sure you do not use hardware compression, even if you plan to use hardware compression in the future. amtapetype writes random data to tape, and random data will expand instead of compressing, therefore you'll get an estimate that's smaller than expected.

Some tapetype definitions are available here.

Changers

This part has been moved to a separate page.

See Changers.

RAIT

RAIT is an acronym for "Redundant Array of Inexpensive Tapes", where data is striped over several tape drives, with one drive writing an exclusive-or-sum of the others which can be used for error recovery. Any one of the data streams can be lost, and the data can still be recovered.

This means that a 3-drive RAIT set will write 2 "data" streams and one "parity" stream, and give you twice the capacity, twice the throughput, and the square of the failure rate (i.e. a 1/100 failure rate becomes 1/10,000, since a double-tape failure is required to lose data).

This means you can back up partitions as large as twice or four times your tape size with Amanda, with higher reliability and speed.

RAIT can also be used to mirror a backup to two drives, even when one of them is a virtual tape and the other is a real tape.

See Rait for a more complete description.

File driver/Disk backups

This driver uses files on disk as virtual tapes. Amanda can write to and read from virtual tapes, just like real tapes. A bunch of virtual tapes can even be manipulated with a changer.

Possible Uses

  • Test installations: You can easily explore the rich features of Amanda on systems without tape drives.
  • Inexpensive installations: Without buying a tape drive you can enjoy the benefits of Amanda and backup to a bunch of internal or external harddisks connected with Firewire or USB. You can create CD/DVD-sized backups which you can burn onto optical disks later.
  • Disk-based installations: You can use the file driver to backup onto a set of virtual tapes hosted on a bunch of hard-disks or a RAID-system. Combined with another Amanda configuration that dumps the virtual tapes to real tapes, you can provide reliable backup with faster tapeless recovery. This is called "disk-to-disk-to-tape" backup by some people today.

See File driver for a more complete description of virtual tapes, and their use.

Server-side and Client-side encryption

This section has been moved to a separate page.

See Encryption.

Custom Compression

  • compress client custom
    • Specify client_custom_compress "PROG"
    • PROG must not contain white space and it must accept -d for uncompress.
  • compress server custom
    • Specify server_custom_compress "PROG"
    • PROG must not contain white space and it must accept -d for uncompress.
  • sample dumptype:
 define dumptype custom-tar {
 global
 program "GNUTAR"
 comment "root partitions dumped with custom compression"
 compress server custom
 server_custom_compress "/usr/bin/my_gzip"
 priority low
}
  • I have tested custom compression using bzip2. Dumps works fine. Amrestore has a glitch on which

the image gets uncompressed correctly and written to a temp file but gets a broken-pipe error. I am investigating the problem.


Tape hardware compression

This section has been moved to a separate page.

See: Hardware compression.

Retrieved from ‘http://wiki.zmanda.com/index.php?title=Backup_server_(old)&oldid=7972’