Backup server (old)

From wiki.zmanda.com
Revision as of 00:36, 12 November 2005 by Paddy (talk | contribs) (→‎File driver)
Jump to navigation Jump to search

Simple Backup server configuration

  • Create the config directory (eg. /usr/local/etc/amanda/confname) and copy the example/ files into that directory. Edit these files to be correct for your site, consulting the amanda(8) man page if necessary. You will also need to create the directory for the log and database files for the configuration to use (eg /usr/local/var/amanda/confname), and the work directory on the holding disk. These directories need to agree with the parameters in amanda.conf. Don't forget to make all these directories writable by the dump user!

Make sure that you specify the no-rewind version of the tape device in your amanda.conf file. This is a frequently encountered problem for new sites.

Note that you might want to temporarily set the option "no-record" in all your dumptypes when first installing AMANDA if you'd like to run tests of AMANDA in parallel with your existing dump scheme. AMANDA will then run but will not interfere with your current dumpdates. However, you don't want to run with "no-record" under normal operations.

  • Put AMANDA into your crontab. Here's a sample:
/etc/crontab


0 16 * * 1-5 /usr/local/sbin/amcheck -m confname
45 0 * * 2-6 /usr/local/sbin/amdump confname

This is for SunOS 4.x, which has a per-user crontab; most other systems also require a userid on each cron line. See your cron(8) for details. With these cron lines, AMANDA will check that the correct tape is in the drive every weekday afternoon at 4pm (if it isn't, all the operators will get mail). At 12:45am that night the dumps will be run.

  • Put the AMANDA services into your /etc/services file. Add entries like:
/etc/services


amanda		10080/udp
amandaidx	10082/tcp
amidxtape	10083/tcp
You may choose a different port number if you like, but it must match that in the services file on the client hosts too.
If you are running NIS (aka YP), you have to enter the AMANDA service into your NIS services database. Consult your NIS documentation for details.
You may use the `patch-system' script, from client-src, in order to modify this file. Run it with a `-h' argument for usage.
  • If you are going to use the indexing capabilities of AMANDA, follow one of the following steps:
If your server uses inetd, then add these lines to your inetd.conf on the tape server host:
/etc/inetd.conf
amandaidx stream tcp nowait $USER $AMINDEXD_PATH amindexd
amidxtape stream tcp nowait $USER $AMIDXTAPED_PATH amidxtaped
where $AMINDEXD_PATH and $AMIDXTAPED_PATH are the complete paths to where the amindexd and amidxtaped executables (usually :libexec_dir/amindexd and libexec_dir/amidxtaped), and USER is the AMANDA user.
You may use the `patch-system' script, from client-src, in order to modify this file. Run it with a `-h' argument for usage.
If your tape server uses xinetd instead of inetd, then you have to add the following two files to your xinetd-configuration (usually :/etc/xinetd.d) and edit the paths:
/etc/xinetd.d/amandaidx
service amandaidx
{
       socket_type		= stream
       protocol		= tcp
       wait			= no
       user			= $USER
       group			= $GROUP
       groups 			= yes
       server			= $AMINDEXD_PATH/amindexd 
}
/etc/xinetd.d/amidxtape
service amidxtape
{
       socket_type		= stream
       protocol		= tcp
       wait			= no
       user			= $USER
       group			= $GROUP
       groups 			= yes
       server			= $AMIDXTAPED_PATH/amidxtaped
}
  • If your tape server uses Dan Bernstein's daemontools http://cr.yp.to/daemontools.html) instead of (x)inetd, you have to create amandaidx and amidxtape services by hand.
    • Create service directories:
mkdir -p $prefix/etc/amanda/supervise/amandaidx
mkdir -p $prefix/etc/amanda/supervise/amidxtape
    • Create service startup files and make them executable:
/etc/amanda/supervise/amandaidx/run
#!/bin/sh
exec /usr/local/bin/setuidgid amanda \
/usr/local/bin/tcpserver -DHRl0 0 10082 \
/usr/local/libexec/amindexd >/dev/null 2>/dev/null
/etc/amanda/supervise/amidxtape/run
#!/bin/sh
exec /usr/local/bin/setuidgid amanda \
/usr/local/bin/tcpserver -DHRl0 0 10083 \
/usr/local/libexec/amidxtaped >/dev/null 2>/dev/null
    • Link service directories into your svscan directory:
cd /service
ln -s $prefix/etc/amanda/supervise/amandaidx .
ln -s $prefix/etc/amanda/supervise/amidxtape .
  • If the tape server host is itself going to be backed up (as is usually the case), you must also follow the client-side install instructions below on the server host, INCLUDING setting up the file .amandahosts so that the server host lets itself in. This is a frequently encountered problem for new sites.

amanda.conf

disklist

tapelist

Exclude lists

There are three ways of excluding data in an AMANDA backup:

  • Exclude an individual item explicitly in the dumptype
  • Utilize an "Exclude List"
  • Do not include the data in the disklist

The only way to exclude files or directories with AMANDA is to use GNU-tar as the dump program (others?). The file system dump programs provided with unix systems (e.g. dump, ufsdump) get data at a raw drive level and generally do not allow exclusion of specific files or directories.

The GNU version of tar, (GNU-tar or gtar), reads its data at a file system, (or higher), level and does include the option to exclude specific files and/or directories. It should be mentioned here that tar will change the access times on files. Tar has the ability to preserve the access times however, doing so effectively disables incremental backups since resetting the access time alters the inode change time, which in turn causes the file to look like it needs to be archived again.

For the purpose of this document an AMANDA backup configuration named "exclude-test" will be used. The machine that contains the tape drive which receives data to be archived will be referred to as "SERVER". The machine that data is being archived from will be referred to as "CLIENT". These two systems are usually different machines but are not required to be, and may be the same machine. Parts of this setup are on the server and some are on the client. 

When AMANDA attempts to exclude a file or directory it does so relative to the area being archived. For example if /var 
is in your disklist   and you want to exclude /var/log/somefile, then your exclude file would contain ./log/somefile. 
You may use one exclude file in multiple dumptypes without any restriction.

Before We Begin

  • The first step that should be taken is to verify that backups are currently working. Connect to SERVER and run amcheck as your AMANDA user, to verify that there are no errors in the current setup.
$ amcheck -cl CLIENT

Output should look something like below for success: 

       AMANDA Tape Server Host Check
       -----------------------------

       /path/to/holding-disk: 4771300 KB disk space available, that's plenty.  
       AMANDA Backup Client Hosts Check
       -------------------------------- 
       Client check: 1 host checked in 0.084 seconds, 0 problems found.
  • Next make sure that GNU-tar is the dump program currently in use. The easiest way to tell if your dumptype is using gnutar is to run the following:
$ amadmin exclude-test disklist CLIENT

Among all the output is the "program" value currently in use. This value is also specified with the "program" option in the dumptype. If the dumptype has the line "program GNUTAR" your setup should be ready to exclude data.

If GNU-tar is not in use add the line "program GNUTAR" to the dumptype, and then run amcheck again to verify that backups should work. The capitalization of GNUTAR is required.

The dumptype should look something like: 

define dumptype exclude-test {
      comment "test dumptype for documentation"
      priority high
      program "GNUTAR"
}

Choosing an exclude mechanism

If the need is to exclude only one file or directory then the easiest way to accomplish this is to exclude an individual item explicitly in the dumptype. If the need is to exclude multiple files or directories then use an Exclude List.

Exclude Mechanisms

Exclude an individual item explicitly in the dumptype

The easiest way to exclude a file or directory is to specify it with the "exclude" option in the dumptype. This option accepts an argument of the file or directory to be excluded. AMANDA allows only one exclude option in any dumptype at a time. 

UPDATE: Recent AMANDA-releases bring the option "exclude append" which enables the administrator to define more than 
one exclusion-pattern within one dumptype without using a exclude-list. Please look at the amanda.conf manpage for details.

Any path specified to be excluded must be encapsulated with quotes. Continuing with our example from above /var/log/somefile and using the same dumptype as above, the dumptype would now look like: 

define dumptype exclude-test {
     comment "test dumptype for documentation"
     priority high
     program "GNUTAR"
     exclude "./log/somefile"
}

Next run amcheck again to verify that there are no problems with the revised AMANDA configuration. If the data is not being excluded as expected please see the Troubleshooting section below. This completes the setup of excluding an individual item in the dumptype.

Utilize an Exclude List

An exclude list is a file that resides on the CLIENT machine and contains paths to be excluded, one per line. This file can be in any location on the CLIENT so long as the same path is specified in the dumptype. Some find /usr/local/etc/amanda an appropriate location, but it is up to you. I personally like to have a subdirectory for exclude files but it is up to you where you place this file.

The exclude file may also be placed in the area being archived. This is an easy way to have a different exclusion file for each disklist entry without needing separate dumptype definitions. To use this technique, enter a path relative to the area being archived as the exclude file below instead of an absolute path.

Connect to CLIENT and create the exclude directory as root. For example: 

$ mkdir -p /usr/local/etc/amanda/exclude
$ cd /usr/local/etc/amanda/exclude

Next create the exclude list for AMANDA to use. You can name the exclude file anything you wish it to be. Create a file, and in this file place all paths to files and directories that are to be excluded. Keeping with the /var example, assume that /var/log/XFree86.0.log, and /var/log/maillog need to be excluded. Remember that all paths are relative. The exclude list would look like:

./log/XFree86.0.log ./log/maillog

Make sure that permissions are restricted on this file. Run the following as root, where exclude-filename is the name of the file you just created. For example: 

$ chmod 644 /usr/local/etc/amanda/exclude/exclude-filename

This concludes the necessary configuration on the client.

Connect to SERVER and cd to the exclude-test AMANDA configuration directory. Edit the AMANDA configuration file e.g. amanda.conf. Add an entry similar to the following line, to the dumptype for the client in question, where the exclude-filename is the file that was created on CLIENT in the step above including the quotes. For example: 

exclude list "/usr/local/etc/amanda/exclude/exclude-filename"

The new dumptype should look something like:

define dumptype exclude-test{ 

  comment "test dumptype for documentation"
  priority high
  program "GNUTAR"
  exclude list "/usr/local/etc/amanda/exclude/exclude-filename"

}

Save the file. Run amcheck again to verify that there are no problems with the revised AMANDA configuration. If amcheck succeeds then run amdump to verify the data is being excluded correctly. If the data is not being excluded as expected please see the Troubleshooting section below. This completes the setup of an exclude list.

Do not include the data in the disklist

AMANDA uses disklist entries to define which directories or partitions should be archived. This allows us to exclude data by just not placing the data in question in the disklist. Assume that there is a disk mounted on /example. The directory /example has five subdirectories "a", "b", "c", "d", and "e". The directories "a", "b", and "c" need to be archived, while "d" and "e" should not. This can be accomplished by not specifying "d" and "e" in the disklist. Using the same dumptype and host in the above examples the disklist would contain:

CLIENT /examples/a exclude-test CLIENT /examples/b exclude-test CLIENT /examples/c exclude-test

Run amcheck to verify that AMANDA is working correctly. If the data is not being excluded as expected please see the Troubleshooting section below. This completes the setup of using a disklist to exclude data. Expression

Quiz: what is the difference between the following entries in an exclude list? 

./foo
./foo/
./foo/*

case 1 : directory ./foo won't be in the backup image (that's what you want) case 2 : matches nothing (don't use it) case 3 : directory ./foo will be in the backup image but nothing below it.

Wildcard Expansion

AMANDA has the ability to use wildcard expansion while excluding data as implemented by tar(1). The only places that wildcard expansion is allowed is in the "exclude" option in the dumptype, or in the exclude list. Some simple examples:

./*.log
Exclude any file or directory that ends in ".log" e.g. ppp.log, XFree86.0.log
*/*log*
Exclude any file or directory with the string "log" e.g. logfile, maillog, syslog, ppp.log, XFree86.0.log
./*cron*.gz
Exclude any file or directory that starts with string "cron" and ends in ".gz" e.g. cron.1.gz, cron.2.gz, log/cron.1.gz
./log.?
The question mark can be used to specify a single character. e.g. log.1, log.2, etc


Troubleshooting

If you find that you are having trouble getting the exclude patterns to match correctly, check out this really cool script written by John R. Jackson.

ftp://gandalf.cc.purdue.edu/pub/amanda/gtartest-exclude

This script allows you to test your patterns before placing them in an exclude list or in the dumptype. Instructions on how to run the script are included in the script.

Broken gnutar?

There are versions of GNU-tar that do not correctly exclude data. Version 1.12 (plus the AMANDA patches from http://www.amanda.org) are known to work correctly, as does version 1.13.19 (and later). Anything else is questionable. 

UPDATE:Using GNU-tar 1.13.25 is recommended.

Connect to CLIENT and run a ps command and look to see exactly how the tar command is running. Look in the output for the --exclude or --exclude-from options in the running tar process. For example: 

$ ps ax | grep tar 

?        R      0:37 /bin/tar --create --directory /var
--listed-incremental /var/lib/amanda/gnutar-lists/CLIENTvar_0.new
--sparse --one-file-system --ignore-failed-read --totals --file
/dev/null --exclude-from=/usr/local/etc/amanda/exclude-test/exclude.var
.

In the above output notice the string "--exclude-from=". The string following the "=" is the exclude file currently in use. If the string was "--exclude" then the string following the "=" is the file or directory that is currently set to be excluded.

Device configuration

Tapetypes

Tapetype definitions are specified in amanda.conf configuration file. The tapetype definition provides AMANDA how much it is supposed to be able to store in a tape (length), how much space is wasted at the end of a dump image with the EOF mark (filemark) and how fast the tape unit is (speed).

The most inportant parameter is length, since AMANDA may decide to delay a backup if length is too small, but, if it is too large, AMANDA may end up leaving dumps in the holding disk or having to abort some dump.

Filemark is important if you have many disks, particularly with small incremental backups. The space wasted by so many filemarks may add up and considerably modify the available tape space.

The speed is currently unused.

AMANDA provides the amtapetype utility to calculate the size of a tape, to generate a "tapetype" entry for your amanda.conf.

Specifying the appropriate tape device, but beware that it may take many hours to run (it fills the tape twice ...). Make sure you do not use hardware compression, even if you plan to use hardware compression in the future. amtapetype writes random data to tape, and random data will expand instead of compressing, therefore you'll get an estimate that's smaller than expected.

Some tapetype definitions are available here.

Changers

RAIT

Currently it is only integrated with the chg-manual changer script

RAIT is an acronym for "Redundant Array of Inexpensive Tapes", where data is striped over several tape drives, with one drive writing an exclusive-or-sum of the others which can be used for error recovery. Any one of the data streams can be lost, and the data can still be recovered.

This means that a 3-drive RAIT set will write 2 "data" streams and one "parity" stream, and give you twice the capacity, twice the throughput, and the square of the failure rate (i.e. a 1/100 failure rate becomes 1/10,000, since a double-tape failure is required to lose data).

Similarly, a 5-drive RAIT set will give you 4 times the capacity, 4 times the throughput (with sufficient bus bandwidth), and the square of the failure rate.

This means you can back up partitions as large as four times your tape size with AMANDA, with higher reliability and speed.

Using a RAIT

If you have several tape devices on your system [currently either 3 or 5 drive sets are supported] you tell AMANDA to use them as a RAIT by listing them as a single tape device using /bin/csh curly-brace-and-comma notation, as in:

tapedev = "rait:/dev/rmt/tps0d{4,5,6}n"


which means that /dev/rmt/tps0d4n, /dev/rmt/tps0d5n, and /dev/rmt/tps0d6n are to be treated as a RAIT set. You can now mount three tapes, and label them with amlabel, etc.

Also, you want to create a new tape-type entry, which lists an n-drive RAIT set, for this RAIT-set. So if you were using an entry like: 

define tapetype EXB-8500 {
       comment "Exabyte EXB-8500 drive on decent machine"
       length 4200 mbytes
       filemark 48 kbytes
       speed 474 kbytes
}


You would want to make a new one like: 

define tapetype EXB-8500x3 {
       comment "Exabyte EXB-8500 3 drive stripe on decent machine"
       length 8400 mbytes
       filemark 200 kbytes
       speed 948 kbytes
}

and change your tapetype entry to: tapetype EXB-8500x3 to tell AMANDA about the multiple drive set.

Disaster Recovery

To assist in disaster recovery (as well as changer scripts) the AMANDA package now also includes amdd, which is a simple dd(1) replacement which supports (only) the "if=xxx", "of=xxx", "bs=nnn[kMb]" "skip=nnn" and "count=nnn" options, but which can read and write RAIT tapesets.

Using amdd and your usual AMANDA unpack instructions will suffice for disaster recovery from RAIT tape-sets.


File driver/Disk backups

Stefan G. Weichinger, November - December, 2003 ; minor updates in April, 2005.

Introduction

Since release 2.4.3 AMANDA supports the usage of a output driver called "file". See man amanda, section OUTPUT DRIVERS, for more information on its implementation. As the name suggests, this driver uses files as virtual (or file) tapes. Once created and labeled, these file tapes can be selected and changed with the standard tape-changer-interface of the AMANDA server.

Possible Uses

  • Test installations: You can easily explore the rich features of AMANDA on systems without tape drives.
  • Inexpensive installations: Without buying a tape drive you can enjoy the benefits of AMANDA and backup to a bunch of harddisks. You can create CD/DVD-sized backups which you can burn onto optical disks later.
  • disk-based installations: You can use the file-driver to backup onto a set of file tapes hosted on a bunch of hard-disks or a RAID-system. Combined with another AMANDA-configuration that dumps the file tapes to real tapes, you can provide reliable backup with faster tapeless recovery. This is called "disk-to-disk-to-tape"-backup by some people today.

Setup

This guide assumes you have setup the basic AMANDA-services as described in Amanda Installation Notes

The configuration in this HOWTO is called "daily". The file tapes are also called vtapes in this document, which stands for "virtual tapes".

Please be sure to understand the differences between holding disks and file tapes. The two serve different purposes; holding disks allow for parallelism of multiple DLE's being backed up while file tapes are a replacement for physical tapes.

Before beginning you will need to decide on (a) dedicated part(s) of your hard disk(s) for your file tape storage. While this space could be spread among several file systems and hard disks, I recommend to dedicate at least a specific partition, better a specific physical harddisk to the task of keeping your vtapes. The use of a dedicated disk will speed things up definitely.

The disk space you dedicate for your vtapes should NOT be backed up by AMANDA. Also, for performance reasons there should be NO holding disks on the same partition as the vtapes, preferably not even on the same physical drive.

If you only have one harddisk, it will work out, too, but you will suffer low performance due to massive head-moving in your harddisk, resulting from copying data between the filesystems.

  • Prepare the filesystem(s) used for the tapes. Decide on where to put your files, create the appropriate partition(s) and

filesystem(s) and mount them. In our example we have the dedicated partition hdc1, mounted on /amandatapes for vtape storage. 

$ mount
[...]
/dev/hdc1 on /amandatapes type reiserfs (rw)
[...]

Make sure there is space left. Determine the amount of space you will use. 

$ df -h /amandatapes
Filesystem      Size  Used  Avail  Use%   Mounted on
/dev/hdc1        20G    0G    20G    0%   /amandatapes

In our example we have 20GB diskspace left on /amandatapes.

  • Determine length and number of tapes. After deciding on the number of vtapes you want to create, evenly allocate the

available space among them.

Look at the following rule of thumb: As many filesystems exhibit dramatically reduced performance when they are nearly full I have chosen to allocate only 90% of the available space. So we have: 

     (Available Space * 0.9) >= tapelength * tapecycle

This is a very conservative approach to make sure you don´t suffer any performance drop due to a nearly-full-filesystem. As it is uncommon for AMANDA to fill, or almost fill an entire tape you may also wish to use more space than that. So you could determine possible combinations of tapelength/tapecycle with the more general formula: 

     Available Space >= tapelength * tapecycle

In our example we take the conservative approach:

  • 20 GB * 0.9 = 18 GB to use

and so we could create the following combinations:

  • 18 GB = 18 GB * 1
  • 18 GB = 9 GB * 2
  • 18 GB = 6 GB * 3
  • 18 GB = 3 GB * 6
  • 18 GB = ......... you get the picture.

Using only one tape is generally considered a bad idea when it comes to backup, so we should use at least 3 tapes (for testing purposes), better 6 or more tapes.

  • 18 GB = 3 GB * 6

so we get the value 3 GB for the tapelength if we want to use 6 tapes.

  • Create a tapetype definition. Add a new tapetype definition similar to the following to your amanda.conf. I named my definition "HARD-DISK". Choose whatever name you consider appropriate.
define tapetype HARD-DISK {
      comment "Dump onto hard disk"
      length 3072 mbytes 	# specified in mbytes to get the exact size of 3GB
}

You don´t have to specify the parameter speed (as it is commonly listed in tapetype definitions and reported by the program amtapetype). AMANDA does not use this parameter right now.

There is also an optional parameter filemark, which indicates the amount of space "wasted" after each tape-listitem. Leave it blank and AMANDA uses the default of 1KB.

  • Think about tapechangers.

As you will use a set of vtapes, you have to also use a kind of vtape-changer. There are several tape-changer-scripts included in the AMANDA. Read more about tape-changer-scripts in AMANDA Tape Changer Support. Right now there are two scripts that can be used with vtapes. These scripts take different approaches to the handling of tapes.

  • The script chg-multi handles many drives with a tape in each drive.
  • The script chg-disk handles a library with one drive and multiple tapes.

So with vtapes you could look at it this way: chg-multi simulates multiple tape drives with one tape in each drive. chg-disk simulates one tape-library with multiple tapes in. As chg-multi exists for a much longer time than chg-disk, it is still used in many AMANDA-vtape-installations.

Contrary to chg-multi, which is a generic changer-script that must be somewhat adjusted to the use of the file-driver, chg-disk offers exactly the behavior needed for handling vtapes. IMHO the approach is much more logical, so I recommend to use chg-disk in new AMANDA-vtape-installations. 

     To use chg-disk you need to have at least amanda-2.4.4p1-20031202.

Choose the one that fits your way of vtape-handling and -maintenance. In this HOWTO I only cover the use of chg-disk. Usage of chg-multi is pretty similar and will maybe covered in a later version of this document.

  • Set up your tape-config. In the general section of amanda.conf, you have to set the parameters tapecycle , tapetype , tpchanger , changerfile , tapedev , rawtapedev and changerdev.
$ vi /usr/local/etc/amanda/daily/amanda.conf
...

tapecycle 6
tapetype HARD-DISK
tpchanger "chg-disk"
changerfile "/usr/local/etc/amanda/daily/changer"
tapedev  "file:/amandatapes/daily"

This reflects the use of your defined tapetype. The parameter tapecycle tells AMANDA how much tapes can be used, Set this value according to the number of tapes you want to use. The parameter tapetype , points to the tapetype definition you have created before. The parameter tpchanger tells AMANDA to use the generic tape-changer-script to handle the vtapes. You can think of it as a virtual tape-changer-device.

The parameter changerfile is used to give chg-disk the "prefix" for the "%s-changer, %s-clean, %s-slot" files it needs. Use something like "changer" in your config-dir. Please note that this file does NOT have to exist, but it won't hurt anyway. The parameter tapedev tells the chg-disk-script where the root-dir for your vtapes is.

In our example the vtape-files go to /amandatapes. To separate multiple configurations, we decided to use subdirectories according to the configuration name "daily". 

     The parameter changerdev is NOT needed with chg-disk as it is not parsed by chg-disk.
  • Create the virtual tapes.
     Gene Heskett has committed a shell-script which creates and labels the vtapes in one step. Stefan G. Weichinger will 
     generalize this script and contribute it, this script will just read your settings in amanda.conf and create the 
     appropriate vtape-directories.

Now you have to create the tape-directories. chg-disk needs a directory structure like: 

slot_root_dir -|

|- info |- data -> slot1/ |- slot1/ |- slot2/ |- ... |- slotn/ 

where 'slot_root_dir' is the tapedev 'file:xxx' parameter and 'n' is the tapecycle parameter.

So in our example we do: 

$ mkdir /amandatapes/daily 
for the 'slot_root_dir' and
   $ mkdir /amandatapes/daily/slot1
   $ mkdir /amandatapes/daily/slot2
     ....

for the virtual slots that will later contain the vtapes. If you have many vtapes to create and their names follow a pattern you may be able to do them all with a single loop such as: 

$ for n in 1 2 3 4 5 6 7 8 9 10 11 12
> do
>    mkdir /amandatapes/daily/slot${n}
> done

Create the info-file: 

$ touch /amandatapes/daily/info

and link the first slot to the data-file (to "load" the vtape into the first slot): 

$ ln -s /amandatapes/daily/slot1 /amandatapes/daily/data

Make sure the AMANDA-user has write-permissions on these directories: 

$ chown -R amanda_user /amandatapes
$ chgrp -R amanda_group /amandatapes
$ chmod -R 750 /amandatapes
  • Label the virtual tapes. As the virtual tapes are handled just like physical tapes by the AMANDA-Server they have to be

labeled before use. 

$ amlabel daily daily1 slot 1
 ....
$ amlabel daily daily2 slot 2
 ....

If you have many vtapes to label and their names follow a pattern you may be able to do them all with a single loop such as: 

$ for n in 1 2 3 4 5 6 7 8 9 10 11 12
> do
>    amlabel daily daily${n} slot ${n}
> done

Label all your created tapes according to the "labelstr"-parameter in your amanda.conf. Consult the amlabel-man-page for details.

  • Test your setup with amcheck. Run amcheck daily (or, more general, amcheck <config>) and look for anything AMANDA complains about.

A proper output looks like: 

$ amcheck daily
Amanda Tape Server Host Check
--
Holding disk /amhold: 6924940 KB disk space available,
that's plenty
amcheck-server: slot 2: date 20031115 label daily02
(exact label match)
NOTE: skipping tape-writable test
Tape daily02 label ok
Server check took 0.377 seconds

Recheck your files if errors occur.

Recovery

Recovering files from vtapes is very similar to recovering files from a "real" tapechanger. Make sure you read the chapter Restore. I will simply paste a amrecover-session here (provided by JC Simonetti, author of chg-disk): 

# /usr/local/amanda/sbin/amrecover woo
AMRECOVER Version 2.4.4p3. Contacting server on backupserver.local ... 
220 backupserver AMANDA index server (2.4.4p3) ready.
200 Access OK
Setting restore date to today (2004-10-08)
200 Working date set to 2004-10-08.
Scanning /BACKUP2/holding...
Scanning /BACKUP/holding...
200 Config set to woo.
200 Dump host set to backupserver.local.
Trying disk /tmp ...
$CWD '/tmp/RECOVER' is on disk '/tmp' mounted at '/tmp'.
200 Disk set to /tmp.
Invalid directory - /tmp/RECOVER
amrecover> sethost backupserver.local
200 Dump host set to backupserver.local.
amrecover> setdisk /
200 Disk set to /.
amrecover> cd /etc
/etc
amrecover> add passwd
Added /etc/passwd
amrecover> list
TAPE B3_14 LEVEL 0 DATE 2004-09-26
       /etc/passwd
amrecover> extract

Extracting files using tape drive file:/BACKUP2/slots/ on host
backupserver.local. The following tapes are needed: B3_14

Restoring files into directory /tmp/RECOVER
Continue [?/Y/n]? Y

Extracting files using tape drive file:/BACKUP2/slots/ on host
backupserver.local. Load tape B3_14 now
Continue [?/Y/n/s/t]? Y
. /etc/passwd
amrecover> quit
200 Good bye.

Nothing spectacular? The trick is this: When AMANDA asks you 

Load tape B3_14 now Continue [?/Y/n/s/t]?

you have to run the following in a second terminal: 

$ amtape woo slot 14
amtape: changed to slot 14 on file:/BACKUP2/slots/

This step is necessary to load the proper tape into your virtual changer. Let me express this in a more general way:

When amrecover prompts for the tape it needs to restore the files you requested, you have to "load" the tape it requests. The recommended way to do this is to use amtape. The options that make sense in this context are: 

# amtape
Usage: amtape <conf> <command>
       Valid commands are:
               [...]
               slot <slot #>        load tape from slot <slot #>
               [...]
               label <label>        find and load labeled tape
               [...]

If you know which slot contains the requested tape (for example, if you have tape daily01 in slot 1, tape daily02 in slot 2, and so on) you may use the first option. If you just know the label of the tape you need, use the second option.

To continue the upper example: 

amtape woo slot 14 	        # option 1 OR
amtape woo label B3_14 	# option 2

amtape will return something like: 

amtape: label B3_14 is now loaded.

After this you can return to your amrecover-session and continue restoring your files.

Please be aware of the fact reported by JC Simonetti: " I have never never used the "settape" command of amrecover [with chg-disk] since there's some problems with it (tape not loaded correctly, or impossible to change from tape to tape when restoring data shared accross multiple tapes...) "

Server side encryption

This is based on a AMANDA-vtape-setup with the AMANDA-release 2.4.5. As this is still in the testing-stage, I have 
coded the home-dir of the AMANDA-user into my scripts (/var/lib/amanda). 

***This page needs updating***

Additional packages needed

Setup

  • Configure and compile aespipe:
tar -xjf aespipe-v2.3b.tar.bz2
cd aespipe-v2.3b
./configure
make
make install
  • Generate and store the gpg-key for the AMANDA-user:
# taken from the aespipe-README
head -c 2925 /dev/random | uuencode -m - | head -n 66 | tail -n 65 | \
gpg --symmetric -a > /var/lib/amanda/.gnupg/am_key.gpg

This will ask for a passphrase. Remember this passphrase as you will need it in the next step.

Store the passphrase inside the home-directory of the AMANDA-user and protect it with proper permissions: 

echo my_secret_passphrase > ~amanda/.am_passphrase
chown amanda:disk ~amanda/.am_passphrase
chmod 700 ~amanda/.am_passphrase

We need this file because we don't want to have to enter the passphrase manually everytime we run amdump. We have to patch bz2aespipe to read the passphrase from a file. I have called that file ~amanda/.am_passphrase. 

     Store the key and the passphrase in some other place as well, without these information you can't access any tapes that 
     have been encrypted with it (this is exactly why we are doing all this, isn't it? ;) ).
  • Create the wrapper for GNU-tar: /usr/local/libexec/amgtar
#!/bin/sh
#
# Original wrapper by Paul Bijnens
#
# crippled by Stefan G. Weichinger
# to enable gpg-encrypted dumps via aespipe

GTAR=/bin/tar
AM_AESPIPE=/usr/local/bin/amaespipe
AM_PASSPHRASE=/var/lib/amanda/.am_passphrase
LOG=/dev/null
LOG_ENABLED=1

if [ "$LOG_ENABLED" = "1" ]
then 
LOG=/var/log/amanda/amgtar.debug
date >> $LOG
echo "$@" >> $LOG
fi

if [ "$3" = "/dev/null" ]
then

echo "Estimate only" >> $LOG $GTAR "$@" 

else

echo "Real backup" >> $LOG $GTAR --use-compress-program="$AM_AESPIPE" "$@" 3< $AM_PASSPHRASE 

fi

rc=$?
exit $rc
  • Copy the wrapper-script bz2aespipe, which comes with the aespipe-tarball, to /usr/local/bin/amaespipe and edit it this way:
#! /bin/sh

# FILE FORMAT
# 10 bytes: constant string 'bz2aespipe'
# 10 bytes: itercountk digits
# 1 byte: '0' = AES128, '1' = AES192, '2' = AES256
# 1 byte: '0' = SHA256, '1' = SHA384, '2' = SHA512, '3' = RMD160
# 24 bytes: random seed string
# remaining bytes are bzip2 compressed and aespipe encrypted

# These definitions are only used when encrypting.
# Decryption will autodetect these definitions from archive.
ENCRYPTION=AES256
HASHFUNC=SHA256
ITERCOUNTK=100
WAITSECONDS=1
GPGKEY="/var/lib/amanda/.gnupg/am_key.gpg"
FDNUMBER=3

if test x$1 = x-d ; then
   # decrypt
   n=`head -c 10 - | tr -d -c 0-9a-zA-Z`
   if test x${n} != xbz2aespipe ; then
       echo "bz2aespipe: wrong magic - aborted" >/dev/tty
       exit 1
   fi
   itercountk=`head -c 10 - | tr -d -c 0-9`
   if test x${itercountk} = x ; then itercountk=0; fi
   n=`head -c 1 - | tr -d -c 0-9`
   encryption=AES128
   if test x${n} = x1 ; then encryption=AES192; fi
   if test x${n} = x2 ; then encryption=AES256; fi
   n=`head -c 1 - | tr -d -c 0-9`
   hashfunc=SHA256
   if test x${n} = x1 ; then hashfunc=SHA384; fi
   if test x${n} = x2 ; then hashfunc=SHA512; fi
   if test x${n} = x3 ; then hashfunc=RMD160; fi
   seedstr=`head -c 24 - | tr -d -c 0-9a-zA-Z+/`
   #aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${encryption} -H ${hashfunc} -S "${seedstr}" -C ${itercountk} -d | bzip2 -d -q
   aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${encryption} -H ${hashfunc} -S "${seedstr}" -C ${itercountk} -d 
else
   # encrypt
   echo -n bz2aespipe
   echo ${ITERCOUNTK} | awk '{printf "%10u", $1;}'
   n=`echo ${ENCRYPTION} | tr -d -c 0-9`
   aesstr=0
   if test x${n} = x192 ; then aesstr=1; fi
   if test x${n} = x256 ; then aesstr=2; fi
   n=`echo ${HASHFUNC} | tr -d -c 0-9`
   hashstr=0
   if test x${n} = x384 ; then hashstr=1; fi
   if test x${n} = x512 ; then hashstr=2; fi
   if test x${n} = x160 ; then hashstr=3; fi
   seedstr=`head -c 18 /dev/urandom | uuencode -m - | head -n 2 | tail -n 1`
   echo -n ${aesstr}${hashstr}${seedstr}
   #bzip2 | aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -w ${WAITSECONDS}
   aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -w ${WAITSECONDS}
fi
exit 0

or apply this small patch (bz2aespipe.patch) 

@@ -15,3 +15,5 @@
 ITERCOUNTK=100
-WAITSECONDS=10
+WAITSECONDS=1
+GPGKEY="/var/lib/amanda/.gnupg/am_key.gpg"
+FDNUMBER=3

@@ -36,3 +38,4 @@
     seedstr=`head -c 24 - | tr -d -c 0-9a-zA-Z+/`
-    aespipe -e ${encryption} -H ${hashfunc} -S "${seedstr}" -C ${itercountk} -d | bzip2 -d -q
+    #aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${encryption} -H ${hashfunc} -S "${seedstr}" -C ${itercountk} -d | bzip2 -d -q
+    aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${encryption} -H ${hashfunc} -S "${seedstr}" -C ${itercountk} -d
 else
@@ -52,3 +55,4 @@
     echo -n ${aesstr}${hashstr}${seedstr}
-    bzip2 | aespipe -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -T -w ${WAITSECONDS}
+    #bzip2 | aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -w ${WAITSECONDS}
+    aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -w ${WAITSECONDS}
 fi

Patch changes:

  • Decreased WAITSECONDS: No need to wait for 10 seconds to read the passphrase.
  • Removed bzip2 from the pipes: AMANDA triggers GNU-zip-compression by itself, no need to do this twice (slows down things, blows up size).
  • Added options -K and -p: This enables aespipe to use the generated gpg-key and tells it the number of the file-descriptor to read the passphrase from.

You may set various parameters inside bz2aespipe. You may also call bz2aespipe with various command-line-parameter to choose the encryption-algorithm, hash-function etc. . For a start I have chosen to call bz2aespipe without command-line-options.

As described in How to use a wrapper you have to run configure again with the option --with-gnutar=/usr/local/libexec/amgtar, after that recompile and reinstall AMANDA. These steps are described in the mentioned document.

Plans

There are several wishes:

  • Ability to switch encryption inside a dumptype. This HOWTO describes a method that enables/disables encryption for the whole installation. You might remove the amgtar-wrapper and simply link to plain GNU-tar again to disable encryption, but be aware that you also disable decryption with this step. You will hit problems when you then try to restore encrypted tapes.
  • Ability to switch encryption-parameters inside a dumptype. Choice of algorithm, hash-functions etc. I don't know if it makes sense to put it into a dumptype or if it would be enough to configure it once inside amaespipe (I assume the latter).
  • All this leads to the need to code this into AMANDA itself: new dumptype-options and corresponding calls to GNU-tar etc. inside client-src/sendbackup-gnutar.c.

Compression

Retrieved from ‘http://wiki.zmanda.com/index.php?title=Backup_server_(old)&oldid=1486’