View source for How To:Split DLEs With Exclude Lists

{{How To Header}}
This article describes a way to split your [[DLE]]s to produce smaller individual dumps.  This is often useful when you only have a few DLEs, and one of them is much larger than the rest: without this technique, the tape usage spikes when Amanda runs a level 0 for that DLE.  For example, many small businesses have one very large NFS-mounted <tt>/home</tt>.  Using this technique, <tt>/home</tt> can be split by subdirectory, e.g., <tt>/home/[a-o]*</tt> and <tt>/home/[p-z]*</tt>.  A more extended example is provided below.

With Amanda < 2.5.0 the need arises to virtually split partitions into parts so that each part remains smaller than a tape. This will lead to smaller single dumps so that Amanda can continue the backup of other dumps on the next tape, if need be.

If you have a lot of large dumps and Amanda 2.5.0 or later, then [[How To:Split Dumps Across Tapes |tape spanning]] may be more useful than this technique.

= Example =

Let's assume that you have two large partitions, holding several directories and subdirectories like the following:

 /home/
 /home/andrew
 /home/anne
 /home/brian
 [...]
 /home/wanda
 /home/zeus
 
 /share/
 /share/financial
 /share/financial/company-a
 /share/financial/company-b
 /share/reports

You created entries in your [[disklist]] to back them up:

 largehost /home        my-dumptype
 largehost /share       my-dumptype

That worked fine, but now on each of the partitions, there's too much data to fit onto a single tape. You will have to split up the partitions to back them up.

= Splitting it up, the easy part =

Considering the /home partition, it becomes obvious that you could split it alphabetically. You would create disklist entries to back up each subdirectory by using wildcards. You also want to catch anything under /home that doesn't fit into the wildcard specifications, so there:

 largehost /home/./a-c /home {
   my-dumptype
   include "./[a-c]*"
 }       1
 largehost /home/./d-f /home {
   my-dumptype
   include "./[d-f]*"
 }       1
 [...]
 largehost /home/./w-z /home {
   my-dumptype
   include "./[w-z]*"
 }       1
 # catch all the rest of /home:
 largehost /home/./_rest_ /home {
   my-dumptype
   exclude append "./[a-z]*"
 }       1

Notice the use of the 'include' and 'exclude' options. They do what you would guess, i. e. include the files specified, or exclude them. The tricky part is to use different disk names ('/home/./a-c', '/home/./d-f', ...) on the same partition/mount point ('/home'). You can omit specifying the spindle.

The disk name can be any string.  In the example we used a notation "/mountpoint/./mnemonic".
The part before the "/./" is the moint point and the part after the "/./" is a mnemonic for the include/exclude pattern.  The files on the backup are relative to the mount point.
Note that gnutar actually does not need that the toplevel directory is a mount point.
Any top level directory will do, as we see in the next example.

* NOTE: When using 'program "GNUTAR"' in the dumptype, include and exclude require that the Amanda user have read access down to the level of the item being included or excluded in order to accomplish globbing. 'program "APPLICATION"' with 'application "app_amgtar"' (see: {{man|8|amgtar}}) does not have this limitation and should be used in preference to 'program "GNUTAR"'. Globbing failures with 'program "GNUTAR"' will show up in the Amanda reports under STRANGE DUMP DETAILS with a gnutar error such as, "/bin/tar: ./datadir/19*: Warning: Cannot stat: No such file or directory". In this case, the Amanda user does not have read access to "datadir" and cannot expand the "19*". In the case of an include, the result will be that the items represented by "19*" will not be backed up.

= Splitting it up, the more complicated part =

The /share partition is a little more complicated to split because you have to deal with the subdirectories. We assume that there's so much data in some of the directories that you have to treat them by diving into the subdirectories. You need to catch /share, /share/financial, /share/financial/company-a, /share/financial/company-b and /share/reports:

 largehost /share/financial/./company-a /share/financial {
   my-dumptype
   include "./company-a"
 }       2
 largehost /share/financial/./company-b /share/financial {
   my-dumptype
   include "./company-b"
 }       2
 largehost /share/financial/./_rest_ /share/financial {
   my-dumptype
   exclude append "./company-a"
   exclude append "./company-b"
 }       2
 #
 largehost /share/./reports /share {
   my-dumptype
   include "./reports"
 }       2
 #
 largehost /share/./_rest_ /share {
   my-dumptype
   exclude append "./financial"
   exclude append "./reports"
 }       2

In this example, it is important to notice that each subdirectory is refered to from its immediate parent directory. It doesn't work otherwise.

Also be sure to specify a catchall using excludes for each top level directory as last DLE.
This avoids risking that you forget to add a DLE when someone creates a directory with a name
that did not fit into one of the "include" categories.

To use many exclude/include arguments on one line, you must put quotes around each argument:

 exclude append "./financial" "./reports"

Currently (Amanda version 2.4.x and 2.5.0) do not allow spaces in the exclude/include pattern:

 exclude append "./Documents and Settings"    # Invalid in 2.5.0 or earlier

You'll get an obscure error message about "wrong number of arguments" in that case.
There is no escape character to protect the space.
As workaround you can use wildcards instead:

 exclude append "./Documents?and?Settings"    # workaround space in pattern

This limitation is fixed in version 2.5.1 and later.

= See also =

* [http://wiki.zmanda.com/man/amanda.conf.5.html#idm140480245628336 amanda.conf(5), DUMPTYPE SECTION section]

* There are some important notes in the article on [[Exclude and include lists]].
{{Languages}}