How To:Split DLEs With Exclude Lists: Difference between revisions
m (→Example: links) |
Paul.bijnens (talk | contribs) (using a selfdocumenting nice name as diskname) |
||
Line 1: | Line 1: | ||
This article describes a way to split your [[DLE]]s to produce smaller individual dumps. This is often useful when you only have a few DLEs, and one of them is much larger than the rest: without this technique, the tape usage spikes when Amanda runs a level 0 for that DLE. For example, many small businesses have one very large NFS-mounted <tt>/home</tt>. Using this technique, <tt>/home</tt> can be split by subdirectory, e.g., <tt>/home/[a-o]*</tt> and <tt>/home/[p- | This article describes a way to split your [[DLE]]s to produce smaller individual dumps. This is often useful when you only have a few DLEs, and one of them is much larger than the rest: without this technique, the tape usage spikes when Amanda runs a level 0 for that DLE. For example, many small businesses have one very large NFS-mounted <tt>/home</tt>. Using this technique, <tt>/home</tt> can be split by subdirectory, e.g., <tt>/home/[a-o]*</tt> and <tt>/home/[p-z]*</tt>. A more extended example is provided below. | ||
With Amanda < 2.5.0 the need arises to virtually split partitions into parts so that each part remains smaller than a tape. This will lead to smaller single dumps so that Amanda can continue the backup of other dumps on the next tape, if need be. | With Amanda < 2.5.0 the need arises to virtually split partitions into parts so that each part remains smaller than a tape. This will lead to smaller single dumps so that Amanda can continue the backup of other dumps on the next tape, if need be. | ||
Line 34: | Line 34: | ||
Considering the /home partition, it becomes obvious that you could split it alphabetically. You would create disklist entries to back up each subdirectory by using wildcards. You also want to catch anything under /home that doesn't fit into the wildcard specifications, so there: | Considering the /home partition, it becomes obvious that you could split it alphabetically. You would create disklist entries to back up each subdirectory by using wildcards. You also want to catch anything under /home that doesn't fit into the wildcard specifications, so there: | ||
largehost / | largehost /home/./a-c /home { | ||
my-dumptype | my-dumptype | ||
include "./a*" | include "./[a-c]*" | ||
} 1 | } 1 | ||
largehost / | largehost /home/./d-f /home { | ||
my-dumptype | my-dumptype | ||
include "./ | include "./[d-f]*" | ||
} 1 | } 1 | ||
[...] | [...] | ||
largehost / | largehost /home/./w-z /home { | ||
my-dumptype | my-dumptype | ||
include "./w | include "./[w-z]*" | ||
} 1 | } 1 | ||
# catch all the rest of /home: | # catch all the rest of /home: | ||
largehost / | largehost /home/./_rest_ /home { | ||
my-dumptype | my-dumptype | ||
exclude append "./[a-z]*" | exclude append "./[a-z]*" | ||
} 1 | } 1 | ||
Notice the use of the 'include' and 'exclude' options. They do what you would guess, i. e. include the files specified, or exclude them. The tricky part is to use different disk names ('/ | Notice the use of the 'include' and 'exclude' options. They do what you would guess, i. e. include the files specified, or exclude them. The tricky part is to use different disk names ('/home/./a-c', '/home/./d-f', ...) on the same partition/mount point ('/home'). You can omit specifying the spindle. | ||
The disk name can be any string. In the example we used a notation "/mountpoint/./mnemonic". | |||
The part before the "/./" is the moint point and the part after the "/./" is a mnemonic for the include/exclude pattern. The files on the backup are relative to the mount point. | |||
Note that gnutar actually does not need that the toplevel directory is a mount point. | |||
Any top level directory will do, as we see in the next example. | |||
= Splitting it up, the more complicated part = | = Splitting it up, the more complicated part = | ||
Line 63: | Line 65: | ||
The /share partition is a little more complicated to split because you have to deal with the subdirectories. We assume that there's so much data in some of the directories that you have to treat them by diving into the subdirectories. You need to catch /share, /share/financial, /share/financial/company-a, /share/financial/company-b and /share/reports: | The /share partition is a little more complicated to split because you have to deal with the subdirectories. We assume that there's so much data in some of the directories that you have to treat them by diving into the subdirectories. You need to catch /share, /share/financial, /share/financial/company-a, /share/financial/company-b and /share/reports: | ||
largehost / | largehost /share/financial/./company-a /share/financial { | ||
my-dumptype | my-dumptype | ||
include "./company-a" | include "./company-a" | ||
} 2 | } 2 | ||
largehost / | largehost /share/financial/./company-b /share/financial { | ||
my-dumptype | my-dumptype | ||
include "./company-b" | include "./company-b" | ||
} 2 | } 2 | ||
largehost / | largehost /share/financial/./_rest_ /share/financial { | ||
my-dumptype | my-dumptype | ||
exclude append "./company-a" | exclude append "./company-a" | ||
Line 77: | Line 79: | ||
} 2 | } 2 | ||
# | # | ||
largehost / | largehost /share/./reports /share { | ||
my-dumptype | my-dumptype | ||
include "./reports" | include "./reports" | ||
} 2 | } 2 | ||
# | # | ||
largehost / | largehost /share/./_rest_ /share { | ||
my-dumptype | my-dumptype | ||
exclude append "./financial" | exclude append "./financial" | ||
Line 89: | Line 91: | ||
In this example, it is important to notice that each subdirectory is refered to from its immediate parent directory. It doesn't work otherwise. | In this example, it is important to notice that each subdirectory is refered to from its immediate parent directory. It doesn't work otherwise. | ||
Also be sure to specify a catchall using excludes for each top level directory as last DLE. | |||
This avoids risking that you forget to add a DLE when someone creates a directory with a name | |||
that did not fit into one of the "include" categories. | |||
To use many exclude/include arguments on one line, you must put quotes around each argument: | To use many exclude/include arguments on one line, you must put quotes around each argument: | ||
Line 96: | Line 102: | ||
Currently (Amanda version 2.4.x and 2.5.0) do not allow spaces in the exclude/include pattern: | Currently (Amanda version 2.4.x and 2.5.0) do not allow spaces in the exclude/include pattern: | ||
exclude append "./Documents and Settings" # Invalid | exclude append "./Documents and Settings" # Invalid in 2.5.0 or earlier | ||
You'll get an obscure error message about "wrong number of arguments" in that case. | You'll get an obscure error message about "wrong number of arguments" in that case. | ||
Line 103: | Line 109: | ||
exclude append "./Documents?and?Settings" # workaround space in pattern | exclude append "./Documents?and?Settings" # workaround space in pattern | ||
This limitation is fixed in version 2.5.1 and later. | |||
= See also = | = See also = | ||
There are some important notes in the article on [[Exclude and include lists]]. | There are some important notes in the article on [[Exclude and include lists]]. |
Revision as of 08:56, 6 May 2008
This article describes a way to split your DLEs to produce smaller individual dumps. This is often useful when you only have a few DLEs, and one of them is much larger than the rest: without this technique, the tape usage spikes when Amanda runs a level 0 for that DLE. For example, many small businesses have one very large NFS-mounted /home. Using this technique, /home can be split by subdirectory, e.g., /home/[a-o]* and /home/[p-z]*. A more extended example is provided below.
With Amanda < 2.5.0 the need arises to virtually split partitions into parts so that each part remains smaller than a tape. This will lead to smaller single dumps so that Amanda can continue the backup of other dumps on the next tape, if need be.
If you have a lot of large dumps and Amanda 2.5.0 or later, then tape spanning may be more useful than this technique.
Example
Let's assume that you have two large partitions, holding several directories and subdirectories like the following:
/home/ /home/andrew /home/anne /home/brian [...] /home/wanda /home/zeus /share/ /share/financial /share/financial/company-a /share/financial/company-b /share/reports
You created entries in your disklist to back them up:
largehost /home my-dumptype largehost /share my-dumptype
That worked fine, but now on each of the partitions, there's too much data to fit onto a single tape. You will have to split up the partitions to back them up.
Splitting it up, the easy part
Considering the /home partition, it becomes obvious that you could split it alphabetically. You would create disklist entries to back up each subdirectory by using wildcards. You also want to catch anything under /home that doesn't fit into the wildcard specifications, so there:
largehost /home/./a-c /home { my-dumptype include "./[a-c]*" } 1 largehost /home/./d-f /home { my-dumptype include "./[d-f]*" } 1 [...] largehost /home/./w-z /home { my-dumptype include "./[w-z]*" } 1 # catch all the rest of /home: largehost /home/./_rest_ /home { my-dumptype exclude append "./[a-z]*" } 1
Notice the use of the 'include' and 'exclude' options. They do what you would guess, i. e. include the files specified, or exclude them. The tricky part is to use different disk names ('/home/./a-c', '/home/./d-f', ...) on the same partition/mount point ('/home'). You can omit specifying the spindle.
The disk name can be any string. In the example we used a notation "/mountpoint/./mnemonic". The part before the "/./" is the moint point and the part after the "/./" is a mnemonic for the include/exclude pattern. The files on the backup are relative to the mount point. Note that gnutar actually does not need that the toplevel directory is a mount point. Any top level directory will do, as we see in the next example.
Splitting it up, the more complicated part
The /share partition is a little more complicated to split because you have to deal with the subdirectories. We assume that there's so much data in some of the directories that you have to treat them by diving into the subdirectories. You need to catch /share, /share/financial, /share/financial/company-a, /share/financial/company-b and /share/reports:
largehost /share/financial/./company-a /share/financial { my-dumptype include "./company-a" } 2 largehost /share/financial/./company-b /share/financial { my-dumptype include "./company-b" } 2 largehost /share/financial/./_rest_ /share/financial { my-dumptype exclude append "./company-a" exclude append "./company-b" } 2 # largehost /share/./reports /share { my-dumptype include "./reports" } 2 # largehost /share/./_rest_ /share { my-dumptype exclude append "./financial" exclude append "./reports" } 2
In this example, it is important to notice that each subdirectory is refered to from its immediate parent directory. It doesn't work otherwise.
Also be sure to specify a catchall using excludes for each top level directory as last DLE. This avoids risking that you forget to add a DLE when someone creates a directory with a name that did not fit into one of the "include" categories.
To use many exclude/include arguments on one line, you must put quotes around each argument:
exclude append "./financial" "./reports"
Currently (Amanda version 2.4.x and 2.5.0) do not allow spaces in the exclude/include pattern:
exclude append "./Documents and Settings" # Invalid in 2.5.0 or earlier
You'll get an obscure error message about "wrong number of arguments" in that case. There is no escape character to protect the space. As workaround you can use wildcards instead:
exclude append "./Documents?and?Settings" # workaround space in pattern
This limitation is fixed in version 2.5.1 and later.
See also
There are some important notes in the article on Exclude and include lists.