Amanda::Xfer - the transfer architecture
use Amanda::MainLoop; use Amanda::Xfer qw( :constants ); use POSIX;
my $infd = POSIX::open("input", POSIX::O_RDONLY, 0); my $outfd = POSIX::open("output", POSIX::O_CREAT|POSIX::O_WRONLY, 0640); my $xfer = Amanda::Xfer->new([ Amanda::Xfer::Source::Fd->new($infd), Amanda::Xfer::Dest::Fd->new($outfd) ]); $xfer->start(sub { my ($src, $xmsg, $xfer) = @_; print "Message from $xfer: $xmsg\n"; # use stringify operations if ($msg->{'type'} == $XMSG_DONE) { Amanda::MainLoop::quit(); } }, 0, 0); Amanda::MainLoop::run();
See http://wiki.zmanda.com/index.php/XFA for background on the transfer architecture.
A new transfer is created with Amanda::Xfer->new()
, which takes
an arrayref giving the transfer elements which should compose the
transfer.
The resulting object has the following methods:
Start this transfer. It transfer $size bytes starting from offset $offset.
$offset must be 0. $size is only supported by Amanda::Xfer::Source::Recovery.
A size of 0 transfer everything to EOF.
Processing takes place asynchronously, and messages will
begin queueing up immediately. If $cb
is given, then it is installed as the
callback for messages from this transfer. The callback receives three
arguments: the event source, the message, and a reference to the controlling
transfer. See the description of Amanda::Xfer::Msg
, below, for details.
There is no need to remove the source on completion of the transfer - that is handled for you.
cancel()
Stop transferring data. The transfer will send an XMSG_CANCEL
,
"drain" any buffered data as best it can, and then complete normally
with an XMSG_DONE
.
get_status()
Get the transfer's status. The result will be one of $XFER_INIT
,
$XFER_START
, $XFER_RUNNING
, or $XFER_DONE
. These symbols are
available for import with the tag :constants
.
repr()
Return a string representation of this transfer, suitable for use in debugging messages. This method is automatically invoked when a transfer is interpolated into a string:
print "Starting $xfer\n";
get_source()
Get the Amanda::MainLoop event source through which messages will
be delivered for this transfer. Use its set_callback
method to
connect a perl sub for processing events.
Use of this method is deprecated; instead, pass a callback to the start
method. If you set a callback via get_source
, then you must remove
the source when the transfer is complete!
The individual transfer elements that compose a transfer are instances
of subclasses of Amanda::Xfer::Element. All such objects have a
repr()
method, similar to that for transfers, and support a similar
kind of string interpolation.
Note that the names of these classes contain the words "Source", "Filter", and "Dest". This is merely suggestive of their intended purpose -- there are no such abstract classes.
Amanda::Xfer::Source::Device->new($device);
This source reads data from a device. The device should already be
queued up for reading ($device->seek_file(..)
). The element
will read until the end of the device file.
Amanda::Xfer::Source::Fd->new(fileno($fh));
This source reads data from a file descriptor. It reads until EOF, but does not close the descriptor. Be careful not to let Perl close the file for you!
Amanda::Xfer::Source::Holding->new($filename);
This source reads data from a holding file (see Amanda::Holding).
If the transfer only consists of a Amanda::Xfer::Source::Holding
and an Amanda::Xfer::Dest::Taper::Cacher
(with no filters), then the source
will call the destination's cache_inform
method so that it can use
holding chunks for a split-part cache.
Amanda::Xfer::Source::Random->new($length, $seed);
This source provides length bytes of random data (or an unlimited
amount of data if length is zero). $seed
is the seed used to
generate the random numbers; this seed can be used in a destination to
check for correct output.
If you need to string multiple transfers together into a coherent sequence of random numbers, for example when testing the re-assembly of spanned dumps, call
my $seed = $src->get_seed();
to get the finishing seed for the source, then pass this to the source constructor for the next transfer. When concatenated, the bytestreams from the transfers will verify correctly using the original random seed.
Amanda::Xfer::Source::Pattern->new($length, $pattern);
This source provides length bytes containing copies of pattern. If length is zero, the source provides an unlimited number of bytes.
Amanda::Xfer::Source::Recovery->new($first_device);
This source reads a datastream composed of on-device files. Its constructor takes a pointer to the first device that will be read from; this is used internally to determine whether DirectTCP is supported.
The element sense $XMSG_READY
when it is ready for the first start_part
invocation. Don't do anything with the device between the start of the
transfer and when the element sends an $XMSG_READY
.
The element contains no logic to decide which files to assemble into the datastream; instead, it relies on the caller to supply pre-positioned devices:
$src->start_part($device);
Once start_part
is called, the source will read until $device
produces an
EOF. As each part is completed, the element sends an $XMSG_PART_DONE
Amanda::Xfer::Msg, with the following keys:
size bytes read from the device duration time spent reading fileno the on-media file number from which the part was read
Call start_part
with $device = undef
to indicate that there are no more
parts.
To switch to a new device in mid-transfer, use use_device
:
$dest->use_device($device);
This method must be called with a device that is not yet started, and thus must
be called before the start_part
method is called with a new device.
Amanda::Xfer::Source::DirectTCPListen->new();
This source is for use when the transfer data will come in via DirectTCP, with
the data's source connecting to the data's destination. That is, the
data source is the connection initiator. Set up the transfer, and after
starting it, call this element's get_addrs
method to get an arrayref of ip/port pairs,
e.g., [ "192.168.4.5", 9924 ]
, all of which are listening for an incoming
data connection. Once a connection arrives, this element will read data from
it and send those data into the transfer.
my $addrs = $src->get_addrs();
Amanda::Xfer::Source::DirectTCPConnect->new($addrs);
This source is for use when the transfer data will come in via DirectTCP, with
the data's destination connecting to the the data's source. That is, the
data destination is the connection initiator. The element connects to
$addrs
and reads the transfer data from the connection.
$xfp = Amanda::Xfer::Filter::Process->new([@args], $need_root);
This filter will pipe data through the standard file descriptors of the
subprocess specified by @args
. If $need_root
is true, it will attempt to
change to uid 0 before executing the process. Note that the process is
invoked directly, not via a shell, so shell metacharcters (e.g., 2>&1
)
will not function as expected. This method create a pipe for the process
stderr and the caller must read it or a hang may occur.
$xfp->get_stderr_fd()
Return the file descriptor of the stderr pipe to read from.
Amanda::Xfer::Filter::Xor->new($key);
This filter applies a bytewise XOR operation to the data flowing through it.
Amanda::Xfer::Dest::Device->new($device, $cancel_at_eom);
This source writes data to a device. The device should be ready for writing
($device->start_file(..)
). On completion of the transfer, the file will
be finished. If an error occurs, or if $cancel_at_eom
is true and the
device signals LEOM, the transfer will be cancelled.
Note that this element does not apply any sort of stream buffering.
Amanda::Xfer::Dest::Buffer->new($max_size);
This destination records data into an in-memory buffer which can grow up to
$max_size
bytes. The buffer is available with the get
method, which
returns a copy of the buffer as a perl scalar:
my $buf = $xdb->get();
Amanda::Xfer::Dest::DirectTCPListen->new();
This destination is for use when the transfer data will come in via DirectTCP,
with the data's destination connecting to the data's source. That is,
the data destination is the connection initiator. Set up the transfer, and
after starting it, call this element's get_addrs
method to get an arrayref
of ip/port pairs, e.g., [ "192.168.4.5", 9924 ]
, all of which are listening
for an incoming data connection. Once a connection arrives, this element will
write the transfer data to it.
my $addrs = $src->get_addrs();
Amanda::Xfer::Dest::DirectTCPConnect->new($addrs);
This destination is for use when the transfer data will come in via DirectTCP,
with the data's source connecting to the the data's destination. That
is, the data source is the connection initiator. The element connects to
$addrs
and writes the transfer data to the connection.
Amanda::Xfer::Dest::Fd->new(fileno($fh));
This destination writes data to a file descriptor. The file is not closed after the transfer is completed. Be careful not to let Perl close the file for you!
Amanda::Xfer::Dest::Null->new($seed);
This destination discards the data it receives. If $seed
is
nonzero, then the element will validate that it receives the data that
Amanda::Xfer::Source::Random
produced with the same seed. No
validation is performed if $seed
is zero.
This is the parent class to Amanda::Xfer::Dest::Taper::Cacher
and
Amanda::Xfer::Dest::Taper::DirectTCP
. These subclasses allow a single
transfer to write to multiple files (parts) on a device, and even spread those
parts over multiple devices, without interrupting the transfer itself.
The subclass constructors all take a $first_device
, which should be
configured but not yet started; and a $part_size
giving the maximum size of
each part. Note that this value may be rounded up internally as necessary.
When a transfer using a taper destination element is first started, no data is
transfered until the element's start_part
method is called:
$dest->start_part($retry_part);
where $device
is the device to which the part should be written. The device
should have a file open and ready to write (that is,
$device->start_file(..)
has already been called). If $retry_part
is
true, then the previous, unsuccessful part will be retried.
As each part is completed, the element sends an $XMSG_PART_DONE
Amanda::Xfer::Msg
, with the following keys:
successful true if the part was written successfully eof recipient should not call start_part again eom this volume is at EOM; a new volume is required size bytes written to volume duration time spent writing, not counting changer ops, etc. partnum the zero-based number of this part in the overall dumpfile fileno the on-media file number used for this part, or 0 if no file was used
If eom
is true, then the caller should find a new volume before
continuing. If eof
is not true, then start_part
should be called
again, with $retry_part = !successful
. Note that it is possible
for some destinations to write a portion of a part successfully,
but still stop at EOM. That is, eom
does not necessarily imply
!successful
.
To switch to a new device in mid-transfer, use use_device
:
$dest->use_device($device);
This method must be called with a device that is not yet started.
If neither the memory nor disk caches are in use, but the dumpfile is
available on disk, then the cache_inform
method allows the element
to use that on-disk data to support retries. This is intended to
support transfers from Amanda's holding disk (see
Amanda::Xfer::Source::Holding
), but may be useful for other
purposes.
$dest->cache_inform($filename, $offset, $length);
This function indicates that $filename
contains $length
bytes of
data, beginning at offset $offset
from the beginning of the file.
These bytes are assumed to follow immediately after any bytes
previously specified to cache_inform
. That is, no gaps or overlaps
are allowed in the data stream described to cache_inform
.
Furthermore, the location of each byte must be specified to this
method before it is sent through the transfer.
$dest->get_part_bytes_written();
This function returns the number of bytes written for the current part to the device.
Amanda::Xfer::Dest::Taper::Splitter->new($first_device, $max_memory, $part_size, $expect_cache_inform);
This class splits a data stream into parts on the storage media. It is for use
when the device supports LEOM, when the dump is already available on disk
(cache_inform
), or when no caching is desired. It does not cache parts, so
it can only retry a partial part if the transfer source is calling
cache_inform
. If the element is used with devices that do not support LEOM,
then it will cancel the entire transfer if the device reaches EOM and
cache_inform
is not in use. Set $expect_cache_inform
appropriately based
on the incoming data.
The $part_size
and $first_device
parameters are described above for
Amanda::Xfer::Dest::Taper
.
Amanda::Xfer::Dest::Taper::Cacher->new($first_device, $max_memory, $part_size, $use_mem_cache, $disk_cache_dirname);
This class is similar to the splitter, but caches data from each part in one of
a variety of ways to support "rewinding" to retry a failed part (e.g., one that
does not fit on a device). It assumes that when a device reaches EOM while
writing, the entire on-volume file is corrupt - that is, that the device does
not support logical EOM. The class does not support cache_inform
.
The $part_size
and $first_device
parameters are described above for
Amanda::Xfer::Dest::Taper
.
If $use_mem_cache
is true, each part will be cached in memory (using
$part_size
bytes of memory; plan accordingly!). If $disk_cache_dirname
is defined, then each part will be cached on-disk in a file in this directory.
It is an error to specify both in-memory and on-disk caching. If neither
option is specified, the element will operate successfully, but will not be
able to retry a part, and will cancel the transfer if a part fails.
Amanda::Xfer::Dest::Taper::DirectTCP->new($first_device, $part_size);
This class uses the Device API DirectTCP methods to write data to a device via DirectTCP. Since all DirectTCP devices support logical EOM, this class does not cache any data, and will never re-start an unsuccessful part.
As state above, $first_device
must not be started when new
is called.
Furthermore, no use of that device is allowed until the element sens an
$XMSG_READY
to indicate that it is finished with the device. The
start_part
method must not be called until this method is received either.
Messages are simple hashrefs, with a few convenience methods. Like
transfers, they have a repr()
method that formats the message
nicely, and is available through string interpolation:
print "Received message $msg\n";
The canonical description of the message types and keys is in
xfer-src/xmsg.h
, and is not duplicated here. Every message has the
following basic keys.
The message type -- one of the xmsg_type
constants available from
the import tag :constants
.
The transfer element that sent the message.
The version of the message. This is used to support extensibility of the protocol.
Additional keys are described in the documentation for the elements
that use them. All keys are listed in xfer-src/xmsg.h
.
This page was automatically generated Tue Feb 21 19:14:01 2012 from the Amanda source tree, and documents the most recent development version of Amanda. For documentation specific to the version of Amanda on your system, use the 'perldoc' command.