XFA/Writing Transfer Elements

This page tries to take you step-by-step through the process of designing a new transfer element.

Which Xfer Mechanisms?

Begin my envisioning all of the ways your element could operate in a transfer. First, does the element more naturally deal with file descriptors, or buffers? Second, does it make sense to "pull" data from upstream, or should upstream "push" data to the element? Consult XFA/Xfer Mechanisms.

For example:

• a filter which forks and executes another application can be implemented most easily with file descriptors both for input and output.
• A destination which packetizes and transmits data over the network is probably best implemented with buffers, so that the element can frame each buffer for transmission.

Make all sensible pairs of mechanisms available, but do not write code that is duplicates work the glue elements will perform. In the first example above, adding PUSH_BUFFER, with a method that calls write() on the pipe file descriptor, would accomplish exactly the same thing as the PUSH_BUFFER->WRITE_FD element glue.

EOF?

Each mechanism specifies what an EOF looks like. How will you handle incoming EOFs? Can you generate an EOF for your downstream element if necessary?

Does your element have an "active" component that can indicate when it has received an EOF (so that the XFA knows when the transfer is complete)?

Cancellation?

The XFA specifies a fairly complex dance to smoothly cancel a transfer. The complexity stems from the heavy caching that can take place along the length of a transfer, both by Amanda components and by the operating system. All of this data must be "drained", so that buffers can be properly freed, before the cancellation is complete. The architecture does this by injecting early EOFs into the datastream wherever possible, then waiting until each element has received such an EOF. This EOF is only sent after the cancelled flag is set on each element; elements must properly recognize this as a cancellation and not a successful EOF.

If your element has an active thread which can be interrupted somehow, then the cancel method should perform such an interruption.

That thread must be ready at all times to stop its operation, generate an EOF for the downstream element (if possible), and (if expecting an EOF) drain any buffered data from upstream until the EOF is received. In particular, this means that any threads which are blocking on a condition variable must be awakened (the condition variable signalled) and must exit properly. This leads to a lot of code that looks like

while (!condition_I_care_about && !elt->cancelled) {
    g_cond_wait(self->some_cond, self->some_mutex);
}
if (elt->cancelled) {
    handle_cancellation();
    return;
}

and the cancel method must carefully acquire the appropriate mutexes (without deadlock!) and signal the appropriate condition variables.

Read Up

Read the header files xfer.h and xfer-element.h, which explain the implementation in authoritative detail.

Code

There is a good deal of boilerplate code involved in writing a GObject class. Probably the easiest way to create a new transfer element is to copy a similar, existing element and perform a global search-and-replace. If your element is suitable for use in server and client installations, place it in xfer-src/. If it is only suitable to the server, use server-src/ or, if it's device-related, device-src/. Add the prototype for your class's constructor to xfer-src/xfer-elements.h, server-src/xfer-server.h, or device-src/xfer-device.h, as appropriate.

Add a new class to perl/Amanda/Xfer.swg or perl/Amanda/XferServer.swg, as appropriate, and add exhaustive POD documentation to Xfer.swg.

Add tests to installcheck/Amanda_Xfer.pl or installcheck/Amanda_Xfer_serveronly.pl, exercising all of the functionality of your new element.