Device API

From wiki.zmanda.com
Revision as of 15:50, 25 May 2008 by Dustin (talk | contribs) (→‎Specification: remove big copy of old device.h, more changes)
Jump to navigation Jump to search

The Device API is a clean interface between Amanda and data-storage systems. It provides a tape-like model -- a sequence of bytestreams on each volume, identified only by their on-volume file number -- even of non-tape devices.

Background

Device API Features

The Device API also adds a number of new features not previously available in Amanda. These include:

  • Device properties allow drivers to describe themselves (and the devices and media they control), as well as allow arbitrary user settings to propagate down to the driver.
  • Smart locking means unrelated accesses can be performed without issue, while conflicting accesses wait for one another.
  • Appending to volumes is supported for capable devices.
  • A device can describe its supported blocksize range to the Amanda core, instead of the other way around.
  • Deleting parts of volumes (without erasing the whole thing) is supported for capable devices.
  • The amount of free space on a device can be reported to the Amanda core, where the device supports it (e.g., VFS Device)

History

The Device API is designed to replace the ancient tapeio subsystem (sometimes called the vtape API or Virtual Tape API), originally introduced to support virtual tapes. The original design of the tapeio system (located in tape-src/) was to abstract tape-related functionality into a separate library. Thus, different devices (tape, vtape and RAIT) were all phrased in terms of tape operations: Rewind, fast-forward, read a block, etc. Operations were assumed to have the same semantics as a UNIX tape device. Furthermore, the API revealed a file descriptor, with the assumption that other parts of Amanda could perform operations such as stat() or dup2() on it.

The Device API clears up a large number of limitations and device assumptions that the tapeio system made, allowing native support for new devices (such as CDs and DVDs) as well as new device-related functionality (such as parallel access, partial recycling, and appending).

The Device API does not change (or even address) media formats.

Limitations of tapeio

Important limitations inherent in the design of tapeio include:

  • Exposes a UNIX file descriptor
  • Operates in terms of UNIX operations
  • Does not store information in a portable (yet device-specific) way.
  • Not reentrant: Access to the device is not permitted from multiple threads or processess.
  • Assumes devices can be opened and closed at will.
  • RAIT driver performs operations in series (rather than parallel).

Tapeio Operations

Tapeio included the following operations:

  • tape_access(), tape_open(), tape_stat(), tapefd_close(), tapefd_read(), tapefd_write(): Act like their UNIX equivalents
  • tapefd_rewind(), tapefd_unload(): Do what you expect.
  • tapefd_fsf(): Seeks forward a certain number of filemarks.
  • tapefd_weof(): Writes a filemark.
  • tapefd_resetofs(): Workaround for some buggy kernels.
  • tapefd_status(): Prints status to stdout (!).

In addition, all tapefd_ commands have matching tape_ commands, which work on an unopened device.

What the Device API is not

The Device API does not distinguish between random-access and linear-access media: The seek operation may take a long time for some devices, or it may be instantaneous for others. It does, however, distinguish between concurrent devices and exclusive devices: Concurrent devices may be accessed by multiple readers (and sometimes multiple writers) simultaneously, while exclusive devices cannot.

Although the Device API deals in on-medium headers and blocks, it is otherwise agnostic to media format issues. Changes to the Amanda media format, including split vs traditional formats, can be made without change to the Device API.

Specification

The Device API is implemented in the device-src/ directory. It relies heavily on GLib's type system, so developers unfamiliar with the GObject system are encouraged to consult the relevant documentation

Instantiation

Broadly speaking, the C interface revolves around a single glib virtual class -- Device -- which is actually implemented as one of many possible subclasses. Subclasses are identified with device-name prefixes, e.g., tape: for the Tape Device or s3: for the S3 Device. device_open takes a device name, selects a Device subclass based on the prefix, instantiates the subclass, and passes the remainder of the device name to the subclass's factory function.

Perl Interface

The Device API is available from perl as Amanda::Device. Most of the discussion here applies equally well, regardless of the language in use.

Using the Device API

If you are writing an application that makes use of the Device API, whether in Perl or C, consult device-src/device.h for the most up-to-date information.

TODO

Error Handling

TODO

Properties

Device properties provide a bidirectional means of communication between devices and their users. A device provides values for some properties, which other parts of Amanda can use to adjust their behavior to suit the device. For example, Amanda will only attempt to append to a volume if the device's properties indicate that it supports this activity. Some devices have additional properties that can be set to control its activity. For example, the S3 Device requires that the users' keys be given via properties.

The Amanda configuration interface for properties is not completely developed. Currently, global device_property parameters are examined by device_get_startup_properties_from_config(), which means that individual properties cannot be specified for individual devices.

Standard Properties

The Device API includes a number of standard properties, and allows devices to declare and register other custom properties as may be appropriate. Standard properties include (TODO: list these below, and list individual properties on pages for individual devices):

  • Concurrency: Does the device support concurrent readers and/or writers?
  • Streaming: Does the device require (or desire) streaming data?
  • Compression: Does the device support it, and what compression rate has been yielded.
  • Blocksize: What block size(s) are supported by the device? This property is settable for devices that support a variety of block sizes, or can be left unset for variable block size.
  • Device UUID: Returns a unique identifier for this piece of hardware.
  • Media access mode: What is the access paradigm for this volume? Can be Read-Only, Write-Once-Read-Many (WORM), Read-Write, or Write-Once-Read-Never.
  • Feature support: Does the device support partial deletion or appending to volumes?

Implementing a new device

To add a new device to the Device API, do the following:

  • Implement a subclass of one of the existing Device classes named above. In particular, implement the various virtual functions provided for in device.h.
    • Using the FdDevice class for devices that have a unix file discriptor will spare you from implementing device_read_block() and device_write_block(), but there are other virtual functions in FdDevice that you (may) need to reimplement.
    • Note that in addition to the (mostly virtual) functions discussed above, there are some additional protected functions that require implementation.
  • Make sure your device calls device_add_property() to register both its dynamic and static properties.
  • Arrange to call the register_device() function to register the device, so it will recieve relevant calls to device_open(). The easiest way to do this is to write a initialization function, and add a call to your function from device_api_init() in device.c.

Users new to GLib will need to read up on the GObject system first, to see how GLib provides virtual abstract classes in C and how to implement a new subclass. Additionally, the existing device drivers may serve as a useful template.

Properties Interface

Most code related to device-independent properties is in property.h and property.c. Since property values are passed around as a GValue, all value types must be registered in the GLib type system.

Every abstract property has a DevicePropertyBase, which refers to a DevicePropertyId. The Base structure is the same for all properties of all devices; it holds information about the property outside other context. Each property has a unique DevicePropertyId, though the ID of a property is only guaranteed within a single run of a program: Between runs, IDs might change, so it's best to refer to properties by name outside of a particular program.

When a device starts up, it creates a DeviceProperty structure for each supported property. This structure refers to the DevicePropertyBase, but also holds information about when the property may be accessed: Not all properties may be set or gotten at any time. The PropertyAccessFlags field access holds this information: it is the bitwise OR of any of the various PROPERTY_ACCESS_GET_ and PROPERTY_ACCESS_SET_ values defined. Note that there are five distinct "time periods" as far as access is concerned:

  • BEFORE_START: Before device_start() has been called; i.e., before any permanant action may have been taken.
  • BETWEEN_FILE_WRITE: When in write mode, but not actually writing a file. Specifically, after device_start() has been called with ACCESS_WRITE or ACCESS_APPEND, but outside of a pair of calls to device_start_file() and device_finish_file().
  • INSIDE_FILE_WRITE: In the middle of writing a file. Specifically, after device_start_file() has been called, but before the coresponding call to device_finish_file().
  • BETWEEN_FILE_READ: When in read mode, but not actually reading a file. This means after device_start() has been called with ACCESS_READ, but either before a call to device_seek_file, or after a call to device_read_block returns EOF (provided no intervening call to device_seek_file() or device_seek_block() since the EOF).
  • INSIDE_FILE_READ: When in read mode, and while actually reading a file. This means after a call to device_seek_block(), but before a call to device_read_block() returns EOF.

Adding a new property requires modifying both property.h and property.c, as C code in property.c is required to register the properties and generate their IDs. See those files for the symbolic names for the standard properties described above.

Future Directions

Future directions for the Device API include:

  • new devices
  • runtime loading of device modules as shared objects
  • more flexible device configuration
Retrieved from ‘http://wiki.zmanda.com/index.php?title=Device_API&oldid=5887’