Testing

From The Open Source Backup Wiki (Amanda, MySQL Backup, BackupPC)
Jump to navigationJump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Why Test?

Amanda is a critical application, and one in which an unanticipated failure, such as silent data corruption in dumps, may have extremely high costs for users. It also has a long development history, which has led to very strict expectations of its behavior.

We have added tests to help us detect and correct any bugs, regressions, or misbehaviors we introduce while adding new functionality. Of course, no set of tests can completely characterize Amanda's behavior, but the more tests we run, the better the chances of catching problems early.

Amanda's testing is structured by Automake and Test::Harness, and for technical reasons, comes in two parts.

What to Test

The short answer: everything you can think of.

The long answer is that tests are cheap, and where the benefits are greater than the costs, you should add a test.

  • Write tests for any behaviors that could be affected by changes you are making to the codebase. This includes not just expected behavior changes, but any potential unexpected behavior changes that you can think of.
  • Write tests for any Amanda program to which you make significant changes. This includes trying out all command-line options and combinations thereof, and any other permutations of inputs that might have different results.
  • Write thorough tests for all Amanda perl modules. If necessary, add code to the module itself to support testing. This ensures that the behavior of these modules is well-understood and dependable.

Sometimes, however, it may not be worth writing a test, or may even be impossible:

  • The current testing frameworks run on a single machine, as a non-root user, and assume little about that machine. Testing network protocols, such as security API implementations, is therefore difficult or impossible.
  • Tests involving large quantities of data will waste more developer time and/or drive space than they will save. Such tests can usually be scaled down to make them reasonable, for example by using 512K or 1M vtapes.

Make Check

Running make check in a configured source directory does the following:

  • syntax-checks all shell and perl scripts, if the relevant interpreters support it
  • runs any C-only test programs
  • runs a quick check that threading is allowed within a perl executable

These tests are limited in range, because they cannot assume that any part of Amanda is installed, and they cannot do anything that might affect an installed Amanda.

Many packaging systems, such as Gentoo's portage, can run these checks automatically, and they provide a nice way to catch simple errors early in the build process.

Make Installcheck

make installcheck runs the more interesting tests. As its name suggests, this target runs tests on an installed Amanda. This has the following advantages:

  • All of the compiled-in paths in Amanda (e.g., ${libexecdir}) point to the correct files and directories.
  • Perl modules and extensions are organized in the directory structure that perl expects.
  • The environment is realistic enough to allow authentic dumps and restores, with little or no emulation of a real system required.

While the installchecks endeavor to co-habitate with any existing Amanda configuration, we do not recommend that they be run on a production installation, nor even with root privileges. The installchecks blithely overwrite any existing amandates or amanda-client.conf. Furthermore, by their very nature, they are testing untrusted code. You have been warned. See below for advice on running installchecks without root.

Prerequisites

Beyond the requirements for Amanda, running installchecks also requires Test::More, which only began shipping with Perl in 5.8.0. This can be installed from http://cpan.org, or via your OS's package-distribution system. On most RPM-based distros, it is in the perl-Test-Simple package.

Running Installcheck without Root

At Zmanda, our automated testing system runs installchecks without requiring root access. This allows us to limit the entire test process to a single user and a single subdirectory. After the checks are complete, the system can be restored to its previous state by simply removing that directory. Here are the necessary configure flags to accomplish this:

  • --without-force-uid - remove all checks for and changes to the current userid
  • --disable-installperms -- don't try to chown/chgrp files after installing them
  • --prefix=$PWD/prefix - use a subdir of the current directory as an install prefix
  • --with-tmpdir=$PWD/tmp - use a different subdir of the current directory as a temporary directory
  • --without-amperldir - put perl modules under $prefix/lib/amanda/perl, instead of in perl's 'installsitearch'
  • --with-single-userid - expect to run all parts of Amanda as a single user

all at once now, for ease of cut-and-paste:

./configure --without-force-uid --disable-installperms --prefix=$PWD/prefix --with-tmpdir=$PWD/tmp --without-amperldir --with-single-userid

Then, as a user other than amanda or root, run

make check install
make CLOBBER_MY_CONFIG=OK installcheck

to run all tests. The CLOBBER_MY_CONFIG=OK is a signal that you understand that things may get hairy, and that you're not running with permissions that could clobber anything important (that's why this isn't run as root or amanda). The installchecks will clobber $PWD/../prefix/etc/amanda/amanda-client.conf and $PWD/../prefix/var/amanda/amandates, which won't be a problem in this case.

Infrastructure

Installchecks are perl scripts, run by Perl's Test::Harness. All of the scripts are in the installcheck/ subdirectory, and are enumerated in the Makefile.am in that directory. Scripts which use Amanda perl libraries contain the line

use '@amperldir@';

before using any of the Amanda::* modules. This adds the path to Amanda's modules to perl's search path.

Contact

For any questions or comments on the testing architecture, please connect with the developers on the [email protected] mailing list.