Selfcheck request failed



This article is a part of the Troubleshooting collection.

Problem

In Amanda 2.5.0 and later:

 Amanda Backup Client Hosts Check
 --------------------------------
 WARNING: 192.168.15.245: selfcheck request failed: timeout waiting for ACK
 Client check: 1 host checked in 30.097 seconds, 1 problem found

Another variation of the same message:

WARNING: client.company.com: selfcheck request failed: client.company.com: did not resolve to client.company.com

In Amanda 2.4.4 and earlier the error message was worded differently:

AMANDA backup client hosts check
--------------------------------
Warning: selfcheck request timed out.  Host down?
Client check: 1 host checked in 30.051 seconds, 1 problem found

Solution

This is usually a client configuration error, although there are many possibilities for misconfiguration.

Backing Up Older Amanda Clients (pre-2.5.1)

You can backup older Amanda client using a Amanda 2.5.1 and later Server however you must use a auth "bsd" setting as the older Amanda clients can only use udp datagrams. If this is not correct you will get errors such as

  selfcheck request failed: recv error: Connection reset by peer

for a amcheck on the server and an error such as

  Transport endpoint is not connected

in the /tmp/amanda/amandad*.debug files on the client.

To back up disks on the older clients you can override a global auth "bsdtcp" setting in special dumptype entry in "amanda.conf" for use with older clients.

Check if amanda services and .amandahosts are configured correctly

Correct xinetd and .amandahosts configuration are available at Amanda 2.5.1 configuration file changes. For earlier releases, see Amanda Quickstart page.

Check if TCP wrappers (tcpd) is configured correctly

If you are using tcpd wrapper for amanda inetd entries (as the following example), hosts.allow(5) have to modified to allow amanda connections.

Example: inetd configuration entry using tcpd:

amanda dgram udp wait amandabackup /usr/sbin/tcpd /usr/lib/amanda/amandad 

hosts.allow file:

amandad: ALL : ALLOW
amindexd: ALL : ALLOW
amidxtape: ALL : ALLOW

Access to amanda processes should be restricted to only Amanda clients.

Check inetd/xinetd/daemontools configuration on the client

If amandad was not started, check your inetd/xinetd/daemontools configuration.

• Make sure you have added the Amanda services to /etc/services (or the NIS services map).
• Make sure you signalled (x)inetd to reread its configuration (some systems may need rebooting).
• Check the inetd man-page for possible differences between the standard inetd.conf format and the one in your system. For example, you will need to specify 'amandad' once again, as the first argument (argv[0]), with openbsd-inetd.
• Pay special attention to typos in inetd.conf; error messages will probably appear in /var/adm/messages or /var/log/messages if you have typed the amandad program name incorrectly.
• If you are building Amanda binaries on your own, make sure the dump user that has been specified at configure-time (--with-user=USERNAME) is listed in the (x)inetd config file.
• Check whether the dump user has permission to run amandad, as well as any shared libraries amandad depends upon, by running the specified amandad command by hand, as the Amanda user. It should just time-out after 30 seconds waiting for a UDP packet. If you type anything, it will abort immediately, because it can't read a UDP packet from the keyboard.
• The only_from parameter in xinetd configuration should be correctly defined (it should be set to amanda server)
• Check, if inetd is running (ps -ef | grep inetd). If not, start it manually.

Use "netstat -a | grep amanda" to verify there is some program listening on the amanda/udp port (usually 10080). Another tool that can used for verifying that amandad is listening to the udp port is lsof.

Check for unwritable debug directory

Locate the AMANDA_DBGDIR directory (usually /tmp/amanda) and find a file named amandad.<DATETIME.debug> in the directory. When amandad starts, the debug file will be created for the process.

If the debug file does not exist, the Amanda client process, amandad, has not been started properly. Go through the checklist for inetd/xinetd/daemontools in the section above.

It is also possible that the debug directory (/tmp/amanda) is not writeable by the amanda backup user (example: amandabackup)

Verify the owner and permissions of /tmp/amanda directory. It should be owned by the user that is specified in inetd/xinetd configuration and the directory permissions should be 700 (drwx------).

Also check the permissions of the parent directory (usually /tmp: permissions 1777, drwxrwxrwt). If the amandabackup user does not have write access in the parent directory, you must create the debug directory yourself, and set ownership/permissions manually.

You may erase the directory and run amcheck again: the directory is created automatically.

If you are using Cygwin Amanda client, the /tmp/amanda - Amanda debug directory is created by amcheck command with owner being the user who installed Cygwin. The directory should be owned by the Amanda backup user.

Slow NFS-server

If Amanda programs are NFS auto-mounted on the client, some clients may fail to mount the Amanda binaries in time for the check.

Firewall/TCP-wrapper settings

Firewall between backup server and client can cause selfcheck to timeout if the firewall is not configured correctly.

Like most services started from (x)inetd, the TCP-wrapper on the client has to be configured to allow the server to come in.

Failing DNS service

Name services on the Amanda client is not configured correctly or is not working. If the Amanda client cannot resolve server name(FQDN) then this error message is seen. Either of the following two solutions can be used to fix the problem:

• Check the forward and reverse name resolution on the Amanda client. Make sure the Amanda client is able to connect to the Amanda server using the Amanda server FQDN.

NOTE: The note on correct/existing reverse DNS resolution is very important. Sometimes it works when you just use the raw IP address.

• Add amanda server name (FQDN) entry in /etc/hosts of client machines.
• Check if the nsswitch.conf has files before dns for hosts.

Aliases on the network interface

If there are IP aliases on the network interface that is being used for backup, replace the SRC with the correct IP address or use the network interface without IP aliases. For example :

bond0 -> 192.168.18.7
bond0:0 -> 192.168.18.8

If Amanda backup is using the bond0:0 interface and the SRC route uses 192.168.18.7 IP address, the amcheck will fail with this error. To fix the problem, use 192.168.18.7 for Amanda backup and SRC route should use 192.168.18.7.

Virtual machine interaction with NIS

If your client is a Virtual Machine, and /tmp/ directory is locked/engaged by some process. e.g NIS client of VM fetching NIS server.

Remedy: Stop the NIS service, ensure "ls /tmp" or ls "/var/tmp" responds promptly as other directories, then run amcheck.

auth "bsdtcp"

When following the steps from The 15-Minute Backup Solution, I also get the error:

WARNING: host.corp.com: selfcheck request failed: timeout waiting for ACK
Client check: 1 host checked in 30.024 seconds, 1 problem found

when I failed to do the following instructions in my amanda.conf:

Go to the “define dumptype global” section in the amanda.conf file and add the auth "bsdtcp" line right before the last “}” bracket. This is done to enable “BSDTCP” authentication.