DAS Disaster Discovery

Introduction

What happens if your DAS servers are destroyed by fire or flood? What happens if they are stolen? How do you handle this? First, we will examine a few disaster scenarios. Then, we will look at the steps necessary to restore critical DAS files that include user information and Kerberos data.

Loss of the DAS Slave Server

Fortunately, if the DAS slave is destroyed, a new one can be rebuilt following the original installation instructions for DAS-S. While this is going on, the DAS master server will handle all authentication and name service lookups.

Loss of the DAS Master Server

If DAS-M is destroyed, or the NIS/Kerberos files are corrupted, the system should be taken offline and DAS-S will handle all of the DAS functions. In a worst-case scenario, you will have to build DAS-M from scratch. Although you can recover the Kerberos database or the NIS maps from DAS-S, we will assume that you have been a good sysadmin and regularly saved your backups to another server or some type of removable media. As long as you have recent backups available, you will be able to restore your authentication and account information. The backup information also includes the backup DAS configuration files that you made during the initial setup. You can use the backup config files for a speedy rebuild.

Assuming that your DAS-M server has to be completely rebuilt, you will follow all of the installation instructions except for the creation of the Kerberos database. Here are the step-by-step instructions for restoring from the following backup files, which were located on floppy disk:

313-03.das-m.kerb5.gz
313-03.das-m.nis.tar.gz

Remember that the original location for these files on DAS-M is the /home/Backups directory, where the files are automatically saved once an hour.

Step 1: Extract the backups to the new DAS-M

[root@das-m Backups]# mount /mnt/floppy
[root@das-m Backups]# cd /mnt/floppy
[root@das-m floppy]# ls -l
total 4
-rwxr-xr-x    1 root     root         2355 Nov 10 09:50 313-03.das-m.kerb5.gz
-rwxr-xr-x    1 root     root         1362 Nov 10 09:50 313-03.das-m.nis.tar.gz
[root@das-m floppy]# cp -v 313*gz /root
`313-03.das-m.kerb5.gz' -> `/root/313-03.das-m.kerb5.gz'
`313-03.das-m.nis.tar.gz' -> `/root/313-03.das-m.nis.tar.gz'
[root@das-m floppy]# cd /root
[root@das-m root]# gunzip 313*gz
[root@das-m root]# ls -al 313*
-rwxr-xr-x    1 root     root         6068 Nov 10 09:58 313-03.das-m.kerb5
-rwxr-xr-x    1 root     root        10240 Nov 10 09:58 313-03.das-m.nis.tar
[root@das-m root]# tar -xvf 313-03.das-m.nis.tar
group.nis
hosts.nis
passwd.nis

Step 2: Restore the /etc/passwd, /var/yp/hosts, and /etc/group files

This assumes that you have already created the non-root local account with the same UID and given that account a password. Whenever we manipulate /etc/group or /etc/passwd, we need to be very careful. We will first backup those two files and we will make sure to keep a root terminal or window open while we are testing this.

[root@das-m root]# cp -v /etc/passwd /etc/passwd.org
`/etc/passwd' -> `/etc/passwd.org'
[root@das-m root]# cp -v /etc/group /etc/group.org
`/etc/group' -> `/etc/group.org'
[root@das-m root]# cp -v /root/hosts.nis /var/yp/hosts
`/root/hosts.nis' -> `/var/yp/hosts'

[root@das-m root]# pico passwd.nis
[root@das-m root]# cat passwd.nis >> /etc/passwd
[root@das-m root]# pico group.nis
[root@das-m root]# cat group.nis >> /etc/group

[root@das-m root]# ls -al /etc/passwd
-rw-r--r--    1 root     root         2066 Nov 10 10:19 /etc/passwd
[root@das-m root]# ls -al /etc/group
-rw-r--r--    1 root     root          701 Nov 10 10:19 /etc/group

Once this is done, you can run the nisupdate script or the make -C /var/yp command to update the NIS maps. Make sure that you can still login as root and as the non-root local user before closing the root terminal.

[root@das-m root]# nisupdate
make: Entering directory `/var/yp'
gmake[1]: Entering directory `/var/yp/kerb.org'
gmake[1]: `ypservers' is up to date.
gmake[1]: Leaving directory `/var/yp/kerb.org'
gmake[1]: Entering directory `/var/yp/kerb.org'
Updating passwd.byname...
Updating passwd.byuid...
Updating group.byname...
Updating group.bygid...
Updating hosts.byname...
Updating hosts.byaddr...
gmake[1]: Leaving directory `/var/yp/kerb.org'
make: Leaving directory `/var/yp'

Step 3: Restore the Kerberos database

[root@das-m root]# /etc/init.d/krb5kdc stop
Stopping Kerberos 5 KDC:                                   [  OK  ]
[root@das-m root]# /etc/init.d/kadmin stop
Stopping Kerberos 5 Admin Server:                          [  OK  ]

[root@das-m root]# /usr/kerberos/sbin/kdb5_util load /root/313-03.das-m.kerb5

You can use the -verbose flag after kdb5_util to see all of the principals that are being restored. If you look at the /var/kerberos/krb5kdc directory, you will see the restored database files. They start with "principal":

[root@das-m root]# ls -l /var/kerberos/krb5kdc
total 44
-rw-------    1 root     root           51 Oct 20 09:38 kadm5.acl
-rw-------    1 root     root           22 Sep  9 14:34 kadm5.acl.org
-rw-------    1 root     root          244 Sep 22 10:06 kadm5.keytab
-rw-------    1 root     root          326 Sep 22 09:51 kdc.conf
-rw-------    1 root     root          694 Sep  9 14:35 kdc.conf.org
-rw-------    1 root     root        16384 Nov 10 10:41 principal
-rw-------    1 root     root         8192 Nov 10 10:41 principal.kadm5
-rw-------    1 root     root            0 Nov 10 10:41 principal.kadm5.lock
-rw-------    1 root     root            0 Nov 10 10:41 principal.ok

Recreate the stash file, using the same password that you used when you created the original Kerberos principals' database. Without the password, the database is useless. Here is how you do it:

[root@das-m root]# /usr/kerberos/sbin/kdb5_util stash
kdb5_util: Cannot find/read stored master key while reading master key
kdb5_util: Warning: proceeding without master key
Enter KDC database master key:
[root@das-m krb5kdc]# ls -al /var/kerb*/k*/.k5*
-rw-------    1 root     root           30 Nov 10 10:53 /var/kerberos/krb5kdc/.k5.KERB.ORG

Since this is a new install, you will also need to extract the host key into the keytab again for DAS-M:

[root@das-m sbin]# kadmin.local
Authenticating as principal root/[email protected] with password.
kadmin.local:  ktadd host/das-m.kerb.org
Entry for principal host/das-m.kerb.org with kvno 5, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/das-m.kerb.org with kvno 5, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.

Step 4: Bring DAS-M back online and test

Now that you have restored the database and recreated the stash file, you should be able to start up your Kerb5 services and test the system:

[root@das-m krb5kdc]# /etc/init.d/krb5kdc start
Starting Kerberos 5 KDC:                                   [  OK  ]
[root@das-m krb5kdc]# /etc/init.d/kadmin start
Starting Kerberos 5 Admin Server:                          [  OK  ]

[root@das-m krb5kdc]# kadmin -p super/admin
Authenticating as principal super/admin with password.
Enter password:
kadmin:  listprincs
K/[email protected]
[email protected]
[email protected]
host/[email protected]
host/[email protected]
host/[email protected]
[email protected]
kadmin/[email protected]
kadmin/[email protected]
kadmin/[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
krbtgt/[email protected]
super/[email protected]
user/[email protected]
[email protected]
kadmin:  quit
[root@das-m krb5kdc]# kinit -p kitty
Password for [email protected]:
[root@das-m krb5kdc]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: [email protected]
 
Valid starting     Expires            Service principal
11/10/03 10:58:17  11/10/03 20:58:17  krbtgt/[email protected]
 
 
Kerberos 4 ticket cache: /tmp/tkt0
klist: You have no tickets cached
[root@das-m krb5kdc]# kdestroy

You will now want to make sure that user "kitty" can login from a DAS client, and that replication is working between the DAS-M and DAS-S.

Loss of Both DAS Servers

The first thing to determine is whether or not the failure is a network failure. If it is not a network failure, then you will need to notify your users and begin restoring/rebuilding your DAS servers. In the meantime, you may need to reconfigure the DAS clients through the /etc/nsswitch.conf file and the various /etc/pam.d config files to not attempt to use NIS or Kerberos. Other temporary changes may be necessary, as well.

Conclusion