The following document outlines how to recover from a system crash under SCO XENIX Operating System Release 2.3 (AT/MC/GT). TABLE OF CONTENTS Section I.....................Introduction Section II....................Initial Preparations a. Booting from a floppy b. Cleaning the file system c. Mounting the Hard Drive Section III...................Defining the Problem a. At bootup the system hangs and I get no error message. b. I get the message: "NO OS" c. I get the message: "STAGE 1 BOOT FAILURE ERROR LOADING /BOOT" d. I get the message: "STAGE 1 BOOT FAILURE NOT A DIRECTORY" e. I get the message: "SRMOUNT: PANIC CANNOT OPEN CVTV7SUPERB() YET" f. I get the message: "/XENIX NOT FOUND" g. My System Hangs at the Letter "Z" h. I get the message: "Can't exec /etc/init" or "Can't exec /etc/inir" i. I get the message "No Space on dev 1/40" Section IV....................Restoring from Tape c. Recreating the /dev directory b. Restoring the root file system c. Restoring the root file system w/o a boot/root floppy Section V.....................Contacting SCO Technical Support SECTION I - Introduction The purpose of the following document is to explain how to recover from a system crash when you cannot boot from your hard disk under SCO XENIX Operating System Release 2.3 (AT/MC/GT). The same basic methods will also apply to earlier releases of the SCO XENIX System V Operating System. If, after proceeding through the steps outlined in this document, you are still having problems and need to contact SCO Support Services, please refer to Section V of this document. SECTION II - Initial Preparations First you must either boot from the N1 diskette that comes with the SCO XENIX Operating System Release 2.3, or from a special "bootable/rootable" floppy that you created before your system crashed. Please refer to your SCO XENIX System Administrators Guide, page 16-13, for instructions on creating a "bootable/rootable" floppy. (a) Insert the floppy and turn on the machine as if you were reinstalling the operating system. SCO XENIX will take a moment to load into memory. - Using the N1 floppy: Continue through to the "Hard Disk Initialization" message. If you are using 48tpi or 135tpi diskettes, the procedure will be different. Refer to Appendix A of this docu- ment. At the prompt "During installation you may choose to overwrite all or part of the present contents of your hard disk. Do you wish to continue?," press the key. After the "install aborted" message, you should see the message "Proceed to step 3". - Using a Bootable/Rootable floppy: If you are using a bootable/rootable floppy, press at the Boot: prompt. After various boot messages, you will see a "#" prompt. b. CLEANING THE FILE SYSTEM At the prompt, either "" or "#", type the command: /bin/fsck -s /dev/hd0root If fsck asks for a SCRATCH file, press the key. When you get a prompt, issue the command: /etc/mknod ramdisk b 31 26 Run fsck again and when it asks you for a scratch file, enter "ramdisk." If fsck completes within a few seconds without going through its normal phases, or if you see error messages, such as "UNKNOWN FILE SYSTEM VERSION 65535" or "CLEANING NON SYSTEM 3 FILE SYSTEM," the file system is damaged beyond repair and you will need to reinstall your operating system files. c. MOUNTING THE HARD DRIVE If the file system was successfully cleaned, it can now be mounted on the floppy by typing: /etc/mount /dev/hd0root /mnt If you get the message: "mount: cannot access special file" you will probably have to reinstall your operating system. If you were able to mount your hard disk, you are now ready to begin your crash recovery. First you must determine where in the boot procedure your system fails. What are your symptoms? SECTION III - Defining the Problem a. At bootup the system hangs and I get no error message: The masterboot block has either been corrupted or removed. Boot the the system as in Section II and at the prompt "Boot :" enter: hd(40)xenix This will cause the kernel on the hard disk to be booted. Once the kernel has loaded, enter System Maintenance mode and enter the command: dparam -w This will cause a copy of /etc/masterboot to be copied to the disk. b. I get the message "NO OS." The boot block on your hard disk has probably been corrupted. Follow steps in Section II to boot from the floppy and mount the hard drive. Then type: /bin/dd if=/mnt/etc/hdboot0 of=/mnt/dev/hd0a /bin/dd if=/mnt/etc/hdboot1 of=/mnt/dev/hd0a bs=1k seek=1 /etc/haltsys Now boot from the hard disk. If this boot fails, your last resort is to restore your root file system from your backup tape. c. I get the message "STAGE 1 BOOT FAILURE ERROR LOADING /BOOT." Follow the steps in Section II above and then issue the commands: /bin/cp /boot /mnt/boot /etc/umount /dev/hd0root /etc/haltsys Boot from the hard disk. d. I get the message "STAGE 1 BOOT FAILURE NOT A DIRECTORY." This is an unrecoverable error. You need to reinstall. e. I get the message "SRMOUNT: PANIC CANNOT CVTV7SUPERB() YET." This is usually a non-recoverable error. It indicates severe file system corruption. You can attempt to recover by following steps in Section IId. Use the following command: /bin/fsck -c /dev/hd0root Should this fail it may be possible to make a replacement super- block on another XENIX system. WARNING: An improperly created superblock copy, or one that is improperly used, may cause extreme damage to your file system that makes recovery impossible. (1) Find the size of your root file system on the original machine by issuing the following command: /etc/divvy -b 1 -c 1 When the table is shown, subtract the starting block entry on the first line from the ending block entry and adding '1' to it. This is the size of the root file system. (2) Create a file system in the /tmp directory on another machine by issuing the command: /etc/mkfs /tmp/filesys SIZE where SIZE is the size of the root file system as determined in step (1). If there is not enough space on the disk, this will not work! (3) Mount the emergency boot/root floppy onto this system by issuing the command: /etc/mount /dev/install /mnt (4) Copy the superblock from this file system into a temporary file on the emergency floppy by issuing the command: /bin/dd if=/tmp/filesys of=/mnt/tmp/superblk bs=1K count=2 NOTE: It is extremely important that the values for 'bs' and 'count' are as indicated. (5) Unmount the emergency boot/root floppy onto this system by issuing the command: /etc/umount /dev/install (6) Boot and root your system, then mount your hard disk using the instructions in Section II above. (6) Copy the superblock into the root directory of your hard disk by issuing the command: /bin/dd if=/tmp/superblk of=/dev/hd0root bs=1K count=2 If this is unsuccessful, you must reinstall. f. I get the message "/XENIX NOT FOUND." /xenix is corrupted or missing. First, boot off the hard disk and enter "xenix.old" at the Boot prompt, as follows: Boot :xenix.old If your boot is successful, enter: rm /xenix mv /xenix.old /xenix You may have device drivers missing from your new /xenix. If they are third party drivers and you need to relink them, refer to the installation instructions of the drivers. Also refer to Chapter 7 of the SCO XENIX System Administrators Guide. If your boot was unsuccessful, you can copy /xenix from the floppy. Follow the steps in Section II above and then enter: /bin/cp /xenix /mnt/xenix If you copied /xenix from the N1 floppy, also enter the command: /etc/brand serno key /mnt/xenix where "serno" is the serial number and "key" is the activation key supplied with your SCO XENIX Operating System Release 2.3 diskettes. This is the same serial number and activation key that you used during installation. Finally, enter: /etc/haltsys Boot from the hard disk and relink any drivers that you had previously installed. g. My system hangs at the letter "Z." Certain system files are probably corrupted. Follow steps in Section II; then enter: /mnt/bin/rm /mnt/dev/console /mnt/dev/mem /mnt/dev/kmem /mnt/etc/mknod /mnt/dev/console c 3 1 /mnt/etc/mknod /mnt/dev/mem c 4 0 /mnt/etc/mknod /mnt/dev/kmem c 4 1 /etc/haltsys Remove your floppy diskette and reboot from the hard drive when you see one of the following messages: ** Safe to Power Off ** -or- ** Hit Any Key to Reboot ** If your boot fails, follow the instructions in Section II. Then follow the steps in step "g." below. h. I get the message "CAN'T EXEC /ETC/INIT" or "CAN'T EXEC /ETC/INIR." Your files /etc/init and /etc/inir might be corrupted. Follow the steps in Section II and then enter: /bin/cp /etc/init /mnt/etc/init /bin/cp /etc/inir /mnt/etc/inir /etc/haltsys Remove your floppy diskette and reboot from the hard drive when you see the following message: ** Safe to Power Off ** -or- ** Hit Any Key to Reboot ** i. I get the Message: "No space left on dev 1/40" This message may have more than one cause, but it means that the root file system is nearly 100% full. The first step is solving this problem is to create enough free space to move around in. In some cases, this message scrolls on the screen and won't allow any input. It may also occur that, although the message keeps scrolling, it may be possible to log in. If logging in is not possible, you will have to boot the system and mount the hard disk as in Section II. Once logged in, the first step is to truncate the files /etc/wtmp and /usr/adm/messages. At the prompt, enter the commands: # > /etc/wtmp # > /usr/adm/messages If the messages continue, remove all files in the /tmp directory and check the /dev directory for any large files. This is done by issuing the following command: ls -s /dev | sort -n This will sort the files in the /dev directory by size. At the end of the list there should be only a few entries with size larger than zero (0). If there are any files whose sizes are excessively large (larger than 2), it is quite possible that the wrong device was used when making a backup or using tar. If this still has not solved the problem, it is possible that an excessively large file resides elsewhere on the disk. The easiest way to locate large files on your disk is by using the very versatile find(C) command with the 'size' option. For example, to locate files greater than 1 megabyte, the command would be: find / -size +2000 -print where "/" designates that the search starts at the root directory; "+2000" locates files greater than 2000 blocks long (using 512 byte blocks); "-print" causes the current complete path to be printed. Should these be the result of an improper backup or tar command, ensure that proper backups exist prior to removing them. SECTION IV - Restoring from Tape WARNING: Steps b & c will destroy the current contents of your hard disk. Make sure that you have exhausted the above methods and have reliable backups before attempting the following procedures. a. Recreating the /dev directory When backups are done with tar, no device files are backed up. Therefore it will be necessary to recreate the minimum devices before restoring a tar backup. (1) First, you must boot from the N1 diskette that comes with the SCO XENIX Operating System Release 2.3. Follow the instructions in Section II in order to clean your hard disk root file system, and mount it on the floppy. (2) Enter the commands: cd /mnt echo * NOTE: The string "" is a prompt from the operating system. Do not type it in. (3) If the second command gives a response of "*", then the mount of the hard disk may have failed. Should the mount have been successful but the root file system be empty, the echo * command will also give the same response. This can be verified by first creating a file in the /mnt directory by issuing the command: touch foo This will create a file called 'foo' in the /mnt. This can be verified by entering: echo * This will show the single file foo in the /mnt directory. (4) Now return to the root directory by issuing the command: cd / Then mount the hard disk using the steps in Section IIc. Now re-enter the commands: cd /mnt echo * If the second command gave a response of '*' then the mount was successful and your root file system is empty. If this is the case, you will have to re-install the operating system. If the response was 'foo', then the mount was unsuccessful. If you received the message: "mount: cannot access special file" Then you will proably have to reinstall your operating system. (3a) If your SCO XENIX system came on 5-1/4" floppy disks, enter the command: /etc/fixperm -cs -dRTS /etc/perms/inst This will re-create your device directory on the hard disk. Ignore any messages about files not being empty. Enter the command: /etc/haltsys Reboot your system. You should have a minimal /dev directory. /etc/fixperm -cs -dRTS /etc/perms/inst (3b) If your SCO XENIX system came on 3-1/2" floppy disks, enter the commands: /bin/mkdir /mnt/dev (Ignore any error messages from this command.) /etc/mknod /mnt/dev/rroot c 1 40 /etc/mknod /mnt/dev/root b 1 40 /etc/mknod /mnt/dev/swap b 1 41 /etc/mknod /mnt/dev/rswap c 1 41 PATH=/mnt/bin:/mnt/etc export PATH cd /dev find . -print | cpio -pd /mnt/dev If the find(C) and cpio(C) commands are not available on your hard disk, you can also try the command: copy -rom /dev /mnt/dev This process will recreate enough of your /dev directory on the hard disk that you will be able to boot the system in single-user mode. At this point, you can enter the command: # cd / # /etc/fixperm -cs -dRTS /etc/perms/inst This will give you a minimal /dev directory. NOTE: Any special device files which have been added to your system (such as tape devices, MultiView devices, or any third party devices) will not be re-created by this procedure. SEE ALSO: copy(C), cpio(C), find(C), fixperm(ADM), mknod(C). Another item in this database entitled "How do I restore the special files in /dev?". b. RESTORING THE ROOT FILE SYSTEM WITH A BOOT/ROOT FLOPPY If you have a tape backup unit, and had previously created a backup of your root file system using the backup command, then booting the system from a customized bootable/rootable floppy will enable you to restore your hard disk's root file system from your tape backups. Any other file systems can be restored after rebooting from the hard disk. Boot from the boot/root floppy and enter: /etc/divvy -b 1 -c 1 The divvy command will bring up the file system division table which will show the size of your file systems in blocks. It should look similar to this: +---------------+------------------+---+-------------+------------+ | Name | New File System? | # | First Block | Last Block | | hd0root | no | 0 | 0| 39023| | swap | no | 1 | 39024| 41250| | u | no | 2 | 41251| 41502| | | no | 3 | -| -| | | no | 4 | -| -| | | no | 5 | -| -| | | no | 6 | 41503| 41512| | hd01 | no, exists | 7 | 0| 41759| +---------------+------------------+---+-------------+------------+ 41513 blocks for divisions, 247 blocks reserved for the system Record this information. When you run divvy from the floppy, the divisions will either not have names, or they will have false ones. The root file system is represented by division 0. Select [c]reate division for hd0root. If the named division does not appear or is false, select [n]ame division. Once completed, [q]uit from the divvy menu and select [i]nstall division setup shown. BEWARE: This option will overwrite the root file system and will provide no chance of retrieval. You will then see "Making Filesystems" prompt. Proceed with the following commands to reload from tape: /bin/restor fr /dev/rct0 /dev/hd0root /bin/fsck -s /dev/hd0root /etc/haltsys Replace /dev/rct0 with the name of your tape drive if you normally access it with a different name. NOTE: If the "restor" command returns with an error message such as "cannot open /dev/hd0root" make sure that your XENIX partition is active. If it is, then issue the commands in step 4 and try to restore again. If the restore still fails, you will need to install the basic RunTime System from your SCO XENIX Operating System Release 2.3 distribution floppies and then re-issue the "restor" command. Boot from the hard disk. If you have other file systems, they may not have been damaged. If they were damaged you can restore them. For example: /bin/restor fr /dev/rct0 /dev/u will restore a /dev/u file system. Make sure you are in System Maintenance mode when you restore. c. Restoring the root file system without a boot/root floppy. First install the basic RunTime System and the Link Kit on the hard disk. Next, run mkdev tape to configure your tape drive and reboot the system. Follow the instructions in the SCO XENIX System Administrators Guide, page 16-13, to create a Boot/Root floppy. When you have finished creating the Boot/Root, follow the instructions in step 12. V - Contacting SCO Technical Support Before you contact SCO Technical Support Services, you need to have specific information for the SCO Support Engineer. Please have the following information available: 1. The exact version of the SCO XENIX System V Operating System you are running. 2. The type and model of machine you are using. 3. A complete description of the hardware configuration. 4. An exact description of the error message you receive and the context in which you receive it. Also, you will need to be able to answer the following questions in detail: 1. What was the machine doing when it crashed? 2. What was the sequence of events that preceded the crash? 3. Has this problem occurred more than once? How often? 4. Did you install any third party device drivers or additional hardware recently? APPENDIX A How to recover /xenix and /boot using the 48 or 135tpi distribution of the SCO XENIX System V Operating System. 1. If the machine has not been shut down, it is possible to mount the N1 diskette and copy over a new version of the files. a. Insert the N1 floppy into drive fd0. b. Log in as root and type the following commands: /etc/mount -r /dev/fd048 /mnt (or /dev/fd0135ds9 for 3.5") /bin/cp /mnt/xenix /xenix /bin/cp /mnt/boot /boot /etc/brand _serial_number _activation_key /xenix /etc/umount /dev/fd048 (or /dev/fd0135ds9 for 3.5") When issuing these commands, make sure to use the serial number and activation key included with your SCO XENIX Operating System Release 2.3. c. Remove the N1 floppy. 2. If the machine will not boot, follow the instructions below. a. Insert the N1 floppy into drive fd0. b. Turn on the power to the machine. c. At the Boot: prompt, type the command that applies to the version you are running: For 2.1.* (48tpi) ================= fd(4,0)xenix rootdev=hd(40,0) pipedev=hd(40,0) swapdev=hd(40,0) For 2.2.* (48tpi) ================= fd(4)xenix root=hd(40) pipe=hd(40) swap=hd(41) For 2.2.* or 2.3.* (135tpi) =========================== fd(64)xenix root=hd(40) pipe=hd(40) swap=hd(41) d. DO NOT type D. Use the root password to enter System Maintenance mode. e. Type the following commands: /etc/mount -r /dev/fd048 /mnt (or /dev/fd0135ds9 for 3.5") /bin/cp /mnt/boot /boot /bin/cp /mnt/xenix /xenix /etc/brand _serial_number _activation_key /xenix /etc/umount /dev/fd048 (or /dev/fd0135ds9 for 3.5") /etc/haltsys Make sure you use the serial number and activation key included with your SCO XENIX System V Operating system in the command above. f. Remove the N1 floppy and press any key to reboot the machine. NOTE: If there were any drivers in the lost kernel, a new kernel will have to be created. The commands below will create a new kernel: cd /usr/sys/conf ./link_xenix ./hdinstall /etc/haltsys Reboot the machine. SEE ALSO: fsck( ), mknod( ), mount(), dd( ), haltsys( ), brand( ) init( ), inir( ), fixperm( ), divvy ( ), restor( ).