It’s been several years since I last described a linux rescue in my earlier post “Rescuing Susie“. And, by now, that’s a little outdated. In today’s post, I’ll go over the basic principles of rescuing a linux system.
I’ll be describing what works with opensuse. But I have also used this method with Ubuntu and with Debian.
Simple problems can often be fixed by booting any live linux media. Typically, you boot the live system, then open a root shell. From there, you can mount a partition and use a text editor to adjust configuration files.
Sometimes it is possible to use an alternative method of booting into your system. For example, I dealt with a recent problem by booting another linux system also installed on my disk. I then modified the boot menu of that bootable system so that I could use it to boot into the system being repaired. That’s always easier than a full rescue.
Full rescue mode
For the hard problems, you will need to use full rescue mode. To accomplish this, you will need a bootable linux system that is as similar as possible to the system you are repairing. Often the media that you originally used for installing linux is a good choice.
What is most important about the system you use for rescue, is that it use the same architecture. If you are repairing a 64-bit system, then you will need to boot live media of a 64-bit system for repairs. Similarly, to repair a 32-bit system, use 32-bit live media. For opensuse, I often use the live rescue CD for the same opensuse version and architecture.
Here are the basic steps, after you have suitably booted your computer with rescue media:
Mount the root file system
I typically mount as “/mnt”, since that directory is pretty standard on live media. For example to solve problems with my recently installed Leap 42.1 milestone system, I used:
# mount /dev/sdb6 /mnt
Obviously, change that “/dev/sdb6” to the appropriate device.
If your root file system is encrypted, or part of an encrypted LVM, you must first deal with that. I’ll discuss that later in this post.
If you are using a separate “/boot” partition, you must also mount that. It might require a command similar to
# mount /dev/sda1 /mnt/boot
You can look in “/etc/fstab” (really “/mnt/etc/fstab”) to find the device to mount for “/boot”, or to check whether there is a separate “/boot”. Note that you must mount it at “/boot” relative to where you mounted the root file system (hence at “/mnt/boot” for my example. If you do not use a separate “/boot” file system, then ignore this step.
On a UEFI system, you will also need to mount “/boot/efi”. Again, you can check “/etc/fstab” to find which device to mount. You should mount it at “/mnt/boot/efi”. Ignore this step on a non-UEFI system.
Mount other system partitions
Typically, repairs need “/dev”, “/proc” and “/sys”. It is easiest to mount these with “–bind” mounts, as in
# mount --bind /dev /mnt/dev # mount --bind /proc /mnt/proc # mount --bind /sys /mnt/sys
The “–bind” mounts create a logical connection between the newly mounted file system and the corresponding file system from the rescue media that you have booted.
Optional — mount “/home”
If your repairs are related to “/home”, you should mount that. Otherwise it might not be necessary. It never hurts.
“chroot” into the system
Use the command:
# chroot /mnt
(change “/mnt” to wherever you mounted the root file system).
At this stage, you are using the kernel of the rescue media that you booted, but you are using commands (software) from the system that you are rescuing. All paths now look to be the relative to the mount point.
Now you can use whatever commands need to be run on the system that you are repairing. This might be simple, such as rebuilding the grub2 boot menu, or a bit more complex (such as remaking the “initrd” file). Or you might even need to install the booting setup.
You will need to be using command line commands here. GUI (graphic) programs are unlikely to work.
Finally exit the “chroot” enviroment (with “exit”), then reboot to test whether your repairs have solved the problem.
Encrypted file systems
There are additional steps if you are using crypto. If the root file system is encrypted, or part of an encrypted LVM, then you must first make that available.
To access an encrypted partition, assuming LUKS encryption, use:
# cryptsetup luksOpen /dev/sdxy virtual_name
Here, replace “/dev/sdxy” with the appropriate device for the encrypted partition. The virtual name is the name you will use to refer to the virtual unencrypted file system. While the virtual name could be anything, it is best to use the same name that was used when your system was setup.
I will sometimes access the encrypted file system, then look in “/etc/crypttab” to find out which virtual name was used (it is the first column of “/etc/crypttab”). Then I’ll undo the decryption with “cryptsetup luksClose”, and redo it with the correct virtual name.
If you don’t use the correct virtual name, that will sometimes confuse things. For the name that you use might get into grub menus or “initrd” files.
In case of an encrypted LVM, after opening the encrypted partition, use
# vgchange -a y
to make the volumes visible. Then
# ls /dev/mapper
will tell you what is available. You can undo that LVM access with “vgchange -a n” after unmounting everything.