Failing hard drive. Need to rescue system.
The hard drive in my home server system – a somewhat older Dell Optiplex 7010 which is a good, reliable box – started giving SMART errors which usually indicates impending drive failure. My initial reaction is always to boot the system from a rescue disk.
Nowadays rescue disks are not used so much as are rescue USB flash drives (also called pen drives or USB keys). They are so much more convenient and offer far better performance than optical media used to. You still see a lot of terminology (such as the term disk) from the time when recuse disks were on optical media and even before when they could be on floppy disks (tomsrtbt anyone?).
History
For a long time Knoppix was the king of rescue systems but over time it is being supplanted by other systems. One reason for this is that now most distros provide installers that also double as live systems. They are intended to allow the user to try the distro before installing it.
These installers were originally intended to be burned onto read-only optical media and can be installed onto USB flash drives using special USB installer utilities. When installed onto USB flash drives were still read-only. Initially there was support for limited persistence in order to maybe save desktop preferences or personal files, but not system-wide persistence necessary to really run a full-blown, customized server.
Because of my need to use a rescue system on my home server with its slowly dying drive I’ve been intensely investigating the suitability of various live systems and the various utilities used to create them.
Persistence is like a real system
As mentioned above, most rescue images were intended to be run from read-only media meaning that any changes made while the system was running would not be saved. They lacked persistence or had only limited persistence..
Not having persistence is fine if you are only using a rescue system to perform a selected task. For example you can run a non-persistent Clonezilla live rescue system to clone drives. Or you may run a Knoppix live system to perform some recovery tasks or to make offline backups or something.
With system-wide persistence a live system will be more like a regular system in that configuration and other changes will be saved to the system. Obviously you need this if you want to actually run a full-fledged server from a live USB system. Beyond that, persistence is just nice to have: It’s nice to have your custom changes, configurations, data, etc. saved and not lost once the live USB system is rebooted.
Knowing this, because of the failing hard drive in my home server, I decided to look into creating a persistent live USB flash-based system that would be suitable to set up as a permanent server.
mkusb is an amazing tool
In my investigation I discovered a truly amazing utility called mkusb that stands out above the rest. It’s so outstanding I’m writing this post about it because you have to try it! mkusb is able to turn any live USB system into a persistent one. Here is a list of some live USB systems:
Debian Live
Ubuntu Desktop (Ubuntu desktop images contain a live system by default)
Lubuntu Desktop
Kali Linux
Daily live images have more recently-updated packages
However I’ve found that installing what are called daily live images of stable releases of the above distros are best because they are the most up-to-date. Here are links to some recommended daily live images:
Debian unofficial non-free live images including firmware
Lubuntu 16.04 Xenial daily live
Basically a daily live image is like installing the standard release and then running an update on all the packages. It ensures you’re getting the most current or close-to-current packages on your system which is important for security and possibly also for performance reasons on a live system.
mkusb is intended to be run from a live USB system
Here is the info page for mkusb for creating persistent live USB drives. mkusb itself needs to be run from a live Linux system. The reason for this is that some of the packages required by mkusb may conflict with one another and not be able to be installed together on a non-live system.
Create a regular live USB system for running mkusb
So first make a regular (non-persistent) live system. You can use any of the live systems mentioned above for this. If you’re on Windows a really great tool for making a live USB system from a downloaded ISO file is Universal USB Installer.
Note: Other persistent systems are not the same
Universal USB Installer and some other USB installers are also able to create persistent live systems for Debian and Ubuntu-related distros however the type of persistent system is different from what mkusb creates. It requires extra setup be done to define what parts of the filesystem are persistent. This is not done by default. It also requires that every time the system is booted a command-line option be manually entered in order to actually boot into persistent mode, which is not the default boot mode. This means that if you’re running such a persistent live USB system as a server and there’s a power failure and the system reboots, it will not reboot into persistent mode.
mkusb avoids all this. The persistent live USB systems it creates are truly persistent and do not require any boot options. They will boot by default into a persistent mode (and may also offer other modes as available options).
Install mkusb
To install mkusb once you are up and running on a live system just open a terminal window and add the PPA archive for it:
sudo add-apt-repository ppa:mkusb/ppa sudo apt update sudo apt install mkusb mkusb-nox usb-pack-efi
Run mkusb
Once installed run the command to create a persistent live USB drive:
mkusb
Make sure you have a target USB drive inserted in the system and an ISO image of a live system available. For inexpensive but good-performing basic workhorse drives see this post. pendrivelinux.com also has a section on recommended flash drives.
Some notes on setting up a persistent live USB system
Below are some notes on system configuration for a persistent live USB system.
Remember that you’re running off a USB flash drive so be reasonable with what you do. I prefer Lubuntu because it is a lightweight distro yet looks good and has very good defaults.
Set up Bluetooth
There should be a menu option under settings or preferences for Bluetooth configuration. If you want to use Bluetooth audio install pulseaudio-module-bluetooth and then run sudo killall pulseaudio and also service bluetooth restart
On some systems if you don’t see the Bluetooth configuration menu option in the menu you may have to install blueman to get the utility (the name of the utility is blueman-manager but once installed it should automatically show up in the menu under either Settings or Preferences).
Clean up after upgrading packages to save space
If you run apt update && apt upgrade to upgrade the system you can wipe the upgrade package cache afterwards with apt-get clean.
Shut stuff off you don’t need for better performance
If you’re using the system as a server then after doing setup it is recommended to log out of the window manager and then stop the X server from a terminal using sudo service lightdm stop Disable it completely with systemctl disable lightdm This assumes the lightdm display manager is installed. You can use this command to see what display manager is installed: dpkg -l |grep ‘dm ‘
Setting the hostname via /etc/rc.local
Because /etc/hostname and fstab are not persistent you can use the following method to to set the hostname and add entries to fstab:
Edit /etc/rc.local and comment out the line:
exit 0
and add the following lines to set the hostname:
myHostname="server1" hostname $myHostname echo $myHostname > /etc/hostname sed -ri -e "s/(127.0.1.1).+/\1 $myHostname/" /etc/hosts
Mounting drives and setting up Samba for network file sharing
Below are example lines of how to set up custom mounts and add entries to fstab. The following lines would go in /etc/rc.local (note that the fstab entry below is only an example):
cat >> /etc/fstab <<EOF # don't overwrite fstab, just append UUID="76B5712A37C1F503" /mnt/mystorage ntfs-3g noatime,rw 0 0 EOF mount /mnt/mystorage
Install Samba with apt install samba and then edit /etc/samba/smb.conf to add stanzas for drives you want to share on your network. Here’s an example stanza to share a drive named mystorage:
[mystorage]
comment = /mnt/mystorage
browseable = yes
path = /mnt/mystorage
writeable = yes
You also need to add a user for Samba:
useradd smith smbpasswd smith -a
And also set the perms of the mount so that it’s accessible to users:
chmod 777 /mnt chmod 777 /mnt/mystorage
You can test the config of Samba with the testparm command and (re)start Samba with:
/etc/init.d/samba restart
or with:
service samba restart
or with:
service smbd restart service nmbd restart
This works but there may be limitations
In trying out multiple live USB systems I’ve noticed that performance can degrade pretty quickly under certain conditions. I’ve tried to upgrade packages on systems built of older images, which required that many packages had to be downloaded and installed, and the upgrade process eventually came to a crawl and the system became useless. This was my experience with an older Kali image which required a large number of packages to upgrade and also with Knoppix. Using a current daily live image based on a distribution’s stable branch will drastically reduce the number of packages that need to be upgraded and is highly recommended.
Lubuntu rocks
With Lubuntu 16.04 Xenial daily live I’ve had good success with setting up and running a full-fledged server that is performing well.
Even if not used as a server, creating a persistent live Linux USB flash system is very useful and you can at least use it to run a computer which might otherwise have become unusable. You can take it with you and because it is persistent any configuration changes and settings you make will be saved.
And as a server it’s amazing that you can have everything conveniently on a USB flash drive which costs only a few dollars. You can customize the server build however you want and easily clone copies using a tool like Clonezilla or using the dd command. The idea of transcending large, bulky, failure-prone hard drives in favor of inexpensive, small USB flash drives is really cool.
References:
help.ubuntu.com: How to install mkusb in Ubuntu 14.04.1 LTS and newer versions
help.ubuntu.com: Persistent live systems
help.ubuntu.com: mkusb/persistent/lubuntu
launchpad.net: MKUSB team
wiki.ubuntu.com: LiveUsbPendrivePersistent
pendrivelinux.com Really good general resource for Live USB systems.
Leave a Reply