This guide contains useful tips for cloning and provisioning KVM/QEMU virtual machines under Debian. Since this guide uses libvirt it may also work with other VM backends such as Virtualbox although there is no guarantee.
Required Components
This guide relies on the following tools:
virt-clone provided by the virtinst Debian package
virt-sysprep provided by the libguestfs-tools Debian package
virsh provided the libvirt-clients Debian package
In addition, to create the template virtual machine, you can use either:
virt-install tool to create a VM from the command-line, provided by the virtinst Debian package
-or-
virt-manager graphical tool to create and manage VMs, provided by the Debian virt-manager package
Virtual Machine Uses
Being able to easily create virtual machine clusters has a lot of uses. Systems administrators can use VMs to set up clusters of machines for testing. Likewise students can use clusters of VMs for learning.
For example Saltstack recommends to use salt-vagrant-demo which uses Vagrant to provision a small cluster of machines for learning. But such a cluster can easily be created and managed using the tools below.
Other possible projects include setting up HAProxy clusters, database clusters such as MariaDB/Galera, an Nginx proxy with backend servers, etc.
If you have a machine with a good amount of RAM (16G recommended, 32G is better) and fast storage (NVMe storage is recommended) it is no problem to run a small VM cluster or even several.
Important Reference
The Debian KVM wiki page is an important resource for information on KVM.
Set up a Network Bridge on the VM Host
On the VM host machine you will likely want to set up a network bridge as that is the most useful way to provide networking to VM guests. VM guests that connect to the bridge interface will appear as regular hosts on the same network as the VM host.
The KVM wiki page above has an example for how to configure a network bridge with ifupdown via /etc/network/interfaces.
Instead of ifupdown you may use systemd’s networkd directly to create a bridge which I found to be very easy. Note that if you are going to configure your interfaces with networkd then you should uninstall ifupdown or at least remove any interfaces that are configured via networkd from the ifupdown configuration.
Finally, when you configure the template VM below be sure to select the bridge device of the host machine for the network configuration.
Create a Template VM
Create a template virtual machine either using the virt-manager graphical VM management tool or with the virt-install command (refer to the Debian KVM wiki for examples). The template VM will be used to create clones.
The amount of configuration done to the template VM depends upon how it is to be used. In my case, I want the template VM to be as close as possible to the minimal Debian installation default. Essentially it is like a raw cloud VM image and it saves a lot of time from having to run through the full Debian installation process each time I want to create a new VM. However, if you’re creating a server template for an organization or other purpose, you may want to include a baseline of tools, utilities, and/or other custom configuration options.
For me the most important baseline after a minimal Debian install is:
apt install openssh-server openssh-client rsync sudo vim vim-tiny- nano-
This will install OpenSSH, rsync, sudo, and regular Vim and will remove vim-tiny and nano. If you create any users remember to add them to the sudo group:
addgroup user sudo
The Python example near the bottom of this guide also configures Saltstack on the cloned VMs, thus it requires that salt-minion be installed in the template VM. For info on installing it see here.
In the process outlined below we clone a VM from the template VM and make sure that the clone is a assigned a unique IP address and set of SSH host keys and contains no residual stuff from the template VM such as log files or temp files.
If you only want the guest VMs to use DHCP for their network address then you don’t need to do anything because that is the Debian installation default configuration for the primary network interface.
If however you want to assign a static IP address to the primary network interface of VM guests then on the template VM edit /etc/network/interfaces and assign a static IP. For example:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
allow-hotplug enp1s0
iface enp1s0 inet static
address 192.168.0.210/24
gateway 192.168.0.1
Cloning and Provisioning
Let’s say a a template VM named vm-template has been created. Use virt-clone to create a new VM named vm1:
virt-clone --original vm-template --name vm1 --auto-clone
Introducing virt-sysprep
From the virt-sysprep man page:
Virt-sysprep can reset or unconfigure a virtual machine so that clones can be made from it. Steps in this process include removing SSH host keys, removing persistent network MAC configuration, and removing user accounts. Virt-sysprep can also customize a virtual machine, for instance by adding SSH keys, users or logos. Each step can be enabled or disabled as required.
The default operations performed by virt-sysprep are intended to wipe clean the newly provisioned VM by clearing log, temp, and backup files, SSH keys, and any unique machine or system IDs. The idea is to start with a completely fresh system. For example, you would not want a newly-provisioned guest VM to have the same SSH keys as the template machine, nor would you want it to contain any log or temporary files from the template VM.
Use virt-sysprep to provision the newly-cloned VM:
virt-sysprep -d vm1 --hostname vm1 --edit /etc/network/interfaces:'s/192.168.0.210/192.168.0.215/' --firstboot-command 'dpkg-reconfigure openssh-server'
To see a full list of everything that virt-sysprep clears run virt-sysprep --list-operations. All the operations with a ‘*’ are enabled by default and will be run. For example, to get a list of only the default enabled options:
virt-sysprep --list-operations | egrep ' \* '
Beware with virt-clone that if any operation is explicitly enabled then all other default enabled options will be automatically disabled. ‘Operation’ with virt-clone has a specific meaning. If you intend to explicitly enable any operation, you must also explicitly enable all other operations you want performed. In the example above we do not explicitly enable any operations and thus use the default enabled ones.
For Debian VMs we can tell virt-sysprep to generate a new set of SSH host keys via the --firstboot-command 'dpkg-reconfigure openssh-server' flag.
The –edit flag is extremely useful and is used here to edit the contents of /etc/network/interfaces in the VM. It changes the static IP address from the template’s value of 192.168.0.210 to an assigned value of 192.168.0.215. –edit uses a Perl regex to accomplish its task.
Python Script
Here is a Python script to provision VMs from a VM template machine. This script requires that salt-minion is installed in the template VM. In addition to configuring the hostname and IP address it also sets the IP address of the salt master in the salt minion configuration as well as the minion’s ID.
#!/usr/bin/python3
#
# This script provisions new KVM virtual machines from a template virtual
# machine
#
import shutil
virtclonecmd = shutil.which('virt-clone')
if not virtclonecmd:
sys.stderr.write("\nThe virt-clone binary was not found in $PATH\n"
"Please make sure the virtinst package is installed (Debian)\n"
"or that you have the binary otherwise installed.\n"
"This program will now exit.\n")
exit(1)
virtsysprepcmd = shutil.which('virt-sysprep')
if not virtsysprepcmd:
sys.stderr.write("\nThe virt-sysprep binary was not found in $PATH\n"
"Please make sure the libguestfs-tools package is installed (Debian)\n"
"or that you have the binary otherwise installed.\n"
"This program will now exit.\n")
exit(1)
from subprocess import run
# Template VM hostname and IP address
tName = 'bullseye-base'
tIP = '192.168.0.120'
# IP address of the salt master
saltmst = '192.168.0.10'
# hostname and IP address of VMs to create
hosts = {
'bullseye1': '192.168.0.125',
'bullseye2': '192.168.0.126'
}
# For each host and IP defined above, create a clone VM for the host
# and assign the IP. virt-sysprep ensures the cloned image is fresh
# and doesn't contain leftover log, temp, or other files or data from
# the template VM from which it is cloned.
for host, ip in hosts.items():
# run() accepts a list as its first argument
# each item in the list is a token from the string of the command + arguments
# use Python f-strings (template strings) where applicable
# list of virt-clone command and arguments
clonecmd = [
f'{virtclonecmd}',
'--original', f'{tName}',
'--name', f'{host}',
'--auto-clone'
]
# run the command
run(clonecmd)
# list containing virt-sysprep command and arguments
sysprepcmd = [
f'{virtsysprepcmd}',
'-d', f'{host}',
'--hostname', f'{host}',
'--edit', f'/etc/network/interfaces:s/{tIP}/{ip}/',
'--edit', f'/etc/hosts:s/{tName}/{host}/g',
'--edit', f'/etc/salt/minion:s/#master: salt/{saltmst}/',
'--edit', f'/etc/salt/minion_id:s/{tName}/{host}/',
'--firstboot-command', 'dpkg-reconfigure openssh-server'
]
# run the command
run(sysprepcmd)
Launch and Manage VMs with virsh
The VMs can now be launched and managed with virsh:
virsh list --all # list all VMs virst start bullseye1 # start bullseye1 VM virst shutdown bullseye1 # gracefully shutdown bullseye1 virsh destroy bullseye1 # force shutdown virsh undefine --remove-all-storage bullseye1 # completely remove bullseye1 (use with caution)
Going Further
There is potentially a lot more that can be done. One possibility is to set up a VDE virtual switch for a VM cluster.
virt-sysprep has a lot of options to explore.
References and Thanks
libguestfs.org/virt-sysprep.1.html
https://wiki.debian.org/QEMU (has useful section on setting up VDE)
cyberciti.biz: How to reset a KVM clone virtual Machines with virt-sysprep on Linux
Leave a Reply
You must be logged in to post a comment.