This document was taken from a document I wrote for my company's internal wiki. Several concepts in it are specific to our implementation (such as the config system), while other concepts are just basic UML concepts. Hostnames have been changed to protect the guilty.

User Mode Linux Guide

The uml1.domain.company.com server in the Reno office is a User Mode Linux (UML) host server. User Mode Linux (not to be confused with Unified Modeling Language, the more popular definition of "UML") is a virtualization technology that lets you run multiple virtualized servers on a single physical server.

Table of contents

Automatically Started UML Guests

Any guest that has ONBOOT="yes" in the guest config file (more on this later) will be started by the startup script (/etc/init.d/uml).

Accessing the Guest's Console

Each guest is started in an individual screen session. To see what screen sessions are currently running:

host# screen -ls
There is a screen on:
        5112.test1      (Detached)
        5240.test2      (Attached)
2 Sockets in /var/run/screen/S-root.

To attach to a detached screen session (for example, test1):

host# screen -r test1

If a screen session is currently attached, most likely it's because another user is attached to it. However, to force a detach from the other person and an attach for you:

host# screen -d -r test2

To detach yourself from a currently-attached screen, type Ctrl-a, then d.

Creating a New Guest

Create the Guest Directory

Guests are stored in the directory /home/uml/guests. First, create a new guest directory. The guest directory name is the guest's identifier, and should be the short name of intended hostname of the new guest. For example, if you want to create a guest named newguest.domain.company.com:

host# mkdir /home/uml/guests/newguest

Copy Base OS Image

Base (skeleton) OS images are stored in /home/uml/skel. If you want Sarge as a base OS, you would do:

host# cp /home/uml/skel/sarge.img /home/uml/guests/newguest/os.img

Expand the OS Image File

Next, the image has to be expanded from the base size (usually around 500MB). If you want a 10GB partition image:

host# resize2fs -p /home/uml/guests/newguest/os.img 10G

Create a Swap Image

Create and format a new swap partition, say 256MB:

host# dd if=/dev/zero of=/home/uml/guests/newguest/swap.img bs=1M count=256
host# mkswap /home/uml/guests/newguest/swap.img

Create a Config File

Next, create a config file at /home/uml/guests/newguest/uml.conf. Here is a sample file:

ONBOOT="yes"
KERNEL="linux-2.6-um"
OPTS="mem=192M ubda=os.img ubdb=swap.img root=/dev/ubda devfs=nomount"
NETWORKING="yes"
MAC=""

ONBOOT

Required
Default: ONBOOT="no"
Sample: ONBOOT="yes"

If set to "yes", the guest will be started next time the UML server itself is booted.

KERNEL

Required
No default
Sample: KERNEL="linux-2.6-um"

The UML kernels (as indicated by a trailing "-um", such as linux-2.6.14.3-bs2-um) are stored in /home/uml/kernels. You do not need to specify the path, just the filename. The symlinks linux-2.4-um and linux-2.6-um point to the most recent respective kernels.

OPTS

Required
No default
Sample: OPTS="mem=192M ubda=os.img ubdb=swap.img root=/dev/ubda devfs=nomount"

This is where most of the UML options are placed. mem is the amount of memory to allocate to the guest (the trailing "M" is important). ubda should be set to the OS image (which should be named os.img). ubdb should be set to the swap image (swap.img). You can specify up to 8 images (up to ubdh). root in most cases must be set to /dev/ubda. devfs=nomount is needed on kernel 2.6.12 and below (including 2.4 kernels), but does not hurt anything if it's used on later kernels, so just go ahead and always specify it. Any other regular Linux kernel options can be appended, such as "single init=/bin/sh".

NETWORKING

Optional
Default: NETWORKING="no"
Sample: NETWORKING="yes"

If set to "yes", an eth0 device is created that is bridged to the normal network. By default, the OS images request a DHCP address, and will receive an address within 192.168.0.0/25. You can set up networking in the guest like you would on a physical server.

MAC

Optional
Default: (randomly assigned)
Sample: MAC="00:08:93:E5:24:87"

Only used if NETWORKING is enabled. If not specified, a pseudo-random, pseudo-permanent address will be created automatically. This random MAC address is based on the guest ID and UML server hostname, and shouldn't change unless either the guest ID or the UML server hostname changes. However, if you are creating a permanent guest, you should assign it a permanent MAC address:

host# randmac 
00:08:93:65:f5:b7

MAC addresses NEED to be unique within a network (and theoretically should be unique globally). The 00:08:93 prefix is chosen because it is designated "private" by the IEEE, and should not conflict with any existing physical network cards.

BRIDGE

Optional
Default: BRIDGE="br0"
Sample: BRIDGE="br0"

Only used if NETWORKING is enabled. This is the bridge the guest's networking associates with. You do not need to be concerned with this.

FORCEHALT

Optional
Default: FORCEHALT="no"
Sample: FORCEHALT="yes"

FORCEHALT should be enabled when the guest's /etc/inittab has not been modified to do a halt instead of a reboot when its CAD handler is run. By default, the UML host shutdown script will send guests a CAD signal, and wait for the guest to gracefully halt (up to 5 minutes). But without the guest modification, the guest will simply gracefully reboot. If FORCEHALT is enabled, the shutdown script will simply send a force halt signal rather than a CAD.

Manually Starting a Guest

If you specified ONBOOT="yes" above, the guest will boot the next time the UML server itself boots. However, you probably want to boot it now. Use the uml_start command:

host# uml_start newguest

The guest will start in the screen session, which you will automatically be attached to. Again, to detach, hit Ctrl-a, then d. If you don't want to automatically attach, use the -d flag:

host# uml_start -d newguest
A screen session for newguest has been created (screen -r newguest).

Once in the Guest...

Once you are inside the guest, it'll look nearly identical to a physical Linux server. Skeleton images are standard Debian installs, with the task-company, task-company-secure, and task-company-www tasks installed, and a www user created. You can reboot/halt the server using the same shutdown/reboot/halt/poweroff commands, just like on a physical server. When a UML guest is halted, the screen session will simply disappear, and you will be dropped down to a command prompt.

guest# cat /proc/cpuinfo 
processor       : 0
vendor_id       : User Mode Linux
model name      : UML
mode            : skas
host            : Linux uml1.domain.company.com 2.6.14.4-skas3-v8.2
  #1 Mon Dec 12 14:24:55 PST 2005 i686
bogomips        : 3486.51
guest# uname -a
Linux basesarge.company.com 2.6.14.4-um-2-bs2 #2 Mon Dec 12 20:03:20 PST 2005 i686 GNU/Linux

basesarge.company.com is the hostname in the skeleton Sarge image. You'll probably want to change that too.

guest# cat /proc/cmdline 
mem=64M ubda=os.img ubdb=swap.img root=/dev/ubda
  devfs=nomount eth0=tuntap,test1,00:08:93:e5:26:87

The eth0=tuntap,test1,00:08:93:e5:26:87 section is automatically computed when you have NETWORKING="yes" specified in the config file.

guest# lsmod
Opening /proc/modules: No such file or directory

The UML guest kernels are monolithic, as is UML's nature. Nothing to worry about.

guest# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/ubda             1.5G  433M 1003M  31% /
tmpfs                  31M  4.0K   31M   1% /dev/shm

Recommended New Guest Modifications

Password

The default root password on a new base install is "the" default throwaway password we always use. Change it immediately once you deploy a new guest.

guest# passwd root

Hostname

The default hostname on a newly-deployed guest is either basewoody.company.com or basesarge.company.com.

guest# hostname newguest.domain.company.com
guest# echo newguest.domain.company.com >/etc/hostname

SSH Host Key

The SSH host key is generated during install, and if it isn't changed during each deployment, there is a slight security risk.

guest# rm /etc/ssh/ssh_host_*key*
guest# dpkg-reconfigure ssh

uml_mconsole

From the UML host, you can launch a utility called uml_mconsole to do a couple tasks:

host# uml_mconsole newguest
(newguest) help
OK Commands: 
    version - Get kernel version 
    help - Print this message 
    halt - Halt UML 
    reboot - Reboot UML 
    config <dev>=<config> - Add a new device to UML;  
        same syntax as command line 
    config <dev> - Query the configuration of a device 
    remove <dev> - Remove a device from UML 
    sysrq <letter> - Performs the SysRq action controlled by the letter 
    cad - invoke the Ctl-Alt-Del handler 
    stop - pause the UML; it will do nothing until it receives a 'go' 
    go - continue the UML after a 'stop' 
    log <string> - make UML enter <string> into the kernel log
    proc <file> - returns the contents of the UML's /proc/<file>
    stack <pid> - returns the stack of the specified pid

Additional local mconsole commands:
    quit - Quit mconsole
    switch <socket-name> - Switch control to the given machine
    log -f <filename> - use contents of <filename> as UML log messages
    mconsole-version - version of this mconsole program
(newguest) 

I won't go through all of the commands, but:

cad

Sends a CAD signal to the guest. This is the same as doing a three finger salute on a physical server. However, on Company UML guests, the CAD handler has been modified from the expected "reboot" funcionality to halt instead. See below for why this is done.

halt

IMMEDIATELY halts the guest. This is the same as pulling the power cord on a physical server.

reboot

IMMEDIATELY reboots the guest. This is (almost) the same as pushing the reset button on a physical server. The difference is the orignal guest kernel you booted is simply reloaded from memory. The net effect is if you upgrade the guest kernel, you must run "halt" from within the guest, then start it manually again for the changes to take effect.

UML and Rebooting

By default, a UML guest functions almost exactly like a real server, with regards to rebooting and halting. However, there are a couple things to be aware of:

  • When you reboot, the same kernel you loaded is reloaded from memory, even if you upgraded the kernel.
  • By default, there is no way to gracefully shut down the guest from the host. This means if the host is shut down/rebooted, the guests are all forcefully killed.

To get around the second item, the guest's /etc/inittab is modified to change what happens when a CAD signal is received by the guest. By distro default, a CAD is interpreted as a request to reboot. Instead, this is changed to a request to halt. This way, the /etc/init.d/uml shutdown section can simply send a CAD signal to the guest and wait for the guest to gracefully shut down. Note that if there is no way to modify the CAD behavior (I can't think of a time when this is actually the case though), you can set FORCEHALT="yes" in the guest's uml.conf to tell the host shutdown script to not bother with a CAD, and instead immediately kill the guest. Of course, if that is the case, you must remember to gracefully halt the guest from within the guest itself before you shutdown/reboot the host.

Converting a Physical Server to a UML Guest

A physical host can be converted to a UML guest rather easily. Set up the new guest as detailed earlier in this guide, but instad of copying a skeleton OS image, you will create and format a new image:

host# dd if=/dev/zero of=os.img bs=1M count=15360
host# mke2fs -j os.img

Create a temporary mount point on the host and mount the image loopback:

host# mkdir /mnt/newguest
host# mount -o loop os.img /mnt/newguest

Copy the data from the physical server into the mount point:

host# rsync -avzP -e ssh --numeric-ids --exclude=/proc --exclude=/sys root@physhost:/ /mnt/newguest/
host# mkdir /mnt/newguest/{proc,sys}

Chroot into the mount point:

host# chroot /mnt/newguest

Create the UML device entires:

host-chroot# cd /dev
host-chroot# cat >MAKEUBD <<EOT
#!/bin/bash
device=ubd
major=98
minor=0
for u in a b c d e f g h; do
  dev=$device$u
  mknod $dev b $major $minor
  chmod 660 $dev
  chgrp disk $dev
  for i in $(seq 1 15); do
    mknod $dev$i b $major $(($minor + $i)) 
    chmod 660 $dev$i
    chgrp disk $dev$i
  done
  minor=$(($minor + 16))
done
EOT
host-chroot# bash MAKEUBD
host-chroot# rm MAKEUBD

Edit /etc/fstab, remove all instances of hd*/sd*, and replace with:

/dev/ubda       /               ext3    defaults        0       1
/dev/ubdb       none            swap    sw              0       0

Edit /etc/inittab, modify the "getty" section to comment out all ttys, and in its place add tty0. You should be left with something like this:

0:2345:respawn:/sbin/getty 38400 tty0
#1:2345:respawn:/sbin/getty 38400 tty1
#2:23:respawn:/sbin/getty 38400 tty2
#3:23:respawn:/sbin/getty 38400 tty3
#4:23:respawn:/sbin/getty 38400 tty4
#5:23:respawn:/sbin/getty 38400 tty5
#6:23:respawn:/sbin/getty 38400 tty6

Additionally, edit the ctrlaltdel handler and replace the reboot flag with the halt flag. You should be left with something like this:

ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -h now

Edit /etc/securetty, and add tty0 as a secure tty:

# Standard consoles
tty0
tty1
tty2
tty3
...

Exit the chroot and umount the image:

host-chroot# exit
host# umount /mnt/newguest

At this point, you should have a functional UML guest image. When editing uml.conf, do not simply clone the MAC address from the old physical server's ethernet card, as you may use that physical box sometime in the future, and MAC address conflicts are nasty. Simply generate a new permanent MAC address using the "randmac" utility.

UML Server Information

One day I will be fired or hit by a bus, and someone else will invariably have to maintain the UML server. Administration isn't hard, and is pretty well integrated with Debian, but there are some things to be aware of.

UML Networking

Networking is done between several UML guests by a series of tunnels connected together by an ethernet bridge. First of all, you need the bridge-utils package.

host# apt-get install bridge-utils

Next, look at /etc/network/interfaces:

auto br0
iface br0 inet dhcp
  bridge_ports eth0

This interface can be configured like a normal Debian ethernet interface (IE, static IP instead of DHCP), but notice the bridge_ports. This creates a bridge called br0 that has eth0 bridged to it. Then dhclient is called on br0, not eth0. But since eth0 is bridged to br0, it still finds an IP from the wired network.

Next, when a UML guest boots, a tunnel is created with the same name as the UML guest. This is handled in /usr/local/bin/uml_tunwrap:

BR="$1"
INT="$2"
shift 2

tunctl -t ${INT} >/dev/null 2>&1
ifconfig ${INT} 0.0.0.0 up
brctl addif ${BR} ${INT} >/dev/null 2>&1
"$@"
ifconfig ${INT} 0.0.0.0 down
brctl delif ${BR} ${INT} >/dev/null 2>&1
tunctl -d ${INT} >/dev/null 2>&1

Argument 1 is the bridge to attach to (br0) and argument 2 is the new tunnel to create. All further arguments are treated as a command to execute after creating the tunnel and before tearing it down. Note that the tunnel must be "up", but not have an IP address. Note: tunctl is provided by the uml-utilities package.

host# apt-get install uml-utilities

Memory Usage (tmpfs)

Without any host modification, the only thing in the guest that is run out of "real" memory is the kernel itself. All other memory usage on the guest is mapped to the host via a hidden, deleted file in /tmp. The temp file does not show up in a "ls /tmp", but "lsof" can see it:

linux-2.6 21423     root  DEL       REG       0,13                 230878 /tmp/vm_file-aoBxao
linux-2.6 21423     root    3u      REG       0,13    67108865     230878 /tmp/vm_file-aoBxao (deleted)

You can see how this can be a problem. The net effect is most memory used in the guest is actually treated the same as "swap" on the host. How do we solve this? tmpfs to the rescue! tmpfs is essentially a dynamic ramdisk, and is mounted like a normal filesystem. Put this in /etc/fstab:

tmpfs           /tmp            tmpfs   size=5000M      0       2

Then simply mount /tmp (though it would be a good idea to make sure no processes are using /tmp first). Note the size field. tmpfs works the same as the normal Linux VM memory system. If you create a file in a tmpfs filesystem, it is initially placed in memory. After awhile of non-usage, it is swapped out to disk using the same swap as the host. Therefore, the "size" field on the mount must be less than the combined memory+swap of the host, but can be more than simply the host's memory. In this case, the host has 1GB main memory and 6GB of swap, so the 5GB tmpfs is well within the limit (7GB). The remaining 2GB is a nice buffer zone for the host itself.

Swap Partitions

While not UML-specific, remember that Linux has a limit of 2GB per swap partition. If you want more than 2GB swap, you must use multiple swap partitons:

/dev/sda2       none            swap    sw              0       0
/dev/sda5       none            swap    sw              0       0
/dev/sda6       none            swap    sw              0       0

Each one of those partitions are 2GB, giving a total of 6GB swap.

Other Scripts/Utilities

/usr/local/bin/uml_start

This is where the config file is assembled and the screen session is created. The screen session calls uml_tunwrap (as explained above).

/etc/init.d/uml

This script searches through config files in /home/uml/guests for configs with ONBOOT="yes", and starts them during boot. It will also forcibly kill any running guests during shutdown. The init script should be run at the end of boot and the beginning of shutdown.

host# update-rc.d uml defaults 99 00

UML Kernels

Host Kernel SKAS Patch

The host does not need to be patched in order to run UML guests. Without any patches, the guest kernel runs in "Tracing Thread" (TT) mode. However, this mode is slow and slightly insecure. So, we patch the host kernel to add Separate Kernel Address Space (SKAS) support, which increases guest performance by roughly an order of magnitude. SKAS patches are available from a UML developer called Blaisorblade (http://www.user-mode-linux.org/~blaisorblade/patches/).

Guest Kernels

UML guests are available for both 2.6 and 2.4 kernels. Guest (-um) support was added to the vanilla Linux kernel in 2.6.9, but there are many backports available, also at Blaisorblade (http://www.user-mode-linux.org/~blaisorblade/patches/)'s patches site. The guest patches take the name linux-2.N.NN-bsN-um, where -bsN is the patch version. You will also notice that there are -bs patches for kernels beyond 2.6.9. These are optional, but recommended stability patches for things not present in the vanilla kernel.

Your best bet when upgrading a guest kernel is to use the config file from an older guest kernel, and do a make oldconfig. The config file is actually built into the kernel, and can be extracted by "linux --showconfig", without having to boot the guest kernel.

Here is a sample set of commands for building a 2.4 kernel. It is important to note that any kernel "make" command must have "arch=UM".

host# wget http://www.kernel.org/pub/linux/kernel/v2.4/linux-2.4.28.tar.bz2
host# wget http://www.user-mode-linux.org/~blaisorblade/\
  patches/guest/uml-2.4.28-bs2/uml-2.4.28-bs2.patch.bz2
host# tar jxf linux-2.4.28.tar.bz2
host# cd linux-2.4.28
host# bunzip2 -c ../uml-2.4.28-bs2.patch.bz2 | patch -p1
host# /path/to/linux-2.4-old-um --showconfig > .config
host# make ARCH=um oldconfig
host# make ARCH=um dep
host# make ARCH=um linux
host# strip linux

Kernel 2.6 is similar, except the "make ARCH=um dep" command is not specified.