Starting new domains on Xen 4.0.1 from command line

So, I have issues with my installation of Xen4.0.1 as the graphics driver keeps crashes during boot up, resulting in a hung machine. For now, I am using a work around suggested by my colleagues David Wolinsky and Pierre St. Juste, namely to disable the graphics display module (using "mv /etc/init/gdm.conf /etc/init/gdm.disabled") and hence I only have command line access to my domain-0 at this point.

At this point, Domain-0 boots up just fine, and xm and xend seem to be working. I now need to start Virtual domains using only the command line. (Later I will explore ways to do the same thing using APIs like Virt-Manager, or its console variant Virt-install)

Creating an Image

The first step is to create an image that will hold the user-domain (domU) virtual disk. This can be just a raw zero-filled file and so we can use the dd command here:

dd if=/dev/zero of=/var/lib/xen/images/domain1.img oflag=direct bs=1M seek=2047 count=1


The above command will create /var/lib/xen/images/domain1.img file of 2048MB although the actual data blocks are allocated in a lazy fashion. To reserve all the data blocks right away, get rid of the seek option from above. So do:

dd if=/dev/zero of=/var/lib/xen/images/domain1.img oflag=direct bs=1M count=2048


This will avoid data block allocation problems if the volume that holds the image is full. It is important to check that the image has the correct security context (permissions), otherwise access to the virtual disk will be denied to the user domain system. You can check this as follows:

ls -l /var/lib/xen/images/domain1.img
-rw-r--r-- 1 root root 2147483648 2010-10-13 10:23 /var/lib/xen/images/domain1.img


Preparing a Xen configuration file for the installation
Xen uses a configuration file per domain. The configuration for the domains is different because we have to provide installation kernels, initial ram-disk and possibly some boot parameters. The domainU installation initrd image and kernel for my machine architecture (64-bit x86) can be downloaded from this page. You can put them in some sensible directory and name them appropriately.

root@Xen-Open:~# mkdir /home/user/x86_64_domU_images
root@Xen-Open:~# cd /home/user/x86_64_domU_images
root@Xen-Open:~/x86_64_domU_images# wget http://archive.ubuntu.com/ubuntu/dists/lucid/main/installer-amd64/current/images/netboot/xen/initrd.gz
root@Xen-Open:~/x86_64_domU_images# wget http://archive.ubuntu.com/ubuntu/dists/lucid/main/installer-amd64/current/images/netboot/xen/vmlinuz
root@Xen-Open:~/x86_64_domU_images# ls
initrd.gz vmlinuz
root@Xen-Open:~/x86_64_domU_images# gunzip -c initrd.gz > initrd.img
root@Xen-Open:~/x86_64_domU_images# ls
initrd.gz initrd.img vmlinuz


In this example, the kernel and the initrd image were named /home/user/x86_64_domU_images/vmlinuz and /home/user/x86_64_domU_images/initrd.img respectively. With the images in place, we can now create the installation configuration file named /etc/xen/domain1.cfg

kernel = "/home/user/x86_64_domU_images/vmlinuz"
ramdisk = "/home/user/x86_64_domU_images/initrd.img"
name = "domain1"
memory = "256"
disk = [ 'file://var/lib/xen/images/disk.img,xvda,w', ]
vif = [ '' ]
dhcp="dhcp"
netmask="255.255.240.0"
gateway="10.5.144.1"
root="/dev/xvda ro"
vcpus=1
extra="4"
on_reboot = 'destroy'
on_crash = 'destroy'

Couple of points to note:

1. The netmask and gateway values to be used can be determined by using the route command on your control domain. If you encounter problems in getting the network to start, you could try using methods suggested here.



2. The [ 'file://var/lib/xen/images/disk.img,xvda,w', ] line here does not work if you use "tap:aio" instead of "file". Also for some reason, if you replace "xvda" with "xvda1" or something else, it does not work either. If you do any of those things you will get the following error:
   

   
Error message:

Error informing the kernel about modifications to partition
/dev/xvda1p1 -- Invalid argument. This means Linux won't know about
any changes you made to /dev/xvda1p1 until you reboot -- so you
shouldn't mount it or use it in any way before rebooting.


Once your config file is ready, you can start creating a domain with:

sudo xm create -c /etc/xen/domain1.cfg


And now, to access your domain, do:

sudo xm domain1


This will open an interactive installation window. Everything is self explanatory in general. If you get an error to the effect that the Ubuntu repository cannot be reached, check your control domain network and here is the fix.

Once your domain is properly created it will ask permission to reboot and will automatically get destroyed in the process because of the last two lines in the configuration command line, namely:

on_reboot = 'destroy'
on_crash = 'destroy'


So now you need to change your configuration file to the following:

name ="domain1"
memory ="256"
disk =[ 'file://var/lib/xen/images/disk.img,xvda,w', ]
vif=[ '' ]
dhcp="dhcp"
netmask="255.255.240.0"
gateway="10.5.144.1"
vcpus=1
on_reboot="restart"
on_crash="restart"
bootloader="/usr/bin/pygrub"



And now just use the xm create command to create your new domain. Finally in order to automatically start and stop your domains when domain-0 starts, move the location of the configuration file to /etc/xen/auto.

At this point, it is a good idea to create a copy of your virtual machine image. To create new virtual machines using the same image all you have to do is to change the configuration file (name, location of the disk image etc) to point to the copy and start the new machines... waaa la !

References:

1. Creating and installing CentOS 5 Xen domain instance
2. Error during guided partitioning

How to install Xen4.0 on Ubuntu 10.04 LTS

This post provides documentation about how to get Xen4.0 (the opensouce version) up and running on an Ubuntu 10.04 LTS (Long Term Support) 64-bit machine.

Start of with a fresh new installation of Ubuntu 10.04 LTS. Install the required packages first:

apt-get install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial build-essential make gcc libc6-dev zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev libjpeg62-dev iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml libx11-dev

On a 64-bit machine, you will also need this additional package:

apt-get install gcc-multilib

The next step is to get Xen4.0.1

cd /usr/src/
wget http://bits.xensource.com/oss-xen/release/4.0.1/xen-4.0.1.tar.gz

Un-tar and make:

tar xf xen-4.0.1.tar.gz
cd xen-4.0.1
make xen
make tools
make stubdom

At this point you should have the Xen hypervisor/tools binaries in "dist/" directory ready for installation. Now run:

make install-xen
make install-tools
make install-stubdom


The next step is to download and compile a dom0 kernel, and to update the grub file. We can do this using the following commands:

make prep-kernels


This will create a build-linux-2.6-pvops_x86_64 directory in /usr/src/ which has by a .config file in it. You need to modify the .config file to suit your requirements. You can have a look at mine here. (Source Boris Derzhavets).
You could use make menuconfig from inside the build-linux-2.6-pvops_x86_64 to make alterations to your .config file.

cd build-linux-2.6-pvops_x86_64
make menuconfig
make
chmod g-s /usr/src/ -R
make deb-pkg
dpkg -i ../linux-image-2.6.32.21-xen_*.deb
depmod 2.6.32.21-xen
update-initramfs -c -k 2.6.32.21-xen


At this point enable Xend at boot:

update-rc.d xend defaults 20 21
update-rc.d xendomains defaults 21 20


When I typed in the last two lines, I got warning messages saying:

update-rc.d: warning: xend start runlevel arguments (2 3 4 5) do not match LSB
Default-Start values (3 4 5)
update-rc.d: warning: xend stop runlevel arguments (0 1 6) do not match LSB
Default-Stop values (0 1 2 6)


Apparently these messages are benign and should not cause any issues in the working of Xen Daemon. Now create the Grub2 entry for your new kernel:

vim /etc/grub.d/40_custom


Following is what I entered to in the 40_custom file. My installation still has issues because the graphics driver keeps crashing, and I think that the grub file has something to do with it.

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.

menuentry 'Xen 4.0.1, Ubuntu 10.4 kernel 2.6.32.21' {
recordfail
insmod ext2
set root='(hd0,1)'
multiboot (hd0,1)/boot/xen-4.0.1.gz dummy=dummy dom0_mem=832M
module (hd0,1)/boot/vmlinuz-2.6.32.21-xen dummy=dummy root=/dev/sda1 nopat ro nomodeset console=tty0 earlyprintk=xen
module (hd0,1)/boot/initrd.img-2.6.32.21-xen
}


Update grub and reboot... And choose the correct grub option from the boot-up menu.

update-grub
reboot


Obtaining GUI access to Xen Domains Using TightVNC

This is a step by step tutorial on how to install a Xen Domain on Xen 3.2 (kernel version 2.6.24-24-xen) and how to get graphical user interface working on it.

I am using a rather ancient version of the Xen kernel because I am short of time at the moment and this is something I have worked with before. But hopefully, I should be able to get the same thing done on a more recent Xen kernel soon.

So to install a new domain from the command line, do:

sudo xen-create-image --hostname=xen_7 --size=1Gb --swap=256Mb --ide --ip=10.5.155.7 --netmask=255.255.240.0 --gateway=10.5.159.255 --force --dir=/home/xen --memory=256Mb --arch=i386 --kernel=/boot/vmlinuz-2.6.24-16-xen --initrd=/boot/initrd.img-2.6.24-16-xen --install-method=debootstrap --dist=hardy --mirror=http://archive.ubuntu.com/ubuntu/


Most of the parameters are self-explanatory. I am using static IP address (10.5.155.7) on my VM. The kernel (vmlinuz-2.6.24-16-xen) and the initial ram disk (initrd.img-2.6.24-16-xen) should be present in your boot directory. Ususally the gateway parameter and the netmask parameter would be the same for your Domain-0 and user domains. To see how much memory is available for the new domain that you are creating use the df or free commands before hand.

Once the new domain has been created you will see a new configuration file in the /etc/xen/ directory. This file needs to be edited a little bit as follows:

# Configuration file for the Xen instance xen_7, created
# by xen-tools 3.8 on Thu Sep 2 19:12:55 2010.
#

#
# Kernel + memory size
#
kernel = '/boot/vmlinuz-2.6.24-16-xen'
ramdisk = '/boot/initrd.img-2.6.24-16-xen'
memory = '256'
vcpu = '2'

#
# Disk device(s).
#
root = '/dev/hda2 ro'
disk = [

'tap:aio:/home/xen/domains/xen_7/swap.img,hda1,w',
'tap:aio:/home/xen/domains/xen_7/disk.img,hda2,w',

The text in red shows the text that has been changed/added. The vcpu parameter can be used if your want your domain to have more than one virtual processor. Later when your machine has booted you can check that this is indeed the case from the /proc/cpuinfo file.

So now we are all set to boot up our machine. In the Domain-0 terminal, type:

sudo xm create /etc/xen/xen_7.cfg


You could monitor the booting up and working of your new domain from Domain-0 using xentop and

sudo xm list


To access your machine, use:

sudo xm console xen_7


Login as root and set your new password using passwd command. If the network on your new domain is not functioning for some reason, look at this post for possible ways to correct the problem.

So at this point you have a functioning domain that you can access via the terminal. But now we want to get GUI access using tightVNC. To do this, a truck load of software needs to be installed. I have used the dpkg --get-selections option to generate a list of all the software that I needed on my machine. Use the dpkg --set-selection command followed by dselect (Select the "install" option on the interface opened by dselect) to install all the packages in the list.

Now the client/remote machine from which you want to access the graphical user interface of your domains needs to have VNCviewer ( sudo apt-get install vncviewer) installed.

On the host (guest domain) terminal, do:

Xvfb :0 -screen 0 800x600x16 &
export DISPLAY=:0
x11vnc -display :0&
xterm&


You can check that the Xvfb and xterm processes have started from the list of running processes. (ps uax)

On the client machine, do vncviewer 10.5.155.6:0 . You should see the GUI interface at this point. To test that the graphics are working, try:

glxgears

Well, that's it! You should see an animation of a bunch of gears rotating on your screen.... Which means that you have a GUI access to your machine!

How to get Virtual Manager to work for a Xen host

So, continuing with the theme of Xen, Virtual Machine Manager or virt-manager for short, is a piece of software that I decided to check out because I wanted graphical access to my virtual machines. A more formal description as given on their website is:

"Virt-manager is a desktop user interface for managing virtual machines. It presents a summary view of running domains, their live performance & resource utilization statistics. The detailed view graphs performance & utilization over time. Wizards enable the creation of new domains, and configuration & adjustment of a domain's resource allocation & virtual hardware. An embedded VNC client viewer presents a full graphical console to the guest domain"

You need to download virt-manager on the machine from which you wish to access your virtual machines. I am using a machine running Ubuntu 10.04 which I will call the client machine in the description below.

My Virtual Machines (VMs) were created on top of Xen 3.2 hypervisor, Earlier I had some issues getting the network up and running on my VMs, the resolution of which I have described in an earlier post. My hypervisor machine which I will call my host machine is running kernel 2.6.24-24-xen.

So here is the procedure that I followed to get virt-manager to work:

1. I used the Synaptic Package Manager (System->Administration->Synaptic Package Manager to download virt-manager and all the dependencies on my client machine. Alternatively, one could also do sudo apt-get install virt-manager


2. On my host machine, I downloaded and installed libvirt-bin and ssh-askpass

   sudo apt-get install libvirt-bin

   sudo apt-get install ssh-askpass



3. Now to access the virt-manager console from my client machine I used Application->System Tools->Virtual Machine Manager


4. On my host machine, I started the libvirtd daemon:

   sudo libvirtd start



5. At this point I tried to connect to my host machine using File->Add Connection...; I chose my specific hypervisor (Xen) and Connection as Remote Tunnel over SSH (I am not sure that this is the one that I was supposed to use, but it did work, eventually!)
   
   At this point, I got the following error on a pop-up on my client machine and on the console of my host machine:

   libvir: Xen Daemon error :internal error failed to connect to xend

   After some effort, the following fix worked:

   
   
   1. Open the xend configuration file

      sudo vim /etc/xen/xend-config.sxp

   
   
   2. In my previous post I had talked about adding a temporary bridge between physical network device and the virtual TAP device. I had to let the xend daemon become aware of this. So under the line:

      (network-script network-bridge)

      I added:

      (network-script 'network-bridge netdev=tmpbridge')

      (To recall what my network device is called, I did a ifconfig. It was the one whose description contains the domain-0's IP.
   
   
   3. I was not done with this file yet. I needed to add the following line somewhere in the file:

      (xend-unix-server yes)

   
   
   4. I saved and closed the file. Restarted the host machine's network (sudo /etc/init.d/networking restart), restarted the xend daemon (sudo xend restart), and finally restarted the libvirtd daemon( sudo libvirtd start)
      

   
   Now when I tried to add the connection using the setting mentioned above, it went through, asked me for a password and I would see a table giving the details of the CPU utilization and some other statistics of my VMs. Following is a screen shot of what I can see so far:
   
   
   

No network on Xen user domain

Today I installed Xen 3.2 for my work and created a guest VM using the this link as a guide. However I found that the network on my VM didn't work. The problem as Pierre explained was that the link between my physical network device and the virtual TAP device was not set up.

This was in spite of the fact that I re-started the xend daemon on my machine and used the static IP address method that was recommended in the blog mentioned above.So it seems that the xm create and xend scripts are not doing what they are supposed to do!

David Wolinsky came up with the following fix:
On the console of your control domain, do the following:

$ brctl addif tmpbridge eth0
$ ifup tmpbridge
$ ifconfig eth0 0.0.0.0 promisc
$ brctl show
$ dhclient tmpbridge

Now when you do an ifconfig on you domain-0, the tmpbridge interface should show up with the IP that was originally assigned to eth0.

And since eth0 is set to "promiscuous" mode, it will accept all network packages coming to it.

Now using the route command find the default gateway used by your domain-0. The following terminal snapshot is provided to clarify which IP I am talking about:


user@Domain0:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.5.144.0 * 255.255.240.0 U 0 0 0 tmpbridge
default 10.5.144.1 0.0.0.0 UG 0 0 0 tmpbridge



In the last line, the default gateway is highlighted in red. Make a note of this IP.
Now open the console of your user domain and check the route:


root@UserDomain:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.5.144.0 * 255.255.240.0 U 0 0 0 eth0


As you can see, in my case the default gateway was not set. To do so use the route add default gw command and the IP we obtained from the default gateway of domain-0.


root@UserDomain:~# route add default gw 10.5.144.1


Verify using route, and then try pinning something:


root@UserDomain:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.5.144.0 * 255.255.240.0 U 0 0 0 eth0
default 10.5.144.1 0.0.0.0 UG 0 0 0 eth0
root@UserDomain:~# ping google.com
PING google.com (72.14.253.104) 56(84) bytes of data.
64 bytes from mia04s03-in-f104.1e100.net (72.14.253.104): icmp_seq=1 ttl=56 time=15.5 ms
64 bytes from mia04s03-in-f104.1e100.net (72.14.253.104): icmp_seq=2 ttl=56 time=14.0 ms
64 bytes from mia04s03-in-f104.1e100.net (72.14.253.104): icmp_seq=3 ttl=56 time=13.9 ms



Dadaaa! Many thanks to Pierre and David!
As a side note, if you are adding more domains, you do not need to repeat all the commands on domain-0, as its interface is already in the "promiscuous" mode. Only run the dhclient again. You will however need to set up the default gateway in every new domain that you create.