Starting new domains on Xen 4.0.1 from command line

So, I have issues with my installation of Xen4.0.1 as the graphics driver keeps crashes during boot up, resulting in a hung machine. For now, I am using a work around suggested by my colleagues David Wolinsky and Pierre St. Juste, namely to disable the graphics display module (using "mv /etc/init/gdm.conf /etc/init/gdm.disabled") and hence I only have command line access to my domain-0 at this point.

At this point, Domain-0 boots up just fine, and xm and xend seem to be working. I now need to start Virtual domains using only the command line. (Later I will explore ways to do the same thing using APIs like Virt-Manager, or its console variant Virt-install)

Creating an Image

The first step is to create an image that will hold the user-domain (domU) virtual disk. This can be just a raw zero-filled file and so we can use the dd command here:

dd if=/dev/zero of=/var/lib/xen/images/domain1.img oflag=direct bs=1M seek=2047 count=1


The above command will create /var/lib/xen/images/domain1.img file of 2048MB although the actual data blocks are allocated in a lazy fashion. To reserve all the data blocks right away, get rid of the seek option from above. So do:

dd if=/dev/zero of=/var/lib/xen/images/domain1.img oflag=direct bs=1M count=2048


This will avoid data block allocation problems if the volume that holds the image is full. It is important to check that the image has the correct security context (permissions), otherwise access to the virtual disk will be denied to the user domain system. You can check this as follows:

ls -l /var/lib/xen/images/domain1.img
-rw-r--r-- 1 root root 2147483648 2010-10-13 10:23 /var/lib/xen/images/domain1.img


Preparing a Xen configuration file for the installation
Xen uses a configuration file per domain. The configuration for the domains is different because we have to provide installation kernels, initial ram-disk and possibly some boot parameters. The domainU installation initrd image and kernel for my machine architecture (64-bit x86) can be downloaded from this page. You can put them in some sensible directory and name them appropriately.

root@Xen-Open:~# mkdir /home/user/x86_64_domU_images
root@Xen-Open:~# cd /home/user/x86_64_domU_images
root@Xen-Open:~/x86_64_domU_images# wget http://archive.ubuntu.com/ubuntu/dists/lucid/main/installer-amd64/current/images/netboot/xen/initrd.gz
root@Xen-Open:~/x86_64_domU_images# wget http://archive.ubuntu.com/ubuntu/dists/lucid/main/installer-amd64/current/images/netboot/xen/vmlinuz
root@Xen-Open:~/x86_64_domU_images# ls
initrd.gz vmlinuz
root@Xen-Open:~/x86_64_domU_images# gunzip -c initrd.gz > initrd.img
root@Xen-Open:~/x86_64_domU_images# ls
initrd.gz initrd.img vmlinuz


In this example, the kernel and the initrd image were named /home/user/x86_64_domU_images/vmlinuz and /home/user/x86_64_domU_images/initrd.img respectively. With the images in place, we can now create the installation configuration file named /etc/xen/domain1.cfg

kernel = "/home/user/x86_64_domU_images/vmlinuz"
ramdisk = "/home/user/x86_64_domU_images/initrd.img"
name = "domain1"
memory = "256"
disk = [ 'file://var/lib/xen/images/disk.img,xvda,w', ]
vif = [ '' ]
dhcp="dhcp"
netmask="255.255.240.0"
gateway="10.5.144.1"
root="/dev/xvda ro"
vcpus=1
extra="4"
on_reboot = 'destroy'
on_crash = 'destroy'

Couple of points to note:

1. The netmask and gateway values to be used can be determined by using the route command on your control domain. If you encounter problems in getting the network to start, you could try using methods suggested here.



2. The [ 'file://var/lib/xen/images/disk.img,xvda,w', ] line here does not work if you use "tap:aio" instead of "file". Also for some reason, if you replace "xvda" with "xvda1" or something else, it does not work either. If you do any of those things you will get the following error:
   

   
Error message:

Error informing the kernel about modifications to partition
/dev/xvda1p1 -- Invalid argument. This means Linux won't know about
any changes you made to /dev/xvda1p1 until you reboot -- so you
shouldn't mount it or use it in any way before rebooting.


Once your config file is ready, you can start creating a domain with:

sudo xm create -c /etc/xen/domain1.cfg


And now, to access your domain, do:

sudo xm domain1


This will open an interactive installation window. Everything is self explanatory in general. If you get an error to the effect that the Ubuntu repository cannot be reached, check your control domain network and here is the fix.

Once your domain is properly created it will ask permission to reboot and will automatically get destroyed in the process because of the last two lines in the configuration command line, namely:

on_reboot = 'destroy'
on_crash = 'destroy'


So now you need to change your configuration file to the following:

name ="domain1"
memory ="256"
disk =[ 'file://var/lib/xen/images/disk.img,xvda,w', ]
vif=[ '' ]
dhcp="dhcp"
netmask="255.255.240.0"
gateway="10.5.144.1"
vcpus=1
on_reboot="restart"
on_crash="restart"
bootloader="/usr/bin/pygrub"



And now just use the xm create command to create your new domain. Finally in order to automatically start and stop your domains when domain-0 starts, move the location of the configuration file to /etc/xen/auto.

At this point, it is a good idea to create a copy of your virtual machine image. To create new virtual machines using the same image all you have to do is to change the configuration file (name, location of the disk image etc) to point to the copy and start the new machines... waaa la !

References:

1. Creating and installing CentOS 5 Xen domain instance
2. Error during guided partitioning