The server has a Quad Core Xeon CPU E5440 and 4 GB of RAM. The storage needs are covered by a MD1000 disk array connected to a Perc5/E controller. The controller supports different RAID configurations: 0, 1, 10 and 50. I’ve tried most of them plus different combinations of software raid and hardware plus software raid. In the end I’ve settled on RAID-5 on 3 disks.

As the operating system I’ve installed CentOS 5.1 which is basically a free copy of RHEL 5.1. It comes with Xen 3.0.3 and of course I’ve installed that too. I want to convert the existing servers we have at work to Xen guests and have them running on this new machine. This will let me update OS on all the servers to the same version without bringing the whole office to halt. And once the servers are virtualized it will be much easier to deal with maintenance of the physical servers. Maybe I will even set up some cluster to host all these Xen guests. I didn’t have time to fully explore this idea but it sounds like it could offer further advantages.

So far I was able to convert one physical server to a virtual one with rather short downtime. I followed the instructions described in my Xen p2v post. The virtual server has been running fine for several months already. In terms of performance there are no problems – it helps that the Xen host has more powerful hardware and this server doesn’t need much I/O.

Besides this P2V conversion I’ve also created a Xen guest with Fedora Core 8 to run our CruiseControl server. The previous installation was running on a Fedora Core 5 host. I’ve used the occasion and upgraded CruiseControl to the latest version. Installing CruiseControl took me a long time. Mostly because I couldn’t find proper instructions. I’ve tried few different ones from the CruiseControl website, from IBM and some others. Unfortunately I just couldn’t get it to work. So in the end I’ve just duplicated the setup we had before. It’s not perfect but it works and I don’t want to spend a week on that.

Third Xen guest I’ve created is a very small installation of CentOS 5.1 running with only 64 MB of RAM. It serves as a license server for some applications we use. I’ve minimized the amount of the services running to reduce the memory footprint and the virtual machine uses only 41 MB of RAM.

So far all three guests are running nicely together. I’ll be adding more of them in the future.

Related post(s)

• Xen p2v conversion in seven simple steps (0)
• USB forwarding on Xen (0)
• Setting up UPS on CentOS 5.2 with SELinux, part 2 (1)
• Setting up UPS on CentOS 5.2 with SELinux, part 1 (0)
• What files did I change? (2)

What can you do now? One suggested approach is to use some kind of USB over IP solution. That does involve spending some money though. USB over IP devices are about €300, a well known example are products from Digi International. Unfortunately I didn’t find any device of this type which would work with Linux clients so this looks like a bad direction to go even if it was cheaper. Another solution mentioned is using a software version of USB over IP. Again, they mostly support Windows clients. I was able to find only one that includes a Linux client – USB Server. At this point it’s in beta so it could be unstable – better test it before you use it on a production system. It includes a kernel driver (released under GPL 2) and binary only programs for the client and the server. It’s not a perfect solution since you still don’t know what’s inside of these binaries.

There are also people working on a similar solution in USB/IP Project. According to them “It is still at a *very* early development stage.” and is not stable.

I’ll try out these options and report here my findings. In the meantime if someone has experiences with any of these two solutions please comment and let us know.

Related post(s)

• How to Xenify office servers (0)
• USB forwarding on Xen – it just does not work (3)
• USB forwarding on Xen (0)
• Skype 4.0 for Windows is out: Pros and cons (compared with Linux Skype 2.0) (1)
• Setting up UPS on CentOS 5.2 with SELinux, part 2 (1)

I think I should start with a redundant file system to be free from depending on the 100% availability of a file server. The redundant file system should possibly be clustered so extra redundancy can be provided for the services and load balancing where necessary.

I’m not sure how to go about it in an efficient way. I’d like to hear from anybody who’s done something like this before and is willing to share their experiences.

Related post(s)

• USB in Xen – how to make it work (0)
• Skype 4.0 for Windows is out: Pros and cons (compared with Linux Skype 2.0) (1)
• Setting up UPS on CentOS 5.2 with SELinux, part 2 (1)
• Setting up UPS on CentOS 5.2 with SELinux, part 1 (0)
• Eee PC 901 – aka Big Eee (0)

I’ve recently performed two conversions of physical hosts to Xen guests running on CentOS 5.1 host. The physical hosts were running RHEL 3.7 and CentOS 4.6. It is important to know that after the conversion you end up with a fully virtualized guest – HVM. This has its effect on the performance, especially when it comes to I/O. It is a general limitation of fully virtualized Xen guests and not of this particular P2V method. I don’t recommend using it for virtualizing for example a file server. In such a case your only real choice is to create a new paravirtualized guest and migrate the services. This article won’t be describing that approach.

These instructions are based on the article by Kyle Rankin. I’ve adapted it for Xen and added comments on those things which did cause me trouble. The instructions have been used to virtualize RHEL and CentOS servers to run as Xen guests on CentOS 5.1 but you should be able to use most of them for other distributions.

First step in the conversion is to create LVM volume(s) in the Xen host to be used by the guest. To figure out how much space you need, have a look at the physical server you want to virtualize. Each LVM volume will be visible in the guest as a separate hard disk. Note that HVM guests support only up to four disks. You will partition the disks from inside the guest in step two.

Second step is to create a new HVM guest and boot it from a bootable rescue CD/DVD. You should use a rescue CD which matches the architecture of the physical host being virtualized. In step four you need to run chroot command on the guest and it won’t work if you try for example a 32 bit rescue CD on a 64 bit system. After you have already booted the guest from a rescue CD you need to create partitions to match the physical host. The easiest thing is to match exactly the number and sizes of partitions on the physical system. You can also put several file systems which were on separate partitions on the physical server on a single partition on the guest as long as you prepare enough space.

Third step is to copy files from the physical server to the guest. This can take some time, so depending on whether you want to minimize the downtime you can do it either in one step or in two. Both methods use rsync command. The second method is described by Kyle in his article. The first method is basically the same but you only do the final synchronization. If you use a single transfer method you don’t need to use the --delete parameter for the final (and in this case the only) synchronization. The single transfer method is faster but it might not be available for you if you need to minimize the downtime.

Fourth step is to adjust the boot settings in the guest. Without that the guest won’t boot. To do that make sure all the guest file systems are mounted and chroot into the guest root file system. From there you need to adjust the boot loader configuration. I use GRUB on my systems. For instructions on adjusting LILO you’ll have to look elsewhere.

First you need to change the file /boot/grub/device.map. If you see /dev/sda there you need to replace it with /dev/hda. This should represent your boot drive. Also verify that the guest boot partition is in /etc/mtab. After that you execute grub-install /dev/hda.

Another file which needs modifications is /etc/grub.conf. Pay attention to the lines starting with boot= and splashimage= as they probably need adjusting. The boot parameters need to be updated as appropriate. The important entries are root (hd0,0) and root= kernel parameter. Note that the first one selects the boot partition while the second selects the partition for the root file system. So for example if the boot partition is in /dev/hda3 and root partition is /dev/hda1 these entries should look root (hd0,2) (here the partitions are counted starting from zero) and root=/dev/hda1 (here they are counted from one).

Fifth step is to generate a new initrd file for the guest. First copy the existing initrd file somewhere in case you need it later. Then adjust the /etc/modules.conf or /etc/modprobe.conf file (only one will be present depending on the distribution and OS version). Remove all entries starting with alias scsi_hostadapter. After that you can run mkinitrd command. I recommend you use -v parameter for mkinitrd. This will list all the modules which will be included in the initrd. You will probably need at least libata and ata_piix – even if the physical server uses SCSI disks the HVM guest will use IDE. This depends on the OS of the physical machine you’re virtualizing because it was only important when I was converting the RHEL 3.7 server but not with CentOS 4.6. If they’re missing you can add them using --with= parameter. Another advice in case you need to verify the content of initrd file (probably because the guest doesn’t boot): take into account that older versions of mkinitrd produce just a gzipped ext2 filesystem while the newer ones create a gzipped cpio archive.

Sixth step is to adjust the network settings and mount points for the guest. The configuration files you need to edit to change network settings are in /etc/sysconfig/network-scripts. Most likely you will have at least ifcfg-eth0. RHEL and CentOS usually specify the MAC address of the network adapter there so you need to change it to match with the MAC address of your guest. Otherwise when the guest boots the interface might not get activated. If you want to change the IP address of the guest you can also do it although I’d recommend to leave it for later when you already know the guest works. For the initial boot it should be easier if the guest runs with the same IP address as was used before by the physical server.

You also need to edit file /etc/fstab to match with the partition scheme defined for the guest.

Seventh step is to boot the HVM guest which will be now able to replace the physical server. In order to avoid conflicts you should either power down the physical server, disconnect it from the network or change its IP address(es). After that you can reboot the Xen guest. If you followed the instructions and if the initrd file contains all required modules the system should boot and start all the configured services. If you’re using kudzu it will ask you a few questions about removed/added hardware. Kyle suggests to select “Keep Configuration” for any removed SCSI or network hardware, and select “Ignore” for any added SCSI or network hardware. If you are prompted about any removed video, sound, USB and similar hardware you can safely select “Remove Configuration”. When booting finishes verify that all required service are running and that you have network connectivity from within the guest.

That’s it. You should have now a fully virtualized guest running on the Xen host which has the same configuration as the physical server it replaces. It would be a good idea to perform this process first on a test physical server so you have more time to figure out solutions to any problems you may encounter.

Related post(s)

• Xen and CentOS 5.1 on PowerEdge 2950 (2)
• USB forwarding on Xen – it just does not work (3)
• USB forwarding on Xen (0)
• What files did I change? (2)
• Setting up UPS on CentOS 5.2 with SELinux, part 2 (1)

Unfortunately the Xen guest was not stable. It would crash after half an hour, sometimes after two hours. I tried increasing the amount of RAM allocated to the guest, tried both paravirtualized and fully virtualized guests and the result was the same. The guest would crash.

I tried again to use USB forwarding and pass only a single USB device to the guest. After a few more attempts I noticed that I could “partially” forward the USB device itself to a fully virtualized guest (HVM) using this syntax:

usb=1
usbdevice="host:xxxx:yyyy"

The values for xxxx and yyyy are the vendor ID and product ID for the USB device. They can be found out by using the lsusb command. For example on my laptop lsusb produces this output:

Bus 007 Device 001: ID 0000:0000
Bus 005 Device 003: ID 0b97:7772 O2 Micro, Inc.
Bus 005 Device 002: ID 0b97:7761 O2 Micro, Inc.
Bus 005 Device 001: ID 0000:0000
Bus 004 Device 009: ID 062a:0000 Creative Labs
Bus 004 Device 008: ID 413c:2002 Dell Computer Corp. SK-8125 Keyboard
Bus 004 Device 007: ID 413c:1002 Dell Computer Corp. Keyboard Hub
Bus 004 Device 001: ID 0000:0000
Bus 003 Device 001: ID 0000:0000
Bus 006 Device 001: ID 0000:0000
Bus 002 Device 001: ID 0000:0000
Bus 001 Device 001: ID 0000:0000

So if I wanted to forward the device Dell Computer Corp. SK-8125 Keyboard I’d put the following in the Xen guest configuration file:

usb=1
usbdevice="host:413c:2002"


Ok, so in theory this should have let me see the USB device in Xen guest. Well it did, kind of. One thing I noticed was that I was only able to see some output when running lsusb on the guest when the guest was started from command line using the xm create command. When starting the guest from Virtual Machine Manager GUI I’d see absolutely no USB devices in the guest. Second, even when starting the guest from command line I’d see this kind of output from lsusb:

Bus 001 Device 001: ID 0000:0000

So it looks like something was being forwarded to the guest but it wasn’t the device I wanted to export. It looks like USB pass-through works on Xen only with some USB devices. That matches with what I’ve read at xgu.ru.

The final result – at the moment I’m using the USB device from the physical host. At least until I have time to look at it again.

Related post(s)

• USB forwarding on Xen (0)
• Xen p2v conversion in seven simple steps (0)
• USB in Xen – how to make it work (0)
• Xen and CentOS 5.1 on PowerEdge 2950 (2)
• How to Xenify office servers (0)

To set it up, I followed instructions written by Jón Fairbairn at xensource.com. The pciback module in the CentOS 5.1 kernel doesn’t support the hide parameter so I left that out from my setup. It doesn’t seem to be necessary, anyway, for the driver for the USB controller in the host can be unbound from the PCI slot used by the controller.

To make sure the PCI slot is exported to Xen guests when the host system boots I created a xenpciexport script and placed it in the /etc/init.d/ directory. You can download the code from here xenpciexport.txt

The script xenpciexport is set up to start before xend and xendomains to make sure the PCI forwarding is working before any guest is running. It stops after xend and xendomains. That’s controlled by the line:

# chkconfig: 2345 97 02

you can see in the script header.

After copying the file to /etc/init.d you need to run

chkconfig --add xenpciexport

to create the appropriate links in the /etc/rc*.d/ directories.

To find out the slot number for the USB controller use lspci and lsusb commands. If the script is set up correctly you can plug in the USB device and run

service xenpciexport stop

After that the results of lsusb should include the USB device you want to export. Then run

service xenpciexport start

and now lsusb won’t show this device anymore – it’s hidden from the host and visible to Xen guests.

Before you start the Xen guest which is supposed to have access to the exported device add this line to its configuration file:

pci = [ '0000:00:1d.1' ]

The number in quotes must be the same as the SLOT variable in the xenpciexport script.

After that you need to patch your Xen installation. The version installed in CentOS 5.1 contains a bug which makes it impossible to use PCI pass-through. The fix can be found at CentOS Bug Tracker. Make sure you also fix the file pciif.py as specified in the note. In total there are three files which need to be edited: iopif.py, irqif.py and pciif.py.

Now you can finally start the Xen guest. When it finishes booting log into it and run lsusb there to confirm that the device is visible to the guest. lspci in the guest should also show you the USB controller. It is important to run the xenpciexport script before you start your guest, otherwise the USB device will not be visible to the guest OS.

The Xen guest has now full access to the USB device.

Here you can read more about PCI passthrough on Xen.

Update:
It turns out that this setup was causing the Xen guest to crash. You can read more in a new post.

Related post(s)

• USB forwarding on Xen – it just does not work (3)
• Xen p2v conversion in seven simple steps (0)
• Xen and CentOS 5.1 on PowerEdge 2950 (2)
• USB in Xen – how to make it work (0)
• Setting up UPS on CentOS 5.2 with SELinux, part 2 (1)