Step one: Define the policy module

To define the policy module for nut CGI create a text file nutcgi.te with the following content:

module nutcgi 1.0.8;

require {
type unlabeled_t;
type xend_var_log_t;
type httpd_sys_script_exec_t;
type default_t;
type procmail_t;
type ping_t;
type httpd_t;
type httpd_sys_script_t;
type port_t;
class tcp_socket { write name_connect connect shutdown read create };
class lnk_file { read getattr };
class file append;
class dir search;
class packet { recv send };
}

#============= httpd_sys_script_t ==============
# src="httpd_sys_script_t" tgt="port_t" class="tcp_socket", perms="name_connect"
# comm="upsstats.cgi" exe="" path=""
allow httpd_sys_script_t port_t:tcp_socket name_connect;
# src="httpd_sys_script_t" tgt="httpd_sys_script_t" class="tcp_socket", perms="{ write read create connect shutdown }"
# comm="upsstats.cgi" exe="" path=""
allow httpd_sys_script_t self:tcp_socket { write read create connect shutdown };
# src="httpd_sys_script_t" tgt="unlabeled_t" class="packet", perms="{ recv send }"
# comm="upsstats.cgi" exe="" path=""
allow httpd_sys_script_t unlabeled_t:packet { recv send };

#============= httpd_t ==============
# src="httpd_t" tgt="httpd_sys_script_exec_t" class="lnk_file", perms="{ read getattr }"
# comm="httpd" exe="" path=""
allow httpd_t httpd_sys_script_exec_t:lnk_file { read getattr };

Save the file somewhere.

Step two : Compile the policy module

Execute the command:

checkmodule -M -m nutcgi.te -o nutcgi.mod

This will generate a binary file representing the policy module.

Step three: Create a SELinux policy module package

Execute the command:

semodule_package -o nutcgi.pp -m nutcgi.mod

This will create a SELinux policy module package which can then be installed.

Step four: Install the SELinux policy module package

Execute the command:

semodule -i nutcgi.pp

Now the policy module is installed. You can refresh the web page with UPS status in your web browser http://localhost/cgi-bin/upsstats.cgi

You can also verify that access to this web page from other machines works as intended. SELinux policy module installation is persistent – you don’t have to do it again if the system reboots.

Related post(s)

• Setting up UPS on CentOS 5.2 with SELinux, part 1 (0)
• Skype 4.0 for Windows is out: Pros and cons (compared with Linux Skype 2.0) (1)
• Eee PC 901 – aka Big Eee (0)
• What files did I change? (2)
• Too much success and popularity? (0)

Let’s start with the installation.

First things first – make sure the UPS is running and connected to the computer. Both power and a serial cable of some kind should be connected. The serial connection can be either via RS-232 or USB.

To install the software execute the following command:

# yum install nut nut-client nut-cgi

or if you’re running a 64 bit OS and don’t want 32-bit packages to be installed:

# yum install nut nut-client.x86_64 nut-cgi

Next step: Basic configuration

After the software is installed we need to configure it. Fire up your favourite text editor and open these files:

• /etc/ups/ups.conf
• /etc/ups/upsd.users
• /etc/ups/upsmon.conf
• /etc/ups/hosts.conf (only if you have installed nut-cgi)

In /etc/ups/ups.conf add at the end the following section:

user = nut            # execute the UPS driver as user nut - otherwise the upsd program won't be able to connect to the device

[pw3105]              # name of the UPS device
driver = bcmxcp_usb   # driver used to access the UPS device
port = auto           # serial port where the UPS is connected, bcmxcp_usb driver allows to enter 'auto' here
shutdown_delay = 0    # additional parameter supported by bcmxcp_usb driver, see 'man bcmxcp_usb'

The comments explain what is the purpose of each line. More details can be found in man page for ups.conf. The actual name of the UPS device and parameters for it depend on the UPS model you’re using. I have Powerware 3105 UPS which uses the bcmxcp_usb driver.

Second file to edit is /etc/ups/upsd.users where you should append these lines:

[server]                   # create a user 'server'
password = ups             # with password 'ups'
allowfrom = localhost      # allowing access only from this machine
instcmds = ALL             # user can execute all instant commands
upsmon master              # add actions necessary for a 'upsmon' process to work

Again the purpose of each line is explained in the comments. More information as usual in man page for upsd.users.

If you use different values in /etc/ups/upsd.users you have to make sure that you’ll use the same values also in the next file we’re going to edit, which is /etc/ups/upsmon.conf. Here add these lines at the end:

FINALDELAY 0                                     # don't wait before shutting down, allowed values depend on the UPS model
RUN_AS_USER nut                                  # don't run as root to avoid security issues
MONITOR pw3105@localhost 1 server ups master     # which UPS to monitor

The last line requires a bit of explanation:

• pw3105@localhost is the identifier of the UPS we want to monitor. pw3105 is the same name we have used before in /etc/ups/ups.conf so make sure it matches. @localhost indicates that we’re monitoring a UPS connected directly to the machine on which upsmon program is running. upsmon can also monitor UPSes connected to other machines on the network.
• 1 indicates that this particular UPS feeds 1 power supply on this system. That’s the usual value for this parameter.
• server and ups are the user name and password we have defined previously in /etc/ups/upsd.users
• master indicates relationship with upsd daemon and again must match what is defined in /etc/ups/upsd.users

More explanations can be found in man page for upsmon.conf.

After these modifications verify the permissions and ownership of the configuration files matches this:

# ll /etc/ups/ups.conf /etc/ups/upsd.users /etc/ups/upsmon.conf
-rw-r----- 1 root nut 3735 Jan 22 00:33 /etc/ups/ups.conf
-rw-r----- 1 root nut 2307 Jan 22 00:08 /etc/ups/upsd.users
-rw-r----- 1 root nut 11194 Jan 22 00:09 /etc/ups/upsmon.conf

and then you can start the software:

# service ups start

You should see this output:

Starting UPS driver controller: [ OK ]
Starting upsd: [ OK ]
Starting UPS monitor (master): [ OK ]

And make sure that UPS software will start automatically after system reboots:

# chkconfig nut on

Verify that the UPS is accessible by executing:

# upsc pw3105@localhost

which should print something like this:

driver.name: bcmxcp_usb
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.shutdown_delay: 0
driver.version: 2.2.0-
driver.version.internal: 0.11
output.phases: 1
ups.alarm:
ups.firmware: Cont:00.80 Inve:00.60
ups.model: POWERWARE UPS 500VA
ups.power.nominal: 500
ups.serial:
ups.status: OL
ups.voltage.nominal: 230

The actual values depend on the UPS model.

At this point the system is configured to automatically shut down when the UPS will indicate that the battery is low on power. You should test this scenario on your system to make sure that it will work when you really need it.

Last step: Configure web access to upsmon

The last step is to configure monitoring of UPS status via a web browser. We have already installed nut-cgi package which contains the necessary CGI scripts. This package includes a file /var/www/nut-cgi-bin/upsstats.cgi which you should copy to /var/www/cgi-bin/ directory:

# cp /var/www/nut-cgi-bin/upsstats.cgi /var/www/cgi-bin/

After that modify file /etc/ups/hosts.conf by appending this line:

MONITOR pw3105@localhost "Local UPS"

As before pw3105@localhost is the name of UPS to monitor. “Local UPS” defines the name which will appear in the web page.

Now start your web server if it’s not running yet and (optionally) configure the firewall to allow access to port 80 from other machines.

You can now check status of the UPS via a web browser by typing the following in the address bar http://localhost/cgi-bin/upsstats.cgi

Unfortunately if you’re running with SELinux enabled then you won’t see much:

The reason for this is a missing SELinux policy module. Because of that you’ll see errors in /var/log/audit/audit.log and the web page won’t provide any useful information. How to install the missing SELinux policy module? This will be described in the follow up post coming soon.

Related post(s)

• Setting up UPS on CentOS 5.2 with SELinux, part 2 (1)
• Skype 4.0 for Windows is out: Pros and cons (compared with Linux Skype 2.0) (1)
• Eee PC 901 – aka Big Eee (0)
• What files did I change? (2)
• Too much success and popularity? (0)

What I used to do was to keep a copy of all the relevant files as they were in the previous version and then modify the new installation accordingly. This solution was working for me but I was never sure whether the new configuration was exactly the same and if I hadn’t missed any settings in one of the configuration files.

A few days ago I found a better way to monitor the configuration files for changes. As long as you install all the software as RPMs (which makes sense anyway since it’s easier to deploy the same package on multiple machines or deal with reinstallation) you can use the rpm verification capability which is built into the rpm command.

The way this works is quite simple. For example, I want to know if I changed my firewall settings. Since the firewall I’m using is installed as part of the iptables RPM I need to execute (as root):

# rpm -V iptables

which produces the following output:

SM5....T c /etc/sysconfig/iptables-config

This means that since the RPM was installed the file /etc/sysconfig/iptables-config has changed. Specifically the output indicates that the following has changed:

• size of the file (S),
• permissions (M),
• MD5 checksum (5),
• modification time (T).

The properties which did not change are:

• device major/minor numbers (1st dot), if it did change we would see D
• path to which a symbolic link points to (2nd dot), if it did change we would see L
• user ownership (3rd dot), if it did change we would see U
• group ownership (4th dot), if it did change we would see G

Now I know that only this file from iptables RPM was modified.

If I want to look for all the changes in /etc folder I would execute:

# rpm -Va | grep "\/etc\/"

which produces quite a long output so I won’t list it here. Instead of writing “rpm -V” one can also use “rpmverify”.

RPM verification has a few more uses than just checking for changes you did to your own system. It can also be used to monitor your system for unauthorized changes.

This post was written based on information I have found on Novell website, rpm.org and man page for rpm command.

Related post(s)

• Xen p2v conversion in seven simple steps (1)
• Stop your laptop from succumbing to thermal death (2)
• Native resolution with Fujitsu Siemens Scaleoview L22W-7SD (0)
• Setting up UPS on CentOS 5.2 with SELinux, part 2 (1)
• Setting up UPS on CentOS 5.2 with SELinux, part 1 (0)

The server has a Quad Core Xeon CPU E5440 and 4 GB of RAM. The storage needs are covered by a MD1000 disk array connected to a Perc5/E controller. The controller supports different RAID configurations: 0, 1, 10 and 50. I’ve tried most of them plus different combinations of software raid and hardware plus software raid. In the end I’ve settled on RAID-5 on 3 disks.

As the operating system I’ve installed CentOS 5.1 which is basically a free copy of RHEL 5.1. It comes with Xen 3.0.3 and of course I’ve installed that too. I want to convert the existing servers we have at work to Xen guests and have them running on this new machine. This will let me update OS on all the servers to the same version without bringing the whole office to halt. And once the servers are virtualized it will be much easier to deal with maintenance of the physical servers. Maybe I will even set up some cluster to host all these Xen guests. I didn’t have time to fully explore this idea but it sounds like it could offer further advantages.

So far I was able to convert one physical server to a virtual one with rather short downtime. I followed the instructions described in my Xen p2v post. The virtual server has been running fine for several months already. In terms of performance there are no problems – it helps that the Xen host has more powerful hardware and this server doesn’t need much I/O.

Besides this P2V conversion I’ve also created a Xen guest with Fedora Core 8 to run our CruiseControl server. The previous installation was running on a Fedora Core 5 host. I’ve used the occasion and upgraded CruiseControl to the latest version. Installing CruiseControl took me a long time. Mostly because I couldn’t find proper instructions. I’ve tried few different ones from the CruiseControl website, from IBM and some others. Unfortunately I just couldn’t get it to work. So in the end I’ve just duplicated the setup we had before. It’s not perfect but it works and I don’t want to spend a week on that.

Third Xen guest I’ve created is a very small installation of CentOS 5.1 running with only 64 MB of RAM. It serves as a license server for some applications we use. I’ve minimized the amount of the services running to reduce the memory footprint and the virtual machine uses only 41 MB of RAM.

So far all three guests are running nicely together. I’ll be adding more of them in the future.

Related post(s)

• Xen p2v conversion in seven simple steps (1)
• USB forwarding on Xen (1)
• Setting up UPS on CentOS 5.2 with SELinux, part 2 (1)
• Setting up UPS on CentOS 5.2 with SELinux, part 1 (0)
• What files did I change? (2)

I’ve recently performed two conversions of physical hosts to Xen guests running on CentOS 5.1 host. The physical hosts were running RHEL 3.7 and CentOS 4.6. It is important to know that after the conversion you end up with a fully virtualized guest – HVM. This has its effect on the performance, especially when it comes to I/O. It is a general limitation of fully virtualized Xen guests and not of this particular P2V method. I don’t recommend using it for virtualizing for example a file server. In such a case your only real choice is to create a new paravirtualized guest and migrate the services. This article won’t be describing that approach.

These instructions are based on the article by Kyle Rankin. I’ve adapted it for Xen and added comments on those things which did cause me trouble. The instructions have been used to virtualize RHEL and CentOS servers to run as Xen guests on CentOS 5.1 but you should be able to use most of them for other distributions.

First step in the conversion is to create LVM volume(s) in the Xen host to be used by the guest. To figure out how much space you need, have a look at the physical server you want to virtualize. Each LVM volume will be visible in the guest as a separate hard disk. Note that HVM guests support only up to four disks. You will partition the disks from inside the guest in step two.

Second step is to create a new HVM guest and boot it from a bootable rescue CD/DVD. You should use a rescue CD which matches the architecture of the physical host being virtualized. In step four you need to run chroot command on the guest and it won’t work if you try for example a 32 bit rescue CD on a 64 bit system. After you have already booted the guest from a rescue CD you need to create partitions to match the physical host. The easiest thing is to match exactly the number and sizes of partitions on the physical system. You can also put several file systems which were on separate partitions on the physical server on a single partition on the guest as long as you prepare enough space.

Third step is to copy files from the physical server to the guest. This can take some time, so depending on whether you want to minimize the downtime you can do it either in one step or in two. Both methods use rsync command. The second method is described by Kyle in his article. The first method is basically the same but you only do the final synchronization. If you use a single transfer method you don’t need to use the --delete parameter for the final (and in this case the only) synchronization. The single transfer method is faster but it might not be available for you if you need to minimize the downtime.

Fourth step is to adjust the boot settings in the guest. Without that the guest won’t boot. To do that make sure all the guest file systems are mounted and chroot into the guest root file system. From there you need to adjust the boot loader configuration. I use GRUB on my systems. For instructions on adjusting LILO you’ll have to look elsewhere.

First you need to change the file /boot/grub/device.map. If you see /dev/sda there you need to replace it with /dev/hda. This should represent your boot drive. Also verify that the guest boot partition is in /etc/mtab. After that you execute grub-install /dev/hda.

Another file which needs modifications is /etc/grub.conf. Pay attention to the lines starting with boot= and splashimage= as they probably need adjusting. The boot parameters need to be updated as appropriate. The important entries are root (hd0,0) and root= kernel parameter. Note that the first one selects the boot partition while the second selects the partition for the root file system. So for example if the boot partition is in /dev/hda3 and root partition is /dev/hda1 these entries should look root (hd0,2) (here the partitions are counted starting from zero) and root=/dev/hda1 (here they are counted from one).

Fifth step is to generate a new initrd file for the guest. First copy the existing initrd file somewhere in case you need it later. Then adjust the /etc/modules.conf or /etc/modprobe.conf file (only one will be present depending on the distribution and OS version). Remove all entries starting with alias scsi_hostadapter. After that you can run mkinitrd command. I recommend you use -v parameter for mkinitrd. This will list all the modules which will be included in the initrd. You will probably need at least libata and ata_piix – even if the physical server uses SCSI disks the HVM guest will use IDE. This depends on the OS of the physical machine you’re virtualizing because it was only important when I was converting the RHEL 3.7 server but not with CentOS 4.6. If they’re missing you can add them using --with= parameter. Another advice in case you need to verify the content of initrd file (probably because the guest doesn’t boot): take into account that older versions of mkinitrd produce just a gzipped ext2 filesystem while the newer ones create a gzipped cpio archive.

Sixth step is to adjust the network settings and mount points for the guest. The configuration files you need to edit to change network settings are in /etc/sysconfig/network-scripts. Most likely you will have at least ifcfg-eth0. RHEL and CentOS usually specify the MAC address of the network adapter there so you need to change it to match with the MAC address of your guest. Otherwise when the guest boots the interface might not get activated. If you want to change the IP address of the guest you can also do it although I’d recommend to leave it for later when you already know the guest works. For the initial boot it should be easier if the guest runs with the same IP address as was used before by the physical server.

You also need to edit file /etc/fstab to match with the partition scheme defined for the guest.

Seventh step is to boot the HVM guest which will be now able to replace the physical server. In order to avoid conflicts you should either power down the physical server, disconnect it from the network or change its IP address(es). After that you can reboot the Xen guest. If you followed the instructions and if the initrd file contains all required modules the system should boot and start all the configured services. If you’re using kudzu it will ask you a few questions about removed/added hardware. Kyle suggests to select “Keep Configuration” for any removed SCSI or network hardware, and select “Ignore” for any added SCSI or network hardware. If you are prompted about any removed video, sound, USB and similar hardware you can safely select “Remove Configuration”. When booting finishes verify that all required service are running and that you have network connectivity from within the guest.

That’s it. You should have now a fully virtualized guest running on the Xen host which has the same configuration as the physical server it replaces. It would be a good idea to perform this process first on a test physical server so you have more time to figure out solutions to any problems you may encounter.

Related post(s)

• Xen and CentOS 5.1 on PowerEdge 2950 (2)
• USB forwarding on Xen – it just does not work (6)
• USB forwarding on Xen (1)
• What files did I change? (2)
• Setting up UPS on CentOS 5.2 with SELinux, part 2 (1)

To set it up, I followed instructions written by Jón Fairbairn at xensource.com. The pciback module in the CentOS 5.1 kernel doesn’t support the hide parameter so I left that out from my setup. It doesn’t seem to be necessary, anyway, for the driver for the USB controller in the host can be unbound from the PCI slot used by the controller.

To make sure the PCI slot is exported to Xen guests when the host system boots I created a xenpciexport script and placed it in the /etc/init.d/ directory. You can download the code from here xenpciexport.txt

The script xenpciexport is set up to start before xend and xendomains to make sure the PCI forwarding is working before any guest is running. It stops after xend and xendomains. That’s controlled by the line:

# chkconfig: 2345 97 02

you can see in the script header.

After copying the file to /etc/init.d you need to run

chkconfig --add xenpciexport

to create the appropriate links in the /etc/rc*.d/ directories.

To find out the slot number for the USB controller use lspci and lsusb commands. If the script is set up correctly you can plug in the USB device and run

service xenpciexport stop

After that the results of lsusb should include the USB device you want to export. Then run

service xenpciexport start

and now lsusb won’t show this device anymore – it’s hidden from the host and visible to Xen guests.

Before you start the Xen guest which is supposed to have access to the exported device add this line to its configuration file:

pci = [ '0000:00:1d.1' ]

The number in quotes must be the same as the SLOT variable in the xenpciexport script.

After that you need to patch your Xen installation. The version installed in CentOS 5.1 contains a bug which makes it impossible to use PCI pass-through. The fix can be found at CentOS Bug Tracker. Make sure you also fix the file pciif.py as specified in the note. In total there are three files which need to be edited: iopif.py, irqif.py and pciif.py.

Now you can finally start the Xen guest. When it finishes booting log into it and run lsusb there to confirm that the device is visible to the guest. lspci in the guest should also show you the USB controller. It is important to run the xenpciexport script before you start your guest, otherwise the USB device will not be visible to the guest OS.

The Xen guest has now full access to the USB device.

Here you can read more about PCI passthrough on Xen.

Update:
It turns out that this setup was causing the Xen guest to crash. You can read more in a new post.

Related post(s)

• USB forwarding on Xen – it just does not work (6)
• Xen p2v conversion in seven simple steps (1)
• Xen and CentOS 5.1 on PowerEdge 2950 (2)
• USB in Xen – how to make it work (0)
• Setting up UPS on CentOS 5.2 with SELinux, part 2 (1)