CentOS 5.6 GitCO xen 3.4.3 kernel, libvirt-client and xen-libs update issues

A recent CentOS 5.5 to 5.6 update has caused one of my xen servers to stop functioning... This has happened few times in the past... and I keep forgetting to look before I reboot. Maybe documenting it here will help me remember and help others QUICKLY get past the xend failing to start with the error of:

[2011-04-16 15:34:19 5082] INFO (SrvDaemon:336) Xend changeset: unavailable.
[2011-04-16 15:34:19 5082] ERROR (SrvDaemon:349) Exception starting xend ((13, 'Permission denied'))
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDaemon.py", line 341, in run
servers = SrvServer.create()
File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvServer.py", line 251, in create
root.putChild('xend', SrvRoot())
File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
self.get(name)
File "/usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line 84, in get
val = val.getobj()
File "/usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line 52, in getobj
self.obj = klassobj()
File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__

Here is the deal. We have been using the Xen 3.4.3 repo from Gitco. For some reason yum will not keep the "kernel" line that matches up with the xen install from xen-3.4.3. So, the VERY simple fix (till the next kernel update) is just to make sure that your grub.conf has a line of "kernel /xen.gz-3.4.3" and not whatever gets dumped there by yum. Hope this helps people with this error.

The second issue with the (finally) released CentOS 5.6 update related to using the xen 3.4.3 from GitCO is the inclusion of the libvirt-client-0.7.0-6.el5. This is now an older version with the 5.6 update now out. The error you see from try a yum update is:

Transaction Check Error:
file /usr/bin/virsh from install of libvirt-0.8.2-15.el5.3.x86_64 conflicts with file from package libvirt-client-0.7.0-6.el5.x86_64
file /usr/bin/virt-xml-validate from install of libvirt-0.8.2-15.el5.3.x86_64 conflicts with file from package libvirt-client-0.7.0-6.el5.x86_64
file /usr/lib64/libvirt.so.0 from install of libvirt-0.8.2-15.el5.3.x86_64 conflicts with file from package libvirt-client-0.7.0-6.el5.x86_64
file /usr/share/libvirt/schemas/capability.rng from install of libvirt-0.8.2-15.el5.3.x86_64 conflicts with file from package libvirt-client-0.7.0-6.el5.x86_64

The fix is:

rpm -e --nodeps libvirt-client
yum update

If you accidentally got the old i386 xen-libs, just run:

rpm -e xen-libs.i386
yum update

CentOS 5.x and Firefox 4.0 howto!

Don't be confused. I did not come up with this! I just want to spread the word as much as possible that one very enlightened fellow referenced as "rkl" at forums.mozillazine.org has made it possible for the rest of us to hit the "easy button" and get FireFox 4.0 working on CentOS 5.x (CentOS 5.5 at this time). HERE is the original link that I lucked upon while trying to google it on my own.

This is the a quote of the general (FF4beta version) steps if you are too lazy to click another link:

1. Unpack the Firefox 4.0b9.tar.bz2 somewhere (e.g. /usr/local/firefox). With the "en_GB" release, I throw in a "dictionaries" sub-dir under there with en-GB.aff and en-GB.dic in there (and en-US.aff/.dic soft-linked to the en-GB ones) otherwise, sadly, Firefox 4 uses US spellings on what's supposed to be an en_GB release :-(

2. Download this 32-bit Fedora 9 libstdc++ RPM and unpack it with this command:

rpm2cpio libstdc++-4.3.0-8.i386.rpm | cpio -i --make-directories

3. Move the unpacked shared library into /usr/local/firefox thus:

mv usr/lib/libstdc++.so.6.0.10 /usr/local/firefox/libstdc++.so.6

Note: It's "usr/lib/libstdc++.so.6.0.10" above (i.e. the unpacked tree from the RPM, not the system /usr/lib tree) - do NOT put a leading slash there!

4. Run Firefox 4.0b9 with:

/usr/local/firefox/firefox

rkl references the beta version, but all steps still apply for official release.

I will say that FF4 is very nice! I am very impressed with what I have seen so far!

Basic OMSA install and usage with RHEL or CentOS 5.x

I am a big fan of Dell Poweredge hardware. It's well designed. It runs CentOS rock solid. It just works well. Dell must have figured out that many people like me feel this way (companies really). To their credit, Dell has a very full featured Linux support offering in the OpenManage Server Administrator (OMSA) for free.

To be fair, OMSA has historically had some speed bumps. Full 64-bit install has only recently been possible. And some past upgrades have not been smooth and actually required removing packages and manually removing directories and files in order to upgrade.

Note: make sure "plugins=1" is in the /etc/yum.conf

As root from a terminal prompt:

wget -q -O - http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash
wget -q -O - http://linux.dell.com/repo/community/bootstrap.cgi | bash
wget -q -O - http://linux.dell.com/repo/firmware/bootstrap.cgi | bash
yum install srvadmin-all OpenIPMI OpenIPMI-tools dell_ft_install

Install BIOS and firmware files

yum -y install $(bootstrap_firmware)
update_firmware --yes

Reboot if needed/requested. If you don't reboot, make sure that you exit and log back in to have the srvadmin bin and sbin directories set for you.

Just to make sure some startup services will be started on reboot.

chkconfig dsm_sa_ipmi on
chkconfig ipmi on
srvadmin-services.sh stop && service ipmi start && srvadmin-services.sh start

The web GUI is good. Just point your browser to http://yourhostname_or_IP:1311 and login as root and check it out from there.

The command line tools, however, can offer some very good information when you don't have that option or need some scripting magic. For example, "omreport" can provide MUCH information about your server and it's state. Here is a quickie on controller:

omreport storage controller

Will spew lots of data like:

Controller PERC 5/i Integrated (Embedded)

Controllers
ID : 0
Status : Ok
Name : PERC 5/i Integrated
Slot ID : Embedded
State : Ready
Firmware Version : 5.2.2-0072
Minimum Required Firmware Version : Not Applicable
Driver Version : 00.00.04.17-4.31.z-RH1

Minimum Required Driver Version : Not Applicable
Storport Driver Version : Not Applicable
Minimum Required Storport Driver Version : Not Applicable
Number of Connectors : 2
Rebuild Rate : 30%
BGI Rate : 30%
Check Consistency Rate : 30%
Reconstruct Rate : 30%
Alarm State : Not Applicable
Cluster Mode : Not Applicable
SCSI Initiator ID : Not Applicable
Cache Memory Size : 256 MB
Patrol Read Mode : Auto
Patrol Read State : Stopped
Patrol Read Rate : 30%
Patrol Read Iterations : 41
Abort Check Consistency on Error : Not Applicable
Allow Revertible Hot Spare and Replace Member : Not Applicable
Load Balance : Not Applicable
Auto Replace Member on Predictive Failure : Not Applicable
Redundant Path view : Not Applicable
CacheCade Capable : Not Applicable
Persistent Hot Spare : Not Applicable
Encryption Capable : Not Applicable
Encryption Key Present : Not Applicable
Encryption Mode : Not Applicable
Spin Down Unconfigured Drives : Not Applicable
Spin Down Hot Spares : Not Applicable

To see information on the status of the virtual disk created on your PERC RAID controllers:

omreport storage vdisk

List of Virtual Disks in the System

Controller PERC 5/i Integrated (Embedded)
ID : 0
Status : Ok
Name : vd0
State : Ready
Encrypted : Not Applicable
Layout : RAID-1
Size : 67.75 GB (72746008576 bytes)
Device Name : /dev/sda
Bus Protocol : SAS
Media : HDD
Read Policy : Adaptive Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Enabled

ID : 1
Status : Ok
Name : vd1
State : Ready
Encrypted : Not Applicable
Layout : RAID-10
Size : 1,396.25 GB (1499212021760 bytes)
Device Name : /dev/sdb
Bus Protocol : SATA
Media : HDD
Read Policy : Adaptive Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy : Enabled

The parameter changing mate to omreport is "omconfig". You get to set or change options and can even trigger events like MAKING SURE YOU HAVE VIRTUAL DRIVE PARITY (AKA parity scrub)!

So to verify/validate/fix drive parity issues, add a cron job (and/or run the command now) to make sure that the RAID groups on your PERC RAID controllers are in perfect shape. You don't want any failed disk rebuild surprises!

crontab -e

00 17 * * 0 /opt/dell/srvadmin/bin/omconfig storage vdisk action=checkconsistency controller=0 vdisk=0 > /tmp/omconfig-vdisk0.out 2>&1 || cat /tmp/omconfig-vdisk0.out |mail -s "omconfig issue on `hostname`" admin
30 17 * * 0 /opt/dell/srvadmin/bin/omconfig storage vdisk action=checkconsistency controller=0 vdisk=1 > /tmp/omconfig-vdisk1.out 2>&1 || cat /tmp/omconfig-vdisk1.out |mail -s "omconfig issue on `hostname`" admin

Or better yet, get crazy with a shell script to dynamically ask for *each* controller and virtual disk to be checked. I will leave that up to you to figure out.

Again, just a basic install and use for OMSA. Take the time and go through the links, get the OMSA manual and enjoy.

Digi Portserver II 16 serial port server setup for RHEL or CentOS 5.x

I had an opportunity to test some additional equipment recently. This is the general setup and install and basic configuration for the Digi Portserver II 16 serial port server for CentOS 5.5. All of this is just a step in adding additional capabilities to our Hylafax servers. Additionally, other serial devices are also connected to give us network access to them. This includes serial port consoles and serial admin ports for some power strips to remotely power cycle some equipment if needed. The Digi Portserver II is not new and has been around for a while. This makes it cheap to pickup on eBay and use.

As a note, all commands that install something will need to be done as effective user "root". You can "sudo" your way there or just "su -"; your choice. Please take some time to
sanely setup a build environment on another box (if possible/practical).

This article is a follow up to a previous post on How to get a Digi Edgeport/416 working with RHEL/CentOS 5

Here is the Digi Portserver II overview from Digi. Nice starting point for information.

Here is the Driver download page if the overview is too much reading.

For even less effort you can get the latest stable kernel driver this way:

wget ftp://ftp1.digi.com/support/driver/40002086_P.src.rpm

or grab the beta code like I did:

wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9-29.src.rpm

To build the kernel module, you will need a few prerequisite packages. These are the ones that got me what I needed. You may need more or less depending on what you already had installed:

yum -y install ncurses-devel openssl-devel rpm-build kernel-headers

rpmbuild --rebuild dgrp-1.9-29.src.rpm

Install the built dgrp-1.9-29.i386.rpm with rpm or yum. You could choose to reboot or more simply just run

modprobe dgrp

If you just want the command line setup for the tty ports under /dev/, you can run something like:

/usr/bin/dgrp/config/dgrp_cfg_node init  -v -v -e never 1 IP_OR_HOSTNAME 16

If you prefer a GUI setup and have tk and xorg-x11-xauth at least installed:

/usr/bin/dgrp/config/dgrp_gui

Finding firmware version of your Digi Portserver II to tell if you need a firmware update:

telnet IP/hostname
#> cpconf term

May show something like:

# PortServerII Version 3.1.13 Nov  1 2006

Post dgrp driver setup with udev if you need:
For wide open perms edit /etc/udev/rules.d/10-dgrp.rules and change the line from

KERNEL=="tty_dgrp*", PROGRAM="/usr/bin/dgrp_udev %k", NAME="%c"

to

KERNEL=="tty_dgrp*", PROGRAM="/usr/bin/dgrp_udev %k", NAME="%c", MODE="0666", OPTIONS="last_rule"

For just group perms change to group "uucp" for example, alter original line from

KERNEL=="tty_dgrp*", PROGRAM="/usr/bin/dgrp_udev %k", NAME="%c"

to

KERNEL=="tty_dgrp*", PROGRAM="/usr/bin/dgrp_udev %k", NAME="%c", MODE="0660", GROUP="uucp"

A quick way to see all of the dev files created by the dgrp driver is the default major number 253 is kept:

ls -la /dev/|grep " 253"

Make sure the dgrp kernel module is loaded:

lsmod|grep dgrp

On a sad note, following these instructions will only create a kernel module for the current kernel. You will need to do rebuild on the dgrp package on any kernel update... sorry I have not yet incorporated kmod or DKMS for a dynamic module build. There is hope for me if would just try THIS link, however.

UPDATE: Linux RHEL/CentOS 5 RoundCube webmail setup for latest stable tar install

Roundcube webmail has updated to roundcubemail version 0.5 recently. The previous article is still mostly true with a couple of changes.

The first obvious one is the download link is now HERE
The other is a name change for php53 to php53u to get past an naming issue with RHEL/CENTOS 5.6.

Failed BIOS update on Dell R710 with CentOS 5.5 solved! (Worked around, realy)

Problem: Dell R710 (and likely R610 and others) under CentOS 5.5 with OMSA 6.4 (newest at this time) fails to update BIOS and will not attempt to update any other component:
# update_firmware --yes
...

Found firmware which needs to be updated.

Running updates...
hey, we are supposed to be installing now... :)
loading xml from: /usr/libexec/dell_dup/BIOS_NONE/PIEConfig.xml
loaded.
/ Installing dell_dup_componentid_00159 - 2.2.10 Plugin command is biosie.bin -u update.xml
Output file is update.xml
Plugin timeout is 600
- Installing dell_dup_componentid_00159 - 2.2.10output from the cmd was:

\ Installing dell_dup_componentid_00159 - 2.2.10Could not parse output, bad xml for package: dell_dup_componentid_00159
None
Installation failed for package: dell_dup_componentid_00159 - 2.2.10
aborting update...

The error message from the low-level command was:

Could not parse output, bad xml for package: dell_dup_componentid_00159
Complete!


Closest as I can tell, there is a Dell parser utility that is not understanding the CentOS derived content of the /etc/redhat-release and a possible "distroverpkg=redhat-release" in the /etc/yum.conf.

There are at least 4 ways to work around or fix this issue until Dell can fix the parser problem.

1. Change the contents of the /etc/redhat-release to the official RHEL information

cp -p /etc/redhat-release /etc/redhat-release.org
echo "Red Hat Enterprise Linux Server release 5 (Tikanga)" > /etc/redhat-release

And the BIOS update will work. Just put back your redhat-release.org when your done.

2. Get the official stand alone Dell BIOS Linux update .BIN file for the R710 and update via the normal

bash ./PER710_BIOS_LX_2.2.11_1.BIN

You will need the "compat-libstdc++-33.i386" package installed if not already. Sadly, this option has not proved 100% yet.

3. If you have the *.BIN file and option #2 fails, you could

bash ./PER710_BIOS_LX_2.2.11_1.BIN --extract PER710_BIOS_LX_2.2.11_1
modprobe dell_rbu
smbios-rbu-bios-update -u -f PER710_BIOS_LX_2.2.11_1/payload/*

Of course you will need to replace with *YOUR* downloaded .BIN file for *YOUR* class of system.

4. Bypass some Dell safeguards and update directly

updatedb
locate dell_dup_componentid_00159|grep ".hdr"

If you get only one return value you can short cut by running

dellBiosUpdate -f `locate dell_dup_componentid_00159|grep ".hdr"` -u

Or just figure out which one you want a put it in

dellBiosUpdate -f /usr/share/firmware/dell/dup/system_ven_0x1028_dev_0x0235/dell_dup_componentid_00159_version_2.2.10/R710-020210C.hdr -u

5. Bypass the OS completely with iDRAC card and Lifecycle Controller. In this case the part that you will care about is that:

Unified Server Configurator (USC) aims at local 1-to-1 deployment via a graphical user interface (GUI) for operating system install, updates, configuration, and for performing diagnostics, on single, local servers. This eliminates the need for multiple option ROMs for hardware configuration

You are suppose to be able to update the BIOS from this thing in other words. The LC will download and install all on its' own. It will scan your hardware and update everything it can find that needs updating. You will need to shutdown and boot into the iDRAC and then do all of the updates. Good, but not ideal for normal production servers with the extra downtime. On the other hand, this should not be a huge issue if you applying updates on a scheduled basis. It would still suck (a lot) if you have a large number of servers.

6. Grab a OMSA LiveCD and boot from it. Initial blog entry is HERE The most recent version is running CentOS with some of the mods above including the /etc/redhat-release change (in part). I did have issues with attempting to use OLDER BIOS .BIN files when I had a badly behaving R710 BIOS. Again, the downside is a shutdown for this to happen and not just a simple reboot.

With *ANY* of these options make sure to *REBOOT* your server and not shutdown or your BIOS update will not take into affect.

LPI exam prep help for free...

Like it or not, companies like certifications. Cert's help HR department figure out baseline skills. They make for an easy box to check on an hiring request. It is far more difficult to interview and discover abilities than it is to look for some acronym of ability.

Need another reason?? Learning is fun or at least should be. Especially if your a full time geek. I am amazed on how many new things I learn just looking at some of the prep information. Besides, some of our favourite commands evolve and morph all the time.

That being said, one very good organization is the Linux Professional Institute or just LPI. I'm not saying it's a Cisco cert or RHEL cert but it's a good all around way to get a little more on the old resume. Besides it's a bunch cheaper to get than a lot of the others.

What's even more interesting is that IBM has a very large amount of helpful training information geared directly toward passing LPI group of tests. And best of all, the price is right... free.

The main starting point for IBM is HERE
LPIC-1 certification prep.
LPIC-2 certification prep.
LPIC-3 certification prep.