The tricky problem dilemma

A good sysadmin believes in treating the cause, not the symptom. Unfortunately, pragmatism sometimes gets in the way of that. A recent example: we just rolled out a kernel update to a few of our compute clusters. About 3% of the machines ended up in a troubled state. By troubled, I mean that the permissions on a few directories (/bin, /lib, /dev, /etc, /proc, and /sys) were set to 700, making the machine effectively unusable. For the most part, we didn’t notice this on the affected machines until after they did their post-upgrade reboot, but fortunately we were able to catch a few that hadn’t yet rebooted.

What we found was that / had a sysroot directory and an init file. These are created by the mkinitrd script, which is called by the new-kernel-pkg script, which is in turn called in the postinstall script of the kernel RPM. The relevant part of the mkinitrd script seems to be

    for t in /tmp /var/tmp /root ${PWD}; do
        if [ ! -d $t ]; then continue; fi
        if ! access -w $t ; then continue; fi

        fs=$(df -T $t 2>/dev/null | awk '{line=$1;} END {printf $2;}')
        if [ "$fs" != "tmpfs" ]; then

which creates a working directory in /tmp under normal conditions. However, there seemed to be something that caused / to be used instead of /tmp. Later in the script, several directories are created in $TMPDIR, which correspond to the wrongly-permissioned directories. There’s not a clear indication of why this happens, but if we clean up and reinstall the updated kernel package it doesn’t necessarily repeat itself. After some soul-searching, we decided that it was more important to return the nodes to service than to try to track down an easily-correctable-but-difficult-to-solve problem. We’ll see if it happens again with the next kernel upgrade.

My thoughts on the Mac App Store

This post proves that this is not a newsy blog.

A few weeks ago, I upgraded my MacBook Pro to Mac OS 10.6.6. With this upgrade, came, the desktop equivalent to the App Store that’s been a large part of the success of iOS. My first impression was “this looks like Novia’s Ovi Store” — it shows a lot of applications and very little information. Looking around, it seems pretty easy to use, but I can’t see myself ever using it.

After years of installing software via `yum install $package`. I got some flak on Twitter for saying this, but the flak was crap. First, I wouldn’t expect anyone to read the man pages for a GUI app on any platform. That’s what the built-in documentation is for (and if it doesn’t exist, that’s a serious bug in the program). Secondly, I wasn’t even talking about the interface. It’s more the idea of paying for the software. Not out of greed, but out of the philosophical feelings about FLOSS.

That having been said, I think the App Store is pretty great overall. My big complaint about Mac OS X is the lack of a package management system. The ability to easily keep packages up to date is a serious strength of Linux distributions, and things like MacPorts and Fink don’t really cut it for casual users. I hope that Apple does the un-Apple thing and makes it more accessible to developers. In the meantime, it’s a great and overdue addition.