Using systemd tmpfiles

As part of my regular duties at my day job, I provide some community support on the HTCondor users’ mailing list. At the beginning of the week, someone came to the list with Fedora RPM troubles. I was able to track it down to the fact that the RPM didn’t correctly create some temporary directories.

But why not? It’s been a while since I did any RPM building, but at first glance, the spec file looked fine. The systemd tmpfiles configuration looked good, too. Then someone else found the cause of the problem: according to the Fedora packaging guidelines, a package should write the config file andcreate the directory. It turns out that tmpfiles.d is only processed at boot time (or when manually invoked), so a package that needs a file/directory to exist needs to create it at install time (or require a restart).

I was going to file an RFE, but I found a related one marked as CLOSED->WONTFIX. I understand the reasoning here, so I won’t reopen it to argue. It’s just a surprising behavior if it’s not something you regularly deal with.

GooBuntu: playground arguments against RPM-based distributions

Full disclosure: I’ve been a contributor to the Fedora Project for several years and am nominally the maintainer of Fedora’s RPM Guide, which will probably never actually be released.

Earlier this week, ZDNet reported on Google’s use of Ubuntu on desktops. While I’m sure they chose a distro based on a variety of of factors and the final choice was the one that best met their business needs, the article could stand to include some additional detail. It appears that all RPM-based distributions were immediately disqualified because “packages and apt (Debian’s basic software package programs) are light-years ahead of RPM (Red Hat and SUSE’s default package management system.)”

I have some philosophical disagreements with how Ubuntu’s parent company operates, but I’m a big proponent of the “use what works for you” philosophy. I have no objection to Google using Ubuntu if that’s what works for them. What I do object to is basing the decision on non-reason.

The first problem is that the argument always seems to be apt vs. RPM, but this argument is a non-starter. apt and RPM aren’t at the same layer. Comparing apt and yum is more reasonable. Both tools have advantages and disadvantages. Comparisons can be made using various metrics, but there is no objective measure of “better”, because fitness for use varies by use case. Similarly, RPM and .deb have overlapping-but-not-identical strengths. The philosophy of building packages differs, and some people prefer one method over the other. I’m weakly-rooted enough to find both philosophies compelling.

I’d be willing to grant that RPM, and the tools around it, have improved over the past few years. Even if the RPM ecosystem was formerly terrible, I’d have expected that the man making the decision at Google would base it on something more substantive than what amounts to a religious argument.

LISA ’11: the first half of the week

If you’ve been following me on Twitter, you know I’ve been in Boston for the USENIX Large Installation System Administration (LISA) Conference. Once again, I have the honor of serving on the conference blog team, which means I spend all day sitting in sessions and all evening writing about them. We’re halfway through now, so here’s what I’ve written so far:

You can follow along with the rest of the blog team at

Google Earth gets packaging wrong

A while back, I needed to install Google Earth on my 64-bit Fedora 15 machine. Being a smart guy, I grabbed the 64-bit RPM and installed it. When I launched it, it failed because it needed a library provided by the redhat-lsb package. After installing redhat-lsb, I expected Google Earth to work. But instead, I was greeted with the same error. That’s when I began to realize that it was looking for /lib/ not /lib64/ So I checked the binary. It’s 32-bit.

To me, this seems like the exact wrong way to package this. The best way would be to build a 64-bit version and package that for 64-bit machines. The acceptable way would be to say “sorry, we only offer 32-bit builds of Google Earth”. The bad way is to pretend the 32-bit package is really 64-bit. This does not make installation more convenient, as Google says, it makes it less convenience because it breaks dependency solving. yum dutifully installs the 64-bit redhat-lsb because that’s what the package says it needs. It’s only after the binary is in place that it becomes evident that the 32-bit redhat-lsb package is needed.

If this bothers you, too, I’ve left feedback for Google. Feel free to pile on.

The tricky problem dilemma

A good sysadmin believes in treating the cause, not the symptom. Unfortunately, pragmatism sometimes gets in the way of that. A recent example: we just rolled out a kernel update to a few of our compute clusters. About 3% of the machines ended up in a troubled state. By troubled, I mean that the permissions on a few directories (/bin, /lib, /dev, /etc, /proc, and /sys) were set to 700, making the machine effectively unusable. For the most part, we didn’t notice this on the affected machines until after they did their post-upgrade reboot, but fortunately we were able to catch a few that hadn’t yet rebooted.

What we found was that / had a sysroot directory and an init file. These are created by the mkinitrd script, which is called by the new-kernel-pkg script, which is in turn called in the postinstall script of the kernel RPM. The relevant part of the mkinitrd script seems to be

    for t in /tmp /var/tmp /root ${PWD}; do
        if [ ! -d $t ]; then continue; fi
        if ! access -w $t ; then continue; fi

        fs=$(df -T $t 2>/dev/null | awk '{line=$1;} END {printf $2;}')
        if [ "$fs" != "tmpfs" ]; then

which creates a working directory in /tmp under normal conditions. However, there seemed to be something that caused / to be used instead of /tmp. Later in the script, several directories are created in $TMPDIR, which correspond to the wrongly-permissioned directories. There’s not a clear indication of why this happens, but if we clean up and reinstall the updated kernel package it doesn’t necessarily repeat itself. After some soul-searching, we decided that it was more important to return the nodes to service than to try to track down an easily-correctable-but-difficult-to-solve problem. We’ll see if it happens again with the next kernel upgrade.

Using bookmark synchronization on Google Chrome for Linux and Mac

For a long time, I blamed the sluggish performance of the web browser on my Linux machine at home on the ancientness of the hardware.  However, when my much nicer Linux machine at work showed the same problem, I began to think maybe it was just Firefox.  I’ve been an avid Firefox user for many years, but my loyalty wavers when my browser can’t keep up with my keyboard.  Based on the advice of strangers on the Internet, I decided to give Google’s Chrome browser a try.

Chrome is still a maturing browser, but it is fast and capable.  There’s only one real drawback: bookmark synchronization.  With Firefox, I had been using Xmarks to synchronize my bookmarks, but that’s not currently available for Chrome.  In the “Early Access” builds of the Linux and Mac versions of Chrome, the bookmark sync that the Windows version has is available.  This syncs the bookmarks to your Google Docs account, which makes it rather handy.  However, synchronization is not enabled by default.  To enable it, you have to pass the –enable-sync option at launch time, which is easier said than done.  Fortunately, it’s not too terribly difficult.

Continue reading