Debian “drops” the Linux Standard Base

LWN recently reported on a decision by the Debian community to drop most support for the Linux Standard Base (LSB). The LSB is an attempt to define as standard for compatibility across Linux distributions. Even binaries should JustWork™ on multiple distributions. At work, I take advantage of this: for many packages we use the same binaries across CentOS, Ubuntu, and SLES.

I can’t blame the Debian maintainers for not wanting to continue putting in the effort. The LSB is a large standard set and very few applications have been officially LSB certified. In addition, the LSB’s selection of RPM as the package manager puts the spec at odds with Debian anyway.

Debian’s unwillingness to put effort into keeping up with the LSB doesn’t necessarily mean that it will suddenly become incompatible with other distributions. Debian plans to continue complying with the Filesystem Hierarchy Standard, a subset of the LSB that defines what files and directories go where. I suspect this is the key standard for many people who work across distributions anyway.

In the short term, this seems like a non-story. In the longer term, I wonder what will become of the Linux ecosystem. Running a single distribution is herding cats on the best of days. Coordinating standards across multiple distributions, even with common upstreams, is madness. Among the major distributions, there are basically two camps: Debian/Ubuntu and Fedora/RHEL (and RHEL-alikes). They’ve managed not to drift too far apart, thought I thought systemd would start that process.

To many, “Linux” (as an OS, not a kernel) is a single entity. Others don’t even realize that Ubuntu and Fedora are in any way related. While reality is (sort of) closer to the former currently, I wonder if we’ll get to a point where it’s closer to the latter. Standards are important but are useful only to the degree that they are complied with. Linux has avoided the competing standards problem so far, but will that remain the case?

elementary misses the point

A recent post on the elementary blog about how they ask for payment on download created a bit of a stir this week. One particular sentence struck a nerve (it has since been removed from the post): “We want users to understand that they’re pretty much cheating the system when they choose not to pay for software.”

No, they aren’t. I understand that people want to get paid for their work. It’s only natural. Especially when you’d really like that work to be what puts food on the table and not something you do after you work a full week for someone else. I certainly don’t begrudge developers asking for money. I don’t even begrudge requiring payment before being able to download the software. The developers are absolutely right when they say “elementary is under no obligation to release our compiled operating system for free download.”

Getting paid for developing open source software is not antithetical to open source or free (libre) software principles. Neither the OSI’s Open Source Definition nor the Free Software Foundation’s Free Software Definition necessarily preclude a developer from charging for works. That most software that’s free-as-in-freedom is also free-as-in-beer is true, but irrelevant. Even elementary touts the gratis nature of their work on the front page (talk about mixed messages):

100% free, both in terms of pricing and licensing. But you're a cheater if you take the free option.

100% free, both in terms of pricing and licensing. But you’re a cheater if you take the free option.

Simply put, the developers cannot choose to offer their work for free and then get mad when people take them up on the offer. Worse, they cannot alienate their community by calling them cheaters. Of the money the elementary receives, how much of it goes upstream to the Linux Foundation, the FSF, and the numerous other projects that make elementary possible? Surely they wouldn’t be so hypocritical as to take the work of others for free?

An open source project is more than just coders. It’s more than just coders and funders. A truly healthy project of any appreciable size will have people who contribute in various ways: writing documentation; providing support on mailing lists, fora, etc.; triaging bug reports; filing bug reports; doing design; marketing (including word-of-mouth). This work is important to the project, too, and should be considered an in-kind form of payment.

It’s up to each project to decide what they want in return for the work put in. But it’s up to each project to accept that people will take from all of the choices that are available. If that includes “I get it for free”, then the right answer is to find ways for those people to become a part of the community and contribute how they can.

On Linus Torvalds and communities

This week, the Internet was ablaze with reactions to comments made by Linus Torvalds at Linux.conf.au. Unsurprisingly, Torvalds defended the tone he employs on the Linux kernel mailing list, where he holds no punches. “I’m not a nice person, and I don’t care about you. I care about the technology and the kernel—that’s what’s important to me,” he said (as reported by Ars Technica). He later said “all that [diversity] stuff is just details and not really important.”

The reactions were mixed. Some were upset at the fact that an influential figure like Torvalds didn’t take the opportunity to address what they see as a major issue in the Linux community. Others dismissed those who were upset by pointing to the technical quality of Linux, cultural differences, etc.

I don’t subscribe to the LKML, so most of the posts I’ve seen are generally when someone is trying to point out a specific event (whether a behavior or a technical discussion), and I don’t claim to have a good sense for what that particular mailing list is like. Torvalds and the Linux community have developed a great technical product, but the community needs work.

Speaking to open source communities in general, too many people use the impersonal nature of email to mistake rudeness for directness. Direct and honest technical criticisms are a vital part of any collaborative development. Insults and viciousness are not. Some people thrive in (or at least tolerate) those kinds of environments, but they are incredibly off-putting to everyone else, particularly newcomers.

Open source communities, like any community, need to be welcoming to new members. This allows for the infusion of new ideas and new perspectives: some of which will be obnoxiously naive, some of which will be positively transformative. The naive posts of newcomers can be taxing when you’ve seen the same thing hundreds of times, but everyone has to learn somewhere. The solution is to have a team armed with pre-written responses in order to prevent frustrated emails.

Not being a jerk doesn’t just mean tolerating noobs, though. Communities should have an established code of conduct which addresses both annoying and mean actors. When the code of contact is being repeatedly breached, the violator needs to be nudged in the right direction. When a community is welcoming and actively works to remain that way, it thrives. That’s how it can get the diversity of ideas and grow the technical competency that Linus Torvalds so desires.

How Richard Stallman got me to ponder extremism

This evening, I had the opportunity to attend a speech by a man whose work over the past decades enters into my life on a daily basis. The Network for Computational Nanotechnology at Purdue hosted Richard Stallman, the founder of the GNU Project and the Free Software Foundation. Stallman is a well-known and controversial figure, not only because of his technical work, but also (primarily?) because of his idealism and activism. His un-nuanced views and public lack of tact have driven fans of his work away from the FSF. I went into the talk expecting some pot-stirring. I didn’t expect to walk out deep in thought.

Stallman opened with a discussion of terminology, drawing a distinction between free (for the purposes of this post, free software means libre, not gratis) and proprietary software.  It is an ethical, social, and political distinction, not a technical one. Free software, Stallman argues, is a contribution to society. Proprietary software is morally unjust. Stallman prefers, given the choice between writing proprietary software and doing nothing, that developers do nothing. Even though free software is often available at no cost, encouraging the adoption of free software should be framed as a moral issue, not an economic or practical one. Software as a Service (SaaS) is morally equal to proprietary software in Stallman’s view, regardless of the licensing of the software, because users have no control over it.

During the question-and-answer session at the end, this view brought a heated discussion from NCN director Dr. Gerhard Klimeck. NCN runs nanoHUB, which is effectively SaaS for nanotechnology simulation. Stallman seemed to argue that it was a niche and not really important to discuss. He also semi-adroitly dodged the question of how developers can make money with free software, only asserting that it is being done without providing the [mostly student] audience any insights as to how.

Stallman’s views are based on his personal morality and seem to be absolute. This is what occupied my thoughts on the walk back to my car. Because I largely agree with Stallman, I’ve been inclined to see his extremism as an annoying, but useful thing. By being on the edge, he defines the middle. But why should extremism that I happen to generally agree with be more valid than extremism that I disagree with? While extremism does help define the middle ground, it also poisons reasonable discussion. I admire and appreciate his technical accomplishments, but I think he hurts his own ideological cause.

Fedora 16 released

It’s that time again — a new release of Fedora is here! I’m about to eat my own dog food and upgrade, so while I do that, why don’t you check out the Release Announcement? This release announcement holds a special place in my heart because I mostly wrote it (along with contributions from others, of course!). That’s right, I’ve actually made a contribution. It sets a dangerous precedent, but I found writing the RA quite enjoyable. I’m particularly proud of my Jules Verne references in each of the “What’s New” subsections. Fortunately, we’ve got a little while to come up with “Beefy Miracle”-themed one liners.

So even though I haven’t yet installed it, I’m confident that Fedora 16 is just as great as the last 15 versions. I’ll have some notes about the upgrade process once it finishes.

Log file myopia

I like to consider myself an enlightened sysadmin. I know I’m supposed to think outside the box from 30,000 feet. Still, every so often, my blinders come back on and I tend to be a bit myopic. This is most common when looking at log files in search of clues to an unknown problem.  A recent example was when I was trying to figure out why the Condor startd wasn’t running on a CentOS VM I had set up.

Since I didn’t want to have hundreds of ‘localhost.localdomain’s in the pool, I needed a way to give each VM a unique-ish and relevant name.  The easiest way seemed to be to check against a web server and use the output of a CGI script to set the host name.  Sounds simple enough, but after I put that in place the startd would immediately segfault.

I had no idea why, so I started peeking through the log file for the startd.  Lots of information there, but nothing particularly helpful.  After several hours of cross-eyed log reading and fruitless Googling, I thought I’d give strace a try.  I don’t know much about system-level programming, but I thought something might provide a clue.  Alas, it was not to be.

Eventually, I remembered that there’s a master log for Condor as well, and I decided to look in there.  Well, actually, I had looked in there earlier in the day and hadn’t seen anything that I thought was helpful.  This time I took a closer look and realized that it couldn’t resolve its host name and that’s why it was failing.

A few minutes later and I had changed the network setup to add the hostname to /etc/hosts so that Condor could resolve it’s host name.  A whole day’s worth of effort because I got too focused in on the wrong log file.

Sometimes, Windows wins

It should be clear by now that I am an advocate of free software.  I’m not reflexively against closed software though, sometimes it’s the right tool for the job.  Use of Windows is not a reason for mockery.  In fact, I’ve found one situation where I like the way Windows works better.

As part of our efforts to use Condor for power saving, I thought it would be a great idea if we could calculate the power savings based on the actual power usage of the machines.  The plan was to have Cycle Server aggregate the time in hibernate state for each model and then multiply that by the power draw for the model.  Since Condor doesn’t note the hardware model, I needed to write a STARTD_CRON module to determine this.  The only limitations I had were that I couldn’t depend on root/administrator privileges or on particular software packages being installed. (The execute nodes are in departments across campus and mostly not under my control.)

Despite the lack of useful tools like grep, sed, and awk (there are equivalents for some of the taken-for-granted GNU tools, but they frankly aren’t very good), the plugin for Windows was very easy.  The systeminfo command gives all kinds of useful, parseable information about the system’s hardware and OS.  The only difficult part was chopping the blank spaces off the end of the output. I wanted to do this in Perl, but that’s not guaranteed to be installed on Windows machines, and I had some difficulty getting a standalone-compiled version working consistently.

On Linux, parsing the output is easy.  The hard part was getting the information at all.  dmidecode seems to be ubiquitous, but it requires root privileges to get any information.  I tried lshw, lshal, and the entire /proc tree.  /proc didn’t have the information I need, and the two commands were not necessarily a part of the “base” install.  The solution seemed to be to require the addition of a package (or bundling a binary for lshw in our Condor distribution).

Eventually, we decided that it was more effort than it was worth to come up with a reliable module.  While both platforms had problems, Linux was definitely the more difficult.  It’s a somewhat rare condition, but there are times when Windows wins.

Solving the CUPS “hpcups failed” error

I thought when I took my new job that my days of dealing with printer headaches were over.  Alas, it was not to be.  A few weeks ago, I needed to print out a form for work.  I tried to print to the shared laser printer down the hall.  Nothing.  So I tried the color printer. Nothing again.  I was perplexed because both printers had worked previously, so being a moderately competent sysadmin, I looked in the CUPS logs.  I saw a line in error_log that read printer-state-message="/usr/lib/cups/filter/hpcups failed". That seemed like it was the problem, so I tried to find a solution and couldn’t come up with anything immediately.

Since a quick fix didn’t seem to be on the horizon, I decided that I had better things to do with my time and I just used my laptop to print.  That worked, so I forgot about the printing issue.  Shortly thereafter, the group that maintains the printers added the ones on our floor to their CUPS server.  I stopped CUPS on my desktop and switched to their server and printing worked again, thus I had even less incentive to track down the problem.

Fast forward to yesterday afternoon when my wife tried to print a handbill for an event she is organizing in a few weeks.  Since my desktop at home is a x86_64 Fedora 12 system, too, it didn’t surprise me too much when she told me she couldn’t print.  Sure, enough, when I checked the logs, I saw the same error.  I tried all of the regular stalling tactics: restarting CUPS, power cycling the printer, just removing the job and trying again.  Nothing worked.

The first site I found was an Ubuntu bug report which seemed to suggest maybe I should update the printer’s firmware.  That seemed like a really unappealing prospect to me, but as I scrolled down I saw comment #8.  This suggested that maybe I was looking in the wrong place for my answer.  A few lines above the hpcups line, there was an error message that read prnt/hpcups/HPCupsFilter.cpp 361: DEBUG: Bad PPD - hpPrinterLanguage not found.

A search for this brought me to a page about the latest version of hplip. Apparently, the new version required updated PPD files, which are the files that describe the printer to the print server.  In this case, updating the PPD file was simple, and didn’t involve having to find it on HP’s or a third-party website.  All I had to do was use the CUPS web interface and modify the printer, keeping everything the same except selecting the hpcups 3.10.2 driver instead of the 3.9.x that it had been using.  As soon as I made that change, printing worked exactly as expected.

The lesson here, besides the ever-present “printing is evil” is that the error message you think is the clue might not always be.  When you get stuck trying to figure a problem out, look around for other clues.  Tunnel vision only works if you’re on the right track to begin with.

Grub error 17

Earlier this week, I moved offices (hooray!), which meant I had to move all my stuff.  Since I had just moved a few months ago when I changed jobs, I had things pretty well pared down, but it still took two trips in the Jeep to get my office mate and I moved across campus.  Everything went pretty well, though, until I tried to boot my desktop.  After the BIOS screen, the video went black for several seconds and then an ominous “Error 17” came up.  That was it, no other information.

Doing some research, I found that this is the error you get when grub thinks the partitions are out of order and freaks out.  So what the heck?  I burned a Knoppix CD because I left my boot USB at home and followed the instructions on this site.  One thing I noticed was that my sda2 partition overlapped with the two partitions around it. The last cylinder of sda1 was the same as the first cylinder of sda2, and the last cylinder of sda2 was the same as the first cylinder of sda3.  Since sda2 was just my /tmp partition, it was no big deal to re-size the partition so as to not stop on any toes and then create a new filesystem on it.

I have no idea how that happened, and I would expect that if it was an existing problem it would have made itself known when I rebooted for kernel updates. Maybe the cylinders got shaken up in the move?  I didn’t think they could do that.

Using bookmark synchronization on Google Chrome for Linux and Mac

For a long time, I blamed the sluggish performance of the web browser on my Linux machine at home on the ancientness of the hardware.  However, when my much nicer Linux machine at work showed the same problem, I began to think maybe it was just Firefox.  I’ve been an avid Firefox user for many years, but my loyalty wavers when my browser can’t keep up with my keyboard.  Based on the advice of strangers on the Internet, I decided to give Google’s Chrome browser a try.

Chrome is still a maturing browser, but it is fast and capable.  There’s only one real drawback: bookmark synchronization.  With Firefox, I had been using Xmarks to synchronize my bookmarks, but that’s not currently available for Chrome.  In the “Early Access” builds of the Linux and Mac versions of Chrome, the bookmark sync that the Windows version has is available.  This syncs the bookmarks to your Google Docs account, which makes it rather handy.  However, synchronization is not enabled by default.  To enable it, you have to pass the –enable-sync option at launch time, which is easier said than done.  Fortunately, it’s not too terribly difficult.

Continue reading