Linux and Microsoft: a “deal with the devil”?

When Microsoft and the Linux Foundation announced that Azure certification will require passing a Linux exam, it caused a great disturbance in the Force. The FOSS Force, specifically. In a column, editor-in-chief Christine Hall called the partnership a “deal with the devil.” In a news roundup, Larry Cafiero said “[r]ather than throw the Microsoft that is treading water a life preserver, I still think throwing it an anchor would be more fitting.” Larry is a personal friend of mine, and he and Hall have both been covering open source since before I got my first computer. I can’t just dismiss their opinions out of hand.

Open source enthusiasts have every right to be leery of Microsoft. Former CEO Steve Ballmer famously said Linux is “a cancer” and the company was openly hostile to the Linux project specifically and open source generally for many years. And yet, Microsoft seems to be sincere in its efforts to participate in open source projects (even if it’s still a little bit two-left-footed).

Hall said Microsoft loves Linux “because [Microsoft] can sell it”. So what? Even Red Hat loves being able to sell Linux. Azure CTO Mark Russinovich told the audience at All Things Open this year “ if we don’t support Linux and open source in our cloud then we’ll be a Windows only cloud, and that would not be practical.” Yes, it’s absolutely in Microsoft’s self-interest to play nicely with the open source world. While the Year of the Linux on the Desktop is always just out of reach, Linux is firmly entrenched in the enterprise.

Microsoft may have (as of this writing), roughly 29 times the market capitalization of Red Hat, but it’s obvious that open source has “won”. And yet, elements of the community are stuck in the scrappy underdog mindset. If we want to pretend that we’re a meritocracy, we have to be willing to allow our former enemies to become…if not friends, then at least collaborators. If Microsoft is willing to play by the rules, then let’s let them.

Forget what Hall wrote earlier this month. Let’s go with what she said in October: “However, it might be time to tone down the anti-Microsoft rhetoric a bit and give them a little breathing room. If we give them enough rope, we can see if they hang themselves, or if they use it to strengthen their ties with the open source community.”

Be careful with shell redirection

Continuing on Friday’s theme of “Ben writes about some Linux basics”, I wanted to share a story of a bug I fixed recently. Our internal documentation server at work had been a little flaky. File copies from the build server would sometimes fail and the web server was being really slow. When I logged in, I noticed the root volume was full.

A full disk is a thing that happens sometimes, especially on small volumes, so I went off in search of the culprit. It turns out that the dead.letter file in root’s home directory was large (several gigabytes, if I recall). For a couple of years, the cron job that runs every 5 minutes to update the documentation page had been trying to send email, which failed since the MTA wasn’t configured.

Why was all of this output trying to be sent via email? Because at some point someone set up the cron job with redirection like so:

2>&1  > /dev/null

Let’s take a step back for a moment and explain what that means. There are two main output streams for command line programs: standard output (a.k.a. “STDOUT”) and standard error (a.k.a. “STDERR”). The former is generally regular output, whereas the latter is for the “important stuff,” like error messages. By default when you run a command in the terminal, they both go to your terminal so you can see them. But you might not always want to see them, so you might redirect to a file or to /dev/null.

Back to our woe-beset server. At first glance, you might say “okay, so both STDOUT (1) and STDERR (2) are being sent to /dev/null”. And you would be wrong. STDERR is being sent to wherever STDOUT is being sent, which at the time is still the terminal (or the cron output email), and then STDOUT is being redirected to /dev/null. So what was in place was effectively the same as:

> /dev/null

Changing the order of the redirection to:

> /dev/null 2>&1

Kept the dead.letter file from slowly strangling the life out of the disk.

 

I find find(1) to be useful

I recently shared Tom Limoncelli’s excellent critique of the BSD find(1) man page in the documentation channel at work. One of my coworkers responded with “that’s why I just use mlocate”, and that made me very sad. Sure, mlocate is a great tool if you know there’s a file somewhere that has a particular name (assuming it was created before the last time updatedb was run), but that’s about the best you can do.

There are plenty of examples on how to use find out there, but I haven’t written a “here’s a basic thing about Linux” post in a while, so I’ll add to the pile. find takes, at a minimum, a path to find things in. For example:

find /

will find (and print) every file on the system. Probably not all that useful. You can change the path argument to narrow things down a bit, but that’s still probably not all that useful to you. So let’s throw in some additional arguments to constrain it. Maybe you want to find all the JPEG files in your home directory?

find ~ -name '*jpg'

But wait! What if some of them have an uppercase extension?

find ~ -iname '*jpg'

Aw, but I bet some of the pictures have an extension of .jpeg because 8.3 is so 1985. Well, we can combine them in a slightly ugly fashion:

find ~ \( -iname '*jpeg' -o -iname '*jpg' \)

Oh, but you have some directories that end in jpg? (Why you named a directory “bucketofjpg” instead of “pictures” is beyond me) We can modify it to only look for files!

find ~ \( -iname '*jpeg' -o -iname '*jpg' \) -type f

Or maybe you’d just like to find those directories so you can rename them later:

find ~ \( -iname '*jpeg' -o -iname '*jpg' \) -type d

It turns out you’ve been taking a lot of pictures lately, so let’s narrow this down to ones whose status has changed in the last week.

find ~ \( -iname '*jpeg' -o -iname '*jpg' \) -type f -ctime -7

You can do time filters based on file status change time (ctime), modification time (mtime), or access time (atime). These are in days, so if you want finer-grained control, you can express it in minutes instead (cmin, mmin, and amin, respectively). Unless you know exactly the time you want, you’ll probably prefix the number with a + (more than) or – (less than). The time arguments are probably the ones I use most often.

Maybe you’re running out of disk space, so you want to find all of the gigantic (let’s define that as greater than 1 gigabyte) files in the log directory:

find /var/log -size +1G

Or maybe you want to find all the files owned by bcotton in /data:

find /data -owner bcotton

You can also look for files based on permissions. Perhaps you want to find all of the world-readable files in your home directory to make sure you’re not oversharing.

find ~ -perm -o=r

So far, all we’ve done is print the file paths, which is useful, but sometimes you want to do more. find has a few built in actions (like -delete), but it’s true power comes in giving input for other commands to act on. In the simplest case, you can pipe the output to something like xargs. There’s also the -exec action, which allows you to execute more complicated actions against the output. For example, if you wanted to get the md5sum of all of your Python scripts:

find ~ -type f -name '*.py' -exec md5sum {} \;

(Yes, you could pipe to xargs here, too, but that’s not the point.) Note the \; at the end. That’s very important.

Warning! You can really cause a world of hurt if you’re not careful with the output of find. Files that contain spaces or other special characters might cause unexpected behavior when passed to another command. Be very careful. One way to mitigate your risk is to use -ok instead of -exec. This prompts you before executing each line (but it might get tedious if you have a lot of lines to process). The -ls action escapes special characters, so that might be useful when piping to another program.

This post only begins to scratch the surface of what find can do. Combining tests with boolean logic can give you incredible flexibility to find exactly the files you’re looking for. Have any favorite find expressions? Share them in the comments!

 

 

Debian “drops” the Linux Standard Base

LWN recently reported on a decision by the Debian community to drop most support for the Linux Standard Base (LSB). The LSB is an attempt to define as standard for compatibility across Linux distributions. Even binaries should JustWork™ on multiple distributions. At work, I take advantage of this: for many packages we use the same binaries across CentOS, Ubuntu, and SLES.

I can’t blame the Debian maintainers for not wanting to continue putting in the effort. The LSB is a large standard set and very few applications have been officially LSB certified. In addition, the LSB’s selection of RPM as the package manager puts the spec at odds with Debian anyway.

Debian’s unwillingness to put effort into keeping up with the LSB doesn’t necessarily mean that it will suddenly become incompatible with other distributions. Debian plans to continue complying with the Filesystem Hierarchy Standard, a subset of the LSB that defines what files and directories go where. I suspect this is the key standard for many people who work across distributions anyway.

In the short term, this seems like a non-story. In the longer term, I wonder what will become of the Linux ecosystem. Running a single distribution is herding cats on the best of days. Coordinating standards across multiple distributions, even with common upstreams, is madness. Among the major distributions, there are basically two camps: Debian/Ubuntu and Fedora/RHEL (and RHEL-alikes). They’ve managed not to drift too far apart, thought I thought systemd would start that process.

To many, “Linux” (as an OS, not a kernel) is a single entity. Others don’t even realize that Ubuntu and Fedora are in any way related. While reality is (sort of) closer to the former currently, I wonder if we’ll get to a point where it’s closer to the latter. Standards are important but are useful only to the degree that they are complied with. Linux has avoided the competing standards problem so far, but will that remain the case?

elementary misses the point

A recent post on the elementary blog about how they ask for payment on download created a bit of a stir this week. One particular sentence struck a nerve (it has since been removed from the post): “We want users to understand that they’re pretty much cheating the system when they choose not to pay for software.”

No, they aren’t. I understand that people want to get paid for their work. It’s only natural. Especially when you’d really like that work to be what puts food on the table and not something you do after you work a full week for someone else. I certainly don’t begrudge developers asking for money. I don’t even begrudge requiring payment before being able to download the software. The developers are absolutely right when they say “elementary is under no obligation to release our compiled operating system for free download.”

Getting paid for developing open source software is not antithetical to open source or free (libre) software principles. Neither the OSI’s Open Source Definition nor the Free Software Foundation’s Free Software Definition necessarily preclude a developer from charging for works. That most software that’s free-as-in-freedom is also free-as-in-beer is true, but irrelevant. Even elementary touts the gratis nature of their work on the front page (talk about mixed messages):

100% free, both in terms of pricing and licensing. But you're a cheater if you take the free option.

100% free, both in terms of pricing and licensing. But you’re a cheater if you take the free option.

Simply put, the developers cannot choose to offer their work for free and then get mad when people take them up on the offer. Worse, they cannot alienate their community by calling them cheaters. Of the money the elementary receives, how much of it goes upstream to the Linux Foundation, the FSF, and the numerous other projects that make elementary possible? Surely they wouldn’t be so hypocritical as to take the work of others for free?

An open source project is more than just coders. It’s more than just coders and funders. A truly healthy project of any appreciable size will have people who contribute in various ways: writing documentation; providing support on mailing lists, fora, etc.; triaging bug reports; filing bug reports; doing design; marketing (including word-of-mouth). This work is important to the project, too, and should be considered an in-kind form of payment.

It’s up to each project to decide what they want in return for the work put in. But it’s up to each project to accept that people will take from all of the choices that are available. If that includes “I get it for free”, then the right answer is to find ways for those people to become a part of the community and contribute how they can.

On Linus Torvalds and communities

This week, the Internet was ablaze with reactions to comments made by Linus Torvalds at Linux.conf.au. Unsurprisingly, Torvalds defended the tone he employs on the Linux kernel mailing list, where he holds no punches. “I’m not a nice person, and I don’t care about you. I care about the technology and the kernel—that’s what’s important to me,” he said (as reported by Ars Technica). He later said “all that [diversity] stuff is just details and not really important.”

The reactions were mixed. Some were upset at the fact that an influential figure like Torvalds didn’t take the opportunity to address what they see as a major issue in the Linux community. Others dismissed those who were upset by pointing to the technical quality of Linux, cultural differences, etc.

I don’t subscribe to the LKML, so most of the posts I’ve seen are generally when someone is trying to point out a specific event (whether a behavior or a technical discussion), and I don’t claim to have a good sense for what that particular mailing list is like. Torvalds and the Linux community have developed a great technical product, but the community needs work.

Speaking to open source communities in general, too many people use the impersonal nature of email to mistake rudeness for directness. Direct and honest technical criticisms are a vital part of any collaborative development. Insults and viciousness are not. Some people thrive in (or at least tolerate) those kinds of environments, but they are incredibly off-putting to everyone else, particularly newcomers.

Open source communities, like any community, need to be welcoming to new members. This allows for the infusion of new ideas and new perspectives: some of which will be obnoxiously naive, some of which will be positively transformative. The naive posts of newcomers can be taxing when you’ve seen the same thing hundreds of times, but everyone has to learn somewhere. The solution is to have a team armed with pre-written responses in order to prevent frustrated emails.

Not being a jerk doesn’t just mean tolerating noobs, though. Communities should have an established code of conduct which addresses both annoying and mean actors. When the code of contact is being repeatedly breached, the violator needs to be nudged in the right direction. When a community is welcoming and actively works to remain that way, it thrives. That’s how it can get the diversity of ideas and grow the technical competency that Linus Torvalds so desires.

How Richard Stallman got me to ponder extremism

This evening, I had the opportunity to attend a speech by a man whose work over the past decades enters into my life on a daily basis. The Network for Computational Nanotechnology at Purdue hosted Richard Stallman, the founder of the GNU Project and the Free Software Foundation. Stallman is a well-known and controversial figure, not only because of his technical work, but also (primarily?) because of his idealism and activism. His un-nuanced views and public lack of tact have driven fans of his work away from the FSF. I went into the talk expecting some pot-stirring. I didn’t expect to walk out deep in thought.

Stallman opened with a discussion of terminology, drawing a distinction between free (for the purposes of this post, free software means libre, not gratis) and proprietary software.  It is an ethical, social, and political distinction, not a technical one. Free software, Stallman argues, is a contribution to society. Proprietary software is morally unjust. Stallman prefers, given the choice between writing proprietary software and doing nothing, that developers do nothing. Even though free software is often available at no cost, encouraging the adoption of free software should be framed as a moral issue, not an economic or practical one. Software as a Service (SaaS) is morally equal to proprietary software in Stallman’s view, regardless of the licensing of the software, because users have no control over it.

During the question-and-answer session at the end, this view brought a heated discussion from NCN director Dr. Gerhard Klimeck. NCN runs nanoHUB, which is effectively SaaS for nanotechnology simulation. Stallman seemed to argue that it was a niche and not really important to discuss. He also semi-adroitly dodged the question of how developers can make money with free software, only asserting that it is being done without providing the [mostly student] audience any insights as to how.

Stallman’s views are based on his personal morality and seem to be absolute. This is what occupied my thoughts on the walk back to my car. Because I largely agree with Stallman, I’ve been inclined to see his extremism as an annoying, but useful thing. By being on the edge, he defines the middle. But why should extremism that I happen to generally agree with be more valid than extremism that I disagree with? While extremism does help define the middle ground, it also poisons reasonable discussion. I admire and appreciate his technical accomplishments, but I think he hurts his own ideological cause.

Fedora 16 released

It’s that time again — a new release of Fedora is here! I’m about to eat my own dog food and upgrade, so while I do that, why don’t you check out the Release Announcement? This release announcement holds a special place in my heart because I mostly wrote it (along with contributions from others, of course!). That’s right, I’ve actually made a contribution. It sets a dangerous precedent, but I found writing the RA quite enjoyable. I’m particularly proud of my Jules Verne references in each of the “What’s New” subsections. Fortunately, we’ve got a little while to come up with “Beefy Miracle”-themed one liners.

So even though I haven’t yet installed it, I’m confident that Fedora 16 is just as great as the last 15 versions. I’ll have some notes about the upgrade process once it finishes.

Log file myopia

I like to consider myself an enlightened sysadmin. I know I’m supposed to think outside the box from 30,000 feet. Still, every so often, my blinders come back on and I tend to be a bit myopic. This is most common when looking at log files in search of clues to an unknown problem.  A recent example was when I was trying to figure out why the Condor startd wasn’t running on a CentOS VM I had set up.

Since I didn’t want to have hundreds of ‘localhost.localdomain’s in the pool, I needed a way to give each VM a unique-ish and relevant name.  The easiest way seemed to be to check against a web server and use the output of a CGI script to set the host name.  Sounds simple enough, but after I put that in place the startd would immediately segfault.

I had no idea why, so I started peeking through the log file for the startd.  Lots of information there, but nothing particularly helpful.  After several hours of cross-eyed log reading and fruitless Googling, I thought I’d give strace a try.  I don’t know much about system-level programming, but I thought something might provide a clue.  Alas, it was not to be.

Eventually, I remembered that there’s a master log for Condor as well, and I decided to look in there.  Well, actually, I had looked in there earlier in the day and hadn’t seen anything that I thought was helpful.  This time I took a closer look and realized that it couldn’t resolve its host name and that’s why it was failing.

A few minutes later and I had changed the network setup to add the hostname to /etc/hosts so that Condor could resolve it’s host name.  A whole day’s worth of effort because I got too focused in on the wrong log file.

Sometimes, Windows wins

It should be clear by now that I am an advocate of free software.  I’m not reflexively against closed software though, sometimes it’s the right tool for the job.  Use of Windows is not a reason for mockery.  In fact, I’ve found one situation where I like the way Windows works better.

As part of our efforts to use Condor for power saving, I thought it would be a great idea if we could calculate the power savings based on the actual power usage of the machines.  The plan was to have Cycle Server aggregate the time in hibernate state for each model and then multiply that by the power draw for the model.  Since Condor doesn’t note the hardware model, I needed to write a STARTD_CRON module to determine this.  The only limitations I had were that I couldn’t depend on root/administrator privileges or on particular software packages being installed. (The execute nodes are in departments across campus and mostly not under my control.)

Despite the lack of useful tools like grep, sed, and awk (there are equivalents for some of the taken-for-granted GNU tools, but they frankly aren’t very good), the plugin for Windows was very easy.  The systeminfo command gives all kinds of useful, parseable information about the system’s hardware and OS.  The only difficult part was chopping the blank spaces off the end of the output. I wanted to do this in Perl, but that’s not guaranteed to be installed on Windows machines, and I had some difficulty getting a standalone-compiled version working consistently.

On Linux, parsing the output is easy.  The hard part was getting the information at all.  dmidecode seems to be ubiquitous, but it requires root privileges to get any information.  I tried lshw, lshal, and the entire /proc tree.  /proc didn’t have the information I need, and the two commands were not necessarily a part of the “base” install.  The solution seemed to be to require the addition of a package (or bundling a binary for lshw in our Condor distribution).

Eventually, we decided that it was more effort than it was worth to come up with a reliable module.  While both platforms had problems, Linux was definitely the more difficult.  It’s a somewhat rare condition, but there are times when Windows wins.