When to have a bug tracker

Back in April, I wrote a post about how the traditional “file a bug” or “patches welcome” responses to issues weren’t always helpful. For as long as open source software has been something I think about, I’ve been of the mind that bug trackers are a necessary component of a project. After all, if we’re going to take advantage of the “given enough eyeballs, all bugs are shallow” philosophy, someone the eyeballs have to be able to communicate information about the bugs.

But then “magnus” made a comment that caused me to question that requirement.

A free sw project should not have a bug tracker unless there is enough organization behind to actually service the reports IMO. Unfortunately many hosting sites have the mentality that they throw in one for free in every project since it “can’t hurt”.

The more I thought about this, the more I agree with it. I think there’s active harm in having a bug tracker that is ignored. Without a bug tracker, someone might file a bug via email, or maybe not at all. Worst case, they probably just go find another project that meets their needs. But once the door is open to filing bugs, people have an expectation that the bugs will be fixed. Bugs that sit in the tracker forever don’t make people indifferent, they make people angry.

If your project lacks the resources or the desire to manage bugs, then it’s probably best to not have a bug tracker. When I say “manage” bugs, I don’t necessarily mean fixing them. Bugs need to be triaged, they need to be verified, they need to be fixed. Critically, the people filing the bugs need to feel like their contributions are valuable. Of course, the contributions are sometimes not valuable, but that’s just an opportunity to help someone learn how to become a valuable contributor.

magnus went on to talk about how bug trackers are implemented:

Also I don’t know if putting a bulky database system between the users and the developers, which makes sense in the commerical world, really makes the best use of the strength of the free software community.

Something more lightweight and free form would be better in many cases. An interface something like this very blog commenting interface could be appropriate. Here I just write what I think and I don’t feel any social expectation that you will reply to it (but would be fun if you did).

I think Bugzilla is a great platform for certain large projects. It’s absolutely atrocious for a casual user. And it’s complete overkill for small projects. Simpler interfaces like GitHub’s Issues can often be just as valuable with a lot less friction. Something even simpler like Trello might be suitable for small projects. Whatever the developers will use works, so long as they use it.

File a bug!

The ability for users to submit patches is often touted by advocates of open source software. “Patches welcome!” is a common refrain on mailing lists. Even if someone lacks the ability to diagnose and fix an issue, they can file a bug in the project’s system. Indeed, there’s an unwritten social expectation that users file a bug.

Sadly, these are often just empty words. “Patches welcome” can be a seemingly-polite way of saying “your problem is not important to me. Go solve it yourself and I’ll accept your solution.” And telling a user to go file a bug can be equally dismissive, especially if the bug-filing process is unpleasant.

I certainly don’t mean to imply that all or even most open source developers use these phrases as a polite cover for their disinterest. There are jerks, of course, but I suspect most developers genuinely want bug reports and patches. Last week, a very busy developer replied to a mailing list post with a terse “file a bug.” Now, I happen to know this particular developer and I know he’s keenly interested in fixing bugs. On this day, he was swamped and didn’t have time to get to the issue right away. Suspecting from the original email that the user who reported the bug wasn’t deeply technical, I took the liberty of reproducing the issue and filing a bug with the details.

Would the person who originally reported the issue have filed the bug done so if I hadn’t? We’ll never know, but I do know that he didn’t by the time I did. After creating a Red Hat Bugzilla account, selecting the right options, and filling out the bug report, he’d have to hope the developer meant it and that he bug would get fixed. As anyone who has been around a while can attest, just because a bug is filed, that doesn’t guarantee it will be fixed in the next few years.

One particular bug that I filed sticks in my memory. It was undeniably a bug, I attached a patch to fix it, and yet the bug remained untouched through several Fedora releases. Granted, it was a relatively minor issue, but if I weren’t involved with the project, I’d have been pretty put off by this. Of course, no one owes me a fix, but as Chris Siebenmann noted, if there’s a social obligation to file a bug, there’s also a social obligation to deal with the bug.

This morning,I asked on Twitter if anyone had what they’d call a “positive” bug reporting experience (as opposed to simply a not-negative experience). I was pleasantly surprised when several people chimed in to say they had. Small projects were generally the ones that drew the good responses (Jordan Sissel was mentioned by multiple people).

So what makes for a positive experience? Rapid acknowledgement and resolution were mentioned several times. Those who replied mentioned courteous and helpful interactions (e.g. asking constructive questions, filing an appropriate bug when the original bug was mis-filed). @phrawzty summed it up well: “Most importantly, however: good intentions were assumed. I never felt treated like a dangerous outsider.”

This is an area wide open for study (and I may end up doing just that), but in the meantime projects should consider how they present themselves. Large projects with more resources should make an active effort to devote some time to bug handling as a community outreach effort (I suspect large projects have worse bug experiences, in part due to the fact that nobody owns the process). When you say “patches welcome” or “file a bug”, consider what you’re really saying and if people will take it the way you mean it.

(Thanks to those who provided feedback: @cpj1, Matt SimmonsMatthew General, Patrick Cable@phrawzty, Rick Waldron@SysAdm DreddThomas Ballinger, @Twirrim, and anyone else I failed to mention.)

Getting good reports from users

Earlier this week, Rob Soden asked an excellent question on Twitter: how do you train your non-technical co-workers to file useful/actionable bug reports? Is it possible?

It is, but this question doesn’t just apply to developers. Operations people need useful incident reports from users as well. Users generally don’t provide unhelpful reports maliciously.. In my experience, reports are often the least useful when users are trying to be the most helpful. A healthy relationship between a customer and a service provider means that everyone is working together. Users often don’t know what information is helpful, they just want to go about their work.

A useful report contains the following:

  • What happened and what the desired result is
  • What the user was doing when it happened
  • Any changes that may have happened
  • If the product/service ever worked and if so, when it last worked
  • If the problem is reproducible and if so, what steps are required to reproduce it

A useful report does not contain speculation about what the solution is. But what, Rob asked, if the lesson doesn’t stick? It gets annoying having to ask over and over again. As with any repetitive task, the answer here is to automate it. One option is to have a canned response at the ready. This way, you polish the text once and you never have to worry about irritation creeping in. A better solution is to have a website that explicitly asks for the information you need. This saves a round of back-and-forth and hopefully increases the time to resolution.

The most important part is to modify the process as you learn. Tweak the questions to make it easier on the users. Provide multiple-choice or other defined-response choices when you can to reduce ambiguity. No good process is ever in a final state.