File a bug, but where?

It’s easy to tell someone to file a bug. Knowing where to file it isn’t quite as simple. In a large project like Fedora, there is no One Tracker to Rule Them All.

Red Hat Bugzilla comes the closest. That’s where most software bugs should be filed. Except for Fedora CoreOS which would rather you use their GitHub tracker. And a few other components, which would very much prefer issues be filed upstream.

But for non-software issues, Bugzilla may or may not be the place. Components still exist for documentation and websites, although those are mostly handled via Pagure repos now. (I should really remove those components.) And some teams discuss software features in their Pagure repo for collaboration and then might open a Bugzilla bug if other tools need one.

Of course, knowing which platform is only the start. You also have to know which component (in Bugzilla) or repo (in Pagure) is right. This isn’t always clear when you’re familiar with the inner workings of the project and is entirely opaque to outsiders. A casual user is not going to know (or care) which URLs happen to be owned by the websites team versus the docs team versus who knows what other team.

And that, dear reader, is one of the key features of a centralized tracker like Bugzilla: mis-filed bugs can be trivially passed to the right team. With independent repos, you force users to know your project’s org chart

Of course, Bugzilla can be pretty heavy and is generally more complicated than you need. There’s no right answer. There’s just how you’re willing to balance the tradeoffs.

In general, I think putting software issues (bugs, features, etc) in Bugzilla and process issues in a team’s Pagure repo is the right approach. But ultimately, teams decide for themselves how they will work.

[If you are intrigued by this half-baked post, you’ll enjoy my book on program management for open source projects, coming from The Pragmatic Bookshelf in 2022.]

Book review: A Place to Start

Jim Grey is a pretty ordinary guy. So why bother reading stories of his life? Because he tells them so well.

I first came to know of Jim through his roads website. At some point, I started reading his blog “Down the Road“. I can’t recall how long it’s been now, but it’s probably the better part of a decade, if not from the beginning in 2007. Although we’ve never met, I’ve come to feel like I know him. Not just because we share similar interests and have mutual friends, but because the way he is able to write his personal stories in a way that welcome strangers in.

Jim does not share every detail; his writing respects the privacy of those in his life. Nonetheless, he is able to warmly and openly share his stories in a way that invites to sit down and listen. There is no false modesty, no exaggeration, and no self-importance. Just honest tales of his life shared because he wants to share them.

Down the Road started in 2007 as a way to process and recover from a rough time in his life. Jim’s new (okay, six months old at this point) book A Place to Start collects some of the posts from the first two years of the blog.

The book is split into three parts: stories, essays, and faith. The stories are personal tales from all eras of Jim’s life. Told in no discernible order, they’re more like a conversation than a timeline. The short essays section contains reflections on lessons Jim has learned over the years. The faith section contains a mix of his personal experiences with his Christian faith along with what I would call sermons.

Admittedly, the faith section was the least engaging part for me. Being religiously indifferent myself, I suppose I’m not inclined toward that kind of story. Nonetheless, I do enjoy the way he is able to discuss his faith in a way that does not feel like evangelism. He shares his beliefs and how they color his life; the reader is free to do with that what they will.

Since each chapter is a blog post, they’re all short. This makes A Place to Start a great book for when you can only read in quick bursts. Even if you’ve never heard of Jim Grey before, this book with worth a read.

A Place to Start is available in print and digital forms from Midnight Star Press. I received no compensation for this review.

Other writing: March 2021

What have I been writing when I haven’t been writing here?

Now you see me!

Stuff I wrote

Fedora

Stuff I curated

Fedora

Projects shouldn’t write their own tools

Over the weekend, the PHP project learned that its git server had been compromised. Attackers inserted malicious code into the repo. This is very bad. As a result, the project moved development to GitHub.

It’s easy to say that open source projects should run their own infrastructure. It’s harder to do that successfully. The challenges compound when you add in writing the infrastructure applications.

I understand the appeal. It’s zero-price (to write; you still need the hardware to run it). Bespoke software meets your needs exactly. And it can be a fun diversion from the main thing you’re working on: who doesn’t like going to chase a shiny for a little bit?

Of course, there’s always the matter of “the thing you wanted didn’t exist when you started the project.” PHP’s first release predates the launch of GitHub by 13 years. It’s 10 years older than git, even.

Of course, this means that at some point PHP moved from some other version control system to Git. That also means they could have moved from their homegrown platform to GitHub. I understand why they’d want to avoid the pain of making that switch, but sometimes it’s worthwhile.

Writing secure and reliable infrastructure is hard. For most projects, the effort and risk of writing their own tooling isn’t worth the benefit. If the core mission of your project isn’t the production of infrastructure applications, don’t write it.

Sidebar: Self-hosting

The question of whether or not to write your infrastructure applications is different from the question of whether or not to self-host. While the former has a pretty easy answer of “no”, the latter is mixed. Self-hosting still costs time and resources, but it allows for customization and integration that might be difficult with software-as-a-service. It also avoids being at the whims of a third party who may or may not share your project’s values. But in general, projects should do the minimum that they can reasonably justify. Sometimes that means running your own instances of an infrastructure application. Very rarely does it mean writing a bespoke infrastructure application.

Indiana COVID-19 update: 27 March 2021

I haven’t written an update in nearly two full months. This is only slightly due to laziness. Mostly, it’s because the state’s numbers have been unremarkable. I mean that in a good way. We’ve consistently trended downward in infections (and positivity), hospitalizations, and deaths. However, we seem to have reached the floor and I’m concerned that the early indications suggest an increase. As usual, I am updating my dashboard most days.

Changes in trends

Five days in the past week (including the three most recent) had a daily increase in the hospitalization count. Two days had a 7% or greater daily increase. These are some of the largest increases on record. The hospitalization count is 12% higher than a week ago. Meanwhile, we’ve had a full week of week-over-week increases in positive cases. Only one day in the last 10 showed a week-over-week decline.

Daily and week-over-week changes in COVID-19 hospitalizations in Indiana.

Deaths continue to slowly trend downward, but if the increase in hospitalization holds, expect an uptick in deaths soon. After under-predicting the deaths during the surge in the fall, the Institute for Health Metrics and Evaluation (IHME) models are consistently over-predicting deaths. However, it does not appear that the most recent model run takes the end of the mask mandate (see “Changes in policy”), so it will be interesting to see how they fare in a month.

Observed and forecasted COVID-19 deaths in Indiana.

Changes in behavior

The IHME’s latest policy brief says mobility in Indiana is 8% below the pre-pandemic baseline. This is a big jump from the 20–25% of just a month or so ago. Meanwhile, mask usage has fallen slightly to 72%. Given these, it’s not hard to see why the numbers are picking back up again. These are both trends specifically called out as factoring into the “worst-case scenario.” (Note that the “reference scenario”—or most likely scenario—is what I plot on my dashboard.)

Changes in policy

Governor Holcomb announced earlier this week that the mask mandate will be gone on April 6. This is bad policy. While all Hoosiers 16+ will be eligible for vaccination beginning on March 31, the earliest a newly-eligible person will be fully vaccinated is April 14. And that assumes that the person can get the vaccine that day and that they receive the one-dose Johnson & Johnson vaccine. Recall that the CDC says people reach full vaccination two weeks after receiving the last shot.

In other words, in the impossibly-best case scenario, the mask mandate ends a week before the state’s (adult) population is fully vaccinated. The more likely case is that we don’t hit the 70% threshold for weeks, perhaps months. Holcomb says “Hoosiers know the science” and will continue to wear masks and follow distancing guidelines once the mandate becomes an advisory. I wonder what Hoosiers the governor has been talking to. Considering how many crowded restaurant parking lots and improperly-worn masks I’ve seen in the past week, I don’t believe him.

This is infuriating, because a mask mandate is essentially free. This is particularly true given how little effort the state put in to enforcing it. Ending the mandate early sends the wrong message. We can only hope that vaccinations outpace the virus. We deserve better than this.

The FSF does not represent my views

Earlier this week, Richard Stallman announced that he was rejoining the board of the Free Software Foundation. You may recall that he resigned as president and board member in 2019 after making unacceptable remarks about the sexual assault of a minor. This was not the first instance of unacceptable behavior. The FSF made no real changes to address the issue and now has welcomed Stallman back.

I’m thankful that the people I choose to associate with have universally condemned this as harmful. I wrote in 2012 that I think he hurts his own ideological cause. At the time I wrote the post, I was thinking entirely of his rigid aherence to free software over all else. In truth, the harm he does goes well beyond that. For me, the licensing terms of free and open source software are not as important as the human impact.

As I wrote last month, free and open source software is not the end goal. What good is free software that is used to harm others? And what good is a free software movement that is not willing to include underindexed groups. We cannot tolerate nor enable this sort of behavior.

The Fedora Council spent a lot of time debating our vision statement.

The Fedora Project envisions a world where everyone benefits from free and open source software built by inclusive, welcoming, and open-minded communities.

Fedora PRoject vision

The inclusion of the “built by” is no accident. We want our community to be vibrant and healthy. That cannot happen when bad behavior is allowed to persist.

I think it’s too late for the FSF. They’ve painted themselves into a corner long ago. This only cements that. Still, perhaps with a new slate, the organization can be reborn into something that aids the cause it purports to champion. That is why I have signed the open letter calling for the resignation of Stallman and the entire FSF Board of Directors.

Applying the Potter Stewart rule to release blockers

I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description, and perhaps I could never succeed in intelligibly doing so. But I know it when I see it.

Justice Potter Stewart in Jacobellis v. Ohio

Potter Stewart was talking bout hard-core pornography when he wrote “I know it when I see it”, but the principle also applies to release blockers. These are bugs that are so bad that you can’t release your software until they are fixed.Most bugs do not fall into this category. It would be nice to fix them, of course, but they’re not world-stopping.

Making a predictable process

For the most part, you want your release blockers to be defined by specific criteria. It’s even better if automated testing can use the criteria, but that’s not always possible. The point is that you want the process to be predictable.

A predictable blocker process provides value and clarity throughout the process. Developers know what the rules are so they can take care to fix (or avoid!) potential blockers early. Testers know what tests to prioritize. Users know that, while the software might do something annoying, it won’t turn the hard drive into a pile of melted metal, for example. And everyone knows that the release won’t be delayed indefinitely.

Having a predictable blocker process means not only developing the criteria, but sticking to the criteria. If a bug doesn’t violate a criterion, you can’t call it a blocker. Of course, it’s impossible to come up with every possible reason why you might want to block a release, so sometimes you have to add a new criterion.

Breaking the predictable process

This came up in last week’s Go/No-Go meeting for Fedora Linux 34 Beta. We were considering a bug that caused a delay of up to two minutes before gnome-initial-setup started. I argued that this should be a blocker because of the negative reputational impact, despite the fact that it does not break any specific release criterion. I didn’t want first time users (or worse: reviewers) to turn away from Fedora Linux because of this bug.

QA wizard Adam Williamson argued against calling it a blocker without developing a specific criterion it violates. To do otherwise, he said, makes a mockery of the process. I understand his position, but I disagree. While there are a variety of reasons to have release blockers, I see the preservation of reputation as the most important. In other words, the point of the process is to prevent the release of software so bad that it drives away current and potential users.

Admittedly, that is an entirely subjective opinion. I expect disagreement on it. But if you accept my premise and acknowledge that you can’t pre-write criteria to catch every possible, then it follows that squishy “I know it when I see it” rules are sometimes okay.

Can’t you just make that a criterion?

The best blocking criteria are objective. This aids automated testing and (mostly) avoids arguments about interpretation. But that’s not always possible. Even saying “feature X works” is open to argument over what constitutes “working”.

The challenge lies in how to incorporate “this bug is bad even though there’s no specific rule” without making the process unpredictable. In this case, Adam’s position makes a lot of sense. It’s much easier to write rules to address specific issues and apply them retroactively. Of course, doing that in a go/no-go meeting is perhaps straining the word “predictable” a bit, too.

So what happened?

In the case of this specific bug, we had an escape hatch. The release was likely to be declared no-go for other reasons, so we didn’t need to come to a decision either way. With a candidate fix available, we could just pull that in as a freeze exception and write a new criterion for the next time.

Because of this, I decided not to push my argument. I declined to propose a criterion in-meeting because I wanted to take some time to think about what the right approach is. I also wanted to spend some time thinking about the blocker process holistically. This gives me a blog post to publish (hi!) and some content for a project I’ll be announcing in the near future. In the meantime, I’ve proposed a change to the criteria.

Other writing: February 2021

Where have I been writing when I haven’t been writing here?

Stuff I wrote

Fedora

Stuff I curated

Fedora

Book review: Range

In many parts of society, we ask people to specialize early and go very deep. This is the path to excellence. In Range: why generalists triumph in a specialized world, David Epstein examines the role breadth plays. I should admit my bias up front: I am definitely a width person, not a depth person. So maybe I just agreed with this book because it reinforced the story I tell myself about my success.

But I do think there’s something to this. Throughout my career, I’ve found that the best colleagues are the ones who have academic or work experience outside of the tech industry. It’s not that they’re necessarily better technically, but they grasp the context much more easily. That becomes increasingly important when dealing with novel and poorly-defined problems.

I’ve long understood the value of coursework outside one’s major. Range helped me understand why that value exists. I sometimes heard at my alma mater that “we have a liberal arts school so we can produce well-rounded engineers.” Now I think perhaps we should have fewer major courses and more gen ed courses. (In addition to ethics classes which should be added to all curricula for separate reasons.)

In the context of the current time, with conspiracy theories enjoying a disturbing degree of acceptance, I find Epstein’s emphasis on amateurs a little concerning. Yes, novices sometimes make discoveries that elude the experts. Still, we must be careful not to replace “appeal to authority” with “appeal to lack of authority”.

I didn’t find Epstein’s writing style particularly compelling. This surprised me since he’s a journalist. I suppose books are a different beast. But the arguments were well-reasoned and supported by research. I would recommend this book to anyone thinking about their future career or seeking reinforcement of their past, seemingly-odd, changes in direction.

On the “commercial side of cancel culture”

Writing on his blog last week, Evan Brown said about “morals clauses” in contracts:

These clauses provide the means for the commercial side of cancel culture to flourish.

Evan Brown

Evan is a fellow 812 native and a person for whom I have tremendous respect. I wouldn’t think to argue with him on a matter of law, but this is a matter of culture, so I will.

We’re starting from different points. Evan clearly believes that “cancel culture” is a real concern. I don’t. I’ll grant that there is a possible extreme that would be a problem, but I do not believe we are there or that we are approaching it. What is called “cancel culture” is often “facing accountability for one’s improper actions.”

To that end, I proposed that the “commercial side of cancel culture” could also be called “the free market”. This specific case—Jeep pulling use of an ad featuring Bruce Springsteen after news came out that he recently had a DUI arrest—is a particularly bad example to use to make the point, too. First of all, it is very reasonable that a vehicle company would want to dissociate from someone who recently had a drunk driving arrest. Secondly, the harm is not on the person “canceled” (to the degree that Bruce Springsteen can be canceled).

Springsteen presumably got paid already (although there may be a clawback clause or a payment structure that has not fully completed yet). He doesn’t need the money or the fame. Meanwhile, Jeep produced an ad and purchased air time for it. They probably planned for a much longer run over which to recoup their expenses. I’m not suggesting we feel sorry for Jeep, but I also don’t think we need to shed any tears for The Boss.