Linus’s awakening

It may be the biggest story in open source in 2018, a year that saw Microsoft purchase GitHub. Linus Torvalds replaced the Code of Conflict for the Linux kernel with a Code of Conduct. In a message on the Linux Kernel Mailing List (LKML), Torvalds explained that he was taking time off to examine the way he led the kernel development community.

Torvalds has taken a lot of flak for his style over the years, including on this blog. While he has done an excellent job shepherding the technical development of the Linux kernel, his community management has often β€” to put it mildly β€” left something to be desired. Abusive and insulting behavior is corrosive to a community, and Torvalds has spent the better part of the last three decades enabling and partaking in it.

But he has seen the light, it would seem. To an outside observer, this change is rather abrupt, but it is welcome. Reaction to his message has been mixed. Some, like my friend Jono Bacon, have advocated supporting Linus in his awakening. Others take a more cynical approach:

I understand Kelly’s position. It’s frustrating to push for a more welcoming and inclusive community only to be met with insults and then when someone finally comes around to have everyone celebrate. Kelly and others who feel like her are absolutely justified in their position.

For myself, I like to think of it as a modern parable of the prodigal son. As tempting as it is to reject those who awaken late, it is better than them not waking at all. If Linus fails to follow through, it would be right to excoriate him. But if he does follow through, it can only improve the community around one of the most important open source projects. And it will set an example for other projects to follow.

I spend a lot of time thinking about community, particularly since I joined Red Hat as the Fedora Program Manager a few months ago. Community members β€” especially those in a highly-visible role β€” have an obligation to model the kind of behavior the community needs. This sometimes means a patient explanation when an angry rant would feel better. It can be demanding and time-consuming work. But an open source project is more than just the code; it’s also the community. We make technology to serve the people, so if our communities are not healthy, we’re not doing our jobs.

FPgM report: 2018-30

Inspired by bex’s “Slice of cake” updates, I present to the community this report of what has happened in Fedora Program Management this week.

Schedule

  • REMINDER β€” Software string freeze is July 31.

Changes

Announced

Submitted to FESCo

Approved by FESCo

I am on PTO this week, so anything not immediately obviously pertaining to submitted changes will be taken care of early next week.

FPgM report: 2018-29

Inspired by bex’s “Slice of cake” updates, I present to the community this report of what has happened in Fedora Program Management this week.

Schedule

  • REMINDER β€”Β Self-Contained Change submission deadline is July 24.
  • REMINDER β€” Software string freeze is July 31.

Changes

Announced

Submitted to FESCo

Approved by FESCo

I will be on PTO next week, but I will be checking in daily to shepherd last-minute change submissions.

Solved: ports on ThinkPad Thunderbolt dock doesn’t work with Fedora

I got a new ThinkPad X1 Carbon laptop for work. Of course I immediately installed Fedora 28 on it. Everything seemed to work just fine. But the laptop came with a ThinkPad Thunderbolt dock and when I went to go use it, I noticed the Ethernet port didn’t work. Then I noticed the USB ports didn’t work. But at least the HDMI port worked? (Full disclosure: I didn’t try the VGA port).

It turns out the solution was really simple, but I didn’t find a simple explanation so I’m putting one here. (Comment #17 of Red Hat Buzilla #1367508 had the basic solution. I hope this post becomes a little easier to find.)

The dock uses Thunderbolt which includes some security features. A package called bolt provides a management tool for this. Happily, it’s already in the Fedora 28 repo.

First, I installed it

#Β dnf install bolt

Then I examined the connected device


# boltctl list
● Lenovo ThinkPad Thunderbolt 3 Dock
β”œβ”€ type: peripheral
β”œβ”€ name: ThinkPad Thunderbolt 3 Dock
β”œβ”€ vendor: Lenovo
β”œβ”€ uuid: 00cd2054-ef95-0801-ffff-ffffffffffff
β”œβ”€ status: connected
β”‚ β”œβ”€ authflags: none
β”‚ └─ connected: Fri 29 Jun 2018 03:13:10 PM UTC
└─ stored: no

Finally, I enrolled the device

# boltctl enroll 00cd2054-ef95-0801-ffff-ffffffffffff
● Lenovo ThinkPad Thunderbolt 3 Dock
β”œβ”€ type: peripheral
β”œβ”€ name: ThinkPad Thunderbolt 3 Dock
β”œβ”€ vendor: Lenovo
β”œβ”€ uuid: 00cd2054-ef95-0801-ffff-ffffffffffff
β”œβ”€ dbus path: /org/freedesktop/bolt/devices/00cd2054_ef95_0801_ffff_ffffffffffff
β”œβ”€ status: authorized
β”‚ β”œβ”€ authflags: none
β”‚ β”œβ”€ parent: cf030000-0080-7f18-23d0-7d0ba8c14120
β”‚ β”œβ”€ syspath: /sys/devices/pci0000:00/0000:00:1d.0/0000:05:00.0/0000:06:00.0/0000:07:00.0/domain0/0-0/0-1
β”‚ β”œβ”€ authorized: Fri 29 Jun 2018 03:19:39 PM UTC
β”‚ └─ connected: Fri 29 Jun 2018 03:13:10 PM UTC
└─ stored: yes
β”œβ”€ when: Fri 29 Jun 2018 03:19:39 PM UTC
β”œβ”€ policy: auto
└─ key: no

After that, everything worked as expected. I’d like to thank the people who did the work to discover and implement the fix. I hope this post means a little less Googling for the next person.

LISA wants you: submit your proposal today

I have the great honor of being on the organizing committee for the LISA conference this year. If you’ve followed me for a while, you know how much I enjoy LISA. It’s a great conference for anyone with a professional interest in sysadmin/DevOps/SRE. This year’s LISA is being held in Nashville, Tennessee, and the committee wants your submission.

As in years past, LISA content is focused on three tracks: architecture, culture, and engineering. There’s great technical content (one year I learned about Linux filesystem tuning from the guy who maintains the ext filesystems), but there’s also great non-technical content. The latter is a feature more conferences need to adopt.

I’d love to see you submit a talk or tutorial about how you solve the everyday (and not-so-everyday) problems in your job. Do you use containers? Databases? Microservices? Cloud? Whatever you do, there’s a space for your proposal.

Submit your talk toΒ https://www.usenix.org/conference/lisa18/call-for-participation by 11:59 PM Pacific on Thursday, May 24. Or talk one of your coworkers into it. Better yet, do both! LISA can only remain a great conference with your participation.

HP laptop keyboard won’t type on Linux

Here’s another story from my “WTF, computer?!” files (and also my “oh I’m dumb” files).

As I regularly do, I recently updated my Fedora machines. This includes the crappy HPΒ 2000-2b30DX Notebook PC that I bought as a refurb in 2013. After dnf finished, I rebooted the laptop and put it away. Then while I was at a conference last week, my wife sent me a text telling me that she couldn’t type on it.

When I got home I took a look. Sure enough, they keyboard didn’t key. But it was weirder than that. I could type in the decryption password for the hard drive at the beginning of the boot process. And when I attached a wireless keyboard, I could type. Knowing the hardware worked, I dropped to runlevel 3. The built-in keyboard worked then.

I tried applying the latest updates, but that didn’t help. Some internet searching lead me to Freedesktop.org bug 103561. Running dnf downgrade libinput and rebooting gave me a working keyboard again. The bug is closed as NOTABUG, since the maintainers say it’s an issue in the kernel, which is fixed in the 4.13 kernel release. So I checked to see if Fedora 27, which was released last week, includes the 4.13 kernel. It does, and so does Fedora 26.

That’s when I realized I still had the kernel package excluded from dnf updates on that machine because of a previous issue where a kernel update caused the boot process to hang while/after loading the initrd. I removed the exclusion, updated the kernel, and re-updated libinput. After a reboot, the keyboard still worked. But if you’re using a kernel version from 4.9 to 4.12, libinput 1.9, and an HP device, your keyboard may not work. Update to kernel 4.13 or downgrade libinput (or replace your hardware. I would not recommend the HP 2000 Notebook. It is not good.)

New to Fedora: wordgrinder

Do you ever wish you had a word processor that just processed words? Font selection? Pah! Styling? Just a tiny bit, please. Or maybe you read Scott Nesbitt’s article on Opensource.com and thought “I’d like to try this!” If this sounds like you, then it may interest you to know that WordGrinder is now available on Fedora 25, 26, and Rawhide.

View of WordGrinder in a terminal

WordGrinder

I should clarify that it’s only available on some architectures (x86_64, i686,Β aarch64, andΒ armv7hl). WordGrinder depends on luaJIT which is only available on those platforms.

This is my first new Fedora package, and I have to say I’m kind of proud of myself. I tried to volunteer someone else for it, but he didn’t know how to build RPMs so I ended up volunteering myself. In the process, I had to patch the upstream release to build on Fedora, and then patch my patch to get it to build on Rawhide. In true Fedora fashion, I submitted my patch upstream and it was accepted. So not only did I make a new package available, but I also made an improvement to a project written in a language that I don’t know.

Yay open source!

Disappearing WiFi with rt2800pci

I recently did a routine package update on my Fedora 24 laptop. I’ve had the laptop for three years and have been running various Fedorae the whole time, so I didn’t think much of it. So it came as some surprise to me when after rebooting I could no longer connect to my WiFi network. In fact, there was no indication that any wireless networks were even available.

Since the update included a new kernel, I thought that might be the issue. Rebooting into the old kernel seemed to fix it (more on that later!), so I filed a bug, excluded kernel packages from future updates, and moved on.

But a few days later, I rebooted and my WiFi was gone again. The kernel hadn’t updated, so what could it be? I spent a lot of time flailing around until I found a “solution”. A four-year-old forum post saidΒ don’t reboot. Booting from off or suspending and resuming the laptop will cause the wireless to work again.

And it turns out, that “fixed” it for me. A few other posts seemed to suggest power management issues in the rt2800pci driver. I guess that’s what’s going on here, though I can’t figure out why I’m suddenly seeing it after so long. Seems like a weird failure mode for failing hardware.

Here’s what dmesg and the systemd journal reported:

Aug 01 14:54:24 localhost.localdomain kernel: ieee80211 phy0: rt2800_wait_wpdma_ready: Error - WPDMA TX/RX busy [0x00000068]
Aug 01 14:54:24 localhost.localdomain kernel: ieee80211 phy0: rt2800pci_set_device_state: Error - Device failed to enter state 4 (-5)

Hopefully, this post saves someone else a little bit of time in trying to figure out what’s going on.

Changing how HTCondor is packaged in Fedora

The HTCondor grid scheduler and resource manager follows the old Linux kernel versioning scheme: for release x.y.z, if y is an even number it’s a “stable” series that get bugfixes, behavior changes and major features go on odd-numbered y. For a long time, the HTCondor packages in Fedora used the development series. However, this leads to a choice between introducing behavior changes when a new development HTCondor release comes out or pinning a Fedora release to a particular HTCondor release which means no bugfixes.

This ignores the Fedora Packaging Guidelines, too:

As a result, we should avoid major updates of packages within a stable release. Updates should aim to fix bugs, and not introduce features, particularly when those features would materially affect the user or developer experience. The update rate for any given release should drop off over time, approaching zero near release end-of-life; since updates are primarily bugfixes, fewer and fewer should be needed over time.

Although the HTCondor developers do an excellent job of preserving backward compatibility, behavior changes can happen between x.y.1 and x.y.2. HTCondor is not a major part of Fedora, but we should still attempt to be good citizens.

After discussing the matter with upstream and the other co-maintainers, I’ve submitted a self-contained change for Fedora 25 that will

  1. Upgrade the HTCondor version to 8.6
  2. Keep HTCondor in Fedora on the stable release series going forward

Most of the bug reports against the condor-* packages have been packaging issues and not HTCondor bugs, so upstream isn’t losing a massive testing resource here. I think this will be a net benefit to Fedora since it prevents unexpected behavior changes and makes it more likely that I’ll package upstream releases as soon as they come out.

Fedora 24 upgrade

Fedora 24 was released last week, so of course I had to upgrade my machines. As has become the norm, there weren’t any serious issues, but I hit a few annoyances this time around. The first was due to packages in the RPMFusion repos not being signed. This isn’t Fedora’s fault, as RPMFusion is a completely separate project. And it was temporary: by the time I upgraded my laptop on Sunday night, the packages had all been signed.

Several packages had to be dropped by using the –allowerasing argument to dnf. Mostly these were packages installed from RPMFusion, but there were a couple of Fedora packages as well.

The biggest annoyance was that post-upgrade, I had no graphical login. I had to explicitly start the desktop manager service with:

systemctl enable kdm
systemctl start kdm

kdm had previously been enabled on both machines, but the upgrade nuked that in both cases. It looks like I’m not the only person to hit this:Β https://bugzilla.redhat.com/show_bug.cgi?id=1337546

And now, my traditional meaningless torrent stats!

Here’s my seeding ratios for Fedora 23:

Flavor i686 x86_64
KDE 16.2 35.6
Security 10.3 21.1
Workstation 30.9 46.7
Server 17.5 25.0

The “ratio ratio” as I call it is a comparison of seeding ratios between the two main architectures:

Flavor x86_64:i686
KDE 2.20
Security 2.05
Workstation 1.51
Server 1.43

So what does all of this tell us? Nothing, of course. Just because someone downloads a torrent that doesn’t mean they use it. Still, if we pretend that it’s a proxy for usage, all of the seeding ratios are higher than on the last release day. That tells me that Fedora is becoming more popular (yay!). 64-bit architectures are continuing to be a larger portion of the pie, as well.

Now that I’m starting to build a record of these, I can start reporting trends with the Fedora 25 release.