What do “rolling release” and “stable” mean in the context of operating systems?

In a recent post on his blog, Chris Siebenmann wrote about his experience with Fedora upgrades and how, because of some of the non-standard things he does, upgrades are painful for him. At the end, he said “What I really want is a rolling release of ‘stable’ Fedora, with no big bangs of major releases, but this will probably never exist.”

I’m sympathetic to that position. Despite the fact that developers have worked to improve the ease of upgrades over the years, they are inherently risky. But what would a stable rolling release look like?

“Arch!” you say. That’s not wrong, but it also misses the point. What people generally want is new stuff so long as it doesn’t cause surprise. Rolling releases don’t prevent that, they spread it out. With Fedora’s policy, for example, major changes (should) happen as the release is being developed. Once it’s out, you get bugfixes and minor enhancements, but no big changes. You get the stability.

On the other hand, you can run Fedora Rawhide, which gets you the new stuff as soon as it’s available, but you don’t know when the big changes will come. And sometimes, the changes (big and little) are broken. It can be nice because you get the newness quickly. And the major changes (in theory) don’t all come at once.

Rate of change versus total change

For some people, it’s the distribution of change, not the total amount of change that makes rolling releases compelling. And in most cases, the changes aren’t that dramatic. When updates are loosely-coupled or totally independent, the timing doesn’t matter. The average user won’t even notice the vast majority of them.

But what happens when a really monumental change comes in? Switching the init system, for example, is kind of a big deal. In this case, you generally want the integration that most distributions provide. It’s not just that you get an assortment of packages from your distribution, it’s that you get a set of packages that work together. This is a fundamental feature for a Linux distribution (excepting those where do-it-yourself is the point).

Applying it to Fedora

An alternate phrasing of what I understand Chris to want is “release-quality packages made available when they’re ready, not on the release schedule.” That’s perfectly reasonable. And in general, that’s what Fedora wants Rawhide to be. It’s something we’re working on, particularly with the ability to gate Rawhide updates.

But part of why we have defined releases is to ensure the desired stability. The QA team and other testers put a lot of effort into automated and manual tests of releases. It’s hard to test against the release criteria when the target keeps shifting. It’s hard to make the distribution a cohesive whole instead of a collection of packages.

What Chris asks for isn’t wrong or unreasonable. But it’s also a difficult task to undertake and sustain. This is one area where ostree-based variants like Fedora CoreOS (for servers/cloud), Silverblue (for desktops), and IoT (for edge devices) bring a lot benefit. The big changes can be easily rolled back if there are problems.

Removing unmaintained packages from an installed system

Earlier this week, Miroslav Suchý proposed removing removing retired packages as part of Fedora upgrade (editor’s note: the proposal was withdrawn after community feedback). As it stands right now, if a package is removed in a subsequent release, it will stick around. For example, I have 34 packages on my work laptop from Fedora 28 (the version I first installed on it) through Fedora 31. The community has been discussing this, with no clear consensus.

I’m writing this post to explore my own thoughts. It represents my opinions as Ben Cotton: Fedora user and contributor, not as Ben Cotton: Fedora Program Manager.

What does it mean for a package to be “maintained”?

This question is the heart of the discussion. In theory, a maintained package means that there’s someone who can apply security and other bug fixes, update to new releases, etc. In practice, that’s not always the case. Anyone who has had a bug closed due to the end-of-life policy will attest to that.

The practical result is that as long as the package continues to compile, it may live on for a long time after the maintainer has given up on it. This doesn’t mean that it will get updates, it just means that no one has had a reason to remove it from the distribution.

On the other hand, the mere fact that a package has been dropped from the distribution doesn’t mean that something is wrong with it. If upstream hasn’t made any changes, the “unmaintained” version is just as functional as a maintained version would be.

What is the role of a Linux distribution?

Why do Linux distributions exist? After all, people could just download the software and build it themselves. That’s asking a lot of most people. Even those who have sufficient technical knowledge to compile all of the different packages in different languages with different quirks, few have the time or desire to do so.

So a distribution is, in part, a sharing of labor. By dividing the work, we reduce our own burden and democratize access.

A distribution is also a curated collection. It’s the set of software that the contributors say is worth using, configured in the “right way”. Sure there are a dozen or so web browsers in the Fedora repos, but that’s not the entirety of web browsers that exist. Just as an art museum may have several similar paintings, a distribution might have several similar packages. But they’re all there for a reason.

To remove or not to remove?

The question of whether to remove unmaintained packages then becomes a balance between the shared labor and the curation aspects of a distribution.

The shared labor perspective supports not removing packages. If the package is uninstalled at update, then someone who relies on that package now has to download and build it themselves. It may also cause user confusion if something that previously worked suddenly stops, or if a package that exists on an upgraded system can’t be installed on a new one.

On the other hand, the curation perspective supports removing the package. Although there’s no guarantee that a maintained package will get updates, there is a guarantee that an unmaintained package won’t. Removing obsolete packages at upgrade also means that the upgraded system more closely resembles a freshly-installed system.

There’s no right answer. Both options are reasonable extensions of fundamental purposes of a distribution. Both have obvious benefits and drawbacks.

Pick a side, Benjamin

If I have to pick a side, I’m inclined to side with the “remove the packages” argument. But we have to make sure we’re clearly communicating what is happening to the user. We should also offer an easy opt-out for users who want to say “I know what you’re trying to do here, but keep these packages anyway.”

Cherrytree updates in COPR

For Fedora 31 users, I have updated the cherrytree package in my COPR to the latest upstream release (0.39.2). For Fedora 32 and rawhide users…well, there’s a problem. As you may know, Python 2 has reached end of life. And that means most of Python 2 is gone in Fedora 32. I tried to build the dependency chain in COPR, but the yaks kept getting hairier and hairier. Instead, I’ve packaged the C++ rewrite as cherrytree-future.

cherrytree-future is available for Fedora 31, Fedora 32, and rawhide. I have packages for x86_64 and aarch64 for all three versions and for armhfp on Fedora 31 and 32 (the rawhide builder was out of disk space, oops!).

Because cherrytree-future is still pre-release I intentionally did not have the package obsolete cherrytree. If you’re upgrading from Fedora 31 to Fedora 32, you will first have to remove cherrytree and install cherrytree-future.

I have been using cherrytree-future in the last day and it’s working well for me so far. If you encounter any problems with the package (e.g. a missing dependency), please file an issue on my GitHub repo. If you encounter problems with the program itself, file the bug upstream.

Once upstream cuts an official release of the rewrite, I’ll work on getting it into the official repos.

[solved] Can’t log in to KDE on Fedora 31

Earlier today, I ran dnf update on my laptop, as I do regularly. After rebooting, I couldn’t log in. When I typed in my user name and password, it almost immediately returned to the login screen. Running startx from the command line failed, too. I spent an hour or two trying to diagnose the problem. There were a lot of distracting messages in the xorg log.

The problem turned out to be that the startkde command was no longer on my machine. It seems upgrading from version 5.16 to 5.17 of the plasma-workspace package removes startkde in favor of startplasma-x11. Creating a symlink fixed it as a workaround.

This is reported as bug #1785826, and I’m sure Rex and the rest of the Fedora KDE team will have a suitable fix out soon. In the meantime, creating a symlink appears to be the best way to fix it.

Why the symlink works

When an X session starts, it looks in a few different places to see what should be run. One of those places is /etc/X11/xinit/Xclients. This file checks for a preferred desktop environment. If one isn’t specified, it works through a list trying to find one that works. It does this by looking for the specific desktop environment’s executable.

Since startkde no longer exists, it had no way of checking for KDE Plasma. I don’t have any other desktop environments installed on this machine, so there was no other desktop environment to fallback to. I suspect if GNOME were installed, it would have logged me into GNOME instead, at least when running startx.

So another fix would be to replace instances of startkde with startplasma-x11 in the Xclients file (similarly if you have that file in your home directory). However, this leaves anything else that might check for the existence of startkde in the lurch. (I don’t know if anything does).

There’s probably more options for fixing it out there; this is very much not my area of expertise. I’d have to say that this was the most frustrating issue I’ve had to debug in a long time, in part because it took me a while to even know where the problem was. The fact that moving my ~/.kde directory didn’t result in a new one being created told me that it was pretty early in the process.

What distractions did I see?

In trying to diagnose the issue, I got distracted by a variety of error messages:

  • xf86EnableIOPorts: failed to set IOPL for I/O (Operation not permitted)
  • /dev/fb0: permission denied
  • gkr-pam: unable to locate daemon control file
  • pam_kwallet5: couldn't open file

New to Fedora: z

Earlier this month, I attended Chris Waldon’s session “Terminal Velocity: Work faster in your shell” at All Things Open. He covered several interesting tools, one of which is a project called z. z is a smarter version of the cd command. It keeps track of what directories you change to and uses a combination of the frequency and recency (“frecency”) to make an educated guess about where you wanted to go.

I find this really appealing because I often forget where in the file system I put a directory. And z is written as a shell script, so it’s easy to package and use.

z is now packaged and submitted to rawhide, with updates pending for F31 and F30.

FPgM report: 2018-30

Inspired by bex’s “Slice of cake” updates, I present to the community this report of what has happened in Fedora Program Management this week.

Schedule

  • REMINDER — Software string freeze is July 31.

Changes

Announced

Submitted to FESCo

Approved by FESCo

I am on PTO this week, so anything not immediately obviously pertaining to submitted changes will be taken care of early next week.

FPgM report: 2018-29

Inspired by bex’s “Slice of cake” updates, I present to the community this report of what has happened in Fedora Program Management this week.

Schedule

  • REMINDER — Self-Contained Change submission deadline is July 24.
  • REMINDER — Software string freeze is July 31.

Changes

Announced

Submitted to FESCo

Approved by FESCo

I will be on PTO next week, but I will be checking in daily to shepherd last-minute change submissions.

Solved: ports on ThinkPad Thunderbolt dock doesn’t work with Fedora

I got a new ThinkPad X1 Carbon laptop for work. Of course I immediately installed Fedora 28 on it. Everything seemed to work just fine. But the laptop came with a ThinkPad Thunderbolt dock and when I went to go use it, I noticed the Ethernet port didn’t work. Then I noticed the USB ports didn’t work. But at least the HDMI port worked? (Full disclosure: I didn’t try the VGA port).

It turns out the solution was really simple, but I didn’t find a simple explanation so I’m putting one here. (Comment #17 of Red Hat Buzilla #1367508 had the basic solution. I hope this post becomes a little easier to find.)

The dock uses Thunderbolt which includes some security features. A package called bolt provides a management tool for this. Happily, it’s already in the Fedora 28 repo.

First, I installed it

# dnf install bolt

Then I examined the connected device


# boltctl list
● Lenovo ThinkPad Thunderbolt 3 Dock
├─ type: peripheral
├─ name: ThinkPad Thunderbolt 3 Dock
├─ vendor: Lenovo
├─ uuid: 00cd2054-ef95-0801-ffff-ffffffffffff
├─ status: connected
│ ├─ authflags: none
│ └─ connected: Fri 29 Jun 2018 03:13:10 PM UTC
└─ stored: no

Finally, I enrolled the device

# boltctl enroll 00cd2054-ef95-0801-ffff-ffffffffffff
● Lenovo ThinkPad Thunderbolt 3 Dock
├─ type: peripheral
├─ name: ThinkPad Thunderbolt 3 Dock
├─ vendor: Lenovo
├─ uuid: 00cd2054-ef95-0801-ffff-ffffffffffff
├─ dbus path: /org/freedesktop/bolt/devices/00cd2054_ef95_0801_ffff_ffffffffffff
├─ status: authorized
│ ├─ authflags: none
│ ├─ parent: cf030000-0080-7f18-23d0-7d0ba8c14120
│ ├─ syspath: /sys/devices/pci0000:00/0000:00:1d.0/0000:05:00.0/0000:06:00.0/0000:07:00.0/domain0/0-0/0-1
│ ├─ authorized: Fri 29 Jun 2018 03:19:39 PM UTC
│ └─ connected: Fri 29 Jun 2018 03:13:10 PM UTC
└─ stored: yes
├─ when: Fri 29 Jun 2018 03:19:39 PM UTC
├─ policy: auto
└─ key: no

After that, everything worked as expected. I’d like to thank the people who did the work to discover and implement the fix. I hope this post means a little less Googling for the next person.

HP laptop keyboard won’t type on Linux

Here’s another story from my “WTF, computer?!” files (and also my “oh I’m dumb” files).

As I regularly do, I recently updated my Fedora machines. This includes the crappy HP 2000-2b30DX Notebook PC that I bought as a refurb in 2013. After dnf finished, I rebooted the laptop and put it away. Then while I was at a conference last week, my wife sent me a text telling me that she couldn’t type on it.

When I got home I took a look. Sure enough, they keyboard didn’t key. But it was weirder than that. I could type in the decryption password for the hard drive at the beginning of the boot process. And when I attached a wireless keyboard, I could type. Knowing the hardware worked, I dropped to runlevel 3. The built-in keyboard worked then.

I tried applying the latest updates, but that didn’t help. Some internet searching lead me to Freedesktop.org bug 103561. Running dnf downgrade libinput and rebooting gave me a working keyboard again. The bug is closed as NOTABUG, since the maintainers say it’s an issue in the kernel, which is fixed in the 4.13 kernel release. So I checked to see if Fedora 27, which was released last week, includes the 4.13 kernel. It does, and so does Fedora 26.

That’s when I realized I still had the kernel package excluded from dnf updates on that machine because of a previous issue where a kernel update caused the boot process to hang while/after loading the initrd. I removed the exclusion, updated the kernel, and re-updated libinput. After a reboot, the keyboard still worked. But if you’re using a kernel version from 4.9 to 4.12, libinput 1.9, and an HP device, your keyboard may not work. Update to kernel 4.13 or downgrade libinput (or replace your hardware. I would not recommend the HP 2000 Notebook. It is not good.)

Using the ASUS ZenBook for Fedora

I recently decided that I’d had enough of the refurbished laptop I bought four years ago. It’s big and heavy and slow and sometimes the fan doesn’t work. I wanted something more portable and powerful enough that I could smoothly scroll the web browser. After looking around for good Linux laptops, I settled on the ASUS ZenBook.

Installation

The laptop came with Windows 10 installed, but that’s not really my jam. I decided to boot off a Fedora 26 KDE live image first just to make sure everything worked before committing to installing. Desktop Linux has made a lot of progress over the years, but you never know which hardware might not be supported. As it turns out, that wasn’t a problem. WiFi, Bluetooth, webcam, speakers, etc all worked out of the box.

It’s almost disappointing in a sense. There used to be some challenge in getting things working, but now it’s just install and go. This is great overall, of course, because it means Linux is more accessible to new users and it’s less crap I have to deal with when I just want my damn computer to work. But there’s still a little bit of the nostalgia for the days when configuring X11 by hand was something you had to do.

Use

I’ve had the laptop for a little over a month now. I haven’t put it through quite the workout I’d hoped to, but I feel like I’ve used it enough to have an opinion at this point. Overall, I really like it. The main problem I have is that the trackpad has a middle-click, which is actually pretty nice except for when I accidentally use it. I’ve closed many a browser tab because I didn’t move my thumb far enough over. That’s probably something I can disable in the settings, but I’d rather learn my way around it.

The Bluetooth has been flaky transferring files to and from my phone. but audio is…well I’ve never found Bluetooth audio to be particularly great, but it works as well as anything else.

One other bit of trouble I’ve had is with my home WiFI. I bought a range extender so that I can use WiFi on the back deck and it to use the same SSID as the main router. The directions said you can do this, but it might cause problems. With this laptop, the WiFi connection becomes unusable after a short period of time. Turning off the range extender fixes it, and I’ve had no other problems on other networks, so I guess I know what I have to do.

One thing that really stood out to me is carrying it around in a backpack. This thing is light. I had a few brief moments of panic thinking I had left it behind. I’ve held lighter laptops, but this is a good weight. But don’t worry about the lightness, it still has plenty of electrons to have a good battery life.

Around the same time I bought this, I got a new MacBook Pro for work. When it comes to typing, I like the keyboard on the ZenBook way better than the new MacBook keyboards.

Recommendation

If you’re looking for a lightweight Linux laptop that can handle general development and desktop applications, the ASUS ZenBook is a great choice. Shameless commercialism: If you’re going to buy one, maybe use this here affiliate link? Or don’t. I won’t judge you.