What does it mean for a Linux distribution to be “fresh”?

I recently had a discussion with Luboš Kocman of openSUSE about how distros can monitor their “freshness”. In other words: how close is a distro to upstream? From our perspectives, it’s helpful to know which packages are significantly behind their upstreams. These packages represent areas that might need attention, whether that be a gentle nudge to the maintainer or recruiting additional volunteers from the community.

The challenge is that freshness can mean different things. The Repology project monitors a large number of distributions and upstreams to report on the status. But simply comparing the upstream version number to the packaged version number ignores a lot of very important context.

Updating to the latest upstream version as soon as it comes out is the most obvious definition of “fresh”, but it’s not always the best. Rolling releases (and their users) probably want that. In Fedora, policy is to not do “major updates” within a release. Many other release-oriented distributions have a similar policy, with varying degrees of “major”. Enterprise distributions add another wrinkle: they’ll backport security fixes (and sometimes key features), so the difference in version number doesn’t necessarily tell you what’s missing.

Of course, the upstream’s version number doesn’t necessarily tell you much. Semantic versioning is great, but not everyone uses it. And not everyone that uses it uses it well. If a distribution has version 1.4 and upstream released 1.5, is that a lack of freshness or an intentional decision to avoid mid-release compatibility changes?

I don’t have a good answer. This is a hard problem to solve. Something like Repology may be the best we can do with reasonable effort. But I’d love to have a more accurate view of how fresh Fedora packages are within the bounds of policy.

Cursed house: Jennifer broke the house

This is the fourth (and hopefully final) post in a series of personal stories about how my parents’ house has some really bad luck.

I want to bring the curse house series to an end with a short and amusing story. It was Thanksgiving a decade or so ago. We were at my parents’ house and had just finished eating. When the phone rang, my sister got up to answer it. The caller was asking for my other sister, so Jennifer brought the phone into the dining room. (Kids, this is when people had landlines and cordless phones.)

She stepped into the dining room and CRACK! I felt the floor drop out from under me. It’s hard to say exactly how much, but it was enough to notice and be very worried that I was about to fall into the cellar.

What happened was that the metal pole under the floor joist had finally rusted away after many years and several cellar floods. Jennifer stepped in just the right place at the right time to break it. Fortunately, the house held together well enough that Dad and I could grab a 4-by-4 and use it to support the floor.

But all of these years later, we still tease Jennifer about how she broke the house.

Bo Burnham’s “Inside”

I recently watched Bo Burnham’s new Netflix special. “Inside” is…something. It brings to mind two movies I’ve seen several times. But first, I want to talk about it as art. There’s no doubt in my mind that it’s an excellent work of art. It evoked a lot of emotion, including confusion and discomfort.

The discomfort comes from the intimate feel it has. Burnham has been open about his struggles with mental health. “Inside” is not always clear on how much is Burnham being sincere and how much is him exaggerating reality for effect, as art often does. In this way, it brings to mind Pink Floyd’s “The Wall.”

“The Wall” came about after Roger Waters spat on noisy audience members and expressed a desire to build a wall between the audience and the band. But I seriously doubt that Waters actually wanted to shave off his nipples or create a Nazi-like following. “The Wall” took a sincere feeling and made art out of it. Is “Inside” the same?

I hope so. There are definitely times where it felt sincere enough that I was uncomfortable with the awkwardness of watching Bo Burnham have a breakdown. Was that the goal?

“Inside” also reminds me of the Monkees’ flop of a movie. “Head” is just plain weird. It’s full of mostly unrelated bits that occasionally call back to each other. After a few viewings, I’ve started to feel like I have an idea of what it’s about, just not one I can articulate. It’s easier to express what “Inside” is about, but it’s still a weird mish-mash of content without a coherent arc.

I can’t say that I enjoyed “Inside”, but I did appreciate it as a work of art. Of course, it’s not perfect. It’s a little self-important and masturbatory. But it’s entirely Bo Burnham, and you can’t ask anything more than that.

Cursed house: the other fires

This is the third post in a series of personal stories about how my parents’ house has some really bad luck.

In the first post of this series, I wrote about the fire that nearly destroyed our house before we ever moved in. But that wasn’t the only fire.

The backyard fire

Our first winter after finally being able to live in our new house was eventful. My mom was pregnant with my youngest sister. And the back yard caught on fire. The fire happened shortly before my sister was born, in fact. Since we were served by a volunteer fire department, the response time was not quick. My parents, including a very pregnant Mom, were carrying buckets of water to contain the fire until the Georgetown Township Volunteer Fire Department arrived.

My memory on this is murky, but I think the tanker truck came from Greenville Twp. I don’t remember if the Lafayette Twp. fire department came for this one or not. My parents’ house is right in the corner of Georgetown Twp., so it’s pretty common for multiple departments to respond to a call in their area.

I don’t remember much else about this fire. I know it happened, but I don’t recall how it started or how much it spread. I know it didn’t damage the house, which was surely a great relief for my parents.

The bathroom fire

Fast forward about a decade and a half. I was home from Purdue for the summer. One day I was sitting in the bathroom, doing one what does. The light and the exhaust fan shut off suddenly. Since it wasn’t just the light, I know that it wasn’t an issue of a bad bulb. I hollered for someone to go check the breaker and they flipped it.

It didn’t take long for the light to shut off again. I figured it was time to finish my business so I got out of there as fast as hygiene would allow. We flipped the breaker on again and didn’t think much of it. I was in the dining room talking to whatever family members happened to be in there, when I heard a clattering noise in the bathroom.

I went in to see what caused it and saw the light cover on the floor. How odd. Then I saw it was on fire. Ah, that would explain it. I looked up and saw that the entire fixture was on fire.

As fortune would have it, I had a big fire extinguisher in the trunk of my car, so I ran out and got it. I used it to put out the fire. Then I ran to the shed to get my dad’s ladder so we could check the attic. I couldn’t see any fire.

Meanwhile, someone had called 911 and the fire department was on the way. Since it was around the Fourth of July, the Georgetown VFD was stationed near where some fireworks were going to be set off clear on the other side of the township. So Lafayette also responded.

The fire was out well before either department arrived, but since they have those fancy infrared cameras, it seemed like a good idea to have them make sure nothing was smoldering in the attic. Thankfully it wasn’t.

My parents tended to leave the bathroom exhaust fan on 24/7. It never bothered me until that day. Maybe it’s because I would have had melted, fiery plastic drop on my lap had I been a few minutes later to the bathroom, but I am now firmly in the “the exhaust fan only runs for showers” camp. It doesn’t feel like a traumatic event, but it has clearly changed my behavior.

Well that’s awkward

The bathroom fire is personally notable for another reason: I had a friend who was a volunteer for one of the responding departments. She had decided to leave them and volunteer for the other department. But she hadn’t told the old department yet, so they found out when she showed up on the new department’s truck. I felt bad for her, but it was also pretty amusing.

How I configure sshd at home

My “server” at home isn’t particularly important to the outside world. But by virtue of being on the Internet, it’s subject to a lot of SSH logins. The easiest thing to do is to shut it off from the outside world. But I need to access it when away from home, so that’s not a particularly useful solution.

So what I’ve done is use the SSH daemon’s (sshd) configuration to reduce the risk profile. The first thing I wanted to do is forbid login as root:

PermitRootLogin no

I also don’t want anyone to be able to log in with passwords. “Anyone” is essentially me here, but since I have sudo on the box, if someone is able to figure out my password they are able to get root remotely.

PermitRootLogin no
ChallengeResponseAuthentication no

Finally, I want to restrict remote login to only explicitly-permitted users. I do this with a dedicated Unix group that I call “sshusers”.

AllowGroups sshusers

These are pretty standard changes and not really worth a blog post. But it turns out that sshd has a very flexible configuration. When a client is coming from inside the LAN, I want to enable password authentication. This is particularly helpful when I’m installing a new system and don’t have SSH keys setup yet.

Match Address 192.168.1.*
    PasswordAuthentication yes

Also within the LAN, it’s easier to run Ansible playbooks across machines if the root user can SSH in with a key. So I combine user and address matching to permit key-based root login only from the server with the Ansible playbooks.

Match User root Address 192.168.1.10
    AllowGroups root sshusers
    PermitRootLogin prohibit-password

Finally, I want my ex to be able to access the server in order to access photos, etc. So I set up her account so that she can use an sftp client but can’t log in (not that she would anyway, but it was a fun challenge to set this up).

Match User angie
    ForceCommand internal-sftp
    PasswordAuthentication yes
    PermitTunnel no
    AllowAgentForwarding no
    AllowTcpForwarding no
    X11Forwarding no

Why didn’t you … ?

The configuration above isn’t the only way to secure my SSH server from the outside world. It’s not even necessarily the best. I could, for example, move SSH to a different port, which would cut down on the drive-by attempts significantly. I resisted that in the past because I felt “security through obscurity isn’t security.” But in practice, it can be a layer in a more secure approach. In the past, I also recall some clients I used (particularly on mobile) not having the ability to use a non-default port. If that recollection is correct, it seems to also be outdated now. So basically I’m still on port 22 because of inertia.

I could also set up a VPN server and use that for remote access. That requires an additional service to manage, of course. And it also presents challenges when I’m also connected to a work VPN server. The sshd configuration approach is a simpler way for my needs.

Other writing: May 2021

Stuff I wrote

Fedora

Open Organization

Stuff I curated

Fedora

How to document your job

I was recently talking to a coworker who is in the middle of a large project to document her job. The goal is two-fold: to give teammates the ability to cover for her when she’s out of the office and to improve the onboarding experience for new team members. She has her manager’s support, but it’s a large project. She finds it difficult to make progress, in part because she doesn’t particularly like writing. As a result, the deadline keeps slipping further out.

I won’t claim to be the best documentarian, but I try to always leave my successor with better documentation than my predecessor left me. I’d like to think I’ve succeeded in that. In some cases, the bar was pretty low because there was no documentation at all. At any rate, this post is a collection of things I’ve learned over the years. This is an opinionated post and does not necessarily represent the One True Way to Document Your Job™.

Writing

  • Separate the why and the how. You have two separate audiences: someone filling in for a day or two and someone taking over the role. The information they need is very different. A person covering short-term probably doesn’t need or want to know the whole history of a process. They just want to know the steps they need to take. On the other hand, your replacement will benefit from a greater level of explanation. They’ll want to know why the steps are they way they are. In particular, knowing what didn’t work well in the past is a good reference to help them avoid re-learning that lesson later.
  • Start at a high level. If you cover the broad stuff first, that gives your reader something to work with, even if they need to ask you or someone else to help fill in the details. On the other hand, a detailed list for process A does nothing to help with process B. Starting at a high level also allows you to…
  • Write small chunks. You don’t need to write everything in a day. Starting with higher level concepts allows you to break the role into a few key areas. From there, you can break it down further and further.
  • Limit the work in progress. Similarly, you don’t need to write everything at the same time. if you work one one or two concepts at a time, you can get those sufficiently done and move on to the next. Yes, it’s a bummer if the one of the incomplete concepts is what your coworker needs to cover a sick day, but some fully-complete documentation is generally better than a lot of woefully incomplete documentation.
  • Avoid duplication. The more places the same information is recorded, the more likely it will become out of date. If documentation for something already exists, link to the authoritative source. You can provide some basic context around the link, but minimize the amount of copy/paste you do.
  • Write as you learn. If you’re just getting started, try to write as much as you can as you learn it. That way, you don’t end up assuming knowledge that your reader won’t have. If you’re learning from someone else, this also gives you the opportunity to let them verify it.

Verifying

The first time you write documentation, it will almost always be incomplete. You’ll want to get feedback to help improve it. This turns out to be another benefit of small chunks with limited work in progress. It’s almost like Agile was on to something here!

  • Trade places with a coworker. If you can (this requires management support for sure), swapping jobs with a coworker for a few hours gives you the opportunity to see how the documentation performs in real life. Doing it while you’re still around means that the documentation has been tested before you have a sick day or vacation. If there are problems, you can answer them without being inconvenienced.
  • Get feedback quicky. Even if you don’t swap roles, at least let someone look at it as soon as you’re done writing. This way everything is fresh in your mind as the questions come up.

The mechanics

You may have noticed that I cleverly avoided discussing the technical aspects. Do you write a wiki page? A word processor document? A standalone note taking app (like CherryTree)? Rendered HTML on a docs site? I leave this as an exercise for the reader. They all have benefits and drawbacks, so whatever works best for your team is the right answer. If people can’t find or update the documentation, then why did you bother writing it?

A survey of the kanban tools I use

In a previous post about distinguishing between tasks and projects, I made a brief mention of the different kanban tools I use with a promise to go into further depth. Here is the depth. This post is more about the tools I use and the context in which I use them as opposed to the specific design of the boards themselves.

Trello

Trello is the gold standard when it comes to kanban. It’s free to use (although I’ve paid for a business subscription in the past), has a useful mobile app, and focuses on being good at being a kanban board. Trello was the first kanban board I used and it became my default. I use it to track article ideas for this blog, plan meals, and track online orders.

I wouldn’t mind switching to another tool, just to have one less place to track work, but there are a few things that keep me around. The mobile app is the biggest. Trello’s mobile app works really well. It doesn’t have the full feature set of the web app, but it’s enough to be productive. I’m also a big fan of the calendar view. This helps me plan time-based work like blog posts and dinners. Combined with labels, the calendar view makes it really easy to see if I’m being too repetitive. I also like that Trello makes a visual distinction between complete and incomplete when displaying dates. Relatedly, the package tracking plugin is really nice, too. Plug in a tracking number and the estimated delivery date displays on the card as if it were a due date. The last feature that I love is the automation. I only use it on my writing board, but it comes in handy. I have a rule that when I set a due date on a card, it gets moved from “Ideas” to “Planned”. There’s another one that will mark a card as done when I move it to the “Scheduled” column.

Trello also has a good blog if you’re into that sort of thing.

Todoist

I switched to Todoist when I couldn’t put off migrating away from Wunderlist any longer. As the name implies, it’s primarily a todo list manager (and a darn good one at that), but it recently added support for kanban boards. To try it out, I converted my “laundry” category to a board. It works okay for that purpose.

The mobile app is great for the todo side, but the boards are a little buggy (although the same is true for the web app). Cards don’t always want to move to the column you’re trying to drag them to. But my biggest gripe is that there’s currently no way to cleanly handle recurring tasks. For example, my laundry board is broken out by load (e.g. whites, pants, towels), with tasks scheduled to recur every two weeks. When I mark a task done, I’d like the new incarnation to start in the “dirty” column. Ideally, moving a task to the “folded” column would automatically mark it done.

I’ve found myself using the board feature less and less with my laundry. The issues above combined with the fact that the state is readily apparent means it doesn’t have a lot of value. I like Todoist overall, so I wish I had more reason to use the board feature.

Taiga

I actually use two different Taiga instances: Fedora’s and the public instance. In the Fedora instance, I have a board where I track all of my work as Fedora Program Manager, as well as using it collaboratively with other teams. On the public instance, I’m currently only using it to track progress on my book.

Taiga is designed to be a full-featured project management tool, which gives it a leg up in some ways. User stories on the kanban board can have child tasks with their own state, which is helpful when i need to decompose work. It also has a module for epics, which is useful for aggregating larger work. As an example, I have a card for each chapter on my book board. When my editor gives me things to fix, I add those as tasks. Each of the milestones that the publisher has in their process is an epic.

There are two missing features that keep from moving to Taiga as my main kanban tool. The first is the lack of a calendar view. This would be particularly for the Fedora Magazine editorial board. The second is a sub-optimal (read: basically unusable) mobile experience. I don’t manipulate my boards on my phone a ton, but I do enough that it would be hellish. There are some third-party apps, but they can’t connect via Fedora’s authentication system, so they don’t help me.

I’d also like the ability to mark specific user stories as private, although I concede that doesn’t make a ton of sense in the context of what it’s intended for.

GitHub, GitLab, and Pagure

I combined these three because they’re basically the same to me: nice additions to issue tracking tools. I wouldn’t use either of them primarily as a kanban tool, but it comes in handy as a layer on the primary purpose. GitHub allows you to add both issues and non-issue cards to a board, which has resulted in a very confused me on several occasions. Pagure does not appear to have this problem. I’ve used GitLab’s board feature a little bit, but I don’t feel I’m familiar enough to comment on it.

Cursed house: the smell of wet insulation

This is the second post in a series of personal stories about how my parents’ house has some really bad luck.

You know how scent is associated with memories? The smell of warm cookies reminds you of visiting your grandparents. A warm, salty breeze takes you back to that family vacation when you were a kid. But are you able to see something and then recall the smell? I can!

The summer after my freshman year of high school, my parents decided it was time to have some work done on the house. The first step was replacing the roof. Their house being old (like “parts of the original log cabin still exist” old), the roof was…rough. It had a rafter construction, so the roofers had to take the entire roof apart.

Before they could put the nice, new trusses in place, they had to make the top plate level. This took some Doing™ apparently, and they made slow progress. When Friday rolled around, it rained. The roofers had put tarps flat across the top of the house, but some water soaked through and damaged the drywall ceiling in a few places. No big deal—that’s easy to fix.

So then Monday rolls around. Both of my parents are at work, so I’m home with my sisters on a warm June day in the Ohio Valley. If you don’t know about warm June days in the Ohio Valley, they sometimes have pretty bad storms. Come the afternoon, several tornado warnings have been issued. Being the eldest child (and also a weather weenie), I keep my eye on the TV coverage. My memory is a little fuzzy on this point, but I seem to recall having to get my sisters to shelter at least once.

But the important part here is that it rained. And rained. And rained some more. Officially, Standiford Field recorded 0.86″ of rain that day. At my parents’ house a dozen miles to the northwest, it rained slightly more than that.

The tarps were still on the not-roof, but they were still flat. This meant that water could not run off and fall to the ground, but instead puddled. Slowly, water began seeping through the tarps. And into the house. Not just in one or two places like it had on Friday, but all over.

The living room. The dining room. My parents’ room. My sisters’ room. The bathroom. Water was coming in everywhere. (My bedroom and the kitchen were an addition and had a separate roof, which spared them). For the next few hours, we became a bucket brigade.

Everything we could find, we put to use catching the water falling from the ceiling. Trash cans. Buckets. Our sleds. The water became a steady stream in some places, filling up the small trash cans almost as soon as we could empty them. Meanwhile, severe thunderstorms still threatened.

Eventually the rain stopped and my parents came home from work. It was clear that staying in the house that night was not an option. Everything was wet and the ceiling was falling in the living room. We stayed in a hotel that night, and for the rest of the week. Then we spent three weeks in the basement of some friends. After six months in a rental house, we were able to return to our home.

All of the interior walls and ceilings had to be replaced. The floor (including the floor joists) in two rooms were also replaced. We threw out many of our possessions: books, toys, furniture, clothing. So much had been soaked through. High temperatures were in the 90s the rest of the week, making it oppressively humid in the house.

The smell of wet insulation and drywall is something else. It sticks with you. For years, if I saw a picture of damage, I could smell the insulation as if I were standing there in the middle of the aggressively moist house.

We fired the roofers. Our insurance company sued their insurance company. Life went on. But we never did the addition that we had planned.

Using Element as an IRC client

Like many who work in open source communities, IRC is a key part of my daily life. Its simplicity has made it a mainstay. But the lack of richness also makes it unattractive to many newcomers. As a result, newer chat protocols are gaining traction. Matrix is one of those. I first created a Matrix account to participate in the Fedora Social Hour. But since Matrix.org is bridged to Freenode, I thought I’d give Element (a popular Matrix client) a try as an IRC client, too.

I’ve been using Element almost exclusively for the last few months. Here’s what I think of it.

Pros

The biggest pro for me is also the most surprising. I like getting IRC notifications on my phone. Despite being bad at it (as you may have read last week), I’m a big fan of putting work aside when I’m done with work. But I’m also an anxious person who constantly worries about what’s going on when I’m not around. It’s not that I think the place will fall apart because I’m not there. I just worry that it happens to be falling apart when I’m not there.

Getting mobile notifications means I can look, see that everything is fine (or at least not on fire enough that I need to jump in and help), and then go back to what I’m doing. But it also means I can engage with conversations if I choose to without having to sit at my computer all day. As someone who has previously had to learn and re-learn not to have work email alert on the phone, I’m surprised at my reaction to having chat notifications on my phone.

Speaking of notifications, I like the ability to set per-room notification settings. I can set different levels of notification for each channel and those settings reflect across all devices. This isn’t unique to Element, but it’s a nice feature nonetheless. In fact, I wish it were even richer. Ideally, I’d like to have my mobile notifications be more restrictive than my desktop notifications. Some channels I want to see notifications for when I’m at my desk, but don’t care enough to see them when I’m away.

I also really like the fact that I can have one fewer app open. Generally, I have Element, Signal, Slack, and Telegram, plus Google Chat all active. Not running a standalone IRC client saves a little bit of system resources and also lets me find the thing that dinged at me a little quicker.

Cons

By far the biggest drawback, and the reason I still use Konversation sometimes, is the mishandling of multi-line copy/paste. Element sends it as a single multi-line message, which appears on the IRC side as “bcotton has sent a long message: <url>”. When running an IRC meeting, I often have reason to paste several lines at once. I’d like them to be sent as individual lines so that IRC clients (and particularly our MeetBot implementation), see them.

The Matrix<->IRC bridge is also laggy sometimes. Every so often, something gets stuck and messages don’t go through for up to a few minutes. This is not how instant messaging is supposed to work and is particularly troublesome in meetings.

Overall

Generally, using Element for IRC has been a net positive. I’m looking forward to more of the chats I use becoming Matrix-native so I don’t have to worry about the IRC side as much. I’d also like the few chats I have on Facebook Messenger and Slack to move to Matrix. But that’s not a windmill I’m willing to tilt at for now. In the meantime, I’ll keep using Element for most of my IRC need,s, but I’m not quite ready to uninstall Konversation.