Moving the website to Lektor

Years ago, I moved all of funnelfiasco.com (except the blog, which runs on WordPress) from artisinally hand-crafted HTML to using a static site generator. At the time, I chose a project called “blatter” which used jinja2 templates to generate a site. This gave me the opportunity to change basic information across the whole site at once. Not something I do often, but it’s a pain when I do.

Unfortunately, blatter was apparently quietly abandoned by the developer. This wasn’t really a problem until Python 2 reached end of life. Fedora (reasonably) retired much of the Python 2 ecosystem. I tried to port it to Python 3, but ran into a few problems. And frankly, the idea of taking on the maintenance burden for a project that hadn’t been updated in years was not at all appealing. So I went looking for something else.

I wanted to find something that used jinja2 in order to minimize the amount of work involved. I also wanted something focused on websites, not blogs specifically. It seems like so many platforms today are blog-first. That’s fine, it’s just not what I want. After some searching and a little bit of trial and error, I ended up selecting Lektor.

The good

Lektor is written in (primarily) Python 3 and uses jinja2 templates, so it hit my most important points. It has a command to run a local webserver for testing. In addition, you can set up multiple servers configurations for deployment. So I can have the content sync to my local web server to verify it and then deploy that to my “production” webserver. Builds are destructive, but the deploys are not, which means I don’t have to shoe-horn everything into Lektor.

Another great feature is the ability to programmatically generate thumbnails of images. I’ve made a little bit of use of that for the time being. In the future, especially if I ever go storm chasing again, I can see myself using that feature a lot more.

Lektor optionally supports writing the page content in markdown. I haven’t done this much since I was migrating pre-written content. I expect new content will be much markdownier. Markdown isn’t flexible enough for a lot of web purposes, but it covers some use cases well. Why write HTML when it’s not needed?

Lektor uses databags to provide input data to templates. I do this using JSON files. Complex operations with that are a lot easier than the embedded Python data structures that Blatter supported.

If I were interested in translating my site into multiple languages, Lektor has good support for that (including changing URLs). It also has a built-in admin and editing console, which is not something I use, but I can see the appeal.

The bad

Unlike Blatter, Lektor puts contents and templates in separate files. This makes it a little more difficult to special-case a specific site.

It also has a “one directory, one file” paradigm. Directories can have “attachments”, which can include html files, but they won’t get processed, so they need to stand alone. This is not such an issue if you’re starting from scratch. Since I’m not, it was more of a headache. You can overwrite the page’s slug, but that also makes certain assumptions.

For the Forecast Discussion Hall of Fame, I wanted to keep URLs as-is. That site has been linked to from a lot of places, and I’d hate to break those inbound links. Writing an htaccess file to redirect to the new URLs didn’t sound ideal either. I ended up writing a one-line patch that passed the argument I need to the python-slugify library. I tried to do it the right way so that it would be configurable, but it was beyond my skill to do so.

The big down side is the fact that the development has ground to a halt. It’s not abandoned, but the development activity happens in spurts. Right now it’s doing what I need it to do, but I worry at some point I’ll have to make a switch again. I’d like to contribute more upstream, but my skills are not advanced enough for this.

GitHub should stand up to the RIAA over youtube-dl

Earlier this week, GitHub took down the repository for the youtube-dl project. This came in response to a request from the RIAA—the recording industry’s lobbying and harassment body. youtube-dl is a tool for downloading videos. The RIAA argued that this violates the anticircumvention protections of the Digital Millennium Copyright Act (DMCA). While GitHub taking down the repository and its forks is true to the principle of minimizing corporate risk, it’s the wrong choice.

Microsoft—currently the world’s second-most valuable company with a market capitalization of $1.64 trillion—owns GitHub. If anyone is in a position to fight back on this, it’s Microsoft. Microsoft’s lawyers should have a one word answer to the RIAA’s request: “no”. (full disclosure: I own a small number of shares of Microsoft)

The procedural argument

The first reason to tell the RIAA where to stick it is procedural. The RIAA isn’t arguing that youtube-dl is infringing its copyrights or circumventing its protections. It is arguing that youtube-dl infringes YouTube’s protections. So even if it is, that’s YouTube’s problem, not the RIAA’s.

The factual argument

I have some sympathy for the anticircumvention argument. I’m not familiar with the specifics of how youtube-dl works, but it’s at least possible that youtube-dl circumvents YouTube’s copy protection. This would be a reasonable basis for YouTube to take action. Again, YouTube, not the RIAA.

I have less sympathy for the infringement argument. youtube-dl doesn’t induce infringement more than a web browser or screen recorder does. There are a variety of uses for youtube-dl that are not infringing. Foremost is the fact that some YouTube videos are under a license that explicitly allows sharing and remixing. Archivers use it to archive content. Some people who have time-variable Internet billing use it to download videos overnight.

So, yes, youtube-dl can be used to infringe the RIAA’s copyrights. It can also be used for non-infringing purposes. The code itself does not infringe. There’s nothing about it that gives the RIAA a justification to take it down.

youtube-dl isn’t the whole story

youtube-dl provides a focal point, but there’s more to it. Copyright law is now used to suppress instead of promote creative works. The DMCA, in particular, favors the large rightsholders over smaller developers and creators. It essentially forces sites to act on a “guilty until proven innocent” model. Companies in a position to push back have an obligation to do so. Microsoft has become a supporter of open source, now it’s time to show they mean it.

We should also consider the risks of consolidation. git is a decentralized system. GitHub has essentially centralized it. Sure, many competitors exist, but GitHub has become the default place to host open source code projects. The fact that GitHub’s code is proprietary is immaterial to this point. A FOSS service would pose the same risk if it became the centralized service.

I saw a quote on this discussion (which I can’t find now) that said “code is free, infrastructure is not.” And while projects self-hosting their code repository, issue tracker, etc may be philosophically appealing, that’s not realistic. Software-as-a-Service has lowered the barrier for starting projects, which is a good thing. But it doesn’t come without risk, which we are now seeing.

I don’t know what the right answer is for this. I know the answer won’t be easy. But both this specific case and the general issues they highlight are important for us to think about.

Indiana COVID-19 update: 23 October 2020

I’m not going to sit here and try to come up with new ways to say “this is bad”. Not a lot has changed since the last update: the numbers are all chugging along on trend. There is a change to my dashboard, however.

I realized today that I had been pulling the wrong data from the IHME models. I had been entering the “best case” scenario that includes universal mask wearing and the like. What I should have been pulling from was the reference model. This results in higher predicted values.

The overall impact isn’t that great. The scenarios don’t really diverge for a while, so for the most part the model error graph is unchanged. The future, particularly late November and into December is where you notice a difference. The recent models still under-predict the deaths, but the general trend matches well.

IHME said in their latest update that they didn’t make many changes to the model for the latest run. It’s essentially the same as last week but with more recent data. Unsurprisingly, it’s pretty close to the previous run. Both of those have a lower peak than forecasts from September, but still 50% higher than the spring peak.

It’s worth noting that IHME’s model assumes that states will re-introduce restrictions at a certain point. I’m having a hard time seeing that happening in Indiana—at least not as quickly as IHME’s assumption would have it. I wonder how much of Governor Holcomb’s refusal to even entertain the idea of moving back a phase or several has to do with the election in a week and a half. After the election, he’ll either be a lame duck or he’ll be into his last term (sort of). That takes away much of the political risk.

Indiana COVID-19 update: 17 October 2020

I updated my Indiana COVID-19 dashboard with the latest numbers. It continues to look bad. Hospitalizations are up 15% in the past week. The new daily case record set yesterday is 30% higher than the record set a week ago. We had two days in the last four with 20+ deaths (and bear in mind that the recently daily numbers tend to rise rather significantly in the days that follow).

Most alarming is the latest forecast from the Institute for Health Metrics and Evaluation (IHME). Their 10/15 model forecast is now on the dashboard, and it continues to show a big upswing in fatalities through November and December. The models have been pretty consistent with underpredicting the death count lately, so the big increase in the last two runs is extra worrisome.

In order to get a better sense of the past and possible future, I plotted the observed deaths with each of the model runs I have in the spreadhseet.

Observed and forecast daily COVID-19 deaths in Indiana

While the early September runs were a little hot, none of them really captured the increase we’ve seen over the past few weeks. The last two runs (10/9 and 10/15) are the first two to fully consider the move to Stage 5, I believe. And it’s clear that the forecast is not looking kindly on that.

Indeed, the Governor’s move to Stage 5 looks worse and worse with each passing day. The state health commissioner announced earlier this week that she and her family tested positive.

Indiana COVID-19 update: 10 October 2020

I just updated my Indiana COVID-19 dashboard with today’s numbers. They are not pretty. The state set a record for new cases for the third consecutive day. Today’s increase was “only” 6.5%, which is an improvement on the 24% increase that yesterday’s new record represented.

Positive tests or positive individuals?

The number of tests administered is on the rise, but we’re testing far fewer individuals. In fact, we’re testing about 33% fewer people a day than we did at the peak in late August. That the state is focusing on the total positivitiy rate (5.2% over the last 7 days) as opposed to the rate of positive individuals (9.3% over the last 7 days) strikes me as deceptive.

I attribute the disproportionate increase in tests (compared to people tested) to school systems, at least in part. I know of teachers who have had to take several COVID-19 tests in the past two months in order to return to work after any illness that shares a symptom with COVID-19. While I applaud the schools for taking this seriously, it does lead to some misleading numbers.

Deaths and hospitalizations

On Thursday, Indiana hit 20 daily COVID-19 deaths again. Most recently, this mark was tallied on September 26 (21 deaths). The last time before that was June 14th. We have not had a day with single-digit deaths since September 21. The only stretch longer than that is the 68 days from March 28 through June 5.

Hospitalizations are up dramatically as well, as I mentioned in the last update. The current levels haven’t been seen since late May. Hospitalizations yesterday were 42% higher than on September 9 and 30% than two weeks prior.

Looking forward

The Institute for Health Metrics and Evaluation (IHME) released a new model run late last night. I have added that to the dashboard as IHME 10/9 and hidden the IHME 9/11 lines for readability. A few days in, and this run seems to be over-estimating Indiana deaths so far. This is a welcome relief, since the last month’s worth of runs have been pretty consistently running too low. Given that deaths tend to be reported over the course of several days, the model may end up being more accurate after all. IHME has not published the updated briefing yet, so they may have more to say about the changes in this week’s run.

IHME’s forecast assumes that states will re-implement restrictions when conditions deteriorate to a certain point. Assuming that is accurate, we’re looking at restrictions coming back in mid-to-late November. Under that scenario, the forecast calls for a peak of 66 deaths per day in early December (with a range of 35-105). That would exceed our April peak by 32%.

However, given Governor Holcomb’s decision to move to Stage 5 in the face of materially unimproved circumstances, I don’t know if we can depend on that. If we do nothing, or further ease the few restrictions left, the model suggests we could be losing over a hundred Hoosiers a day in late December.

Other writing: September 2020

What have I been writing when I haven’t been writing here?

Stuff I wrote

Fedora

Stuff I curated

Fedora

Updated observations on Indiana COVID-19 trends

I haven’t made any changes to my Indiana COVID-19 plots since the last update, but I wanted to comment on some of the trends. The Governor announced a week ago today that the state would move to Stage 5 of our response. The reduced restrictions took effect on Saturday.

It’s far too early for drawing any causal effects. Nonetheless, I find it interesting that fate seems to be saying “I’ll show you!” In the last six days, the trend in daily deaths is upward. Saturday, Sunday, and Monday have averaged an increase in 7 deaths over the prior week (although the two-week comparisons are much noisier). The week-over-week change in cases is riding a five-day positive run. This is the first stretch longer than three days since early August. Normally the new case count varies wildly in both directions, so it’s unusual to see a stable run like this.

The state’s dashboard hints at an upward trend in the positive test rate again. Hospitalizations are up 16% (135 patients) in the past week. This trend has continued fairly steadily for the past week and a half.

What concerns me most is the model verification. The last few weeks of IHME forecasts were initially running a bit high, but in the last few days, they’re now under-predicting the daily death counts. This could suggest that the bad scenario predicted for December will be worse than forecast. It also may not. This is a short window, so we’ll have to see how trends hold.

IHME model daily death forecast model error.

As I said at the beginning, these bad trends in Indiana’s data cannot be tied to the move to Stage 5. But it does suggest that it was a bad decision. As my friend Renee wrote today, it’s less that things have improved and more that we’ve just grown accustomed to things being bad.

New entries in the Forecast Discussion Hall of Fame

Well it took approximately forever, but I finally got around to updating the back end tech for my website (post coming soon!). That means I can catch up on a backlog of Forecast Discussion Hall of Fame nominees.

I have added four new entries to the Social Media Foyer: tweets from NWS offices in Atlanta, Paducah, and Pittsburgh, as well as a Facebook post from Nashville. Plus: a Public Information Statement that includes a map, a Santa Claus watch, forecasters in Juneau telling you to go outside, a sweet discussion with no tricks, an “I’m only doing this because I have to”, a not-so-super Mario, and monster wrestling. Because you’ve all been so patient, there’s also a new entry in the ISIXTYFIVE section where he comes to the conclusion that he sucks.

Some of these have been sitting in my inbox since January 2018, so I apologize in being slow to add them. You get what you pay for sometimes.

Graphing Indiana’s COVID-19 situation

Like many of you, COVID-19 has weighed heavily on me in 2020. Part of the weight is the uncertainty of it all. While we seem to have a reasonable knowledge now of how to minimize spread and avoid fatality (not that we necessarily are doing these things. Wear your damn masks, people), that was not the case in the beginning. And while I’m not a virologist or an epidemiologist, I find having a sense of the numbers helps my unease. So early on, I started keeping track of some basic stats for Indiana COVID-19 deaths in a Google spreadsheet. You can take a look at it now. Below, I explain some of the history and some observations.

Initial work

At first, I tracked the deaths by day of report. This led to a noticeable pattern. Deaths dropped Sunday and Monday, since the previous day was a weekend. I assume hospitals were slower to report to the local health departments who were in turn slower to report to the states. To address this I also had a plot that ignored weekends. For both of these, I had a seven-day moving average to smooth out individual bumps in the data. This made it easier to spot trends.

After a while, though, I realized that the deaths reported on any given day could represent deaths that occurred on many days. Realizing this, I cleared out the old data and went through each day on the Indiana COVID-19 dashboard. The state makes it easy to see when past days have new deaths added, so it’s easy to keep that up to date. I plotted the daily deaths on linear and log scales with 7-day moving averages. Those first two graphs have basically remain unchanged since.

Indiana COVID-19 daily deaths through 18 September 2020.

It’s also worth noting that the state’s dashboard has improved dramatically since the early days. This includes a moving average for all of the reported metrics.

Spotting trends

Even without relying on day-of-report for tracking deaths, there seemed to be a rough periodicity to the daily death counts. I won’t try to come up with an explanation. But it was clear that comparing day-to-day didn’t necessarily give an accurate picture. So I started tracking week-over-week and week-over-two-week death counts. This, I figured, gives a better picture of the trend. If the differences are consistently negative, that means we’re heading in the right direction. If the differences are consistently positive, that’s a bad sign.

Indiana COVID-19 death comparisons to previous weeks and two weeks ago through 18 September 2020.

After a while, I decided to start tracking cases in the same way. The state’s dashboard makes this more difficult. The graphs don’t indicate when dates have changed, although in daily checks I’ve routinely observed changes of 5-10 cases as far back as 2-3 weeks. The state does make data available via downloadable spreadsheet, so I’ve started using that instead. It’s just less convenient (especially on a weekend when I am sometimes doing it from my phone).

Model verification

Most recently (as in the last two days), I’ve started tracking the Institute for Health Metrics and Evaluation’s (IHME) forecasts. I’d checked their website pretty regularly in the beginning, but now that we’ve reached a sort of terribleness equilibrium, I haven’t. But given the model trends that are suggesting a really terrible Christmas for a lot of people, I thought it would be worth paying attention to.

George E.P. Box said “all models are wrong, but some are useful”. You don’t earn a meteorology degree without learning this lesson. So in order to see how useful the models are, I’m comparing their forecast to the actual deaths.

This is where it gets pretty squishy. To de-noise the data a little bit, I’m comparing the models to a three-day average. I picked that because that’s what IHME says they use to smooth the observed data on their website. But their smoothed numbers don’t quite match mine, so I don’t really know.

At any rate, IHME seems to update about once every week or so. That graph would get messy pretty quickly. My plan is to keep the four most recent model runs and the first run in prior months just to get a feel on how much the model forecasts are improving. I haven’t gone back to add historical model runs beyond the few I’ve currently added. I may end up doing that at some point, but probably not. I’m not particularly interested in whether or not a model from April correctly predicted December. I care if last week’s forecast looks like it has a good handle on things.

Observations

Indiana’s daily death rate has been remarkably consistent over time. With the exception of early August when we saw a bump, we’ve averaged around 9 deaths per day since late June. This is better than the quick increases we saw in April, when the increases were twice what the totals are now. But considering that early IHME model runs had the rate going to zero in May (if I recall correctly), 10 a day is pretty disheartening.

Hospitals and local officials are a little slow to report deaths. It’s not uncommon for a day’s count to double from the initial report in the days following. It’s gotten to the point where I generally don’t enter a day’s deaths until the next day in order to not skew the end of the graph.

The week-over-week differences in new cases are surprisingly volatile. As recently as a few days ago, there’s a swing from +359 on 14 September to -91 on 15 September in the one week comparisons. The two week comparison went from -376 on 9 September to +445 on 10 September. Just looking at the graph, the volatility has seemingly worsened over time.

The future

I try to update the spreadsheet every day. Generally in the early afternoon, as the state dashboard updates at noon. At the moment, I don’t have any plans to make significant changes to what I track or how I graph it. If I do, I’ll post here. I have briefly considered writing some tooling to graph, parse, and plot all of the input data, but the spreadsheet works well enough for now. I have plenty of other things to occupy my time.

Linux distros should be opinionated

Last week, the upstream project for a package I maintain was discussing whether or not to enable autosave in the default configuration. I said if the project doesn’t, I may consider making that the default in the Fedora package. Another commenter said “is it a good idea to have different default settings per packaging ? (ubuntu/fedora/windows)”

My take? Absolutely yes. As I said in the post on “rolling stable” distros, a Linux distribution is more than an assortment of packages; it is a cohesive whole. This necessarily requires changes to upstream defaults.

Changes to enable a functional, cohesive whole are necessary, of course. But there’s more than “it works”, there’s “it works the way we think it should.” A Linux distribution targets a certain audience (or audiences). Distribution maintainers have to make choices to make the distro meet that audience’s needs. They are not mindless build systems.

Of course, opinions do have a cost. If a particular piece of software works differently from one distro to another, users get confused. Documentation may be wrong, sometimes harmfully so. Upstream developers may have trouble debugging issues if they are not familiar with the distro’s changes.

Thus, opinions should be implemented judiciously. But when a maintainer has given a change due thought, they should make it.