3-4 week forecasts are coming, or not

With a few exceptions, most operational public forecasts only go out to 7-10 days. This is due in large part to the fact that the skill of models gets progressively worse the further out you go. However, a recent article in the Journal of Climate suggests that models may have sufficient skill for forecasts in the 3-4 week range.

I have not read the full paper (yet), but the abstract suggests that this isn’t as great as it might sound at first. The resolution, according to the abstract, is one degree of latitude at 38 degrees. By my calculations, that’s a grid spacing of about 69 miles or 111 kilometers. The grid used by the Global Forecast System (GFS) model is 28 kilometers. Shorter-range and regional models use even smaller grids.

Grid size matters because it effects the scale of weather phenomena that can be modeled. Larger-scale features such as low pressure systems can be captured at this scale, but much gets lost. So you wouldn’t use this forecast to schedule the exact time of your picnic four weeks from now, but it might at least help you pin down a day.

Of course, this is all dependent on the NOAA budget. The Weather Research and Forecasting Innovation Act of 2017, which was passed by Congress and signed by the President, requires additional efforts to improve forecasts at this range. However, the proposed budget from the White House cuts this effort.

Terminate Investment in Mid-Range Weather Outlooks: NOAA requests a decrease of $5,000,000 to terminate all development, testing, and implementation of experimental products to extend operational weather outlooks, including temperature and precipitation outlooks, from 16 days to 30 days.”

The Washington Post‘s Capital Weather Gang blog and Professor Cliff Mass both have excellent writeups on the misalignment between the Weather Research and Forecasting Innovation Act and the proposed budget. The Weather Research and Forecasting Innovation Act of 2017 was a good bill, hopefully the budget adjusts to meet it. If not, the state of American weather forecasting is set to take a dramatic hit.

The free market will never fix technology policy issues

Over the weekend, I was listening to “This Week in Law” episode 384 as I mowed the front yard. While the panel was talking about the public relations fiasco that Unrollme experienced after reports surfaced that they sold user data to Uber. The conversation revolved around the terms of service for free services and how users can protect themselves and their data. Host Denise Howell asked if the free market could address the issue instead of requiring government intervention (e.g. by Federal Trade Commission regulations).

My first thought was “no”, but after giving it further consideration, it’s a more developed “no”. Here’s the thing:

The free market will never fix technology policy issues.

And why is that? An effective free market solution requires informed consumers. But consumers are not informed when it comes to technology services. As one of the panelists mentioned, it would take 76 work days per year to keep up with the privacy policies for the average user. That’s entirely untenable, and it’s probably gotten worse since that article was published five years ago.

One possible solution is a standard TL;DR (too long; didn’t read) format for privacy policies. This would at least allow users to “read” policies in something approaching a reasonable time. I’m not convinced that the industry could develop and widely adopt such a standard. It would probably take at least some degree of nudging from the FTC or other regulatory body. In likelihood, it would probably be more like the FDA’s requirements for nutrition labels.

But let’s assume that a privacy TL;DR standard developed and gained wide adoption without government intervention. It still doesn’t fix the problem. Access to information is not sufficient. Being informed also requires having the sophistication to process the information. Does the average user have the necessary knowledge to understand the implications of the privacy policy?

So a basic premise of a free market solution does not apply. But there’s another issue, too: competition. For services like Unrollme, it’s relatively easy for new competitors to spring up. If Unrollme does unsavory things with data, users can switch. That’s less the case with services like Facebook where the whole point is to be in the same place as your friends. Ask the Google+ team how easy it is to be an effective competitor against Facebook. Where there’s a network effect, there’s an effective lack of choice. Everyone these days wants to develop a network effect because that’s how they hold on to users.

The point of all of this is to say that it will probably never be reasonable to expect market effects to prevent user-hostile terms of service. While the free (or free-ish) market may be effective in some areas, technology policies is not one of them. User protection will only happen by way of regulation or legislation.

Find the solution by changing the problem

Or: what are you actually trying to do?

One thing I’ve noticed is that when I’m trying to solve a problem, the answer often becomes apparent when I stop to consider what it is I’m actually trying to do. This is hardly a new concept, I know. But so many times I’ve caught myself and others get stuck because they’re trying to solve a problem that doesn’t need solving. It’s not so much about “thinking outside the box” as much as it is thinking about what the box is.

One of the better ways I’ve found to do this is to not let customers speak in technical terms. More correctly: get them to describe features in business terms. So instead of saying “I want this page to show me x, y, and z when I click the foo button”, get them to say “I need to be able to tell my boss how much this simulation cost.” It may turn out you can solve that problem in a much easier or more reliable way.

This isn’t limited to technology, either. Half of answering a question is figuring out what’s really being asked. It’s not always the same as the words used. Just yesterday, I started to ask my boss about the setup period for a conference we’ll be at next week. Then I stopped and said “let me ask you my real question: do I have time to grab lunch with a coworker or should I go straight to the venue?” 

Even though I long ago learned that what’s said isn’t always what’s asked, I still find myself on the wrong end when I ask questions. I’ll catch myself asking two or three questions and then getting to the heart of the matter. And of course, if the person I’m talking to doesn’t realize what I’m actually asking, they might give an answer that doesn’t work in the context of my real question.

Does anyone at Twitter use Twitter?

Full disclosure: I own a small number of shares of Twitter.

Earlier this month, Twitter announced deals to bring more live content to the platform. Bloomberg will provide an original stream 24/7 and many other sources will generate technology, news, sports, and other content. Which makes me wonder if anyone at Twitter actually uses Twitter.

There’s something to be said for telling your users what they want instead of letting them tell you. It worked well for Apple, and of course there’s the famous Henry Ford quote about a faster horse. But this doesn’t seem like a product vision so much as grasping for something that might turn around the stock price. Twitter is a great place for near real time conversations about breaking news and live events, but is it the place to watch those? I’m not convinced.

It’s worth noting that Snap is working on similar deals for Snapchat. Snap is coming off a disappointing earnings report (its first since going public) that saw a 25% drop in stock price. Snap is facing a lot of pressure from Instagram, which is adding features that look very similar to Snapchat’s with the added bonus of being a Facebook property.

Facebook has been strong in user-generated live content, but they don’t seem to be that interested in pursuing Content. Given the success of Facebook, this is either a glaring oversight or a wise decision that other social networks might want to take a lesson from.

But getting back to Twitter, I recently joined the “Twitter Insiders” community. They asked for feedback on a potential new threading feature last week. It’s basically native tweetstorms. One of the survey questions asked what I’d call such a feature. I said “Medium”.

Motivations for storm chasing

Maybe I’m not the right person to write this post. Or maybe I would have been had I written it during a time when I was active. (It’s almost six years since the last time I went storm chasing, how much longer can I pretend that it’s a thing I do?) But here on Blog Fiasco, I get to make the rules, and Rule #1 is “Ben gets to write about whatever the hell he feels like writing about.”

At any rate, it seems that storm chasers have one thing in common: we/they really like to criticize the motivations of others. The most common target are the chasers who get in extremely close in order to get the perfect shot for TV. They take risks that most of us won’t (whether or not those risks are justified are left as an exercise for the reader). As a result, they’re dismissed as merely thrill-seekers by the “serious” chasers.

He’s in it for the money, not the science.

As my friend Amos said, “there’s no single explanation for chasing. It’s like trying to count all the reasons tourists visit Paris.” “Serious” chasers like to think they’re doing it for some altruistic reason. That could be scientific research, warning the public, or whatever. These things definitely happen, and they’re very good reasons for participating in an activity, but I doubt it’s what primarily motivates people.

Warning can be done by stationary (or nearly stationary) spotting, which also probably means you’ve developed some kind of relationship with the local authorities or NWS office. Some kinds of scientific research can only happen in situ, but that also requires a degree of discipline that many don’t want. Storm chasing is a very boring hobby that involves sitting on your butt in a car for hours on end in the hopes of seeing something interesting. It takes more than a sense of civic duty for most people.

I used to think I was doing it as a learning exercise or in order to serve the public. At some point I realized I was kidding myself. I chased (and hope to chase again) because I enjoy the thrill of the hunt. Can I figure out what the atmosphere is doing? Can I stay ahead of a dangerous beast while keeping myself safe? I’ll absolutely report severe weather I see, and I’ll share pictures with the NWS and any researchers, but that’s not the primary motivation. Now to get myself back out there…

Why HTCondor is a pretty awesome scheduler

In early March, The Next Platform published an article I wrote about cHPC, a container project aimed at HPC applications. But as I wrote it, I thought about how HTCondor has been addressing a lot of the concerns for a long time. Since I’m in Madison for HTCondor Week right now, I thought this was a good time to explain some of the ways this project is awesome.

No fixed walltime. This is a benefit or a detriment, depending on the circumstances, but most schedulers require the user to define a requested walltime at submission. If the job isn’t done at the end of that time, the scheduler kills it. Sorry about your results, get back in line and ask for more walltime. HTCondor’s flexible configuration allows administrators to enable such a feature if desired. By default users are not forced to make a guess that they’re probably going to get wrong.

Flexible requirements and resource monitoring. HTCondor supports user-requestable CPU, memory, and GPU natively. With partitionable slots, resources can be carved up on the fly. And HTCondor has “concurrency limits”, which allow for customizable resource constraints (e.g. software licenses, database connections, etc).

So many platforms. Despite the snobbery of HPC sysadmins, people do real work on Windows. HTCondor has almost-full feature parity on Windows. It also has “universes” for Docker and virtual machines.

Federation. Want to overflow to your friend’s resource? You can do that! You can even submit jobs from HTCondor to other schedulers.

Support for disappearing resources. In the cloud, this is the best feature. HTCondor was designed for resource scavenging on desktops, and it still supports that as a first-class use case. That means machines can come and go without much hassle. Contrast this to other schedulers where some explicit external action has to happen in order to add or remove a node.

Free as in freedom and free as in beer. Free beer is also the best way to get something from the HTCondor team. But HTCondor is licensed under the Apache 2.0 license, so anyone can use it for any purpose.

HTCondor isn’t perfect, and there are some use cases where it doesn’t make sense (e.g. low-latency), but it’s a pretty awesome project. And it’s been around for over three decades.

Other writing in April 2017

Where have I been writing when I haven’t been writing here? If you don’t want to wait for these monthly summaries, subscribe to my weekly newsletter!


Over on Opensource.com, we had our 7th consecutive million-page-view month. I’m not going to bother mentioning this anymore. I wrote the articles below.

  • Haters gonna hate: 7 ways to deal with criticism – Jason van Gumster and I wrote this Taylor Swift-themed article about dealing with haters when you share your work. One commenter even turned it into a live demo.
  • Creative Commons: 1.2 billion strong and growing – Creative Commons released their annual “State of the Commons” report this week. More and more works use those licenses, and there’s some really interesting projects (like a human skull-shaped 3D-printable candy dish).

Cycle Computing

Meanwhile, I wrote or edited a few things for work, too:

Why I choose the licenses I do

In the discussion around Richard Stallman’s speech at Purdue and my subsequent blog post about it, there was a great focus on the philosophy of various licenses. On Twitter, the point was raised that the GPL restricts the freedom of those making derivative works. The implication being that it imposes selfish restrictions. As a response to that (and because I told “MissMorphine” that I would write on this), I thought it would be a good idea to write about my own license choices and the personal philosophy behind them. Never mind the fact that it’s five years late.

First, my general philosophy. I am a proponent of the free sharing of information, knowledge, and software. I’m also aware that producing these things requires effort and resources, so I’m sympathetic to people who prefer to receive compensation. For myself, I generally opt to make things available. In exchange, I ask that people who would build off my work not restrict the freedoms of others. In order to do that, I must restrict the freedom to restrict freedoms. This is the choice I make for my own work.

Open source licenses (and I use the term broadly here to include the Creative Commons licenses and similar) maximize freedom in one of two ways: they maximize freedom for the next person downstream or they maximize freedom to any level downstream. It’s shouldnt be a surprise to learn that the former is often favored by developers and particularly by commercial entities, as it allows open source code to be used in closed products.

My default licenses are the GNU General Public License version 2 for code and Creative Commons Attribution-ShareAlike 4.0 for non-code. The gist (and to be clear, I’m hand waving a lot of detail here) of these licenses is “do what you want with my work, but give me credit and let others have the same rights.” However, I’ve licensed work under different licenses when there’s a case for it (e.g. contributing to a project or ecosystem that has a different license).

The important thing to remember when choosing a license is to pick one that matches your goals.


Why can’t I develop software the right way?

It’s not because I’m dumb (or not just because I’m dumb).

Readers of this blog will know that I have no shortage of opinions. I know what I think is the Right Way™ to develop software, but I have a hard time doing it when I’m a solo contributor. I’ve never written unit tests for a project where I’m the only (or primary) contributor. I’m sloppy with my git branching and commits. I leave myself fewer code comments and write less documentation. This is true even when I expect others may see it.

A few months ago, I was at the regular coffee meetup of my local tech group. We were talking about this phenomenon. I blurted out something that, upon further reflection, I’m particularly fond of.

Testing is a cultural thing and you can’t develop a culture with one person.

Okay, maybe you can develop a culture with one person, but I can’t. One could argue that a one-person culture is called a “habit”, and I’m open to that argument. But I would counter that they’re not the same thing. A habit is “a settled tendency or usual manner of behavior”, according to my friends at Merriam-Webster. In other words, it’s something you do. A culture, in my view, is a collective set of habits, but also mores. Culture defines not just what is done, but what is right.

Culture relies on peer pressure (and boss pressure, and customer pressure, etc) to enforce the norms of the group. Since much of the Right Way™ in software development (or any practice) is not the easy way, this cultural pressure helps group members to develop the habit. When left alone, it’s too easy to take the short term gains (e.g. not going to the effort of writing tests) even if that involves long term pain (yay regressions!).

If I wrote code regularly at work, or as a part of an open source project that has already developed a culture of testing, I’m sure I’d pick up the habit. Until then, I’ll focus on leading thoughts and let “mostly works” be my standard for code.

“Oh crap, I have to manage”

The logical extension of what I wrote above is that code testing and other excellent-but-also-no-fun practices will not spontaneously develop in a group. Once the culture includes the practice in question, it will be (to a large degree) self-perpetuating. Until then, someone has to apply pressure to the group. This often falls to a technical lead who is probably not particularly fond of putting on the boss hat to say “hey, you need to write tests or this patch doesn’t get accepted”.

Too bad. Do it anyway. That’s one of the key features of project leadership: developing the culture so that the Right Way™ becomes the way things are done.

Climatune: correlating weather and Spotify playlists

I don’t remember how I stumbled upon it, but I recently discovered Climatune, a joint effort between Spotify and Accuweather that presents a playlist to match the current weather. According to Accuweather’s blog post on the topic, the playlists were developed based on an analysis of 85 million streams across 900 cities. This is an incredibly interesting project, even if the Lafayette playlists don’t seem to vary much.

What I like about projects such as Climatune is the reminder that we are still animals affected by our surroundings. When I worked at McDonald’s, we noticed anecdotally that sales of the Filet-o-Fish apparently increased on rainy days. I regret that I never ran daily sales reports to test this observation. I suppose in either case, there was an effect. Either customers ordered more, or we were more aware of the sales.

Correlating weather with other data is hardly a new concept. The most common example is probably comparing Chicago crime reports to the temperature. But researchers have investigated mood-weather correlation from Twitter posts. Several studies have examined the effects of weather on the stock market.

These sorts of studies can be hard, since it’s hard to control for all of the possible factors. But even if we can’t draw statistically sound conclusions, it’s fun to look at the connections. And if weather isn’t your thing, Spotify also has a site that makes a custom playlist that fits the cook time of your Thanksgiving turkey.