Hurricane Lee forecast game

Update: we have a winner!

It took longer than I’d have liked to publish the results of this contest. I was traveling out of the country. But I’d like to congratulate KP on winning the Lee forecast contest.

One thing that I realized after the fact: my changes below made it so only whole numbers could be used for latitude and longitude. I’ve fixed that for next time!

Original post

The prodigal game returns! A technical glitch ruined the Dorian contest in 2019, so we haven’t seen a Funnel Fiasco tropical forecast game since Hurricane Matthew in 2016. But I’m pleased to announce that we’re up and running for Hurricane Lee. You can submit your landfall forecast by 2100 UTC on Wednesday 13 September.

In keeping with tradition, we’re still using the same crappy Perl script I wrote in 2005. Despite the fact that I’ve been putting off a total rewrite for over a decade, I did make a few improvements recently:

  • Numerical fields now require numeric input. If you were hoping to submit “butts” as your wind speed, I’m sorry to disappoint you.
  • Coordinates are constrained to reasonable ranges. I refuse to give in to Kevin’s whining about west being negative numbers. (I believe my exact words to him were “take it up with the Prime Meridian.”) But I was feeling magnanimous so I’ve constrained the latitude to 0–90 degrees north and the longitude to 180 degrees west to 10 degrees east.
  • Similarly, wind speed is now constrained to realistic values. You can’t submit a wind speed less than zero or above 200 miles per hour.
  • Furtherly similar, the time segments can’t be negative or overflow.

So go ahead and submit your forecast by 2100 UTC on Wednesday so you can join in the grand tradition.

Beware weather forecast snake oil

Snake oil salesman are found in every industry and weather forecasting is no different. So how do you identify weather forecast snake oil? One major sign is that the forecaster doesn’t talk about it until after the fact. Another is that you only hear about the successful forecasts. And of course, if it seems to good to be true, there’s a good chance it is.

I recently saw someone talking about severe weather forecasts months out. The man behind these forecasts isn’t just some rando with a website. He has a PhD in meteorology from the University of Oklahoma and is a forecaster at the Storm Prediction Center. So it’s entirely possible that he’s on to something here. But I’m suspicious.

He recently posted about his forecast for March tornadoes. The forecast is ostensibly from three months before the outbreak. I looked through the archives and there was no indication prior to the fact. His website contains no forward-looking forecasts. There’s no methodology. There’s no discussion of busted forecasts.

I don’t know Dr. Cook. I don’t want to say anything about him as a person or a forecaster. But until he shows more transparency on his forecasts, I’m inclined to call it weather forecast snake oil.

Weather forecast accuracy is improving

ForecastWatch recently issued a report on the accuracy of weather forecasts from 2010 through June 2016 (PDF here). While many readers will focus on who was more accurate, what stood out to me was how forecast accuracy has improved. Meteorologists have long “enjoyed” a reputation for inaccuracy — often more due to perception than fact. But those in the know are aware that skill is increasing.

Forecast accuracy over time

ForecastWatch’s U.S. analysis shows a clear — if small — improvement in the average accuracy since 2010.

Average U.S. forecast accuracy from 2010 – June 2016.

The chart above shows the average for all of the forecast sources Forecast Watch analyzed. To be frank, World Weather Online is a stinker, and brings the averages down by a considerable margin. Examining the best and worst forecast shows more interesting results.

Best and worst U.S. forecast accuracy from 2010 – June 2016.

Forecasts get less skillful over time, thanks to subtle inaccuracies in the initial conditions (see also: butterfly effect). That’s obvious in both graphs. What this second chart shows is that the best 6-9 forecast is now roughly as skillful as the worst 3-5 forecast was in 2010. And the best 3-5 day forecast is in the middle of the 1-3 day forecast skill from just a few years ago.

Forecasts are definitely improving. This is due in part to better modeling — both more powerful computers and also the ability to ingest more data. Research and improved tooling helps as well.

Forecasts still bust, of course, and forecasters hate bad forecasts as much as the public does. As I write this, forecasters in North Carolina are dealing with an inaccurate snow forecast (winter weather forecasting sucks due to reasons I explained in a previous post). Missed forecasts can cost money and lives, so it’s good to see a trend of improvement.

Forecast accuracy in your city

The ForecastWatch report breaks down by broad regions: United States, Europe, and Asia/Pacific. But weather is variable on much smaller scales. The ForecastAdvisor tool compares forecasts at the local level giving you the ability to see who does the best for your city. As of early January 2017, AccuWeather had the most accurate forecasts for Lafayette, Indiana, but they only place fourth when considering the past year.

Long range heat wave forecasts

What if I could tell you today that we’d have a major heat wave on June 11? A recently-published study could make that possible. Researchers analyzing heat waves over several decades have found a signal that improves the reliability of long-range heat forecasts. Will it be useful for forecasting specific days? I have my doubts, but we’ll see. They apparently plan to use it quasi-operationally this summer.

The more likely scenario is that it will help improve probabilistic forecasts on a multi-day-to-month scale. For example, the one month and three month outlooks issued by the Climate Prediction Center. There’s clear value in knowing departure from normal conditions over the course of the next few months, particularly for agricultural concerns but also for civic planning. I’m not sure I see much value in knowing now that June 11 will be oppressively hot as opposed to June 4.

While this study got a fair amount of coverage in the weather press, I don’t see that it will have much of an impact to the general public for a while. In fact, if it results in gradual improvement to long-range probabilistic forecasts, the public will probably never notice the impact, even if it turns out to be substantial over the course of several years.

Why the Sunshine app won’t change weather prediction

With $2 million in funding behind it, the Sunshine app hit the iOS App Store on Wednesday. Sunshine promises to disrupt weather forecasting by using crowd-sourced data and providing custom point forecasts. Sadly, that promise will fall flat.

First, I’m pretty wary of weather companies that don’t have a meteorologist on staff. If Sunshine has one, they’re doing a good job of hiding that fact. It’s not that amateurs can’t be good forecasters, but the atmosphere is more complicated than it is often given credit for. The Sunshine team seems to know just enough to say things that sound reasonable but aren’t really. For example, this quote from CEO Katerina Stroponiati.

The more users we have, with phones offering up sensor data and users submitting weather reports, the more accurate we will get. Like an almanac.

Except that almanacs aren’t accurate. Then there’s this quote from their first Medium post.

The reason weather forecasts are inaccurate and imprecise is because traditional weather companies use satellites that can only see the big picture while weather stations are few and far between.

That’s fairly accurate (though it oversimplifies), but they point to a particularly noteworthy busted blizzard forecast as an example of the inaccuracy of traditional forecasts. Snowfall can be impacted greatly by small differences, but blizzards are fairly large-scale systems, and I’m skeptical that Sunshine would have done any better, especially considering that it has no “experience” outside of the Bay Area.

It sounds like Sunshine’s approach is basically a statistical model. That is a valid and often valuable forecast tool, but it has its limits. Sunshine claims a 36% improvement over “weather incumbents” in its trial period (where’s the published study?), but that involved only 200 testers in the San Francisco area. While definite microclimates exist in that region, it’s not exactly known for wild changes in weather. I doubt such an improvement could be sustained across a wider area.

Sunshine relies on crowdsourced reports and the pressure sensor in new iPhones to collect data. Unlike many other parameters, reasonably accurate pressure measurements are not sensitive to placement. A large, dense network of pressure sensors would be of considerable benefit to forecasters, provided the data is made available. However, wind, temperature, and humidity measurements — both at the surface and aloft — are important as well. This is particularly true for severe weather events.\

Crowdsourcing weather observations is nothing new. Projects like CoCoRaHS and mPing have been collecting weather data from the general public for years. The Weather Underground app has crowdsourced observations, and Weather Underground — along with other sites like Weatherbug — has developed a network of privately-owned weather observation stations across the country. The challenge, as it will be with Sunshine’s reports, lies in quality assurance and making the data available to the numerical weahther prediction models.

I hope Sunshine does well. I hope it makes a valuable contribution to the science of weather forecasting. I hope it gets people asking their Congressional delegation why we can’t fund denser surface and upper-air observations. I just don’t expect it will have much of an impact on its own.

Hurricane Joaquin forecast contest begins

Hey! The tropics have awoken and there’s a not-unreasonable chance that the newly-upgraded Hurricane Joaquin will make landfall. Here’s your chance to test your forecast skill: http://funnelfiasco.com/cgi-bin/hurricane.cgi?cmd=view&year=2015&name=joaquin

Submit your forecast by 00 UTC on October 2 (8 PM EDT Thursday). If Joaquin does not make landfall, we’ll just pretend like this never happened. For previous forecast game results, see http://weather.funnelfiasco.com/tropical/game/

Regional weather forecast offices?

Update: The Senate Commerce Committee has amended the bill to remove the regionalization, according to The Washington PostThe bill now focuses on improving how weather hazards are communicated to the public (an effort that is already underway in both the public and private sectors). At the time of this update, it is unclear whether this morning’s post was what convinced Senator Thune to change approaches.


 

Last week, The Washington Post‘s Capital Weather Gang blog reported on a bill being introduced by the chair of the Senate Commerce Committee. The bill, dubbed “The National Weather Service Improvement Act”, would direct the National Weather Service to consolidate the current 122 weather forecast offices around the country down to six. Although the bill itself is light on details, supporters say it is designed to reduce under-staffing during severe weather outbreaks and it is apparently expected to save money (since the bill has explicit provisions for what is to be done with any savings).

The reactions among my meteorologically-inclined friends were mostly dismissive. Tim Cermak’s thoughts were fairly representative:

Others privately suggested the bill was solely political posturing. No one in Congress would want to be the one to shutter the local NWS office. Still, were the bill to become law, several people expressed concern about the loss of local knowledge.

There are some definite benefits to consolidating forecast responsibilities. Forecast area boundaries lead to discontinuous forecast grids and ugly warning polygons. It stands to reason that fair weather forecasts could be handled at the regional level. But what do the local office forecasters do while they wait for a significant weather event?

If I understand the text of the bill correctly, there are no local forecasters. Only the Warning Coordination Meteorologists would remain at local offices (and presumably electronics technicians to repair radars and other equipment). This would mean the loss of local knowledge which can be key during severe weather events. Local forecasters get to know the locations of small towns in their forecast area better than regional forecasters could. This is important not only for writing warning text, but for contextualizing storm reports.

Ubiquitous broadband Internet has removed some of the need for co- or near-location with data sources such as radars. The ham radio spotter networks would have some trouble, but that could be addressed. One concern I haven’t seen raised is what happens when an office has to take shelter. It’s one thing for an office to hand over responsibilities to surrounding offices for a few minutes as a tornado bears down (or a few hours because phone lines are down). This becomes more difficult (though likely more rare) when suddenly one sixth of the forecast offices drops offline.

Most of the thoughts above are not new. Many of them come from the National Academy of Sciences report upon which the Senate bill is supposedly based. The NAS report has a few key differences, though. Most importantly, it describes regional offices as one of several possibilities (another of which is to retain the status quo). The NAS report also explicitly keeps open local WFOs for hazardous weather, marine, and aviation products. The Senate bill (again, based on my reading) implies that these functions would also be regionalized.

Consolidating forecast operations is not without precedent; the U.S. Navy consolidated its various forecast units in 2011. I could not find any follow-up studies that evaluated a change in effectiveness or forecast accuracy. If any such studies were conducted, they may be classified.

It’s no secret that I’m generally a fan of the National Weather Service. I believe it is a steal for taxpayers. That’s not to say that there is no room for improvement. The NWS must be open to constructive criticism and changes that can improve its ability to perform its mission to protect life and property. This bill does not represent that. I’m not familiar enough with the politics to guess at the driving factor, but it seems to cherry pick from the NAS report without regard for the end result.

The amateur weather website hype machine

Word on the street is that a certain amateur meteorology site is starting to tease about a large snowfall event a week or so out. It must be winter again!

I don’t begrudge amateur meteorology sites in general. In the Internet age, there’s a lot that you can teach yourself and plenty of access to raw model data from which to build a forecast. As in most fields, the passionate amateur can be more skillful than the trained professional. Of course, this is generally limited to a specific skill, which is why the better amateur weather sites tend do focus on a particular thing.

Focusing on hyping winter weather events a week or more out is not an area that should be focused on. This is particularly true when the hype is completely unjustified meteorologocially and ends up requiring professional meteorologists in the National Weather Service and local media to spend time telling the public not to believe the “information” that should never have been shared in the first place.

Forecasting the weather is hard. Effectively communicating the uncertainty inherent to that forecast to the public is even harder (and not done nearly enough). Posting an outlier scenario to Facebook is easy. Any site that provides forecasts for public consumption and (somehow) finds a way to get partnerships with legitimate media outlets needs to eschew the easy. Otherwise, it’s simply self-service and not public service.

Never believe year-long forecasts

On my to-do list, this post is titled “Chad Evans, you son of a bitch.” Though the specifics are about the failings of a specific local TV meteorologist, the broader lesson is that weather forecasts longer than about a week aren’t worth the time it takes to make or read them. AccuWeather’s 45-day forecasts have caught some flack for being awful, as everyone expected they would be. Less attention has been paid to verifying the long-range forecasts from WLFI meteorologist Chad Evans.

I decided to take a look at the September 2011 forecast to see how it fared (there’s probably a forecast from September 2012, but I’m too lazy to search for it). As the graphs below show, it’s hard to beat climatology for long-range forecasts. Interestingly, there’s not a noticeable drop in skill over time with temperatures. The precipitation forecast does seem to get worse over the life of the forecast, with the exception of a lucky break in the summer.

Forecast and climatology monthly average temperatures.

Forecast and climatology monthly average temperature errors.

 

Forecast and climatology precipitation total errors.

Forecast and climatology precipitation total errors.

Mr. Evans was smart enough not to include day-by-day specifics, except for Christmas. This year, he claimed  claimed to be 4-0 on his white Christmas forecasts. The forecast called for 1″ or more of snow on Christmas morning. Unfortunately, there was none. Several inches fell the week before, but warm and rainy weather the weekend prior took care of that. Speaking of snowfall, 10″ was forecast for January 2014. In six days, we’ve already passed that, and the snow continues to fall as I write.

In the first two months of the most recent annual forecast, the temperature errors aren’t awful, but the precip forecasts miss the mark pretty hard (though the direction of the error was right in both cases). As the year progresses, you’d expect to see the skill diminish.

Nov Dec
Tmax 9 1
Tmin 5 8
Tavg 3.9 2.1
Precip .73″ (25%) .95″ (38%)
Forecast absolute error

And that’s really the point here: seasonal (or longer) outlooks are really bad at giving specific information. You can sometimes make use of them for trends, but even then they’re not very reliable. I can’t fault a forecaster for busting a forecast, I’ve had plenty of busts. But presenting skill-less forecasts to the public is a disservice to the public and to the reputation of the meteorology profession.

Why I hate winter

Whenever snow appears in the forecast, I’m filled with dread. There are two reasons: 1) I hate shoveling the driveway and 2) people ask me “how much snow are we going to get?” I consider myself a pretty decent severe weather forecaster. It’s my particular area of interest, and I’ve given myself some practice at it. Winter weather forecasting is a whole ‘nother beast.

It’s not just that I don’t like it, or that I haven’t practiced (both are true), but winter weather forecasting is really more challenging. There are a variety of reasons — some scientific, some psychological. The most obvious scientific reason is that temperature matters. A three degree difference doesn’t mean much when the temperature is 80 degrees; the rain will still be rain. When the temperature is in the lower 30s, a three degree difference can be the difference between rain, snow, or some awful mix. Surface temperatures aren’t the only ones that matter; small differences in the low-level air temperatures can have an impact on the precipitation type.

Even when you nail the precipitation type, how much snow do you get? A common rule of thumb is that one inch of rain is equal to 10 inches of snow, but that’s a really awful rule. The snow-to-liquid ratio can vary widely. I’ve measured from 2.7:1 to 30:1 at my house, with the average somewhere around 14:1. With half an inch of liquid, the difference between the rule of thumb and my observed average is 2″ of snow.

Then there’s the psychological aspect. For the most part, people don’t care how much rain falls in a given event. Sure, ridiculous torrents like southern Indiana saw last weekend are noticeable, but how many people can look outside and tell the difference between .25″ and .5″of rain? I bet they can tell the difference between 3″ and 7″ of snow, though. Snow also requires more preparation than rain does. You generally don’t see grocery store bread and milk shelves scoured clean before a rain storm. The highway department doesn’t put in overtime to salt the streets before it rains. Schools very rarely delay or cancel classes because of rain.

Snow draws attention to itself. It’s the biggest prima donna of the non-destructive weather phenomena. The natural result is that people become very sensitive to how well a forecast verifies. Unfortunately, it verifies “not good” all too often. I suppose this is my way of saying “is it spring yet?”