Why the Sunshine app won’t change weather prediction

With $2 million in funding behind it, the Sunshine app hit the iOS App Store on Wednesday. Sunshine promises to disrupt weather forecasting by using crowd-sourced data and providing custom point forecasts. Sadly, that promise will fall flat.

First, I’m pretty wary of weather companies that don’t have a meteorologist on staff. If Sunshine has one, they’re doing a good job of hiding that fact. It’s not that amateurs can’t be good forecasters, but the atmosphere is more complicated than it is often given credit for. The Sunshine team seems to know just enough to say things that sound reasonable but aren’t really. For example, this quote from CEO Katerina Stroponiati.

The more users we have, with phones offering up sensor data and users submitting weather reports, the more accurate we will get. Like an almanac.

Except that almanacs aren’t accurate. Then there’s this quote from their first Medium post.

The reason weather forecasts are inaccurate and imprecise is because traditional weather companies use satellites that can only see the big picture while weather stations are few and far between.

That’s fairly accurate (though it oversimplifies), but they point to a particularly noteworthy busted blizzard forecast as an example of the inaccuracy of traditional forecasts. Snowfall can be impacted greatly by small differences, but blizzards are fairly large-scale systems, and I’m skeptical that Sunshine would have done any better, especially considering that it has no “experience” outside of the Bay Area.

It sounds like Sunshine’s approach is basically a statistical model. That is a valid and often valuable forecast tool, but it has its limits. Sunshine claims a 36% improvement over “weather incumbents” in its trial period (where’s the published study?), but that involved only 200 testers in the San Francisco area. While definite microclimates exist in that region, it’s not exactly known for wild changes in weather. I doubt such an improvement could be sustained across a wider area.

Sunshine relies on crowdsourced reports and the pressure sensor in new iPhones to collect data. Unlike many other parameters, reasonably accurate pressure measurements are not sensitive to placement. A large, dense network of pressure sensors would be of considerable benefit to forecasters, provided the data is made available. However, wind, temperature, and humidity measurements — both at the surface and aloft — are important as well. This is particularly true for severe weather events.\

Crowdsourcing weather observations is nothing new. Projects like CoCoRaHS and mPing have been collecting weather data from the general public for years. The Weather Underground app has crowdsourced observations, and Weather Underground — along with other sites like Weatherbug — has developed a network of privately-owned weather observation stations across the country. The challenge, as it will be with Sunshine’s reports, lies in quality assurance and making the data available to the numerical weahther prediction models.

I hope Sunshine does well. I hope it makes a valuable contribution to the science of weather forecasting. I hope it gets people asking their Congressional delegation why we can’t fund denser surface and upper-air observations. I just don’t expect it will have much of an impact on its own.

Hurricane Joaquin forecast contest begins

Hey! The tropics have awoken and there’s a not-unreasonable chance that the newly-upgraded Hurricane Joaquin will make landfall. Here’s your chance to test your forecast skill: http://funnelfiasco.com/cgi-bin/hurricane.cgi?cmd=view&year=2015&name=joaquin

Submit your forecast by 00 UTC on October 2 (8 PM EDT Thursday). If Joaquin does not make landfall, we’ll just pretend like this never happened. For previous forecast game results, see http://weather.funnelfiasco.com/tropical/game/

New entries in the Forecast Discussion Hall of Fame

Radar estimates of wind speed aren’t always the most reliable for a variety of reasons, but on Friday, forecasters in Memphis, TN opted to believe the radar instead of the surface observation. Maybe because the Millington station was reporting a 343 MPH wind gust.

NWS forecasters are public servants dedicated to preserving life and property. It should come as no surprise that they are sometimes moved by uncontrollable bursts of patriotism. Chris Hattings in Riverton, WY felt very Jeffersonian on Independence Day.

Both of these discussions have been added to the Forecast Discussion Hall of Fame.

Regional weather forecast offices?

Update: The Senate Commerce Committee has amended the bill to remove the regionalization, according to The Washington PostThe bill now focuses on improving how weather hazards are communicated to the public (an effort that is already underway in both the public and private sectors). At the time of this update, it is unclear whether this morning’s post was what convinced Senator Thune to change approaches.


Last week, The Washington Post‘s Capital Weather Gang blog reported on a bill being introduced by the chair of the Senate Commerce Committee. The bill, dubbed “The National Weather Service Improvement Act”, would direct the National Weather Service to consolidate the current 122 weather forecast offices around the country down to six. Although the bill itself is light on details, supporters say it is designed to reduce under-staffing during severe weather outbreaks and it is apparently expected to save money (since the bill has explicit provisions for what is to be done with any savings).

The reactions among my meteorologically-inclined friends were mostly dismissive. Tim Cermak’s thoughts were fairly representative:

Others privately suggested the bill was solely political posturing. No one in Congress would want to be the one to shutter the local NWS office. Still, were the bill to become law, several people expressed concern about the loss of local knowledge.

There are some definite benefits to consolidating forecast responsibilities. Forecast area boundaries lead to discontinuous forecast grids and ugly warning polygons. It stands to reason that fair weather forecasts could be handled at the regional level. But what do the local office forecasters do while they wait for a significant weather event?

If I understand the text of the bill correctly, there are no local forecasters. Only the Warning Coordination Meteorologists would remain at local offices (and presumably electronics technicians to repair radars and other equipment). This would mean the loss of local knowledge which can be key during severe weather events. Local forecasters get to know the locations of small towns in their forecast area better than regional forecasters could. This is important not only for writing warning text, but for contextualizing storm reports.

Ubiquitous broadband Internet has removed some of the need for co- or near-location with data sources such as radars. The ham radio spotter networks would have some trouble, but that could be addressed. One concern I haven’t seen raised is what happens when an office has to take shelter. It’s one thing for an office to hand over responsibilities to surrounding offices for a few minutes as a tornado bears down (or a few hours because phone lines are down). This becomes more difficult (though likely more rare) when suddenly one sixth of the forecast offices drops offline.

Most of the thoughts above are not new. Many of them come from the National Academy of Sciences report upon which the Senate bill is supposedly based. The NAS report has a few key differences, though. Most importantly, it describes regional offices as one of several possibilities (another of which is to retain the status quo). The NAS report also explicitly keeps open local WFOs for hazardous weather, marine, and aviation products. The Senate bill (again, based on my reading) implies that these functions would also be regionalized.

Consolidating forecast operations is not without precedent; the U.S. Navy consolidated its various forecast units in 2011. I could not find any follow-up studies that evaluated a change in effectiveness or forecast accuracy. If any such studies were conducted, they may be classified.

It’s no secret that I’m generally a fan of the National Weather Service. I believe it is a steal for taxpayers. That’s not to say that there is no room for improvement. The NWS must be open to constructive criticism and changes that can improve its ability to perform its mission to protect life and property. This bill does not represent that. I’m not familiar enough with the politics to guess at the driving factor, but it seems to cherry pick from the NAS report without regard for the end result.

Tornado warning false alarm rates

Five Thirty Eight recently ran a post about the false alarm rate of tornado warnings. Tornado warnings fail to verify (i.e. have no tornado) approximately 75% of the time, a number that has held steady for years. This comes as no surprise to meteorologists, and probably not to the general public. What’s disappointing about the article is that it doesn’t address the reason that the false alarm rate hasn’t improved: because it’s not a priority.

The ideal case, of course, is a false alarm rate of zero. While the article quotes the reasoning (“you would rather have a warning out there and have it miss than have an event and not have one out there”), it doesn’t explain why that reasoning leads to a high false alarm rate.

The first reason is that an emphasis on maximizing detection means that in questionable scenarios, forecasters will lean toward issuing a warning instead of not. I’ve been in an office when an unwarned tornado has been reported. The forecasters are not happy about that. They take the National Weather Service mission of protecting life and property seriously. The impact of a false alarm (inconvenience and lost productivity) outweighs the potential loss of life from a missed event.

After inadvertently posting this when I meant to save the draft, a friend commented that the “ideal FAR is actually non-zero if you want lead time.” This leads to the second reason an emphasis on detection increases the false alarm rate. Issuing a tornado warning seconds before the tornado hits is of limited utility. People in the warned area need time to move to safety. The article does point out that lead time has increased steadily for the past few decades. But the more lead time you have, the more likely it is that a warned storm won’t produce a tornado. Tornadoes are exceptional events.

There’s a balance between detection rate and lead time on one side and false alarm rate on the other. Like a seesaw, lowering one side raises the other (if you play with the signs on the numbers, that is). Prudent policy focuses first on detection and then on lead time, so the false alarm rate has to suffer. Improvements in technology and science will hopefully move the fulcrum such that we can lower the false alarm rate without reducing the lead time or probability of detection too much.

Severe weather outlooks on TV

At the end of October, the Storm Prediction Center changed the categories used in severe weather outlooks in order to more clearly communicate risk. These outlooks, like many NWS products, started as a way of communicating information to other meteorologists, emergency managers, et cetera. Though they weren’t designed with public consumption in mind, social science has helped to shape some of the changes. The Internet means that weather products are available to anyone who is looking for them.

What I’ve noticed since then is that not all of the local TV stations have gone along for the ride. A while ago, I asked one of the local meteorologists about this. His station discussed it internally and decided fewer categories made for less viewer confusion. I don’t have any reason to dispute that.

Severe weather outlook from the Storm Prediction Center.

Severe weather outlook from the Storm Prediction Center.

Severe weather outlook from NBC affiliate WXIN.

Severe weather outlook from NBC affiliate WXIN.

Severe weather outlook from CW affiliate WISH.

Severe weather outlook from CW affiliate WISH.

Severe weather outlook from Fox affiliate WXIN.

Severe weather outlook from Fox affiliate WXIN.

My main concern isn’t that the station doesn’t use the same categories as the SPC, but that different stations in the market use different categories. Of course, they should do what they think is in the best interests of their viewers. I’m certainly not suggesting there be mandatory unification. At the same time, I think stations having different risk categories is more confusing to the public than adding “marginal” and “enhanced” categories. Then again, TV weather seems to be one place that people have a specific and unwavering loyalty. Outside of weather weenies, I’m not sure there are too many people who would even notice differences between stations.

Communicating weather safety information

Weather is complicated and hyper-local. The general public often lacks a basic understanding of weather evolution and people are generally bad at risk assessment. These facts combined make it really hard to provide general safety advice. It’s made even harder by the fact that if you give bad advice, you may be responsible for injury or death.

What to do when you’re in a car and a tornado is coming is perhaps the epitome of this issue. The National Weather Service office in Kansas City recently posted a scenario to its Facebook page. I saw some dismay expressed about how many people said they’d keep driving in that scenario. But here’s the kicker, I think that’s (conditionally) the right answer.

In the scenario you’re smack in the middle of a six mile stretch of interstate highway that’s expected to be impacted by a tornado in 15 minutes and you’re at an exit. The overpass is clearly the wrong answer. A very good answer would be to go to one of the gas stations or restaurants in the picture and seek shelter there. A car is about the worst place to be in a tornado, so why did I say “keep driving” is the right answer?

Let’s assume you’re traveling at 60 miles per hour. In three minutes, you’ve reached the edge of the warned area. The tornado won’t reach that area for another 12 minutes. Of course, there’s likely some error in the projection, but even if the forward motion is twice what was stated, you still have a cushion of over three minutes. If, in addition, the danger area is twice as large as stated, you still have 30 seconds. That’s cutting it too close, but we’re being really conservative here.

Now let’s look at all of the underlying assumptions that I made. First, I assume that you can safely travel at normal speed the necessary distance. This means no traffic, accidents, construction zones, or debris from earlier storms. In some places, you’d probably have sufficient visibility to make that determination, but certainly not in all places, and not in the picture shown. Second, I assume that you are just passing through. If you’re 10 minutes from home, it might be tempting to try to get there, but that eats into a lot of your safety buffer. Third, I assume you’re traveling south or that the main part of the supercell (another assumption) does not contain heavy rain or large hail that would slow you down or cause damage/injury on its own.

What would I do in that situation? It would depend on my familiarity with the area, my awareness of the storm type and evolution, and (most importantly), my ability to process it all quickly enough.

What should you do in that situation? See above. The best default answer is to seek shelter in one of the buildings off the exit, but that’s not always the best answer.

NWS TV studios?

The Washington Post’s excellent Capital Weather Gang blog had a post earlier this week about a new addition to the Birmingham, AL National Weather Service office. It seems that WFO Birmingham has added a high-quality setup for video recording. This has some private sector broadcasters worried that the NWS will be horning in on their business.

Let’s be honest: the NWS is hardly a digital juggernaut. Even if they wanted to, I don’t see the NWS doing live streams during severe weather events any time soon. But as times change, so too must the function of the NWS. The old model where the NWS generated products, but the media did most of the dissemination to the public is not suited to the 21st century (despite the fact that many NWS products are still in a 20th century paradigm). The public can easily connect directly with the NWS, and the nobody is served by hamstringing the NWS due to tradition.

The NWS has a mission to protect life and property. In a multimedia world, higher-quality weather briefings, interviews, and educational material advances that mission. If a broadcast meteorologist finds this disconcerting, they should seriously evaluate how much value they’re adding for their audience.