Weather forecast accuracy is improving

ForecastWatch recently issued a report on the accuracy of weather forecasts from 2010 through June 2016 (PDF here). While many readers will focus on who was more accurate, what stood out to me was how forecast accuracy has improved. Meteorologists have long “enjoyed” a reputation for inaccuracy — often more due to perception than fact. But those in the know are aware that skill is increasing.

Forecast accuracy over time

ForecastWatch’s U.S. analysis shows a clear — if small — improvement in the average accuracy since 2010.

Average U.S. forecast accuracy from 2010 – June 2016.

The chart above shows the average for all of the forecast sources Forecast Watch analyzed. To be frank, World Weather Online is a stinker, and brings the averages down by a considerable margin. Examining the best and worst forecast shows more interesting results.

Best and worst U.S. forecast accuracy from 2010 – June 2016.

Forecasts get less skillful over time, thanks to subtle inaccuracies in the initial conditions (see also: butterfly effect). That’s obvious in both graphs. What this second chart shows is that the best 6-9 forecast is now roughly as skillful as the worst 3-5 forecast was in 2010. And the best 3-5 day forecast is in the middle of the 1-3 day forecast skill from just a few years ago.

Forecasts are definitely improving. This is due in part to better modeling — both more powerful computers and also the ability to ingest more data. Research and improved tooling helps as well.

Forecasts still bust, of course, and forecasters hate bad forecasts as much as the public does. As I write this, forecasters in North Carolina are dealing with an inaccurate snow forecast (winter weather forecasting sucks due to reasons I explained in a previous post). Missed forecasts can cost money and lives, so it’s good to see a trend of improvement.

Forecast accuracy in your city

The ForecastWatch report breaks down by broad regions: United States, Europe, and Asia/Pacific. But weather is variable on much smaller scales. The ForecastAdvisor tool compares forecasts at the local level giving you the ability to see who does the best for your city. As of early January 2017, AccuWeather had the most accurate forecasts for Lafayette, Indiana, but they only place fourth when considering the past year.

Measuring HVAC efficiency with degree days

I finally turned my furnace on for the winter this weekend. As I thought about all the money I’ve saved keeping it off for several extra weeks, I was reminded of a discussion I had with a friend earlier this year. He was comparing his electricity bill to the previous year to see how much his new air conditioning unit was saving. Of course, there are a lot of ways to arrive at the wrong answer.

Let’s look at some of the things that affect the dollar amount of a utility bill:

  • The price per unit. Natural gas, propane, and electricity all have costs that vary based on a variety of factors: fuel cost, regulatory requirements, taxes, etc.
  • Weather. Obviously extreme weather will affect how much is consumed, which then affects the final bill.
  • Billing period. At least for my bills, the length of the billing period can vary by one or two days. This is probably due to working days (particularly holidays), and the fact that months are not of equal length.
  • Usage patterns. If you’re out of town for a week in October of this year, but not of last year, your overall usage will probably be down this year. Similarly, if you add, remove, or replace appliances that can have an effect separate from your HVAC system. Or if you switch from working at an office to working from home, you’ll probably see an increase in utility usage.

So how can you see if your new furnace, air conditioner, fancy thermostat, or whatever has made a difference? It’s going to be hard to account for some of the factors above (particularly usage patterns), but the best way is to look at your usage per degree day.

What’s a degree day? It’s a measure of the amount of heating or cooling required. The simplest measure of a heating degree day is to subtract the daily average temperature from 65. For example, if the average temperature for a given day is 55, then you would record 10 heating degree days. Similarly, to calculate cooling degree days, you would subtract 65 from the average temperature. So on a day with a 75 degree average temperature, you would record 10 cooling degree days. The lowest measure of degree days is 0; you would not record negative values. (Note that “average” here means the mathematical average of the high and low, not the climatological normal. If you have more detailed temperature data, you can calculate on an hourly or similar basis to get a more accurate value.)

Utility companies will sometimes include the heating or cooling degree days for the billing period in your bill. If not, you can get values online. Divide your usage for the month (e.g. the kiloWatt hours) by the heating or cooling degree days to get a value you can compare to other bills. If your usage patterns are relatively the same, you’ll now be able to compare year-to-year.

Hurricanes doing laps

As I write this Thursday night, Hurricane Matthew is approaching the east coast of Florida. By the time this post goes live, Matthew will have just made landfall (or made its closet approach to the Florida coast). Hundreds have been killed in Haiti, according to officials there, and I haven’t heard of any updates from Cuba or the Bahamas, both of which were hit fairly hard.

But even as the immediate concerns for Florida, Georgia, and the Carolinas are the primary focus, there’s another though in the mind of meteorologists: a second round.

National Hurricane Center forecast graphic for Hurricane Matthew.

National Hurricane Center forecast graphic for Hurricane Matthew.

If the forecast holds and Matthew loops back around to strike the Bahamas and Florida again, it could exacerbate already devastating damage. It is expected to weaken, so the threat will be more for rain than wind, but with existing widespread damage, it could be significant.

Such an event is not unprecedented, but it is rare. Eduoard and Kyle, both in 2002, did loops over open water, but did not strike the same area twice. Hurricane Esther struck Cape Cod twice in 1961.

From what I’ve been able to find, it looks like 1994’s Hurricane Gordon is the closest analog, but it’s not great. Gordon snaked through the Florida Straights and moved onshore near Fort Myers. The second landfall was near the location of the “seafall” on the Atlantic coast. Gordon’s peak strength was a low-end category 1, not the category 3 or 4 that Matthew will be at landfall (or closest approach).

Matthew is already making its place in history as the strongest storm on record to impact the northeastern Florida coast. Next week, we’ll find out how much gets tacked on.

The Local Storm Report product still has value

Several tornadoes hit central Indiana last month. During the event, a tornado warning was issued for Indianapolis. I saw several local media people tweeting that police had reported a tornado but no Local Storm Report (LSR) had been issued by the National Weather Service. I thought that was rather odd, and mentioned this incongruity in a tweet. It didn’t seem right to me that a tornado could be reported in the 14th-largest city in the United States but have no LSR issued.

Several people replied to tell me that the police report was included in the text of the warning. I did not take kindly to that. While including such information in a warning is great, that’s not what I was after. I specifically wanted an LSR. I was asked if it’s still a relevant product,  so here’s this post.

The Local Storm Report is still a distinctly useful product because it has a defined format. While most people do not consume LSRs directly, the rigid format allows it to be used in a variety of useful ways. For example, a media outlet can parse the incoming LSRs and use the coordinates and type to make a map for TV or web viewing. This can help the audience better understand the type and location of a threat.

Additionally, they’re helpful for downstream experts (other forecast offices, emergency managers, etc.) to know what a storm has produced. I often watch the LSRs issued by the Lincoln, IL or Chicago offices when severe weather is approaching my area to see the ground truth to go along with the warning. Knowing that a storm has (or hasn’t)  produced what the warning advertised can be very helpful in formulating a response to an approaching weather threat.

Apart from the warnings, timely and frequent LSR issuance is one of the most valuable functions of a National Weather Service office during a severe weather event.

But what about social media?

I’m glad you asked. Someone suggested that social media is a better avenue for communicating storm reports, in part because a picture is worth a thousand words. I agree to a point. Seeing a picture of the tornado heading for you is more powerful than words or a radar image. In that sense, social media is better.

But Facebook is awful for real-time information. Twitter is limited in the amount of detail you can include and has a relatively small audience. Plus, social media is hard to automatically parse to reuse the data, unless every forecaster tweets in a prescribed format.

The ideal scenario would be to tie social media into the process of issuing LSRs. As an LSR is generated, the forecaster would have the option of posting the information to the office’s social media accounts (perhaps with a link to the LSR) . If we’re granting wishes, the posting process would also allow for the inclusion of external images.

Until that day comes, I’m going to keep looking to LSRs for verification during severe weather events. And I’ll keep being disappointed when they’re not issued. 

New entry in the Forecast Discussion Hall of Fame

Most entries in the Forecast Discussion Hall of Fame earn the honor with a consistent excellency throughout the entire work. As Hurricane Hermine approached the Florida coast earlier this week, forecasters at the Tallahassee forecast office were focused on the effects of that storm. The fire weather discussion contained a single word, and that’s what landed it as the most recent entry.

It’s worth noting, too, that several subsequent updates to the Area Forecast Discussion left the fire weather section unchanged. I’m glad to see Southern Region Headquarters did not immediately rain bureaucratic hell upon the office. I’m not sure that would be the case in other regions.

Long range heat wave forecasts

What if I could tell you today that we’d have a major heat wave on June 11? A recently-published study could make that possible. Researchers analyzing heat waves over several decades have found a signal that improves the reliability of long-range heat forecasts. Will it be useful for forecasting specific days? I have my doubts, but we’ll see. They apparently plan to use it quasi-operationally this summer.

The more likely scenario is that it will help improve probabilistic forecasts on a multi-day-to-month scale. For example, the one month and three month outlooks issued by the Climate Prediction Center. There’s clear value in knowing departure from normal conditions over the course of the next few months, particularly for agricultural concerns but also for civic planning. I’m not sure I see much value in knowing now that June 11 will be oppressively hot as opposed to June 4.

While this study got a fair amount of coverage in the weather press, I don’t see that it will have much of an impact to the general public for a while. In fact, if it results in gradual improvement to long-range probabilistic forecasts, the public will probably never notice the impact, even if it turns out to be substantial over the course of several years.

To warn or not to warn?

The decision to issue a tornado warning is a difficult one for meteorologists. A timely and accurate warning can save lives, but a false alarm contributes to an already-too-high indifference among the public.

On March 31, thunderstorms moved through Indiana in the mid-afternoon. The Storm Prediction Center had a slight risk for the area, so I had been keeping an eye on them. One cell had shown rotation since it had been in east-central Illinois. After a while, it started to fall apart. Then around 4:40 PM, I happened to glance back over at the radar, and I saw this:

KIND reflectivity and velocity at 4:42 PM Eastern on March 31, 2016.

KIND reflectivity and velocity at 4:40 PM Eastern on March 31, 2016.

Oh yeah, there’s something going on there. I was talking to my friend Kevin and we were wondering why there was no warning. Even if the forecaster didn’t think a tornado warning was justified, a severe thunderstorm warning would have been a good hedge. As it turns out, the storm produced a brief EF1 tornado about a mile east of my house.

I don’t know why no warning was issued. But here’s a different take: should a warning have been issued? No one was injured and the damage that was done couldn’t have been prevented with a 10-15 minute lead time. Should this count as a false negative?

Yeah, probably. Although no one was injured, a small difference in the location could have easily changed that. But it makes me wonder if warning for every tornado is a reasonable goal.

Should meteorologists share velocity images with the public?

This week is Severe Weather Preparedness Week in Indiana. For more information, see the NWS Indianapolis website.

If you’re not sure what I mean when I say “velocity images”, you can learn about how weather radar observes winds here or here.

Meteorologist Joe Moore asked an interesting question on Twitter:

The overwhelming majority of respondents say sharing radar velocity images with the public is okay, at least in certain circumstances. My vote was for limited use.

On the one hand, informing and educating the public is a primary duty of meteorologists in both the public and private sectors. A velocity image can sometimes help make the location of a tornadic threat more clear, particularly in squall line or otherwise messy setups. “This is the bad part and it’s coming right for you” can be a powerful motivator to take shelter. In addition to immediate information, the images can be used to provide more general education. “This is how the radar works” isn’t of immediate value, but it can lead to a better understanding of meteorology in general.

On the other hand, meteorologists tend to be science geeks, which can sometimes lead to giving too many facts with no informative value. In severe weather situations, clear and timely information is paramount. If a television viewer gets confused by the velocity image and focuses on that to the exclusion of the “hey, you should really get to the basement now!” part of the message, sharing the image ends up being harmful.

As Joe said in a reply, “Can you explain it in a tweet? And still be timely/relevant?” Context and information content is key.

Storm chasers as first responders

This week is Severe Weather Preparedness Week in Indiana. For more information, see the NWS Indianapolis website.

Storm spotters have a single purpose: to provide “ground truth” to officials (or media) in order to better protect life and property. Storm chasers have purposes as varied as their motivations. There are no universal rules for chasers, but one comes close: if you encounter injured people, you stop and render assistance.

Knowing that a successful tornado intercept means an increased risk of encountering storm-related injuries, some chasers maintain CPR and first aid certifications. Some carry hefty first aid kits. A smaller number will keep tools in their vehicles. (When I was chasing regularly, I thought about keeping a gas-powered chainsaw in the trunk for such an occasion, but never followed through.)

While preparedness is good, I’m concerned that a small minority will view themselves as first responders. That is an inappropriate role for someone to assume, unless they also happen to be a trained first responder when not chasing. I’ve seen this in others and I’ve seen it in myself. There’s a certain romantic appeal to see yourself as a gallant hero, watching the sky and riding in to save strangers who will never know your name.

That’s a dangerous delusion to buy into. Storm chaser and wizened sage Ams Magliocco wandered aloud about the decisions that sometimes get made by chasers when they come across storm damage:

Amos’s tweet appears to be in response to the attack on Jeff Piotrowski, who was assaulted as he attempted to render assistance (while he happened to be streaming on Periscope). Jeff is an experienced and well-respected chaser, but the decision to live stream attempts to help in the immediate aftermath of a tornado strikes me as attention-seeking and not in the best interests of those who he would assist.

I don’t mean this as a criticism of Jeff specifically. I don’t know him, but I’ve known of him for years and I have a high regard for him. His incident just serves as a dramatic reminder of the possible consequences of chasers as self-appointed first responders.

Too many NWS products?

I’ve written before that the National Weather Service is stuck in a paradigm where the general public isn’t assumed to directly consume most products. NWS text products are largely unchanged from decades ago when most public consumption was filtered through media outlets or emergency management officials. As a result, products are more focused on being technically correct instead of usefully correct.

A prime example is how the National Hurricane Center decided not to issue hurricane warnings as “Superstorm” Sandy made landfall. While technically correct, the decision led to confusion and caused some to minimize the threat.

I suggested on Twitter that people really only want to know three things:

  1. How badly will my stuff be wrecked?
  2. When will be stuff be wrecked?
  3. What should I do about it?

To the extent that the specific threat matters, it only matters as it affects the answers to those three questions. If your roof disappears because of a wind gust instead of a tornado, is it any less gone? Of course, the distinction matters for scientific research purposes, but does the general public care?

When I saw an infographic that Ashley Atney put together, it hit me: the NWS simply has too many warning products. Maps like this drive the point home:

Watch, Warning, Advisory map of Arizona from January 31, 2016. Image capture via Rob Dale (@therobdale).

Watch, Warning, Advisory map of Arizona from January 31, 2016. Image capture via Rob Dale.

This may be heresy, but what if we condensed products? Winter storm, lake effect snow, and blizzard products could all be combined into one, with the product itself describing the details. After all, merely saying “winter storm” versus “blizzard” doesn’t really communicate much anyway. Further heresy would be to combine severe thunderstorm and tornado warnings.

I’m not saying that these (and other condensations) are necessarily the right way to go. There’s a lot of research that would need to be done first in order to make sure that the net effect on the public is positive. But the current system is in need of improvement.