Irma forecast performance review 

Posted: 10:03 am Wednesday, September 13th, 2017

By Kirk Mellish

From the beginning forecasters were showing the uncertainty factor on Irma was high, that’s what those “spaghetti” track charts are all about, and the various model output and their ensembles and consensus projections. They all depict the full RANGE of possibilities, the probabilities and how they are trending and changing.

Certainly in this blog from the git-go I pointed out how the pattern was more complex than normal and that was making the models, especially beyond 3 days less reliable than normal. I also pointed out how I wasn’t trusting specifics beyond 3 days because most of the models were vacillating east to west east to west east to west etc. The 3-day projections were performing very well though.

I’ve pointed out in the past that 20-50 mile variance is perfectly normal and average in weather projections, that is the typical and to be expected margin of error and is considered an accurate forecast because at the current state of the art that’s the best that can be done. The same margin of error applies to timing intensity and amounts of rain ice or snow.

This has been the case for decades but oddly the public, and even many news people and news anchors seem not to grasp that fact.

It’s the reason we give the odds or probability/probabilities. Forecasts are best estimates, not guarantees not promises.

For example, if a forecast calls for a high of 85 five days from now and it tops out at 80, that’s considered an “accurate forecast”, even though it WAS off the specific mark targeted. A forecast variance beyond 5 degrees would be wrong. A forecast hi or lo for today should be off by no more than 3 degrees for example.

If we forecast an inch or snow 0-3 is a normal and to be expected margin of error.

If there’s a 30% chance of precipitation there’s a 70% chance of none. 30+70=100.

Hopefully people get it now. This isn’t anything new. It’s been true since at least the early 1960s.


Screen Shot 2017-09-12 at 7.50.50 AM

Locally my wind and rain forecasts seemed to be under what other sources were showing. I think it worked out. I pointed out in Q&A with the talk show hosts there was no real way to zero in on exactly where the highest winds would end up, because it would be random and chaotic, the earliest indications that worst would generally be around I-20 south, but that places farther north could not be ruled out. That more or less is how it played out.

We begin with what we know about weather behavior by studying past weather history, known as climatology the experts being climatologists.

Irma was a “Cape Verde” Cape Cabo type storm at birth coming off of Africa as a “wave”. This told us this is the type storm that usually becomes very strong, often top category, but that more often than not they don’t hit the United States:

Screen Shot 2017-09-01 at 10.17.18 AM

But if they do, based on Irma’s later location still out to sea, history suggested from that distance an east coast threat if it became a rare case of not re-curving out to sea as is the norm:

Screen Shot 2017-09-13 at 8.16.24 AM

Screen Shot 2017-09-13 at 8.17.29 AM

It’s an early guide and a means of thinking about model outputs and if they seem to fit what’s been seen before or if they look goofy.

You can see that, in the end, IRMA (white path) based on where it came from and where it became a hurricane was an amazing historical outlier! Not quite a black swan event but close.

Screen Shot 2017-09-13 at 11.37.00 AM

That is a limitation of the application of climatology, by definition it is the opposite of extreme, and some weather events like Irma are extreme and deviate far from the mean.

The same can happen to snow, ice, rain and temperature forecasts short term or in long-range summer and winter outlooks. Weather patterns repeat but not exactly.

None-the-less, as the storm neared Florida and the question of where the eye would go was alive and well, climatology gave us a signal that the models projection of an East FL or GA/SC direct hit should be looked upon with some skepticism based on past history and where the storm was then:

Screen Shot 2017-09-13 at 8.45.26 AM

Climatology telegraphed a westward shift because only 5 hurricanes (all pre-1900) made landfall on the SE Georgia coast as a Cat 3 or higher. So this was something to consider when modeling at one pointed suggested just that.

Screen Shot 2017-09-13 at 9.11.20 AM

The GFS American model had a north and east track bias, and by the way the GFS “feeds” or “seeds” many of the other models. Meanwhile the Euro model was superior most but not all of the time, the GFS closed strong in the final 12 hours compared to the Euro.


Screen Shot 2017-09-11 at 1.54.36 PM

Meanwhile, as covered in past blogs, intensity forecasts for hurricanes and tropical storms are nowhere near as accurate or reliable as tracking projections. There is just too much of the fine detail in what causes and regulates the strength of storms that is unknown or poorly understood at this stage of the science, and the intensity of storms is very fickle and often changes suddenly with little or no warning.

In theory if a given high-resolution model has the track pretty much right then the intensity forecast should be also, but this often turns out not to be the case in the real world. And even more so for the global models which can do well with track but not strength.

The models can also give you a head-fake:

Screen Shot 2017-09-13 at 9.12.54 AM

Which is a critical reminder to public and forecasters alike, that any given model and all model output are just a snapshot in time. And only represent a VIRTUAL ATMOSPHERE, a representation of what MIGHT come to be, not the real atmosphere.

Yet from a long way out in distance and time, one of the best models and trend methods was, all along, hinting at a more west track risk:

Screen Shot 2017-09-13 at 9.17.04 AM

Screen Shot 2017-09-13 at 9.00.28 AM

All in all official forecasts were actually quite good as you can see, black squares are the actual storm eye path compared to lighter shaded forecast lines:

Screen Shot 2017-09-13 at 8.22.39 AM

Looks pretty darn close to me, but should I believe my own eyes or social media, gossip, and no-nothing critics?


Red dots are the forecast compared to white path reality.

As warned, while the 2-3 day was near perfect, the 5 day is much more off, the predicting that final north turn when and where toward FL was the most difficult:


5-day projection

Not to forget that for a week forecasters warned not to focus too much on where the center point would go because the storm impacts would include areas far from the center of the storm, even beyond the “Cone of uncertainty”, which represents where 2/3rds of the time the eye will occur within. The cone also representing the most recent 5 year measured average forecast error factor.

CRITICAL TO NOTE… that means the “cone” is NOT specific to that particular storm.

NHC cone of uncertainty

Even the ‘best’ models have error of course:


Screen Shot 2017-09-13 at 9.34.27 AM

Screen Shot 2017-09-13 at 9.52.10 AM

IN FACT, for those interested in such a concept, with ALL the official NHC forecasts (total of 45 starting way back on August 30th just off coast of Africa) the real track did fall within the forecast cone! Forecasters also consistently warned that significant impacts do occur outside the cone.

We don’t have to look too far into the past to find a high-stakes tropical forecast that turned on a dime over a few miles at the beginning of the forecast. Hurricane Joaquin briefly looked like it was going to blast into the East Coast. There was model consensus. Then it didn’t happen. Cases related to crippling megalopolis blizzards and rumors of such are all too common as well. We’ve seen good forecasts with last second changes that made all the difference in Sandy and just last season, Mathew. So this is nothing new.

In the end, IRMA was affected by some dry air entrainment, land interaction and wind shear which weakened it from a high end Cat 5 as it hit Cuba to a 3-4 as it hit Florida. That unforecast weakening may also have had an impact on the wobbles altering the final track it ended up taking. Perhaps if it doesn’t hit Cuba first a more worst case scenario happens. God only knows.

Screen Shot 2017-09-13 at 12.20.01 PM

And their last forecasts were even better, than the 5 day above.

I’d say by most metrics that was a forecasting success, as the minimal loss of life can attest compared to past history of more massive losses.

Screen Shot 2017-09-13 at 10.46.11 AM

Screen Shot 2017-09-13 at 3.36.01 PM

Irma wind in Georgia

Model glossary

Screen Shot 2017-09-14 at 5.23.20 AM

FOLLOW ME on Twitter@MellishMeterWSB