Thursday, July 9, 2009

MWN forecast accuracy vs. Memphis TV stations

I've made a fairly bold claim regarding the accuracy of the MemphisWeather.net forecast for some time now... that it is "the most accurate publicly-available forecast for Memphis and the Mid-South." I've been told a number of times that this is true by fans of my site, so I have some degree of confidence in my work. Empirically, I back that claim up each month with accuracy statistics on every temperature forecast I prepare, comparing it to several computer models that are but one input to my forecast, as well as to the NWS forecasts. I also keep accuracy statistics on my dewpoint forecasts, though since I don't show those in the forecasts themselves and most people know very little about dewpoint (read the linked post for more), I also don't publicize their accuracy (suffice it to say, they are good too).

I don't keep accuracy stats on anything else (precip, cloud cover, wind, etc.) for a couple of reasons: 1) it's too hard to assign a "value" to the accuracy of those conditions without introducing bias, and 2) I figure if I don't forecast rain (or do) and it does the opposite, or I over- or under-forecast cloud cover, it's typically reflected in the accuracy of my temperature forecasts. That's good enough for me.

But my claim is that I produce the most accurate "publicly-available" (meaning free for public consumption) forecast for Memphis. I can't claim this without comparing my forecasts to what most people use for their weather information - local TV stations. So, in May, I did just that. With help from a budding young meteorologist who did the leg work for me (thanks Josh!), I set about to prove that my temperature forecasts are at least as good as the TV stations in the Memphis market. Here are the results, which show the average temperature error (in degrees) out to 60 hours for each station based on the forecasts posted on the web as of late afternoon/evening (the ones they show on TV on the 5pm or 6pm newscasts):

WREG (Channel 3): 2.19 degrees
WMC (Channel 5): 2.23 degrees
FOX (Channel 13): 3.01 degrees
CW/UPN (Ch. 24/30): 2.10 degrees
MemphisWeather.net: 1.98 degrees

Not bad huh? They're all very comparable for the most part. The surprising thing to me was that channels 24/30 (which use the same weather department and forecasts) outperformed the "Big 3" in the Memphis market (at least for this subset of time)! Of course, I wouldn't bother posting this if I couldn't back up my claim (well, maybe I would). My hypothesis was proven and, for May (which also happens to be sweeps month), the MemphisWeather.net forecast was anywhere from 6%-52% more accurate than the local TV stations.

Understand that I am not out to "toot my own horn" with this posting, I simply believe that when one person is responsible for the forecast day in and day out (thus establishing consistency and having continuous knowledge of the current state of the atmosphere), when that person gets to concentrate on a small region all of the time, and has lived in and forecasted the weather for that area for well over a decade, the odds are pretty good that what you get is a superior product. I appreciate all of you who have come to trust MemphisWeather.net and use it on a daily basis for your weather information.

Last note, to set the record straight: I do not see this as a competition with the NWS or the local TV stations - they obviously have a much larger following than this site and all of us do a great job of getting the word out to you when you need it most. Without the NWS, none of this is even possible - they do a fantastic job of living out their mission. Without the TV stations, many more people die each year as a result of the weather. My goal is to work collaboratively with the weather community as a whole to keep you informed and safe. Thanks for your support!

----
Become a fan of MemphisWeather.net on Facebook and follow MWN on Twitter!

2 comments:

Paul Yeager said...

Great job, Erik! It's hard to argue with statistics, and I agree that other parts of the forecast are often too subjective to be quantified with any accuracy.

I'm sure it's too much work, but I'd bet that you'd do even better against some of the Internet weather sites(weather.com, accuweather.com, weatherunderground.com, simpleweather.com, etc.) since Internet sites often just use computer-generated numbers, not forecaster-generated numbers.

MWN/Erik said...

Thanks Paul. Many of those internet weather sites that use automated forecasting are using one of two sources - a particular model or the NWS digital forecasts. Since my monthly model comparisons compare against both of those outlets (except the NWS LAMP forecasts and the RUC), I figure I'm including many other websites' accuracy ratings by association! I'll look into it though.

Thanks for the good word.
--Erik, MemphisWeather.net