It used to be a regular habit of mine to check fivethirtyeight.com daily. Ever since I read Nate Silver’s wonderful book “The Signal and the Noise” I have been a wannabe statistics snob. My actually statistics knowledge might only extend to two courses in total and rudimentary sessions with R and STATA, but I am nonetheless constantly on the lookout for bad statistics and statements begging to be data driven.
The daily habit waned as fivethirtyeight’s success grew. On the one hand, I was pleased to see someone I thought brought real numeracy and insight to a media desperately in need of empiricism, but on the other, its evolution into a “real” news site pushed it towards some of the same shortcomings the innumerate media possess. The need to publish stories every day meant that the site’s writers has to apply Silver’s statistically-minded ethos to every more niche topics. What began with superb modeling and arguments around major events – primarily national elections – began to dig into states, then counties, until it seemed like every school district was being analyzed. This wasn’t a problem in and of itself, but the necessary expansion of confidence intervals and the verbal shrugs that the columnists had to resort to for protection (a la “we can’t draw conclusions from this but hey, here’s 2,000 words on it”) made the noise begin to outweigh the signal. To be sure, my utter disinterest in professional sports didn’t help, as ESPN’s ownership of the website meant an ever-increasing volume of related models and discussion. So noisy has the website become that I check it only ever few weeks.
But.
When I checked the website yesterday – quite possibly for the first time in 2019 – my eye caught on a story by the editor-in-chief himself. “When We Say 70 Percent, It Really Means 70 Percent.” Alarms went off in my head, and the voice of Philip Tetlock began whispering to me. Sure enough, Nate had written a beautiful review of fivethirtyeight’s forecasts, scoring them on calibration and discrimination to deeply impressive results. It reminded me of my participation in Tetlock’s superforecasting IARPA project when I was in college. Why Silver swept this under the rug as a mere “housekeeping” item, without even an image to give it extra real estate on the screen, is beyond me (although being a few days late to its publishing, perhaps it was front and center the day of). The fact that fivethirtyeight has an entire portion of their site tracking their predictions is not only a welcome act of transparency, but a laudable means of holding themselves accountable as well as educating the reader on how they should think about the forecasts of Silver and his team. Digging into the offered data, I noted with interest the S-like fit of political forecasts (they tend to underpredict losers and overpredict winners) but that their track record is strong. The sample size of athletic forecasts dwarfs those of elections and so tends to smooth out the overall sample in fivethirtyeight’s favor, but overall I found Silver’s post refreshing and indeed invigorating. Despite the daily coverage that does more to make fivethirtyeight appear to be one of the media herd rather than separate them from it, the rigorous attention to results and transparent self-examination will make me return to fivethirtyeight for the 2020 elections with great anticipation, and confidence that 70 percent means 70 percent.
smq.world 