All data scientists, aspirant or experienced, should read this book. Indeed, who could write more definitely on the topic than the guy who forecasted successfully so many elections and baseball games? Surely, the book lives to its promises: very well researched (75 pages of references – in my Kindle edition – supporting the 453 pages main story), erudite (you will learn about baseball, poker, chess, but also climate science, economics, earthquakes,…) while at the same time pretty clear and concise, clever and entertaining.
So, why do these predictions fail, or not? My favourite and subjective intakes:
– Have a model, i.e. have a preconceived idea of how the outcome you want to forecast is produced. Increased computing power made possible the huge progresses of weather forecasting over the years (chapter 4), but wouldn’t have delivered as much without precise thermo dynamical models. Purely statistical weather weekly forecasts are just not as accurate as what comes out of models based on thermodynamics (figure 4-6). Forecasting earthquakes with radon levels and alignment of Earth with Venus has failed miserably, because there is no reliable theory linking these events with earthquakes (chapter 5). There is no holy grail, though. Structural models – based on specific theoretical elements – are, in lots of fields, too crude to deliver actionable forecasts: chapter 7 contains interesting insights as to why modelling infection diseases can fail, by being too blunt. The forecaster will most of the time have to resort to reduced models – i.e., purely statistical ones. But anyway, these reduced models should be firmly grounded in some theory linking the outcome with explanatory variables.
– Mix the quantitative with the qualitative, by confronting your quantitative models with what experts have to say. M. Silver is wonderfully expert at rendering the details of how scouts and stats interacted to improve forecasting in baseball (chapter 2), or how eyesight will improve computer models of weather forecasting (chapter 4), or how watching closely 5 basketball games at the same time can earn you millions (chapter 8). The apprentice forecaster should strive to convert into quantitative information (the only one a model can use) the minutiae of qualitative expertise.
– Don’t overfit. In modelling terms: don’t put too many variables in the model. That one is so important. Now that computer power is not an issue anymore and that all sorts of statistical or computing techniques are readily available, it is easy for the practitioner to indulge in overfitting. As the author nicely put is, this is mistaking noise for a signal (chapter 5).
– Communicate uncertainty. The main job of a forecaster is not to produce a forecast. This can be done with dices, crystal balls or more sophisticated techniques. It is to produce the confidence interval associated with his forecast. Not communicating this confidence interval can lead to very earthly and damaging consequences. Nate Silver argues (chapter 6) that the flooding of a city in North Dakota in 1997 could have been prevented, had the US Weather Service communicated about the uncertainty of its forecasts at the time, which it actually does now. And in one of the best pages of his book, Mr Silver shows how the failure[1] of climate scientists to communicate clearly about the accuracy of their forecasts has undermined climate science and damaged the fight against global warming. Table 12-12, which applies the Bayes’s theorem to beliefs on global warming, is a most wonderful example of applied statistics.
This enjoyable reading has one small flaw. While Nate Silver is right to emphasize that Bayesian statistics does not have the place it deserves in curriculums and applied statistics, his plaidoyer on that specific issue is marred by caricature and, I would argue, mistakes. Classical (or frequentist) statistic would, he writes, only focus on sampling error, and never on bias. It would presume normal distribution of the data. Frequentists would be sealed from the real world and not consider the underlying context of their hypotheses (chapter 8, pages 253 and 254 in my Kindle edition). What a pity to suddenly assert this in an otherwise really subtle book. Just to name one, Heckman (2005) is an excellent example of how structural models can be estimated with frequentist statistics. And again Heckman (1979) could be a good account as to why some polls were wrong in the 2008 Democratic primary. And a plain frequentist one.
References:
Heckman, J. (2005): “The scientific model of causality,” Sociological Methodology, 35, 1–97.
Heckman, James J. (1979): ”Sample Selection Bias as a Specification Error”, Econometrica 47, 153-161
[1]Admittedly under the pressure of non-scientific groups
Français
English