The recent presidential election provided quite a lot of fodder for those of us that have a fetish for quantitative data. I’m speaking, of course, of the staggering volume of election polling that took place. I have a healthy skepticism for the science (art?) of public opinion polling, but presidential elections seem to be providing ever increasing amounts of public opinion data on a single topic. And if there is anything that I do like, it’s large quantities of data!
Of course, it wasn’t long before we were informed that, really, in order to understand the implications of this glut of polling data, we needed some system for aggregating and combining the information. RealClearPolitics (RCP) was an early leader here. The other heavy hitters are Pollster.com, 538, the lesser known Sam Wang and quite a few others who I will omit for space.
I became a fan very early on of Mark Blumenthal, blogging as MysteryPollster, and then went on to found Pollster.com with political scientist Charles Franklin. In particular, I appreciate a style that emphasizes a restrained approach to data analysis. They are interested in clearly and cleanly displaying data. Nothing more, nothing less. (There’s lots of other commentary on Pollster.com, but this is the heart of what they do.) Most of the other sites, RCP, 538, etc. want to be oracles; Pollster.com wants to be a resource for public information.
In any case, one aspect of (most) sites like these was that they provided aggregated estimates of the level of support for Obama and McCain, not just nationally, but in each state as well. (Sam Wang is an exception here, as are some of the lesser known sites that take a fully Bayesian approach.) Inevitably, the question arises, who’s model did the best? This has been looked at (see here, here, and here ) by others, basically concluding that there wasn’t much of a difference.
The reason this is important is because the complexity of the approaches taken diverged wildly. I won’t go into the gory details, but RCP simply took (unweighted) averages of the most recent polls. Pollster.com fit non-parametric regression curves to all the polling data and 538….well, yeah. Let’s just say that Nate Silver built a lot of machinery up to tackle this problem.
I was curious about how much these three methodologies diverged in accuracy as well (RCP, 538 and Pollster; Chris Bowers didn’t consider RCP, though he did include Gov and Sen races, which I did not). I won’t clog this post with zillions of graphs, but it suffices to say that there were essentially no differences in the accuracy of the predictions made by any of these sites, but that they all did much better than had we simply picked one poll near the election and used that as our prediction. (If this post generates a lot of discussion, I might be convinced to go back and put up some graphs and extended analysis. At the moment, I’m too lazy, so you’ll have to make do with me summarizing my conclusions.)
What can we learn from all this? First, we should always be aware of the diminishing returns of increasing model complexity. Second, we should be frightened by how easily Nate Silver garnered a reputation as being (essentially) infallible by
- Evincing an air of certitude and
- Using methods far more complex than are necessary
Is it really surprising that his background is that of a quant, and not really that of a statistician, as is commonly believed? (Technically, Wiki tells me that his education was in economics, and that he worked as a financial analyst.) This isn’t meant as a harsh knock on Nate Silver, despite appearances. I mean, his model was really accurate. It just wasn’t any better than RCP or Pollster.com.
The rapid rise of Nate Silver worries me, because it sometimes begins to take on a cult-like atmosphere. Nate Silver doesn’t bother me; if I had as much free time as him, I’d probably spend a lot of it doing something similarly too-complex (and fun!). (Indeed, I read the commentary on his site all the time, although Sean Quinn brought way more to that site than Nate Silver ever did, in my opinion.) What frightens me is how easily many people seem to be convinced that “more complicated” is automatically better. Particularly given how well the quants did with their super complicated models for credit scoring in recent years.
PS – I’m being glib about the quants and credit scoring; all I mean is that “we” (someone!) royally screwed that up. I know it’s complicated and don’t mean to single out any particular group for blame.