The Hanover Collaboration

I recently finished reading The Black Swan by Nassim Taleb (thanks Dad¹!). I’d certainly heard a lot about Taleb, mostly in regards to our recent national financial problems. I have even seen some short interviews he’s done, and he has generally struck me as a smart, albeit overly animated, guy.

I was very disappointed to discover that his book (I haven’t read anything else of his) is really quite bad. I’d call The Black Swan an interesting topic in search of a well researched book. Andrew Gelman, a statistician I have a lot of respect for, was fairly charitable compared with the reviews he links to. That said, after reading the book, I’m much more in agreement with Lindley and Robert.

First I should say what I liked about the book: the observation that “standard” statistical models are increasingly inadequate in the face of modern data is important and timely. This doesn’t mean that the “old” Gaussian methods are useless, just that many of the modern, sexy problems that people want to work on don’t fit that paradigm very well. This is related to my earlier musings. Statistics, as an academic field, is partly to blame for the broadside attack it gets from people like Taleb. The field has, at times, become too enamored with what works mathematically, rather than with models that are useful in the world.

This has led to a growing divide between (1) Statisticians and (2) People Who Do Statistics. The problem is that (1) is contained in (2) but (2) is growing much faster than (1). So the field of statistics has acquired a severe public relations problem. Mainly this comes in the form of not understanding and communicating the level of uncertainty in a model (including caveats related to model assumptions and model shift). Taleb is correct that this popular misconception is a major problem.

However, I think that Taleb gets the cause of this problem exactly backward. He thinks the problem is too many narrow minded, formally trained Statisticians who think the whole world obeys their neat little Gaussian distributions. My experience with Statisticians is exactly the opposite. They are, in my experience, exceedingly cautious. Every time I have observed (or participated) in a statistical consulting exercise, the client spends a lot of energy trying to force the statistician to give them The Answer. The statistician spends a lot of energy trying to convince the client The Answer doesn’t really exist, and instead we have to settle with A Likely Possibility (?) and lots of caveats in the footnotes. (Frequently, I’ve had to explain to people that we can’t really conclude anything from the data. When I say that to Statisticians they nod and shrug. When I say that to the owners of the data, they often freak the fuck out. So how is this the fault of the Statistician, again?)

Most of the “reckless” statistics that I’ve seen has been perpetrated by people who are doing statistics, but aren’t really Statisticians, in that they do not have a graduate degree in statistics. Instead, they usually come from another field that involves a lot of math. Sometimes they have taken graduate level mathematics courses, sometimes not. I’m not claiming that this is a statistical knowledge issue. Instead, I think there is a cultural phenomenon here, where Statisticians, as a group, have developed a culture of cautious data analysis, but have not become very good at spreading this trait except through PhD advising.

I also like descriptions of the “impact of the highly improbably”, to steal from his subtitle. It is an interesting way to think about the world around us. So many of those stories are interesting and thought provoking. And there are certainly lots of stories of improbably shit happening in this book.

Anyway, on to my actual complaints. I could list a lot, but I’m going to focus on only a few.

Many of the modeling/prediction challenges that Taleb raises are indeed interesting. It would be really cool if someone wrote a book about all the research (past and present) that’s been done to attack these issues. Instead, we get Taleb waving his arms wildly in the air as though he’s the first person to realize that, say, the Gaussian distribution isn’t always a good idea, or that sometimes the underlying process you’re modeling can change, rendering your model useless. These are not, as Taleb would have us believe, new and revolutionary ideas. They are old ones that have received lots of work and attention by very smart people. Taleb either doesn’t know about this history, or doesn’t care. It’s true that we arguably aren’t very far along in solving these problems and I’d love to read a book where someone went and interviewed a lot of people who are actually tackling these issues. Taleb apparently couldn’t be bothered.

Taleb attacks the entire notion of inductive reasoning. Sadly, he completely misses the point. Look, suppose you’re busy predicting the future based on your past experience and things are going swimmingly, i.e. you’re getting things basically right. Now suppose either (i) an aspect of the world is suddenly revealed to you or (ii) an aspect of the world actually changes. Both instances will likely cause your model to be (very very) badly wrong.

But in what sane world is this a criticism of inductive reasoning? The fact that you can build a model that does really well for a while, and then the world changes and your model sucks doesn’t reveal any magical flaws in inductive reasoning or modeling in general. That’s just life. Again, Taleb is right to point out that we should do a better job of educating people of these issues, so at least he’s got that going for him.

I won’t make this post any longer with detailed descriptions of how Taleb displays his utter ignorance of statistical theory and practice. It suffices to say that, if you feel it necessary to read The Black Swan, you should not bother reading Chapters 15-19. If you know a lot of statistics you will feel unclean. If you don’t, you will be dumber for having read them.

Finally, moving away from Taleb’s actually arguments, I have to briefly comment on his writing and tone. The former is, in my opinion, fairly bad and the latter is downright egotistical.

I felt that his writing was scattered and unfocused, which made the book feel somewhat incoherent. And his tone is just insufferable. I mean, enough with the French bashing already. (Ha ha. French people are smelly. Look at me, aren’t I funny?!)

¹ I hope you aren’t upset that I’m writing a bad review of my birthday present. This is not equivalent to saying “I didn’t like my birthday present”. I liked it, as a gift. I wanted to read it and am glad you gave it to me, for no other reason that it has given us something interesting to talk about.