Observational Studies [and Self-Selection Bias]

 From "Standard Deviations, Flawed Assumptions, Tortured Data & Other ways to Lie with Statistics" by Gary Smith

Chapter 2, Garbage In, Gospel Out

The author states:

    Self-selection bias is pervasive in "observational data", where we collect data by observing what people do. Because those people chose to do what they are doing, their choices may reflect who they are.

Such mistakes can be avoided by conducting RCTs [Randomized Controlled Trials] in which subjects are split into groups randomly.

The chapter is chock-full of examples of these biases. It's so easy to read these examples, many of which are pervasive in journalism, such as the NYTimes and take it at face-value. We are constantly fooled.

One example in the chapter is when David Leonhardt says "Well, college graduates earn 54 percent more on average than college dropouts, so the degree certainly seems to have economic meaning." [This is such an oft-cited stat so it must be true!] However, this is self-selection bias at work. Smith writes, "Surely, students who choose to go to college and choose to work hard enough to graduate from college are systematically different from college dropouts." So we can't simply conclude and say that the degree is the key to earning more.

Other examples include: (1) surveillance causing people to withdraw from political participation, (2) drinking in college, (3) couch potatoes dying sooner than people who don't watch TV, (4) a survey of people who travel to France and do not think the French are unfriendly => this points to a larger problem that is present with customer-satisfaction surveys. It is very easy to manipulate and show results that favor the company or product using these surveys: e.g. 84% of business travelers from A to B preferred this airline to another airline; Yes, it's easy to show this when surveying the people who actually flew on this airline. You're basically polling your own customers who are likely to pat themselves on the back for making such a wise choice (who wants to look foolish by admitting to a bad choice?) But the implication of the advertisement is that : 84% of ALL business travelers from A to B prefer this airline (and so should you!) ; Very powerful stuff.