Law of Averages & Law of Small Numbers

This is to go over the "Laws" of Averages and Small Numbers as sometimes these get conflated.

Both are not in reality laws; they are actually incorrect biases in thinking that occur so often that the word "Law" was applied as a joke.

Law of Averages

This law is about our tendency to think that things must balance out. So if you flip a fair coin 10x and you've seen 6 heads so far, there is a tendency to think that tails are more likely in the remaining 4 flips to 'balance things out'. In essence, tails are 'coming due'; there needs to be a correction so we can end up 50/50. This 'law' is false, hence not a law. Gary Smith states, "A coin has no control over how it lands...heads and tails are equally likely to appear no matter what happened on the last flip or the last 999 flips". Yet, this belief is so widespread, that another version of it is called "The Gambler's Fallacy" because of the belief of gamblers that luck will swing their way after a streak of bad luck.

Another example Smith writes: "A reader wrote to columnist Marilyn vos Savant saying that he had a lot of job interviews, but no offers. He hoped that the odds of an offer were increasing with every rejection."

Ellenberg writes about the Law of Averages as well in the chapter How much is that in Dead Americans? The Law of Averages not being true seems to be in "conflict with the Law of Large Numbers, which ought to be pushing" a 50-50 split (in the case of a fair coin being flipped). The two laws seem that they should go hand-in-hand. How can one be true and not the other??

There actually is no conflict because it's an illusion. If we flip a fair coin 10x, let's say we've flipped 10 heads in a row.

There are 2 thoughts that may arise:

(1) Something is wrong with the coin, i.e., it is weighted

(2) If the coin is fair, we must start to get tails to correct the imbalance we've observed thus far.

Let's assume the coin is fair, so we can disregard #1. Common sense says #2 is true; however what also is common sense is that "the coin can't remember what happened the first 10 times"! So how can the coin correct itself? Perhaps it's not the coin itself but some divine intervention, like God, or the Universe. Indeed, de Moivre who investigated this phenomenon did raise this.

The reality however is that coins indeed have no memory and all the future flips still have a 50/50 chance of coming up heads. Ellenberg continues, "The way the overall proportion settles down to 50% isn't that fate favors tails to compensate for the heads that have already landed; it's that those first ten flips become less and less important the more flips we make.... That's how the Law of Large Numbers works: not by balancing out what's already happened, but by diluting what's already happened with new data, until the past is so proportionally negligible that it can safely be forgotten."

So there is ONLY the Law of Large Numbers. Faulty reasoning devised the Law of Averages to justify making bad bets...

On a somewhat related topic, Sam L. Savage has a whole book called "The Flaw of Averages" that speaks to the danger of focusing only on central tendencies and ignoring variations. This kind of thinking is easy and has led to innumerable real-world mishaps and disasters. Posts to come on this later.

Law of Small Numbers

A similar "Law" to the "Law of Averages" is the "Law of Small Numbers". This is not a real law; it's a bias or mis-application of probability that is so prevalent that Kahneman and Tversky dubbed it a law. It's a play on the Law of Large Numbers and expresses the fallacy that many people believe that a small sample ought to resemble the population from which it is drawn. So even after a few draws from a fishbowl, it might be tempting to jump to conclusions about all the contents in the fishbowl.

John D. Cook has some good posts on this:

https://www.johndcook.com/blog/2008/01/25/example-of-the-law-of-small-numbers/

The law of large numbers is a mathematical theorem; the law of small numbers is an observation about human psychology...people underestimate the variability in small samples.

This under-estimation of variability in small samples is discussed in my previous blog post on brain cancer deaths (Ellenberg's book). Another example of this phenomenon is the belief in the "hot-hand". If a player makes 3 or 4 shots in a row, there is a belief that the player is 'hot' and observers attach a higher probability to the next few shots being made.