Survivorship bias explained: the WWII bomber and your habits

Find what is uncommon. That has been my mantra for a long time, and it is probably why survivorship bias caught my attention the first time I came across it.

I was scrolling an old social network where people asked questions and others answered them, and a picture of an airplane stopped me. It was the cover image of a post. The plane was covered in dots, scattered across the wings and the fuselage, and clustered nowhere in particular. I did not understand what I was looking at, so I read the post.

The dots were bullet holes.

During the second world war, a group of mathematicians at Columbia University were asked to help the American military reduce bomber losses. The obvious approach was to look at the planes coming back, map where they had been hit, and add armour there. That is what the data showed. The wings and the fuselage took the damage.

Abraham Wald, a Hungarian mathematician working with that group, said the opposite. The armour belonged where the returning planes had no holes at all. Around the engines. Around the cockpit.

The reason is simple once you see it, and almost impossible to see before. Every plane they were measuring had come back. The holes on those planes marked the places a bomber could be hit and still fly home. The planes hit in the engine were not in the sample, because they were at the bottom of the channel.

The data was accurate. It just described a population that had already been filtered.

Where it took me

That post sent me to Daniel Kahneman's Thinking, Fast and Slow, which I read slowly and found uncomfortable in the way good books are. What stayed with me was the realisation that most biases are failures of sampling rather than of intelligence. You reason correctly about the wrong set.

Survivorship bias is the cleanest example. Nobody in that room was being careless. They had real data, carefully collected, and they were about to armour exactly the wrong parts of the plane.

Why I care about data

I work with data for a living. In my research it is energy systems and environmental impacts, and in my apps it is much smaller things. Either way the same thing is true: the model is only as good as the sample, and the sample is almost never what you assume.

But there is a second reason, and it is more personal.

Statistics, machine learning, and now AI are all in the business of finding patterns in numbers. That is the professional version. The everyday version is much smaller and, for me, more useful. When you keep a record of something you do, you eventually see a pattern in it. Usually a slow one.

That slow pattern is what keeps me going. Motivation is nothing if you can't see the results of your effort. What I trust is being able to look back at a year and see for myself that a line moved.

The record I keep

This is part of why we built MyGrowth. I wanted to log habits and then look at the statistics, and I wanted to be able to go back years rather than weeks. I still open it to compare a month against the same month two years ago, which is a strange thing to enjoy, and I do enjoy it.

Every article about someone else's routine is a returning plane. It is a real record of something that worked, and it tells you nothing about the ones that did not come back. Your own log is the only record you will ever have that contains your failures alongside your successes.

It is a small sample. But it is not a filtered one, and that turns out to matter more.