Sometimes financial data just won’t fly
An anecdote from World War II tells us a lot about why most of what passes for investment data is wrong.
Here’s a thumbnail of the story, which I found in number theorist Jordan Ellenberg’s wonderful book “How Not to Be Wrong.” During the war, a group of mathematicians — experts in statistics — was tasked with finding ways to keep more American airplanes in the air. They wanted to reduce the number that didn’t return from missions.
Military officers had gathered and studied bullet holes in the aircraft that returned from missions. One early thought was that the planes should have more armor where they had been hit the most — fuselage, fuel system, the rest of the plane — but not on the engines, which had the smallest number of bullet holes per square foot.
Abraham Wald, a leading mathematician, disagreed. Working with the SRG (Statistics Research Group) in Manhattan, he asked an odd question: Where were the missing bullet holes — the ones that would be all over the engine if bullets were equally distributed?
They were on the missing planes, the ones that had been shot down. So the vulnerable place wasn’t where all the bullet holes were on the returning planes; it was where the bullet holes were on the planes that didn’t return.
Restricting your measurements to a final sample, excluding part of the sample that didn’t survive, creates what statisticians call “survivor bias.” It can cause you to come to conclusions that are entirely wrong.
Now let’s examine mutual funds in this light. At a recent conference at Dimensional Fund Advisors in Austin, I learned that 45 percent of the managed equity funds with 15-year records at the end of 2014 outperformed their target indexes. Not wonderful, but not terrible, either.
But DFA’s James Davis, a vice president in their research group, went on to explain that the group didn’t include the equivalent of the missing planes. It didn’t include the mutual funds that had failed to complete their mission. At the beginning of the period there were 2,711 funds. At the end of the period there were 1,139. Only 42 percent of the starting funds had survived.
Do the arithmetic again, and you get a very different figure. Only 19 percent of the managed funds that started the 15-year period outperformed their index target. Comparable figures for fixed-income funds are even worse — only 8 percent of funds that started the period did better than the index they were supposed to beat.
It looks like a lot of those managed funds had bad pilots. Most either didn’t survive or were beaten by an unmanaged fund that tracked an index. Things appear a lot better than they are when you just measure, and talk about, the surviving funds. An accurate assessment would start with the full group. It would not be limited to the survivors.
In spite of this, virtually all discussion that you and I see, hear or read is based on surviving funds, not original sample groups. In fairness, we can say that this happens because surviving fund data is what is generally available. Even Morningstar’s oft-cited figures are based on surviving funds.
But available or convenient doesn’t mean this is information that should be considered accurate or scientific. Let’s put it this way. The next time you see a professional expert using survivor group data to tout the superiority of managed funds, remember this old joke:
A police officer sees a drunk under a streetlight and asks what he is doing.
“Looking for my car keys,” the drunk answers.
“Where did you lose the keys?” the officer asks.
“Oh, about two blocks away, but the light is much better here,” the drunk says.
The professional expert is closely related to the drunk. Both are looking in the wrong place because it’s convenient.
Non-experts can’t be expected to know about statistical bias. That’s most people. But this discipline has been around for decades. Most of the investment professionals you see on TV, quoted in newspapers and magazines have been exposed to it. They have the education that would expose them — MBA degrees, CFA certifications, etc.
So if you see, hear or read one making this error, what’s the problem? Upton Sinclair best stated it:
“It is difficult to get a man to understand something when his salary depends upon his not understanding it.”