Survivorship Bias

Posted by-Posted inpostTags:, ,

1941. This a World War II story, where as many of those, starts with Nazis hounding a Jew out of Europe and ends with the Nazis regretting it.

As a member of the bomber crew, the Allied Aviation, the chances of surviving during a given mission were probably incredulous. After flying above enemy territory suspended in the air, visible and vulnerable from all directions, with the danger of an entire nation whose only purpose was swatting you like a fly, the possibilities of coming back home were, we will say, optimistic. For that reason, airmen during the war were described as “ghosts already” by the historian Kevin Wilson.

Therefore, American aviation was suffering badly from German air defense. The battle in the skies was becoming dramatically. It was crucial to make some changes and reconsider the problem. So it was time for The Statistical Research Group (SRG), a classified program (like Manhattan project) that yoked the assembled might of American statisticians to the war effort.

It was the most high-powered and influential among any group. The atmosphere created there represented pure intellectuality combined with intensity. W. Allen Allis, the director, expressed in his own words the SRG was “the most extraordinary group of statisticians ever organized, taking into account both number and quality.” (and probably arrogant, e.g. “When we made recommendations, frequently things happened”).

Between them were Frederick Mosteller, later founder of Harvard’s statistics department; Leonard Jimmie Savage, pioneer of decision theory; Norbert Wiener, MIT mathematician and creator of cybernetics; Milton Friedman probably considered as the fourth-smartest person in the room, future Nobel prize in economics. It was a department ridiculously full of talent. But the smartest person there was Abraham Wald, he was the man you preferably wanted at your side.

Wald was born in the former Austrian-Hungarian empire in 1902, started his professional life in Vienna as a pure mathematician and later become more interested in the statistics field. As being a jew and, due to the situation with the nazis in the power, he managed to escape to the United States where he worked as a teacher of Allen Wallis at Columbia. (You can find a more rich and detailed biography about Wald in the memoirs by his two closest friends: Wolfowitz, J. 1952).

Basically, we will say the problem was the Americans didn’t want their planes to get shot down by the Axis fighters, so was their work to increase the probability of an airplane’s survival.

The problem has been set up, the armor of the planes needed to be reinforced. But it was not just a matter of adding protection with no limits, this would lead to massive plane’s weights and they would be less maneuverable, among many issues. The key point was assessing the best spots of the airplane to recover with more protection.

The military brought the SRG some statistical data regarding aircraft damage, specifically, the distribution and number of bullet holes in planes coming back from Europe. This would be something similar to the following figure:

Image 2. Red points represent the bullet marks on the plane. Source

It seems that most of the damage is located on the fuselage, where the rest of hits were along the wings and around the tail gunner. Naturally, the military was expecting to add armor mainly to those zones with more density of impacts (i.e. fuselage and winds). They also saw it as an opportunity to get the same protection with less armor if they concentrate the armor with the greatest need, where more bullet holes were allocated.

But Wald’s opinion was completely the opposite. He argued that what they knew were only about planes coming back, those who were not have returned they had no data at all. Thus, the reason planes were coming back with fewer impacts to the engine is that planes that got hit in the engine were not coming back. The focus should be put into the missing holes, it revealed the locations which needed the additional armor. Therefore, the engines would need to increase their protection. High-density impacts zone meant to present a higher — and therefore acceptable — tolerance. Wallis wrote, in a pleasant way: “The military was inclined to provide protection for those parts that on returning planes showed the most hits. Wald assumed, on good evidence, that hits in combat were uniformly distributed over the planes. It follows that hits on the more vulnerable parts were less likely to be found on returning planes than hits on the less vulnerable parts, since planes receiving hits on the more vulnerable parts were less likely to return to provide data. From these premises, he devised methods for estimating vulnerability of various parts”.

Wald’s analysis of bullet-riddled aircraft in World War II saved the lives of dozens of brave airmen. The knowledge provided by that teach us two key things: (i) You must take into account all the data, this also includes that one which is hidden or unrevealed. (ii) Additionally, learning from failure is built above unsuccessful attempts, it takes careful thought and deliberation.

Now, at this point, you might have heard or read something about the topic. A brief google search could bring us incredible headlines regarding Wald’s heroism and innovative thinking. Just try with something like “survivorship bias aircraft” or whathever related.

Many reasons might explain why this Mathematician has become a legend, appearing in plenty of news, articles, self-help books and other formats. For that reason, I have been so reluctant to name this post as such, although it continues the same. Despite that, we are going to play between the myth and the more plausible version, or at least, to be firmly rigorous and give Wald what he deserves, the truth.

In fact, there is very little material about Wald had to say about this aircraft damage. As a curiosity, he was not allowed to see the classified reports produced by himself, so the joke around there was that the secretaries were required to pull each sheet out of Wald’s hands as soon as he was done.

The history of SRG was described in the autobiographical memoir by his leader, W. Allen Wallis in 1980, although it was declassified many years ago, it has not appeared in the open literature. Also, Wald describing the methods were reprinted by the Center for Naval Analyses in the same year, compounded by eight memorandas with the title “A Method of Estimating Plane Vulnerability Based on Damage of Survivors”. As we have said, tiny evidence is available regarding Wald’s work on aircraft damage. Specifically, two short mentions in Wallis’ memoir and the Wald’s collection on the subject.

Many myths about Wald’s brilliant idea presents the problem as a magician response that Wald came up with, trying to create a false illusion with no maths involved in. But in fact, as a mathematician he was, it can not be said that the solution was free of formulas or formalism. Indeed, he did maths, he did so maths that the memoranda is extremely technical, far from our scope and objective. Another valuable guide was provided at the same time by Mangel and Samaniego for those interested. A short view of the original 100 pages memoranda illustrates what I mean:

Extract from a reprint of “A Method of Estimating Plane Vulnerability Based on Damage of Survivors”

Just in one of the eight memoranda parts it is detailed the problem of airplane vulnerability (Part V), leading to the same conclusion as the simple story but with a different path, a more tedious and elegant way which may be as magical as the initial version. The results are summarized below:

PART V. Subdivision of the Plane Into Several Equi-Vulnerability Areas.

Internet, after all, has treated Wald as many of those magical metaphors where multiple steps, if not all, are jumped directly to conclusions. Nothing new under the sun here, when the only motivation was explaining with an example the title of the article, the survivorship bias. This represents the tendency to view the existing consequences as a result of only the visible and past proofs. Frequently in economics, it is observed as the tendency to assess the existing stocks or funds in the market as a representative sample without regarding those who have gone bust. You are clearly suffering from What You See is All There Is. There is no any diagnosed cure, but being aware of it might help. It can be applied to maths, economics or any situation you can imagine.

The acquire knowledge can be summarized in two main points: (1) Bias world is immense, so it is good just to know something about its phenomenons and causes. The analysis of a problem should be carefully considered taking into account all the possible data, whether it is visible or not. (2) Internet, at the end of the day, is Internet. It is just good to know.

Lastly, an ironic little extract from Walli’s rejoinder:

Finally, a legend that has had some currency concering Hotelling’s work at Dahlgren: For his first visit, Hotelling was told that a station wagon to Dahlgren would leave the main Navy building in Washington at 8:00 a.m. Someone else told him 8:30 a.m. He arrived at 8:15, missing the station wagon by 15 minutes. This led to much joking about statisticians drowning in streams only three feet deep on the average. Actually he had started in time to be confident of arriving by 7:45 a.m., but had encountered an extraordinary delay of more than half an hour. Nevertheless, the legend lives on. I do not expect this note to slay it, any more than I expect my paper to slay the legend about Wald’s work being given a security classification and snatched away from him because he lacked a security clearance. Many will feel with Parson Weems, the originator of the story about George Washington and the cherry tree, that “If it isn’t true, it ought to be.” Or is the remark by the parson merely one that he ought to have made, but didn’t?

References

Leave a comment

Design a site like this with WordPress.com
Get started