Thursday, August 30, 2012

Failure to Learn

In this post we call your attention to a current research paper* and Wall Street Journal summary article** that sheds some light on how people make decisions to protect against risk.  The specific subject of the research involves response to imminent risk of house damage due to hurricanes.  As the author of the paper states, “The purpose of this paper is to attempt to resolve the question of whether there are, in fact, inherent limits to our ability to learn from experience about the value of protection against low-probability, high-consequence, events.” (p.3)  Also of interest is how the researchers used several simulations to gain insight and quantify how the decisions compared to optimal risk mitigation.

Are these results directly applicable to nuclear safety decisions?  We think not.  But they are far from irrelevant.  They illustrate the value of careful and thoughtful research into the how and why of decisions, the impact of the decision environment and the opportunities for learning to produce better decisions.  It also raises the question, Where is the nuclear industry on this subject?  Nuclear managers are making routinely what are probably the most safety significant decisions of any industry.  But how good are these decisions, and what determines their decision quality?  The industry might contend that the emphasis on safety culture (meaning values and traits) is the sine qua non for assuring decisions that adequately reflect safety.  Bad decision?  Must have been bad culture.  Reiterate culture, assume better decisions to follow. Is this right or is safety culture the wrong blanket or just too small a blanket to try to cover a decision process evolving from a complex adaptive system? 

The basic construct for the first simulation was a contest among participants (college students) with the potential to earn a small cash bonus based on achieving certain performance results.  Each participant was made the owner of a house in a coastal area subject to hurricane intrusion.  During the simulation animation, a series of hurricanes would materialize in the ocean and approach land.  The position, track and strength of the hurricane were continuously updated.  Prior to landfall participants had the choice of purchasing protection against damage for that specific storm, either partial or full protection.  The objective was to maximize total net asset; i.e., the value of the house, less any uncompensated damage and less the cost of any purchased protection.

While the first simulation focused on recurrent short term mitigation decisions, in the second simulation participants had the option to purchase protection that would last at least for the full season but had to purchased prior to a storm occurring.  (A comprehensive description of the simulation and test data are provided in the referenced paper.)

The results indicated that participants significantly under-protected their homes leading to actual losses higher than a “rational” approach to purchasing protection.  While part of the losses was due to purchasing protection unnecessarily, most was due to under protection.  The main driver, according to the researchers, appeared to be that participants over relied on their most recent experience instead of an objective assessment of current risk.  In other words, if in a prior hurricane they experienced no damage, either due to the track of the hurricane or because they had purchased protection, they were less inclined to purchase protection for the next hurricane. 

The simulations reveal limitations in the ability to achieve improved decisions in what was, in essence, a trial and error environment.  Feedback occurred after each storm, but participants did not necessarily use the feedback in an optimal manner “due to a tendency to excessively focus on the immediate disutility of cost outlays” (p.10)  In any event it is clear that the nuclear safety decision making environment is “not ideal for learning—…[since] feedback is rare and noisy…” (p.5)  In fact most feedback in nuclear operations might appear to be affirming since rarely do decisions to take short term risks result in bad outcomes.  It is an environment susceptible to complacency more than learning.

The author concludes with a final question as to whether non-optimal decision making, such as observed in the simulations, can be overcome.  He concludes, “This is may be a difficult since the psychological mechanisms that lead to the biases may be hard-wired; as long as we remain present-focused, prone to chasing short-term rewards and avoiding short term punishment, it is unlikely that individuals and institutions will learn to undertake optimal levels of protective investment by experience alone. The key, therefore, is introducing decision architectures that allow individuals to overcome these biases through, for example, creative use of defaults…” (pp. 30-31)

*  R.J. Meyer, “Failing to Learn from Experience about Catastrophes: The Case of Hurricane Preparedness,” The Wharton School, University of Pennsylvania Working Paper 2012-05 (March 2012).

** C. Shea, “Failing to Learn From Hurricane Experience, Again and Again,” Wall Street Journal (Aug. 17, 2012).

No comments:

Post a Comment

Thanks for your comment. We read them all. We'd like to display them under their respective posts on our main page but that's not how Blogger works.