To illustrate the application of the scoring method we used a set of twenty decisions based on issues taken from actual U.S. nuclear operating experience, typically those that were reported in LERs. As a baseline, we scored each issue for safety significance and uncertainty. Each issue identified 3 to 4 decision options for addressing the problem - and each option was annotated with the potential impacts of the decision on budgets, generation (e.g. potential outage time) and the corrective action program. We scored each decision option for its decision balance (how well the decision option balances safety priority) and then identified the preferred decision option for each issue. This constitutes what we refer to as the “preferred decision set”. A pdf file of one example issue with decision choices and scoring inputs is available here.
Our assumption is that the preferred decision set would be established/approved by senior management based on their interpretation of the issues and their expectations for how organizational decisions should reflect safety culture. The set of issues would then be used in a training environment for appropriate personnel. For purposes of this example, we incorporated the preferred decision set into our NuclearSafetySim* simulator to illustrate the possible training experience. The sim provides an overall operational context tracking performance for cost, plant generation and CAP program and incorporating performance goals and policies.
As we indicated in the April 9 post, each decision is evaluated for its safety significance and uncertainty in accordance with quantified scales. These serve as key inputs to determining the appropriate balance to be achieved in the decision. In prior work in this area, reported in our posts dated July 15, 2011 and October 14, 2011 we solicited readers to score two issues for safety significance. The reported scores ranged from 2 to 10 (most scores between 4 to 6) for one issue and ranged 5 to 10 (most scores 6 to 8) for the other issue. This reflects the reality that perceptions of safety significance are subject to individual differences. In the current exercise, similar variations in scoring were expected and led to differences between the trainee’s scores and the preferred decision set. The variation may be due to the inherent subjective nature of assessing these attributes and other factors such as experience, expertise, biases, and interpretations of the issue. So this could be one source of difference in the trainee decision selections versus the preferred set, as the decision process attempts to match action to significance.
Another source could be in the decision options themselves. The decision choice by a trainee could have focused on what the trainee felt was the “best” (i.e., most efficacious) decision versus an explicit consideration of safety priority commensurate with safety significance. Additionally decision choices may have been influenced by their potential impacts, particularly under conditions where performance was not on track to meet goals.
The plots illustrate how decision balance may vary over time, with specific decisions reflecting greater or lesser emphasis on safety. During the first half of the sim the decision balances are in fairly close agreement, reflecting in part that in 5 of 8 cases the actual decisions matched the preferred decisions. However in the second half of the sim significant differences emerge, primarily in the direction of weaker balances associated with the trainee decisions. Again, understanding why these differences emerge could provide insight into how safety culture is actually being practiced within the organization. Chart 3 adds in some additional context.
A variety of training benefits can flow from the decision simulation. Comparisons of actual to preferred decisions provide a baseline indication of how well expected safety balances are being achieved in realistic decisions. Consideration of contributing factors such as goal pressure may illustrate challenges for decision makers. Comparisons of results among and across groups of trainees could provide further insights. In all cases the results would provide material for discussion, team building and alignment on safety culture.
In our post dated November 4, 2011 we quoted the work of Kahneman, that organizations are “factories for producing decisions”. In nuclear safety, the decision factory is the mechanism to actualize safety culture into specific priorities and actions. A critical element of achieving strong safety culture is to be able to identify differences between espoused values for safety (i.e., the traits typically associated with safety culture) and de facto values as revealed in actual decisions. We believe this can be achieved by capturing decision data explicitly, including the judgments on significance and uncertainty, and the operational context of the decisions.
The next step is synthesizing the decision and situational parameters to develop a useful systems-based measure of safety culture. A quantity that could be tracked in a simulation environment to illustrate safety culture response and provide feedback and/or during nuclear operations to provide a real time pulse of the organization’s culture.
* For more information on using system dynamics to model safety culture, please visit our companion website, nuclearsafetysim.com.
** It is possible for some decision options to have the same value of balance even though they incorporate different responses to the issue and different operational impacts.