In our June 15th post, we reported on Wahlström and Rollenhagen’s* concern that trying to measure safety culture could do more harm than good. However, the authors go on to assert that safety culture can and should be assessed. They identify different methods that can be used to perform such assessments, including peer reviews and self assessments. They conclude “Ideally safety culture assessments should be carried out as an interaction between an assessment team and a host organization and it should be aimed at the creation of an awareness of potential safety threats . . . .” (§ 7) We certainly agree with that observation.
We are particularly interested in their comments on safety (performance) indicators, another tool for assessing safety culture. We agree that “. . . most indicators are lagging in the sense that they summarize past safety performance” (§ 6.2) and thus may not be indicative of future performance. In an effort to improve performance indicators, the authors suggest “One approach towards leading safety indicators may be to start with a set of necessary conditions from which one can obtain a reasonable model of how safety is constructed. The necessary conditions would then suggest a set of variables that may be assessed as precursors for safety. An assessment could then be obtained using an ordinal scale and several variables could be combined to set an alarm level.” (ibid.)
We believe the performance indicator problem should be approached somewhat differently. Safety culture, safety management and safety performance do not exist in a vacuum. We advocate using the principles of system dynamics to construct an organizational performance model that shows safety as both input to and output from other, sometimes competing organizational goals, resource constraints and management actions. This is a more robust approach because it can not only show that safety culture is getting stronger or slipping, but why, i.e., what other organizational factors are causing safety culture change to occur. If the culture is slipping, then analysis of system information can suggest where the most cost-effective interventions can be made. For more information on using system dynamics to model safety culture, please visit our companion website, nuclearsafetysim.com.
* Björn Wahlström, Carl Rollenhagen. Assessments of safety culture – to measure or not?Paper presented at the 14th European Congress of Work and Organizational Psychology, May 13-16, 2009, Santiago de Compostela, Spain. The authors are also connected with the LearnSafe project, which we have discussed in earlier posts (click the LearnSafe label to see them.)
That’s a question raised in a paper by Björn Wahlström and Carl Rollenhagen.* Among other issues, the authors question the reliability and validity of safety culture measurement tools, especially the questionnaires and interviews often used to assess safety culture. One problem is that such measurement tools, when applied by outsiders such as regulators, can result in the interviewees trying to game the outcome. “. . . the more or less explicit threat to shut down a badly performing plant will most likely at least in a hostile regulatory climate, bring deceit and delusion into a regulatory assessment of safety culture.” (§ 5.3)
Another potential problem is created by a string of good safety culture scores. We have often said success breeds complacency and an unjustified confidence that past results will lead to future success. The nuclear industry does not prepare for surprises yet, as the authors note, the current state of safety thinking was inspired by two major accidents, not incremental progress. (§ 5.2) Where is the next Black Swan lurking?
Surprise after success can occur on a much smaller scale. After the recent flap at Vermont Yankee, evaluators spent considerable time poring over the plant’s most recent safety culture survey to see what insight it offered into the behavior of the staff involved with the misleading report on leaking pipes. I don’t think they found much. Entergy’s law firm conducted interviews at the plant and concluded the safety culture was and is strong. See the opening paragraph for a possible interpretation.
While safety culture may not be measurable, it is possible to assess it. The authors’ thoughts on how to perform useful assessments will be reviewed in a future post.
* Björn Wahlström, Carl Rollenhagen. Assessments of safety culture – to measure or not?Paper presented at the 14th European Congress of Work and Organizational Psychology, May 13-16, 2009, Santiago de Compostela, Spain. The authors are also connected with the LearnSafe project, which we have discussed in earlier posts (click the LearnSafe label to see them.)
The line of thinking in the Wahlström and Rollenhagen paper and the LearnSafe project appears to provide a strong nudge away from thinking of safety culture in terms of a set of beliefs and values. Or of thinking of safety culture as something apart from the how the multiple, complex decision processes within an organization are occurring.
One could also ask, as did Wahlström and Rollenhagen, if the present interpretations of safety culture are rich enough to serve the need for a requisite variety; i.e. does the concept have the same order of complexity as the plant organization that it is supposed to control? [p.8]
One tool for representing the many factors at work in a given environment is an influence diagram. As Wahlström and Rollenhagen note, “Influence diagrams are often used as the next step in a model building exercise to track dependencies between issues. It is relatively easy for people to identify up-stream causes and down-stream consequences of some specific issue. It is far more difficult to merge these influences to a comprehensive model of some interesting phenomenon, because there are usually very many influences to be traced. Sometimes the influences form loops, which in practice may render the influence diagram more difficult to use for making predictions of how some issue may influence another. When the influences are linear, models are relatively easy to build and validate, but many systems include influences with threshold and saturation effects.” [p. 4, emphasis added] Multiple variables, loops, and threshold and saturation effects are all important constructs in the system dynamics world view.
I thought I would use this question as an entre back into some of Professor Bernhard Wilpert’s work with what became known as the LearnSafe project. The LearnSafe website is worth visiting for insights into this issue and a number of others.
Two of the principal contributors to LearnSafe, Björn Wahlström and Carl Rollenhagen, published some of their interpretations of the study results in a 2004 paper, link below. In the paper they state:
“The data collected in the LearnSafe project provides interesting views on some of the major issues connected to the concept of safety culture. A suggestion generated from the data is that attempts to define and measure safety culture may be counterproductive and a more fruitful approach may be to use the concept to stimulate discussions on how safety is constructed. ” [p. 2]
The contribution of the LearnSafe project comes from the empirical data developed in the surveys and discussions with over 300 nuclear managers. It was found that the term safety culture was not frequently mentioned as a challenge for managing nuclear plants. Instead, much more frequently mentioned were factors that are commonly understood to be part of safety culture. Wahlström and Rollenhagen observe, “This would suggest the interpretation is that safety culture is not a concept for itself, but it is instead ingrained in various aspects of the management activities.” [p. 6]
This observation leads to the question of whether it is useful to put forward safety culture as a top level concept that somehow is responsible for or “produces” safety. Or would it be better to think of it as an organic process that continuously evolves and develops within an organization. This perspective would say that safety culture is more the product of the myriad of decisions and interactions that occur within an organization rather than some set of intrinsic values that is the determinant of those decisions.
As mentioned in a prior post we will be highlighting some of the work of the late Bernhard Wilpert, a leading figure in research on the role of human behavior in high reliability organizations.
Professor Wilpert emphasized the interaction of human, technology, and organizational dynamics. His tools for human factors event analysis have become the standard practice in German and Swiss nuclear plants. He is the author of several leading books including Safety Culture in Nuclear Power Operations; System Safety: Challenges; Pitfalls of Intervention; Emerging Demands for Nuclear Safety of Nuclear Power Operations: Challenge and Response; and Nuclear Safety: A Human Factors Perspective. Professor Wilpert was also a principal contributor to the LearnSafe project conducted in Europe from 2001 – 2004. See the following link for information about the project team and its results and look to us for future posts on the LearnSafe research. Link to LearnSafe project.
In our June 15th post, we reported on Wahlström and Rollenhagen’s* concern that trying to measure safety culture could do more harm than good. However, the authors go on to assert that safety culture can and should be assessed. They identify different methods that can be used to perform such assessments, including peer reviews and self assessments. They conclude “Ideally safety culture assessments should be carried out as an interaction between an assessment team and a host organization and it should be aimed at the creation of an awareness of potential safety threats . . . .” (§ 7) We certainly agree with that observation.
