Assessing Safety Culture

In our June 15th post, we reported on Wahlström and Rollenhagen’s* concern that trying to measure safety culture could do more harm than good. However, the authors go on to assert that safety culture can and should be assessed. They identify different methods that can be used to perform such assessments, including peer reviews and self assessments. They conclude “Ideally safety culture assessments should be carried out as an interaction between an assessment team and a host organization and it should be aimed at the creation of an awareness of potential safety threats . . . .” (§ 7) We certainly agree with that observation.

We are particularly interested in their comments on safety (performance) indicators, another tool for assessing safety culture. We agree that “. . . most indicators are lagging in the sense that they summarize past safety performance” (§ 6.2) and thus may not be indicative of future performance. In an effort to improve performance indicators, the authors suggest “One approach towards leading safety indicators may be to start with a set of necessary conditions from which one can obtain a reasonable model of how safety is constructed. The necessary conditions would then suggest a set of variables that may be assessed as precursors for safety. An assessment could then be obtained using an ordinal scale and several variables could be combined to set an alarm level.” (ibid.)

We believe the performance indicator problem should be approached somewhat differently. Safety culture, safety management and safety performance do not exist in a vacuum. We advocate using the principles of system dynamics to construct an organizational performance model that shows safety as both input to and output from other, sometimes competing organizational goals, resource constraints and management actions. This is a more robust approach because it can not only show that safety culture is getting stronger or slipping, but why, i.e., what other organizational factors are causing safety culture change to occur. If the culture is slipping, then analysis of system information can suggest where the most cost-effective interventions can be made. For more information on using system dynamics to model safety culture, please visit our companion website, nuclearsafetysim.com.

* Björn Wahlström, Carl Rollenhagen. Assessments of safety culture – to measure or not? Paper presented at the 14th European Congress of Work and Organizational Psychology, May 13-16, 2009, Santiago de Compostela, Spain. The authors are also connected with the LearnSafe project, which we have discussed in earlier posts (click the LearnSafe label to see them.)

Safety Culture Dynamics (part 1)

Over the last several years there have been a number of nuclear organizations that have encountered safety culture and climate issues at their plants. Often new leadership is brought to the plant in hopes of stimulating the needed changes in culture. Almost always there is increased training and reiteration of safety values and a safety culture survey to gain a sense of the organizational temperature. It is a little difficult to gauge precisely how effective these measures are - surveys are snapshots in time and direct indicators of safety culture are lacking. In some cases, safety culture appears to respond in the short term to these changes but then loses momentum and backslides further out in time.

How does one explain these types of evolutions in culture? Conventional wisdom has been that culture is leadership driven and when safety culture is deficient, new management can “turn around” the situation. We have argued that the dynamics of safety culture are more complex and are subject to a confluence of factors that compete for the priorities and decisions of the organization. We use simulation models of safety culture to suggest how these various factors can interact and respond to various initiatives. We made an attempt at a simple illustration of what may illustrate the situation at a plant which responds as described above. CLICK ON THIS LINK to see the simulated safety culture dynamic response.

The simulation shows changes in some key variables over time. In this case the time period is 5 years. For approximately the first year the simulation illustrates the status quo prior to the change in leadership. Safety culture was in gradual decline despite nominal attention to actions to reinforce a safety mindset in the organization.

At approximately the one year mark, leadership is changed and actions are taken to significantly increase the safety priority of the organization. This is reflected in a spike in reinforcement that typically includes training, communications and strong management emphasis on the elements of safety culture. Note that following a lag, safety culture starts to improve in response to these changes. As time progresses, the reinforcement curve peaks and starts to decay due to something we refer to as “saturation”. Essentially the new leadership’s message is starting to have less and less impact even though it is being constantly reiterated. For a time safety culture continues to improve but then turns around due to the decreasing effectiveness of reinforcement. Eventually safety culture regresses to a level where many of the same problems start to recur.

Is this a diagnosis of what is happening at any particular site? No, it is merely suggestive of some of the dynamics that are work in safety culture. In this particular simulation other actions that may be needed to build strong, enduring safety culture were not implemented in order to isolate the failure of one-dimensional actions to provide long term solutions. One of the indicators of this narrow approach can be seen in the line on the simulation representing the trust level within the organization. It hardly changes or responds to the other dynamics. Why? In our view trust tends to be driven by the overall, big picture of forces at work and the extent to which they consistently demonstrate safety priority. Reinforcement (in our model) reflects primarily a training and messaging action by management. Other more potent forces include whether management “walks the talk”, whether resources are allocated consistent with safety priorities, whether short term needs are allowed to dominate longer term priorities, whether problems are identified and corrected in a manner to prevent recurrence, etc. In this particular simulation example, these other signals are not entirely consistent with the reinforcement messages, with a net result that trust hardly changes.

More information regarding safety culture simulation is available at the nuclearsafetysim.com website. Under the Models tab, Model 3 provides a short tutorial on the concept of saturation and its effect on safety culture reinforcement.

"SimCity Baghdad" is the title of an intriguing article from The Atlantic magazine (Jan/Feb 2010 issue), and available via the link below.  It describes a new computer game called UrbanSim that allows U.S. Army officers to train in counterinsurgency tactics that are being implemented in Iraq.  As is often the case, the military is at the forefront of applying innovative tools such as simulation to meet new challenges.

As is the case with many simulation applications, a significant impetus for their use is that they are inexpensive and allow people to develop skills without being directly exposed to the consequences of their actions.  As pointed out in the article, “....computer games are cheap and can be played anywhere. And because the students all run the same scenarios, they can compare the efficacy of different approaches.”

The simulation was developed with significant input from soldiers who had returned from Iraq.  It is a type of simulation referred to as “agent-based simulation” that can be quite useful in portraying the dynamics of groups.  The game’s characters are modeled as autonomous agents that react not just to specific actions, but to the climate created by a player’s overall strategy.

These types of simulations are not intended to be predictive tools or to teach a specified series of actions in response to given situations.  “Rather, the intent is to teach commanders new ways of thinking about multiple problems in a fast-changing environment, always reevaluating instead of fixating on one approach...You have to think through the cause and effect of your decisions...”

Many of the benefits of simulation described in this case are the same as for managing nuclear safety culture.  As we pointed out in a prior blog post (“Social Licking”, October 6, 2009) , building and sustaining cultural norms can be significantly influenced by networked relationships - as in a nuclear plant organization - and that individuals are likely to model their behaviors based on the network.  The challenge of course is that the modeled behaviors need to be those that support safety culture.
 
Link to article.

“Culture isn’t just one aspect of the game. It is the game.”

The quote is from Lou Gerstner, retired Chairman of IBM, and appears in an interesting presentation by the management team at Wolf Creek Nuclear Operating Company.  In it they put forth their perspectives on addressing culture change within their organization.  There are many good points in the presentation and several I would like to specifically highlight.

First, the issue of trust is addressed on several slides.  For example, on the Engaged Employees slide (p. 24) it is noted that training in building trust had been initiated and would be ongoing.  A later slide, Effective Leadership Team (p. 31), notes that there was increased trust at the station.  In our thinking about safety management, and specifically in our simulation modeling, we include trust as a key variable and driver of safety culture.  Trust is a subjective ingredient but its importance is real.  We think there are at least two mechanisms for building trust within an organization.  One is through the type of initiatives described in the slides – direct attention and training in creating trust within the management team and staff.  A second mechanism that perhaps does not receive as much recognition is the indirect impact of decisions and actions taken by the organization and the extent to which they model desired safety values.  This second mechanism is very powerful as it reflects reality.  If reality comports with the espoused values, it reinforces the values and builds trust.  If reality is contra to the values, it will undermine any amount of training or pronouncements about trust.

The second point to be highlighted is addressed on the Culture slide in the Epilogue section (p.35).  There it is noted that as an industry we are good at defining the desired behaviors, but we are not good at defining how to achieve a culture where most people practice those behaviors.  We think there is a lot of truth in this and the “how” aspect of building and maintaining a robust safety culture is something that merits more attention.  “Practicing” those behaviors is the subject of our white paper, “Practicing Nuclear Safety Management.”

Link to presentation.

Schrodinger’s Bat

This post follows on the issue of whether safety culture is a concept unto itself or a state that is defined by many constituent actions.  Some of our own thinking about safety culture in developing the nuclearsafetysim website and tools led us to prefer a focus on safety management as opposed to safety culture.  Safety management includes the key “levers” of organizational performance (e.g., resource allocation, problem idenfication and resolution, building of trust, etc.) and the integrated effect of the manipulation of these levers results in a safety culture “value” in the simulation.  Thus all the dynamics flow from actions and decisions to a safety culture resultant, not the reverse.

Dare I put forth a sports analogy?  In baseball there is a defined “strike zone”.   In theory the umpire uses the strike zone to make calls of balls and strikes.  But the zone is really open to interpretation in the dynamic, three dimensional world of pitching and umpiring.  The reality is that the strike zone becomes the space delineated by the aggregate set of balls and strike calls by an umpire.  It relies on the skill of the umpire, his understanding of the strike zone and his commitment to making accurate calls. The linked article provides some interesting data on the strike zone and the psychology of umpires' decisions.

Link to "Schrodinger’s Bat" July 26, 2007.

Primer on System Dynamics

System Dynamics is a concept for seeing the world in terms of inputs and outputs, where internal feedback loops and time delays can affect system behavior and lead to complex, non-linear changes in system performance.

The System Dynamics worldview was originally developed by Prof. Jay Forrester at MIT. Later work by other thinkers, e.g., Peter Senge, author of The Fifth Discipline, expanded the original concepts and made them available to a broader audience. An overview of System Dynamics can be found on Wikipedia.

Our NuclearSafetySim program uses System Dynamics to model managerial behavior in an environment where maintaining the nuclear safety culture is a critical element. NuclearSafetySim is built using isee Systems iThink software. isee Systems has educational materials available on their website that explain some basic concepts.

There are other vendors in the System Dynamics software space, including Ventana Systems and their Vensim program. They also provide some reference materials, available here.

The Economist on Computer Simulation

The Economist has occasional articles on the practical applications of computer simulation. Following are a couple of items that have appeared in the last year.

Agent-based simulation is used to model the behavior of crowds. "Agent-based" means that each individual has some capacity to ascertain what is going on in the environment and act accordingly. This approach is being used to simulate the movement of people in a railroad station or during a building fire. On a much larger scale, each of the computer-generated orcs in the "Lord of the Rings" battle scenes moved independently based on his immediate surroundings.

Link to article.

The second article is a brief review of simulation's use in business applications, including large-scale systems (e.g., an airline), financial planning, forecasting, process mapping and Monte Carlo analysis. This is a quick read on the ways simulation is used to illustrate and analyze a variety of complex situations.

Link to article.

Other informational resources that discuss simulation are included on our References page.

Reading List: Just Culture by Sidney Dekker

Thought I would share with you a relatively recent addition to the safety management system bookshelf, Just Culture by Sidney Dekker, Professor of Human Factors and System Safety at Lund University in Sweden.  In Dekker’s view a “just culture” is critical for the creation of safety culture.  A just culture will not simply assign blame in response to a failure or problem, it will seek to use accountability as a means to understand the system-based contributors to failure and resolve those in a manner that will avoid recurrence.  One of the reasons we believe so strongly in safety simulation is the emphasis on system-based understanding, including a shared organizational mental model of how safety management happens.  One reviewer (D. Sillars) of this book on the amazon.com website summarizes, “’Just culture’ is an abstract phrase, which in practice, means . . . getting to an account of failure that can both satisfy demands for accountability while contributing to learning and improvement.” 


Question for nuclear professionals:  Does your organization maintain a library of resources such as Just Culture or Dianne Vaughan’s book, The Challenger Launch Decision, that provide deep insights into organizational performance and culture?  Are materials like this routinely the subject of discussions in training sessions and topical meetings?

Safety Culture Insights from Simulation (MIT #1)

Starting with this post we are reviewing an interesting paper from the Sloan School of Management at MIT - “Preventing Accidents and Building A Culture of Safety: Insights from a Simulation Model.”  While the paper  approaches organizational safety performance on a generic basis – it is not specific to nuclear facilities - it offers many useful insights that are highly applicable to nuclear organizations.  MIT as an institution is known for its work in systems dynamics and simulation modeling.  These disciplines have been used to analyze a variety of safety and accident environments including NASA and nuclear operations.  Also, as you may have noticed on NuclearSafetySim.com, our development of nuclear safety management simulation models is based on systems dynamics.

Future posts will highlight several of the key insights from this paper and their applicability to issues of nuclear safety management.


Link to paper.

Can Driving and Texting Coexist?

In the July 18, 2009 online edition of The New York Times, there is an interesting example of the use of a simulation game to illustrate the impact of texting on a driver’s ability to drive safety and react to changing road conditions. Upon completion of the game, the player is provided with a quantification of his driving performance with and without the distraction presented by receiving and sending text messages.

I thought this would be interesting to nuclear safety management practitioners for several reasons. First, it is another illustration of how simulation games can provide realistic experiences of situations they may have to manage in real life - without the risks associated with the real life activity.

Second, this game demonstrates the impact of competing priorities (texting and driving) on the ability of the driver to maintain performance at a consistent level. In the nuclear operations world, safety management failures are often associated with the impact of competing priorities or pressures on the ability of personnel to perform reliably. The driving game suggests that there is always some diminution of performance due to the competing priority of texting. Is that true of nuclear safety management or is it possible, with sufficient training and practice, to manage competing priorities?

Link to article.