Modeling Safety Culture (Part 3): Simulation Results 1

As promised in our June 29, 2012 post, we are taking the next step to incorporate our mental models of safety culture and decision making in a simple simulation program.  The performance dynamic we described viewed safety culture as a “level”, and the level of safety culture determines its ability to resist pressure associated with competing business priorities. If business performance is not meeting goals, pressure on management is created which can be offset by sufficiently strong safety culture. However if business pressure exceeds the threshold for a given safety culture level, management decision making can be affected, resulting in a shift of resources from safety to business needs. This may relieve some business pressure but create a safety gap that can degrade safety culture, making it potentially even more vulnerable to business pressure.

It is worth expanding on the concept of safety culture as a “level” or in systems dynamics terms, a “stock” - an analogy might be the level of liquid in a reservoir which may increase or decrease due to flows into and out of the reservoir.  This representation causes safety culture to respond less quickly to changes in system conditions than other factors.  For example, an abrupt cut in an organization’s budget and its pressure on management to respond may occur quite rapidly - however its impact on organizational safety culture will play out more gradually.  Thus “...stocks accumulate change.  They are kind of a memory, storing the results of past actions...stocks cannot be adjusted instantaneously no matter how great the organizational pressures…This vital inertial characteristic of stock and flow networks distinguishes them from simple causal links.”* 

Let’s see this in action in the following highly simplified model.  The model considers just two competing priorities: safety and business.  When performance in these categories differs from goals, pressure is created on management and may result in actions to ameliorate the pressure.  In this model management action is limited to shifting resources from one priority to the other.  Safety culture, per our June 29, 2012 post, is an organization’s ability to resist and then respond to competing priorities.  At time zero, a reduction in authorized budget is imposed resulting in a gap (current spending versus authorized spending) and creating business pressure on management to respond.

Figure 1

Figure 1 shows the response of management.  Actions are initiated very quickly and start to reduce safety resources to relieve budget pressure.  The plot tracks the initial response, a plateauing to allow effectiveness to be gauged, followed by escalation of action to further reduce the budget gap.

Figure 2

Figure 2 overlays the effect of the management actions on the budget gap and the business
pressure associated with the gap.  Immediately following the budget reduction, business pressure rapidly increases and quickly reaches a level sufficient to cause management to start to shift priorities.  The first set of management actions brings some pressure relief, the second set of actions further reduces pressure.  As expected there is some time lag in the response of business pressure to the actions of management.

Figure 3

In Figure 3, the impact of these changes in business pressure and management actions are
accumulated in the safety culture.  Note first the gradual changes that occur in culture versus the faster and sharper changes in management actions and business pressure.  As management takes action there is a loss of safety priority and safety culture slowly degrades. When further escalation of management action occurs it is at a point where culture is already lower, making the organization more susceptible to compromising safety priorities.  Safety culture declines further. This type of response is indicative of a feedback loop which is an important dynamic feature of the system.  Business pressure causes management actions, those actions degrade safety culture, degraded culture reduces resistance to further actions.

We invite comments and questions from our readers.


*  John Morecroft, Strategic Modelling and Business Dynamics (John Wiley & Sons, 2007) pp. 59-61.

Modeling Safety Culture (Part 2): Safety Culture as Pressure Boundary

No, this is not an attempt to incorporate safety culture into the ASME code.  As introduced in Part 1 we want to offer a relatively simple construct for safety culture - hoping to provide a useful starting point for a model of safety culture and a bridge between safety culture as amorphous values and beliefs, and safety culture that helps achieve desired balances in outcomes.

We propose that safety culture be considered “the willingness and ability of an organization to resist undue pressure on safety from competing business priorities”.  Clearly this is a 30,000 foot view of safety culture and does not try to address the myriad ways in which it materializes within the organization. This is intentional since there are so many possible moving parts at the individual level making it too easy to lose sight of the macro forces. 

The following diagram conceptualizes the boundary between safety priorities (i.e., safety culture) and other organizational priorities (business pressure).  The plotted line is essentially a threshold where the pressure for maintaining safety priorities (created by culture) may start to yield to increasing amounts of pressure to address other business priorities.  In the region to the left of the plot line, safety and business priorities exist in an equilibrium.  To the right of the line business pressure exceeds that of the safety culture and can lead to compromises.  Note that this construct supports the view that strong safety performance is consistent with strong overall performance.  Strong overall performance, in areas such as production, cost and schedule, ensure that business pressures are relatively low and in equilibrium with reasonably strong safety culture.  (A larger figure with additional explanatory notes is available here.)



The arc of the plot line suggests that the safety/business threshold increases (requires greater business pressure) as safety culture becomes stronger.  It also illustrates that safety priorities may be maintained even at lower safety culture strengths when there is little competing business pressure.  This aspect seems particularly consistent with determinations at certain plants that safety culture is “adequate” but still requires strengthening.  It also provides an appealing explanation for how complacency can over time erode a relatively strong safety culture . If overall performance is good, resulting in minimal business pressures, the culture might not be “challenged” or noticed even as culture becomes degraded.

Another perspective on safety culture as pressure boundary is what happens when business pressure elevates to a point where the threshold is crossed.  One reason that organizations with strong culture may be able to resist more pressure is a greater ability to manage business challenges that arise and/or a willingness to adjust business goals before they become overwhelming.  And even at the threshold such organizations may be better able to identify compensatory actions that have only minimal and short term safety impacts.  For organizations with weaker safety culture, the threshold may lead to more immediate and direct tradeoffs of safety priorities.  In addition, the feedback effects of safety compromises (e.g., larger backlogs of unresolved problems) can compound business performance deficiencies and further increase business pressure.  One possible insight from the pressure model is that in some cases, perceived safety culture issues may be more a situation of reasonably strong safety culture being over matched by excessive business pressures.  The solution may be more about relieving business pressures than exclusively trying to reinforce culture.

In Part 3 we hope to further develop this approach through some simple simulations that illustrate the interaction of managing resources and balancing pressures.  In the meantime we would like to hear reactions from readers to this concept.

The NRC Chairman, Acta Est Fabula

With today’s announcement the drama surrounding the Chairman of the NRC has played out to its foreseeable conclusion.  The merits of the Chairman’s leadership of the agency are beyond the scope of this blog, but there are a few aspects of his tenure that may be relevant to nuclear safety culture in high performing organizations, not to mention in high places.

First we should note that we have previously blogged about speeches and papers (here, here and here) given by the Chairman wherein he emphasized the importance of safety culture to nuclear safety.  In general we applauded his emphasis on safety culture as being necessary to raise the attention level of the industry.  Over time, as the NRC’s focus became absorbed with the Safety Culture Policy Statement we became less enamored with the Chairman’s satisfaction with achieving consensus among stakeholders as almost an end to itself.  The resultant policy statement with a heavy tilt to attitudes and values seemed to lack the kind of coherence that a regulatory agency needs to establish inspectable results.  As Commissioner Apostolakis so cogently observed, “...we really care about what people do and maybe not why they do it….”

Continuing with that thought, and if the assertions made by the four other Commissioners are accurate, what the Chairman’s did as agency head seems to have included intimidation, lack of transparency, manipulation of resources, and other behaviors not on the safety culture list of traits.  It illustrates, again, how easy it is for organizational leaders to mouth the correct words about safety culture yet behave in a contradictory manner.  We strongly suspect that this is another situation where the gravitational force of conflicting priorities - in this case a political agenda - was sufficient to bend the boundary line between strong leadership and self interest.

Going Beyond SCART: A More Useful Guidebook for Evaluating Safety Culture

Our March 11 post reviewed the IAEA SCART guidelines.  We found its safety culture characteristics and attributes comprehensive but its “guiding questions” for evaluators were thin gruel, especially in the areas we consider critical for safety culture: decision making, corrective action, work backlogs and management incentives.

This post reviews another document that combines the SCART guidelines, other IAEA documents and the author’s insights to yield a much more robust guidebook for evaluating a facility’s safety culture.  It’s called “Guidelines for Regulatory Assessment of Safety Culture in Licensees’ Organisations.”*  It starts with the SCART characteristics and attributes but gives more guidance to an evaluator: recommendations for documents to review, what to look for during the evaluation, additional (and more critical) guiding questions, and warning signs that can indicate safety culture weaknesses or problems.

Specific guidance in the areas we consider critical is generally more complete.  For example, in the area of decision making, evaluators are told to look for a documented process applicable to all matters that affect safety, attend meetings to observe the decision-making process, note the formalization of the decision making process and how/if long-term consequences of decisions are considered.  Goal conflict is explicitly addressed, including how differing opinions, conflict based on different experiences, and questioning attitudes are dealt with, and the evidence of fair and impartial methods to resolve conflicts.  Interestingly, example conflicts are not limited to the usual safety vs. cost or production but include safety vs. safety, e.g., a proposed change that would increase plant safety but cause additional personnel rad exposure to implement.  Evidence of unresolved conflicts is a definite warning flag for the evaluator. 

Corrective action (CA) also gets more attention, with questions and flags covering CA prioritization based on safety significance, the timely implementation of fixes, lack of CA after procedure violations or regulatory findings, verification that fixes are implemented and effective, and overall support or lack thereof for the CAP. 

Additional questions and flags cover backlogs in maintenance, corrective actions, procedure changes, unanalyzed physical or procedural problems, and training.

However, the treatment of management incentives is still weak, basically the same as the SCART guidelines.  We recommend a more detailed evaluation of the senior managers’ compensation scheme or, in more direct language, how much do they get paid for production, and how much for safety?

The intended audience for this document is a regulator charged with assessing a licensee’s safety culture.  As we have previously discussed, some regulatory agencies are evaluating this approach.  For now, that’s a no-go in the U.S.  In any case, these guidelines provide a good checklist for self-assessors, internal auditors and external consultants.


*  M. Tronea, “Guidelines for Regulatory Oversight of Safety Culture in Licensees’ Organisations” Draft, rev. 8 (Bucharest, Romania:  National Commission for Nuclear Activities Control [CNCAN], April 2011).  In addition to being on the staff of CNCAN, the nuclear regulatory authority of Romania, Dr. Tronea is the founder/manager of the LinkedIn Nuclear Safety group.  

Is Safety Culture An Inherently Stable System?

The short answer:  No.

“Stable” means that an organization’s safety culture effectiveness remains at about the same level* over time.  However, if a safety culture effectiveness meter existed and we attached it to an organization, we would see that, over time, the effectiveness level rises and falls, possibly even dropping to an unacceptable level.  Level changes occur because of shocks to the system and internal system dynamics.

Shocks

Sudden changes or challenges to safety culture stability can originate from external (exogenous) or internal (endogenous) sources.

Exogenous shocks include significant changes in regulatory requirements, such as occurred after TMI or the Browns Ferry fire, or “it’s not supposed to happen” events that do, in fact, occur, such as a large earthquake in Virginia or a devastating tsunami in Japan that give operators pause, even before any regulatory response.

Organizations have to react to such external events and their reaction is aimed at increasing plant safety.  However, while the organization’s focus is on its response to the external event, it may take its eye off the ball with respect to its pre-existing and ongoing responsibilities.  It is conceivable that the reaction to significant external events may distract the organization and actually lower overall safety culture effectiveness.

Endogenous shocks include the near-misses that occur at an organization’s own plant.  While it is unfortunate that such events occur, it is probably good for safety culture, at least for awhile.  Who hasn’t paid greater attention to their driving after almost crashing into another vehicle?

The insertion of new management, e.g., after a plant has experienced a series of performance or regulatory problems, is another type of internal shock.  This can also raise the level of safety culture—IF the new management exercises competent leadership and makes progress on solving the real problems. 

Internal Dynamics    

Absent any other influence, safety culture will not remain at a given level because of an irreducible tendency to decay.  Decay occurs because of rising complacency, over-confidence, goal conflicts, shifting priorities and management incentives.  Cultural corrosion, in the form of normalization of deviance, is always pressing against the door, waiting for the slightest crack to appear.  We have previously discussed these challenges here.

An organization may assert that its safety culture is a stability-seeking system, one that detects problems, corrects them and returns to the desired level.  However, performance with respect to the goal may not be knowable with accuracy because of measurement issues.  There is no safety culture effectiveness meter, surveys only provide snapshots of instant safety climate and even a lengthy interview-based investigation may not lead to repeatable results, i.e, a different team of evaluators might (or might not) reach different conclusions.  That’s why creeping decay is difficult to perceive. 

Conclusion

Many different forces can affect an organization’s safety culture effectiveness, some pushing it higher while others lower it.  Measurement problems make it difficult to know what the level is and the trend, if any.  The takeaway is there is no reason to assume that safety culture is a stable system whose effectiveness can be maintained at or above an acceptable level.


*  “Level” is a term borrowed from system dynamics, and refers to the quantity of a variable in a model.  We recognize that safety culture is an organizational property, not something stored in a tank, but we are using “level” to communicate the notion that safety culture effectiveness is something that can improve (go up) or degrade (go down).

Vit Plant Glop

Hanford WTP

DOE’s Waste Treatment Plant at Hanford, the “Vit Plant”, is being built to process a complex mixture of radioactive waste products from 1950s nuclear weapons production.  The wastes, currently in liquid form and stored in tanks at the site, was labeled “gorp” by William Mullins in one of his posts on the LinkedIn Nuclear Safety thread.*  Actually we think the better reference is to “glop”.  Glop is defined at merriam-webster.com as “a thick semiliquid substance (as food) that is usually unattractive in appearance”.  Readers should disregard the reference to food.  We would like to call attention to another source of “glop” accumulating at the Vit Plant.  It is the various reports by DOE and Hanford regarding safety culture at the site, most recently in response to the Defense Nuclear Facilities Safety Board’s (DNFSB, Board) findings in June 2011.  These forms of glop correspond more closely to the secondary definition in m-w, that is, “tasteless or worthless material”.

The specific reports are the DOE’s Implementation Plan (IP)** for the DFNSB’s review of safety culture at the WTP and the DOE's Office of Health, Safety and Security (HSS) current assessment of safety culture at the site.  Neither is very satisfying but we’ll focus on the IP in this post.

What may be most interesting in the DOE IP package are the reference documents including the DNFSB review and subsequent exchanges of letters between the Secretary of Energy and the DNFSB Chairman.  It takes several exchanges for the DNFSB to wrestle DOE into accepting the findings of the Board.  Recall in the Board’s original report it concluded:

“Taken as a whole, the investigative record convinces the Board that the safety culture at WTP is in need of prompt, major improvement and that corrective actions will only be successful and enduring if championed by the Secretary of Energy.” (IP, p. 33)***

In DOE’s initial (June 30, 2011) response they stated:

“Even while DOE fully embraces the objectives of the Board’s specific recommendations, it is important to note that DOE does not agree with all of the findings included in the Board’s report.”  (IP, p. 42)****

It goes on to state that “specifically” DOE does not agree with the conclusions regarding the overall quality of the safety culture.  Not surprisingly this brought the following response in the DNFSB’s August 12, 2011 letter, “...the disparity between the [DOE’s] stated acceptance and disagreement with the findings makes it difficult for the Board to assess the response….”  (IP, p. 46)*****  Note that in the body of the IP (p. 4) DOE does not acknowledge this difference of opinion either in the summary of its June 30 response or the Board’s August 12 rejoinder. 

We note that neither the DNFSB report nor the DOE IP is currently included among the references on Bechtel's Vit Plant website.  One can only wonder what the take away is for Vit Plant personnel — isn’t there a direct analogy between how DOE reacts to issues raised by the DNFSB and how Vit Plant management respond to issues raised at the plant?  Here’s an idea: provide a link to the safetymatters blog on the Vit Plant website.  Plant personnel will be able to access the IP, the DNFSB report and all of our informative materials and analysis.

In fact, reading all the references and the IP leave the impression that DOE believes there is no fundamental safety culture issue.  Their cause analysis focuses on inadequate expectation setting, more knowledge and awareness and (closer to the mark) the conflicting goals emerging in the construction phase (IP, pp. 5-8).  While endlessly citing all the initiatives previously taken or underway, never does the DOE reflect on why these initiatives have not been effective to date.  What is DOE’s answer?  More assessments and surveys, more training, more “guidance”, more expectations, etc.

We do find Actions 1-5 and 1-6 interesting (IP, p. 16).  These will revise the BNI contract to achieve “balanced priorities”.  This is important and a good thing.  We have blogged about the prevalence of large financial incentives for nuclear executives in the commercial nuclear industry and assessments of most, if not all, other significant safety events (BP gulf disaster, BP refinery fire, Upper Big Branch coal mine explosion, etc.) highlight the presence of goal conflicts.  How one balances priorities is another thing and a challenge.  We have blogged extensively on this subject - search on “incentives” to identify all relevant posts.  In particular we have noted that where safety goals are included in incentives they tend to be based on industrial safety which is not very helpful to the issues at hand.  Our favorite quote comes from our April 7, 2011 post re the gulf oil rig disaster and is taken from Transocean’s annual report:

“...notwithstanding the tragic loss of life in the Gulf of Mexico, we [Transocean] achieved an exemplary statistical safety record as measured by our total recordable incident rate (‘‘TRIR’’) and total potential severity rate (‘‘TPSR’’).”

Our advice for the Vit Plant would be as follows.  In terms of expectations, enforcing rather than setting, might be the better emphasis.  Then monitoring and independently assessing how specific technical and safety issues are reviewed and decided.  Training, expectations setting, reinforcement, policies, etc. are useful in “setting the table” but the test of whether the organization is embracing and implementing a strong safety culture can only be found in its actions.  Note that the Board’s June 2011 report focused on two specific examples of deficient decision processes and outcomes.  (One, the determination of the appropriate deposition velocity for analysis of the transport of radioactivity, the other the conservatism of a criticality analysis.)

There are two aspects of decisions: the process and the result.  The process includes the ability to freely raise safety concerns, the prioritization and time required to evaluate such issues, and the treatment of individuals who raise such concerns.  The result is the strength of the decision reached; i.e., do the decisions reinforce a strong safety culture?  We have posted and provided examples on the blog website of decision assessment using some methods for quantitative scoring.

*  The link to the thread is here.  Search for "gorp" to see Mr. Mullins' comment.

**  U.S. Dept. of Energy, “Implementation Plan for Defense Nuclear Facilities Safety Board Recommendation 2011-1, Safety Culture at the Waste Treatment and Immobilization Plant”  (Dec. 2011).

***  IP Att. 1, DNFSB Recommendation 2011-1, “Safety Culture at the Waste Treatment and Immobilization Plant” (June 9, 2011).

****  IP Att. 2, Letter from S. Chu to P.S. Winokur responding to DNFSB Recommendation 2011-1 (June 30, 2011) p. 4.

*****  IP Att. 4, Letter from P.S. Winokur  to S. Chu responding to Secretary Chu’s June 30, 2011 letter (Aug. 12, 2011) p. 1.

Nuclear Industry Complacency: Root Causes

NRC Chairman Jaczko, addressing the recent INPO CEO conference, warned about possible increasing complacency in the nuclear industry.*  To support his point, he noted the two plants in column four of the ROP Action Matrix and two plants in column three, the increased number of special inspections in the past year, and the three units in extended shutdowns.  The Chairman then moved on to discuss other industry issues. 

The speech spurred us to ask: Why does the risk of complacency increase over time?  Given our interest in analyzing organizational processes, it should come as no surprise that we believe complacency is more complicated than the lack of safety-related incidents leading to reduced attention to safety.

An increase in complacency means that an organization’s safety culture has somehow changed.  Causes of such change include shifts in the organization’s underlying assumptions and decay.

Underlying Assumptions

We know from the Schein model that underlying assumptions are the bedrock for culture.  One can take those underlying assumptions and construct an (incomplete) mental model of the organization—what it values, how it operates and how it makes decisions.  Over time, as the organization builds an apparently successful safety record, the mental weights that people assign to decision factors can undergo a subtle but persistent shift to favor the visible production and cost goals over the inherently invisible safety factor.  At the same time, opportunities exist for corrosive issues, e.g., normalization of deviance, to attach themselves to the underlying assumptions.  Normalization of deviance can manifest anywhere, from slipping maintenance standards to a greater tolerance for increasing work backlogs.

Decay

An organization’s safety culture will inevitably decay over time absent effective maintenance.  In part this is caused by the shift in underlying assumptions.  In addition, decay results from saturation effects.  Saturation occurs because beating people over the head with either the same thing, e.g., espoused values, or too many different things, e.g., one safety program or similar intervention after another, has lower and lower marginal effectiveness over time.  That’s one reason new leaders are brought in to “problem” plants: to boost the safety culture by using a new messenger with a different version of the message, reset the decision making factor weights and clear the backlogs.

None of this is new to regular readers of this blog.  But we wanted to gather our ideas about complacency in one post.  Complacency is not some free-floating “thing,” it is an organizational trait that emerges because of multiple dynamics operating below the level of clear visibility or measurement.  

     
*  G.B. Jaczko, Prepared Remarks at the Institute of Nuclear Power Operations CEO Conference, Atlanta, GA (Nov. 10, 2011), p. 2, ADAMS Accession Number ML11318A134.

Regulatory Assessment of Safety Culture—Not Made in U.S.A.

Last February, the International Atomic Energy (IAEA) hosted a four-day meeting of regulators and licensees on safety culture.*  “The general objective of the meeting [was] to establish a common opinion on how regulatory oversight of safety culture can be developed to foster safety culture.”  In fewer words, how can the regulator oversee and assess safety culture?

While no groundbreaking new methods for evaluating a nuclear organization’s safety culture were presented, the mere fact there is a perception that oversight methods need to be developed is encouraging.  In addition, outside the U.S., it appears more likely that regulators are expected to engage in safety culture oversight if not formal regulation.

Representatives from several countries made presentations.  The NRC presentation discussed the then-current status of the effort that led to the NRC safety culture policy statement announced in June.  The presentations covering Belgium, Bulgaria, Indonesia, Romania, Switzerland and Ukraine described different efforts to include safety culture assessment into licensee evaluations.

Perhaps the most interesting material was a report on an attendee survey** administered at the start of the meeting.  The survey covered “national regulatory approaches used in the oversight of safety culture.” (p.3) 18 member states completed the survey.  Following are a few key findings:

The states were split about 50-50 between having and not having regulatory requirements related to safety culture. (p. 7)  The IAEA is encouraging regulators to get more involved in evaluating safety culture and some countries are responding to that push.

To minimize subjectivity in safety culture oversight, regulators try to use oversight practices that are transparent,  understandable, objective, predictable, and both risk-informed and performance-based. (p. 13)  This is not news but it is a good thing; it means regulators are trying to use the same standards for evaluating safety culture as they use for other licensee activities.

Licensee decision-making processes are assessed using observations of work groups, probabilistic risk analysis, and during the technical inspection. (p. 15)  This seems incomplete or even weak to us.  In-depth analysis of critical decisions is necessary to reveal the underlying assumptions (the hidden, true culture) that shape decision-making.

Challenges include the difficulty in giving an appropriate priority to safety in certain real-time decision making situations and the work pressure in achieving production targets/ keeping to the schedule of outages. (p. 16)  We have been pounding the drum about goal conflict for a long time and this survey finding simply confirms that the issue still exists.

Bottom Line

The meeting was generally consistent with our views.  Regulators and licensees need to focus on cultural artifacts, especially decisions and decision making, in the short run while trying to influence the underlying assumptions in the long run to reduce or eliminate the potential for unexpected negative outcomes.

*  IAEA Technical Meeting on Safety Culture Oversight and Assessment, Vienna, Feb. 15-18, 2011.

**  A. Kerhoas, "Synthesis of Questionnaire Survey."

Ultimate Bonuses

Just when you think there is a lack of humor in the exposition of dry, but critical issues, such as risk management, our old friend Nicholas Taleb comes to the rescue.*  His op-ed piece in the New York Times** earlier this week has a subdued title, “End Bonuses for Bankers”, but includes some real eye-openers.  For example Taleb cites (with hardly concealed admiration) the ancient Hammurabi code which protected home owners by calling for the death of the home builder if the home collapsed and killed the owner.  Wait, I thought we were talking about bonuses, not capital punishment.

What Taleb is concerned about is that bonus systems in entities that pose systemic risks almost universally encourage behaviors that may not be consistent with the public good much less the long term health of the business entity.  In short he believes that bonuses provide an incentive to take risks.***  He states, “The asymmetric nature of the bonus (an incentive for success without a corresponding disincentive for failure) causes hidden risks to accumulate in the financial system and become a catalyst for disaster.”  Now just substitute “nuclear operations” for “the financial system”. 

Central to Taleb’s thesis is his belief that management has a large informational advantage over outside regulators and will always know more about risks being taken within their operation.  It affords management the opportunity to both take on additional risk (say to meet an incentive plan goal) and to camouflage the latent risk from regulators.

In our prior posts [here, here and here] on management incentives within the nuclear industry, we also pointed to the asymmetry of bonus metrics - the focus on operating availability and costs, the lack of metrics for safety performance, and the lack of downside incentive for failure to meet safety goals.  The concern was amplified due to the increasing magnitude of nuclear executive bonuses, both in real terms and as a percentage of total compensation. 

So what to do?  Taleb’s answer for financial institutions too big to fail is “bonuses and bailouts should never mix”; in other words, “end bonuses for bankers”.  Our answer is, “bonuses and nuclear safety culture should never mix”; “end bonuses for nuclear executives”.  Instead, gross up the compensation of nuclear executives to include the nominal level of expected bonuses.  Then let them manage nuclear operations using their best judgment to assure safety, unencumbered by conflicting incentives.


*  Taleb is best known for The Black Swan, a book focusing on the need to develop strategies, esp. financial strategies, that are robust in the face of rare and hard-to-predict events.

**  N. Taleb, “End Bonuses for Bankers,” New York Times website (Nov. 7, 2011).

*** It is widely held that the 2008 financial crisis was exacerbated, if not caused, by executives making more risky decisions than shareholders would have thought appropriate. Alan Greenspan commented: “I made a mistake in presuming that the self-interests of organizations, specifically banks and others, were such that they were best capable of protecting their own shareholders” (Testimony to Congress, quoted in A. Clark and J. Treanor, “Greenspan - I was wrong about the economy. Sort of,” The Guardian, Oct. 23, 2008). The cause is widely thought to be the use of bonuses for performance combined with limited liability.  See also J.M. Malcomson, “Do Managers with Limited Liability Take More Risky Decisions? An Information Acquisition Model”, Journal of Economics & Management Strategy, Vol. 20, Issue 1 (Spring 2011), pp. 83–120.

An Anthropologist’s View

Academics in many disciplines study safety culture.  This post introduces to this blog the work of an MIT anthropologist, Constance Perin, and discusses a paper* she presented at the 2005 ANS annual meeting.

We picked a couple of the paper’s key recommendations to share with you.  First, Perin’s main point is to advocate the development of a “significance culture” in nuclear power plant organizations.  The idea is to organize knowledge and data in a manner that allows an organization to determine significance with respect to safety issues.  The objective is to increase an organization’s capabilities to recognize and evaluate questionable conditions before they can escalate risk.  We generally agree with this aim.  The real nub of safety culture effectiveness is how it shapes the way an organization responds to new or changing situations.

Perin understands that significance evaluation already occurs in both formal processes (e.g., NRC evaluations and PRAs) and in the more informal world of operational decisions, where trade-offs, negotiations, and satisficing behavior may be more dynamic and less likely to be completely rational.  She recommends that significance evaluation be ascribed a higher importance, i.e., be more formally and widely ingrained in the overall plant culture, and used as an organizing principle for defining knowledge-creating processes. 

Second, because of the importance of a plant's Corrective Action Program (CAP), Perin proposes making NRC assessment of the CAP the “eighth cornerstone” of the Reactor Oversight Process (ROP).  She criticizes the NRC’s categorization of cross cutting issues for not being subjected to specific criteria and performance indicators.  We have a somewhat different view.  Perin’s analysis does not acknowledge that the industry places great emphasis on each of the cross cutting issues in terms of performance indicators and monitoring including self assessment.**  It is also common to the other cornerstones where the plants use many more indicators to track and trend performance than the few included in the ROP.  In our opinion, a real problem with the ROP is that its few indicators do not provide any reliable or forward looking picture of nuclear safety. 

The fault line in the CAP itself may better be characterized in terms of the lack of measurement and assessment of how well the CAP program functions to sustain a strong safety culture.  Importantly such an approach would evaluate how decisions on conditions adverse to quality properly assessed not only significance, but balanced the influence of any competing priorities.  Perin also recognizes that competing priorities exist, especially in the operational world, but making the CAP a cornerstone might actually lead to increased false confidence in the CAP if its relationship with safety culture was left unexamined.

Prof. Perin has also written a book, Shouldering Risks: The Culture of Control in the Nuclear Power Industry,*** which is an ethnographic analysis of nuclear organizations and specific events they experienced.  We will be reviewing this book in a future post.  We hope that her detailed drill down on those events will yield some interesting insights, e.g., how different parts of an organization looked at the same situation but had differing evaluations of its risk implications.

We have to admit we didn’t detect Prof. Perin on our radar screen; she alerted us to the presence of her work.  Based on our limited review to date, we think we share similar perspectives on the challenges involved in attaining and maintaining a robust safety culture.


*  C. Perin, “Significance Culture in Nuclear Installations,” a paper presented at the 2005 Annual Meeting of the American Nuclear Society (June 6, 2005).

** The issue may be one of timing.  Prof. Perin based her CAP recommendation, in part, on a 2001 study that suggested licensees’ self-regulation might be inadequate.  We have the benefit of a more contemporary view.  

*** C. Perin, Shouldering Risks: The Culture of Control in the Nuclear Power Industry, (Princeton, NJ: Princeton University Press, 2005).

Decisions….Decisions

Safety Culture Performance Measures

Developing forward looking performance measures for safety culture remains a key challenge today and is the logical next step following the promulgation of the NRC’s policy statement on safety culture.  The need remains high as safety culture issues continue to be identified by the NRC subsequent to weaknesses developing in the safety culture and ultimately manifesting in traditional (lagging) performance indicators.

Current practice has continued to rely on safety culture surveys which focus almost entirely on attitudes and perceptions about safety.  But other cultural values are also present in nuclear operations - such as meeting production goals - and it is the rationalization of competing values on a daily basis that is at the heart of safety culture.  In essence decision makers are pulled in several directions by these competing priorities and must reach answers that accord safety its appropriate priority.

Our focus is on safety management decisions made every day at nuclear plants; e.g., operability, exceeding LCO limits, LER determinations, JCOs, as well as many determinations associated with problem reporting, and corrective action.  We are developing methods to “score” decisions based on how well they balance competing priorities and to relate those scores to inference of safety culture.  As part of that process we are asking our readers to participate in the scoring of decisions that we will post each week - and then share the results and interpretation.  The scoring method will be a more limited version of our developmental effort but should illustrate some of the benefits of a decision-centric view of safety culture.

Look in the right column for the links to Score Decisions.  They will take you to the decision summaries and score cards.  We look forward to your participation and welcome any questions or comments.

DNFSB Goes Critical

Hanford WTP

The Defense Nuclear Facilities Safety Board (DNFSB)issued a “strongly worded” report* this week on safety culture at the Hanford Waste Treatment and Immobilization Plant (WTP).  The DNFSB determined that the safety culture at the WTP is “flawed” and “that both DOE and contractor project management behaviors reinforce a subculture at WTP that deters the timely reporting, acknowledgement, and ultimate resolution of technical safety concerns.”

For example, the Board found that “expressions of technical dissent affecting safety at WTP, especially those affecting schedule or budget, were discouraged, if not opposed or rejected without review” and heard testimony from several witnesses that “raising safety issues that can add to project cost or delay schedule will hurt one's career and reduce one's participation on project teams.”

Only several months ago we blogged about initiatives by DOE regarding safety culture at its facilities.  In our critique we observed, “Goal conflict, often expressed as safety vs mission, should obviously be avoided but its insidiousness is not adequately recognized [in the DOE initiatives]."  Seems like the DNFSB put their finger on this at WTP.  In fact the DNFSB report states:

“The HSS [DOE's Office of Health, Safety and Security] review of the safety culture on the WTP project 'indicates that BNI [Bechtel National Inc.] has established and implemented generally effective, formal processes for identifying, documenting, and resolving nuclear safety, quality, and technical concerns and issues raised by employees and for managing complex technical issues.'  However, the Board finds that these processes are infrequently used, not universally trusted by the WTP project staff, vulnerable to pressures caused by budget or schedule [emphasis added], and are therefore not effective.” 

The Board was not done with goal conflict. It went on to cite the experience of a DOE expert witness:

“The testimony of several witnesses confirms that the expert witness was verbally admonished by the highest level of DOE line management at DOE's debriefing meeting following this session of the hearing.  Although testimony varies on the exact details of the verbal interchange, it is clear that strong hostility was expressed toward the expert witness whose testimony strayed from DOE management's policy while that individual was attempting to adhere to accepted professional standards.”

This type of intimidation need not be, and generally is not, so explicit. The same message can be sent through many subtle and insidious channels which are equally effective.  It is goal conflict of another stripe - we refer to it as “organizational stress” - where the organizational interests of individuals - promotions, performance appraisals, work assignments, performance incentives, etc. - create another dimension of tension in achieving safety priority.  It is just as real and a lot more personal than the larger goal conflicts of cost and schedule pressures.

*  Defense Nuclear Facilities Safety Board, Recommendation 2011-1 to the Secretary of Energy "Safety Culture at the Waste Treatment and Immobilization Plant" (Jun 9, 2011).

Another Brick in the Wall for BP et al

Yesterday the National Academy of Engineering released their report* on the Deepwater Horizon blowout.  The report includes a critical appraisal of many decisions made during the period when the well was being prepared for temporary abandonment, decisions that in the aggregate decreased safety margins and increased risks.  This Washington Post article** provides a good summary of the report.

The report was written by engineers and scientists and has a certain “Just the facts, ma’am” tone.  It does not specifically address safety culture.  But we have to ask: What can one infer about a culture where the business practices don’t include “any standard practice . . . to guide the tradeoffs between cost and schedule and the safety implications of the many decisions (that is, a risk management approach).”  (p. 15)

We have had plenty to say about BP and the Deepwater Horizon accident.  Click on the BP label below to see all of our related blog entries.

*  Committee for the Analysis of Causes of the Deepwater Horizon Explosion, Fire, and Oil Spill to Identify Measures to Prevent Similar Accidents in the Future; National Academy of Engineering; National Research Council, “Interim Report on Causes of the Deepwater Horizon Oil Rig Blowout and Ways to Prevent Such Events” (2010).

**  D. Cappiello, “Experts: BP ignored warning signs on doomed well,” The Washington Post (Nov 17, 2010).  Given our blog’s focus on the nuclear industry, it’s worth noting that, in an interview, the committee chairman said, “the behavior leading up to the oil spill would be considered unacceptable in companies that work with nuclear power or aviation.”

Human Beings . . . Conscious Decisions

In a  New York Times article* dated November 8, 2010, there was a headline to the effect that Fred Bartlit, the independent investigator for the presidential panel on the BP oil rig disaster earlier this year had not found that “cost trumped safety” in decisions leading up to the accident.  The article noted that this finding contradicted determinations by other investigators including those sponsored by Congress.  We had previously posted on this subject, including taking notice of the earlier findings of cost trade-offs, and wanted to weigh in based on this new information.

First we should acknowledge that we have no independent knowledge of the facts associated with the blowout and are simply reacting to the published findings of current investigations.  In our prior posts we had posited that cost pressures could be part of the equation in the leadup to the spill.  On June 8, 2010 we observed:

“...it is clear that the environment leading up to the blowout included fairly significant schedule and cost pressures. What is not clear at this time is to what extent those business pressures contributed to the outcome. There are numerous cited instances where best practices were not followed and concerns or recommendations for prudent actions were brushed aside. One wishes the reporters had pursued this issue in more depth to find out ‘Why?’ ”

And we recall one of the initial observations made by an OSHA official shortly after the accident as detailed in our April 26, 2010 post:

“In the words of an OSHA official BP still has a ‘serious, systemic safety problem’ across the company.”

So it appears we have been cautious in reaching any conclusions about BP’s safety management.  That said, we do want to put into context the finding by Mr. Bartlit.  First we would note that he is, by profession, a trial lawyer and may be both approaching the issue and articulating his finding with a decidedly legal focus.  The specific quotes attributed to him are as follows:

“. . . we have not found a situation where we can say a man had a choice between safety and dollars and put his money on dollars” and “To date we have not seen a single instance where a human being made a conscious decision to favor dollars over safety,...”

It is not surprising that a lawyer would focus on culpability in terms of individual actions.  When things go wrong, most industries, nuclear included, look to assign blame to individuals and move on.  It is also worth noting that the investigator emphasized that no one had made a “conscious” decision to favor cost over safety.  We think it is important to keep in mind that safety management and failures of safety decision making may or may not involve conscious decisions.  As we have stated many times in other posts, safety can be undermined through very subtle mechanisms such that even those involved may not appreciate the effects, e.g., the normalization of deviance.  Finally we think the OSHA investigator may have been closer to the truth with his observation about “systemic” safety problems.  It may be that Mr. Bartlit, and other investigators, will be found to have suffered from what is termed “attribution error” where simple explanations and causes are favored and the more complex system-based dynamics are not fully assessed or understood in the effort to answer “Why?”  

* J.M. Broder, "Investigator Finds No Evidence That BP Took Shortcuts to Save Money," New York Times (Nov 8, 2010).

NRC Safety Culture Workshop

The information from the Sept 28, 2010 NRC safety culture meeting is available on the NRC website.  This was a meeting to review the draft safety culture policy statement, definition and traits.

As you probably know, the NRC definition now focuses on organizational “traits.”   According to the NRC, “A trait . . . is a pattern of thinking, feeling, and behaving that emphasizes safety, particularly in goal conflict situations, e.g., production vs. safety, schedule vs. safety, and cost of the effort vs. safety.”*  We applaud this recognition of goal conflicts as potential threats to effective safety management and a strong safety culture.

Several stakeholders made presentations at the meeting but the most interesting one was by INPO’s Dr. Ken Koves.**  He reported on a study that addressed two questions:

• “How well do the factors from a safety culture survey align with the safety culture traits that were identified during the Feb 2010 workshop?
• Do the factors relate to other measures of safety performance?” (p. 4)

The rest of this post summarizes and critiques the INPO study.

Methodology

For starters, INPO constructed and administered a safety culture survey.  The survey itself is interesting because it covered 63 sites and had 2876 respondents, not just a single facility or company.  They then performed a principal component analysis to reduce the survey data to nine factors.  Next, they mapped the nine survey factors against the safety culture traits from the NRC's Feb 2010 workshop, INPO principles, and Reactor Oversight Program components and found them generally consistent.  We have no issue with that conclusion. 

Finally, they ran correlations between the nine survey factors and INPO/NRC safety-related performance measures.  I assume the correlations included in his presentation are statistically significant.  Dr. Koves concludes that “Survey factors are related to other measures of organizational effectiveness and equipment performance . . . .” (p. 19)

The NRC reviewed the INPO study and found the “methods, data analyses and interpretations [were] appropriate.” ***

The Good News

Kudos to INPO for performing this study.  This analysis is the first (only?) large-scale attempt of which I am aware to relate safety culture survey data to anything else.  While we want to avoid over-inferring from the analysis, primarily because we have neither the raw data nor the complete analysis, we can find support in the correlation tables for things we’ve been saying for the last year on this blog.

For example, the factor with the highest average correlation to the performance measures is Management Decision Making, i.e., what management actually does in terms of allocating resources, setting priorities and walking the talk.  Prioritizing Safety, i.e., telling everyone how important it is and promulgating safety policies, is 7th (out of 9) on the list.  This reinforces what we have been saying all along: Management actions speak louder than words.

Second, the performance measures with the highest average correlation to the safety culture survey factors are the Human Error Rate and Unplanned Auto Scrams.  I take this to indicate that surveys at plants with obvious performance problems are more likely to recognize those problems.  We have been saying the value of safety culture surveys is limited, but can be more useful when perception (survey responses) agrees with reality (actual conditions).  Highly visible problems may drive perception and reality toward congruence.  For more information on perception vs. reality, see Bob Cudlin’s recent posts here and here.

Notwithstanding the foregoing, our concerns with this study far outweigh our comfort at seeing some putative findings that support our theses.

Issues and Questions

The industry has invested a lot in safety culture surveys and they, NRC and INPO have a definite interest (for different reasons) in promoting the validity and usefulness of safety culture survey data.  However, the published correlations are moderate, at best.  Should the public feel more secure over a positive safety culture survey because there's a "significant" correlation between survey results and some performance measures, some of which are judgment calls themselves?  Is this an effort to create a perception of management, measurement and control in a situation where the public has few other avenues for obtaining information about how well these organizations are actually protecting the public?

More important, what are the linkages (causal, logical or other) between safety culture survey results and safety-related performance data (evaluations and objective performance metrics) such as those listed in the INPO presentation?  Most folks know that correlation is not causation, i.e., just because two variables move together with some consistency doesn’t mean that one causes the other but what evidence exists that there is any relationship between the survey factors and the metrics?  Our skepticism might be assuaged if the analysts took some of the correlations, say, decision making and unplanned reactor scrams, and drilled into the scrams data for at least anecdotal evidence of how non-conservative decision making contributed to x number of scrams. We would be surprised to learn that anyone has followed the string on any scram events all the way back to safety culture.

Wrapping Up

The INPO analysis is a worthy first effort to tie safety culture survey results to other measures of safety-related performance but the analysis is far too incomplete to earn our endorsement.  We look forward to seeing any follow-on research that addresses our concerns.


*  “Presentation for Safety Club Public Meeting - Traits Comparison Charts,” NRC Public Meeting, Las Vegas, NV (Sept 28, 2010) ADAMS Accession Number ML102670381, p. 4.

**  G.K. Koves, “Safety Culture Traits Validation in Power Reactors,” NRC Public Meeting, Las Vegas, NV (Sept 28, 2010).

***  V. Barnes, “NRC Independent Evaluation of INPO’s Safety Culture Traits Validation Study,” NRC Public Meeting, Las Vegas, NV (Sept 28, 2010) ADAMS Accession Number ML102660125, p. 8.

Survival of the Safest

One of our goals with SafetyMatters is bringing thought provoking materials to our readers, particularly materials they might not otherwise come across.  This post is an example from the greater business world and the current state of the U.S. economy.  Once again it is based on some interesting research from professors at Yale University* and described in an article in the New York Times.**

“Corporate managers struggling to preserve their companies and protect their core employees have inadvertently contributed to a vicious cycle of rising unemployment and plummeting national morale. If we are to break out of this downward spiral, we first need to understand the problem…professional managers throughout the business world see it as their job to keep work-force morale high. But, paradoxically, the actions they take for their own workplaces often make the overall crisis more severe.”

These issues have been the subject of research by Yale economics professor Truman Bewley.  While his specific focus is on labor markets and how wages respond (or don’t respond) to periods of reduced demand, some of the insights channel directly into the current issues of safety culture at nuclear plants. 

Bewley’s approach was to interview hundreds of corporate managers at length about the driving forces for their actions.  The article goes on to describe how corporate managers respond to recessions by protecting their most important staff, but paradoxically these actions tend to produce unforeseen and often counter-productive results. 

The description of how actions result in unintended consequences is emblematic of the complexity of business systems, where dynamics and interdependencies are not always seen or understood by the managers tasked with achieving results.  Nuclear safety culture exists in such a complex socio-technical system and requires more than just “leadership” to assure long term sustainability. 

This brings us to the first part of Dr. Bewley’s approach - his focus on identifying and understanding the driving forces for managers’ actions.  We see this as precisely the right prescription for improving our understanding of nuclear safety culture dynamics, particularly in cases where safety culture weaknesses have been observed.  A careful and penetrating look at why people don’t act in accordance with safety culture principles would do much to identify the types of factors, such as performance incentives, cost and schedule pressures, etc. that may be at work in an organization.  Driving forces are not necessarily different from root causes - a term more familiar in the nuclear industry - but I tend to prefer it because it explicitly reminds us that safety culture is dynamic, and results from the interaction of many moving parts.  Currently the focus of the industry, and the NRC for that matter, is on safety culture “traits”.  Traits are really the results or manifestations of safety culture and thus build out the picture of what is desired.  But they do not get at what factors actually produce strong safety culture in the first place.

As an example we refer you to a comment we posted on a Nuclear Safety Culture group thread on LinkedIn.com.  Dr. Bill Corcoran initiated a thread asking for proposals of safety culture traits that were at least as important as those in the NRC strawman.  Our response proposed:

 “The compensation structure in the corporation is aligned with its safety priorities and does not create real or perceived conflicts in decisions affecting nuclear safety.” ***

While this was proposed as a “trait” in response to Bill’s request, it is clearly a driving force that will enable and support strong safety culture behaviors and decisions.

* To read about other interesting work at Yale, check out our August 30, 2010 post.

** Robert J. Shiller, "The Survival of the Safest," New York Times (Oct 2, 2010).

*** The link to the thread (including Bob's comment) is here.  This may be difficult for readers who are not LinkedIn members to access.  We are not promoting LinkedIn but the Nuclear Safety Culture group has some interesting commentary.

BP's New Safety Division

It looks like oil company BP believes that creating a new, “global” safety division is part of the answer to their ongoing safety performance issues including most recently the explosion of Deepwater Horizon oil rig in the Gulf of Mexico.  An article in the September 29, 2010 New York Times* quotes BP’s new CEO as stating “safety and risk management [are] our most urgent priority” but does not provide many details of how the initiative will accomplish its goal.  Without seeming to jump to conclusions, it is hard for us to see how a separate safety organization is the answer although BP asserts it will be “powerful”. 

Of more interest was a lesser headline in the article with the following quote from BP’s new CEO:

“Mr. Dudley said he also plans a review of how BP creates incentives for business performance, to find out how it can encourage staff to improve safety and risk management.”

We see this as one of the factors that is a lot closer to the mark for changing behaviors and priorities.  It parallels recent findings by FPL in its nuclear program (see our July 29, 2010 post) and warning flags that we had raised in our July 6 and July 9, 2010 posts regarding trends in U.S. nuclear industry compensation.  Let’s see which speaks the loudest to the organization: CEO pronouncements about safety priority or the large financial incentives that executives can realize by achieving performance goals.  If they are not aligned, the new “division of safety” will simply mean business as usual.

*  The original article is available via the iCyte below.  An updated version is available on the NY Times website.

Games Theory

In the September 15, 2010 New York Times there is an interesting article* about the increasing recognition within school environments that game-base learning has great potential.  We cite this article as further food for thought about our initiatives to bring simulation-based games to training for nuclear safety management.

The benefits of using games as learning spaces is based on the insight that games are systems, and systems thinking is really the curriculum, bringing a nuanced and rich way of looking at real world situations. 

“Games are just one form of learning from experience. They give learners well-designed experiences that they cannot have in the real world (like being an electron or solving the crisis in the Middle East). The future for learning games, in my view, is in games that prepare people to solve problems in the world.” **

“A game….is really just a “designed experience,” in which a participant is motivated to achieve a goal while operating inside a prescribed system of boundaries and rules.” ***  The analogy in nuclear safety management is to have the game participants manage a nuclear operation - with defined budgets and performance goals - in a manner that achieves certain safety culture attributes even as achievement of those attributes comes into conflict with other business needs.  The game context brings an experiential dimension that is far more participatory and immersive than traditional training environments.  In the NuclearSafetySim simulation, the players’ actions and decisions also feedback into the system, impacting other factors such as  organizational trust and the willingness of personnel to identify deviations.  Experiencing the loss of trust in the simulation is likely to be a much more powerful lesson than simply the admonition to “walk the talk” burned into a Powerpoint slide.

* Sara Corbett, "Learning by Playing: Video Games in the Classroom," New York Times (Sep 15, 2010).

** J.P. Gee, "Part I: Answers to Questions About Video Games and Learning," New York Times (Sep 20, 2010).

*** "Learning by Playing," p. 3 of retrieved article.