Safety Management and Competitiveness

Jean-Marie Rousseau

We recently came across a paper that should be of significant interest to nuclear safety decision makers.  “Safety Management in a Competitiveness Context” was presented in March 2008 by Jean-Marie Rousseau of the Institut de Radioprotection et de Surete Nucleaire (IRSN).  As the title suggests the paper examines the effects of competitive pressures on a variety of nuclear safety management issues including decision making and the priority accorded safety.  Not surprisingly:

“The trend to ignore or to deny this phenomenon is frequently observed in modern companies.” (p. 7)

The results presented in the paper came about from a safety assessment performed by IRSN to examine safety management of EDF [Electricite de France] reactors including:

“How real is the ‘priority given to safety’ in the daily arbitrations made at all nuclear power plants, particularly with respect to the other operating requirements such as costs, production, and radiation protection or environmental constraints?” (p. 2)

The pertinence is clear as “priority given to safety” is the linchpin of safety culture policy and expected behaviors.  In addition the assessment focused on decision-making processes at both the strategic and operational levels.  As we have argued, decisions can provide significant insights into how safety culture is operationalized by nuclear plant management. 

Rousseau views nuclear operations as a “highly complex socio-technical system” and his paper provides a brief review of historical data where accidents or near misses displayed indications of the impact of competing priorities on safety.  The author notes that competitiveness is necessary just as is safety and as such it represents another risk that must be managed at organizational and managerial levels.  This characterization is intriguing and merits further reflection particularly by regulators in their pursuit of “risk informed regulation”.  Nominally regulators apply a conceptualization of risk that is hardware and natural phenomena centric.  But safety culture and competitive pressures also could be justified as risks to assuring safety - in fact much more dynamic risks - and thus be part of the framework of risk informed regulation.*  Often, as is the case with this paper, there is some tendency to assert that achievement of safety is coincident with overall performance excellence - which in a broad sense it is - but notwithstanding there are many instances where there is considerable tension - and potential risk.

Perhaps most intriguing in the assessment is the evaluation of EDF’s a posteriori analyses of its decision making processes as another dimension of experience feedback.**   We quote the paper at length:

“The study has pointed out that the OSD***, as a feedback experience tool, provides a priori a strong pedagogic framework for the licensee. It offers a context to organize debates about safety and to share safety representations between actors, illustrated by a real problematic situation. It has to be noticed that it is the only tool dedicated to “monitor” the safety/competitiveness relationship.

"But the fundamental position of this tool (“not to make judgment about the decision-maker”) is too restrictive and often becomes “not to analyze the decision”, in terms of results and effects on the given situation.

"As the existence of such a tool is judged positively, it is necessary to improve it towards two main directions: - To understand the factors favouring the quality of a decision-making process. To this end, it is necessary to take into account the decision context elements such as time pressure, fatigue of actors, availability of supports, difficulties in identifying safety requirements, etc.
- To understand why a “qualitative decision-making process” does not always produce a “right decision”. To this end, it is necessary to analyze the decision itself with the results it produces and the effects it has on the situation.” (p. 8)

We feel this is a very important aspect that currently receives insufficient attention.  Decisions can provide a laboratory of safety management performance and safety culture actualization.  But how often are decisions adequately documented, preserved, critiqued and shared within the organization?  Decisions that yield a bad (reportable) result may receive scrutiny internally and by regulators but our studies indicate there is rarely sufficient forensic analysis - cause analyses are almost always one dimensional and hardware and process oriented.  Decisions with benign outcomes - whether the result of “good” decision making or not - are rarely preserved or assessed.  The potential benefits of detailed consideration of decisions have been demonstrated in many of the independent assessments of accidents (Challenger, Columbia, BP Texas Oil Refinery, etc.) and in research by Perin and others. 

We would go a step further than proposed enhancements to the OSD.  As Rousseau notes there are downsides to the routine post-hoc scrutiny of actual decisions - for one it will likely identify management errors even in the absence of a bad decision outcome.  This would be one more pressure on managers already challenged by a highly complex decision environment.  An alternative is to provide managers the opportunity to “practice” making decisions in an environment that supports learning and dialogue on achieving the proper balances in decisions - in other words in a safety management simulator.  The industry requires licensed operators to practice operations decisions on a simulator for similar reasons - why not nuclear managers charged with making safety decisions?


*  As the IAEA has noted, “A danger of concentrating too much on a quantitative risk value that has been generated by a PSA [probabilistic safety analysis] is that...a well-designed plant can be operated in a less safe manner due to poor safety management by the operator.”  IAEA-TECDOC-1436, Risk Informed Regulation of Nuclear Facilities: Overview of the Current Status, February 2005.

**  EDF implemented safety-availability-Radiation-Protection-environment observatories (SAREOs) to increase awareness of the arbitration between safety and other performance factors. SAREOs analyze in each station the quality of the decision-making process and propose actions to improve it and to guarantee compliance with rules in any circumstances [“Nuclear Safety: our overriding priority” EDF Group‟s file responding to FTSE4Good nuclear criteria] 

***  Per Rousseau, “The OSD (Observatory for Safety/Availability) is one of the “safety management levers” implemented by EDF in 1997. Its objective is to perform retrospective analyses of high-stake decisions, in order to improve decision-making processes.” (p. 7)

High Reliability Organizations and Safety Culture

On February 10^th, we posted about a report covering lessons for safety culture (SC) that can be gleaned from the social science literature. The report's authors judged that high reliability organization (HRO) literature provided a solid basis for linking individual and organizational assumptions with traits and practices that can affect safety performance. This post explores HRO characteristics and how they can influence SC.

Our source is Managing the Unexpected: Resilient Performance in an Age of Uncertainty* by Karl Weick and Kathleen Sutcliffe. Weick is a leading contemporary HRO scholar. This book is clearly written, with many pithy comments, so lots of quotations are included below to present the authors' views in their own words.

What makes an HRO different?

Many organizations work with risky technologies where the consequences of problems or errors can be catastrophic, use complex management systems and exist in demanding environments. But successful HROs approach their work with a different attitude and practices, an “ongoing mindfulness embedded in practices that enact alertness, broaden attention, reduce distractions, and forestall misleading simplifications.” (p. 3)

Mindfulness

An underlying assumption of HROs is “that gradual . . . development of unexpected events sends weak signals . . . along the way” (p. 63) so constant attention is required. Mindfulness means that “when people act, they are aware of context, of ways in which details differ . . . and of deviations from their expectations.” (p. 32) HROs “maintain continuing alertness to the unexpected in the face of pressure to take cognitive shortcuts.” (p. 19) Mindful organizations “notice the unexpected in the making, halt it or contain it, and restore system functioning.” (p. 21)

It takes a lot of energy to maintain mindfulness. As the authors warn us, “mindful processes unravel pretty fast.” (p. 106) Complacency and hubris are two omnipresent dangers. “Success narrows perceptions, . . . breeds overconfidence . . . and reduces acceptance of opposing points of view. . . . [If] people assume that success demonstrates competence, they are more likely to drift into complacency, . . .” (p. 52) Pressure in the task environment is another potential problem. “As pressure increases, people are more likely to search for confirming information and to ignore information that is inconsistent with their expectations.” (p. 26) The opposite of mindfulness is mindlessness. “Instances of mindlessness occur when people confront weak stimuli, powerful expectations, and strong desires to see what they expect to see.” (p. 88)

Mindfulness can lead to insight and knowledge. “In that brief interval between surprise and successful normalizing lies one of your few opportunities to discover what you don't know.” (p. 31)**

Five principles

HROs follow five principles. The first three cover anticipation of problems and the remaining two cover containment of problems that do arise.

Preoccupation with failure

HROs “treat any lapse as a symptom that something may be wrong with the system, something that could have severe consequences if several separate small errors happened to coincide. . . . they are wary of the potential liabilities of success, including complacency, the temptation to reduce margins of safety, and the drift into automatic processing.” (p. 9)

Managers usually think surprises are bad, evidence of bad planning. However, “Feelings of surprise are diagnostic because they are a solid cue that one's model of the world is flawed.” (p. 104) HROs “Interpret a near miss as danger in the guise of safety rather than safety in the guise of danger. . . . No news is bad news. All news is good news, because it means that the system is responding.” (p. 152)

People in HROs “have a good sense of what needs to go right and a clearer understanding of the factors that might signal that things are unraveling.” (p. 86)

Reluctance to simplify

HROs “welcome diverse experience, skepticism toward received wisdom, and negotiating tactics that reconcile differences of opinion without destroying the nuances that diverse people detect. . . . [They worry that] superficial similarities between the present and the past mask deeper differences that could prove fatal.” (p. 10) “Skepticism thus counteracts complacency . . . .” (p. 155) “Unfortunately, diverse views tend to be disproportionately distributed toward the bottom of the organization, . . .” (p. 95)

The language people use at work can be a catalyst for simplification. A person may initially perceive something different in the environment but using familiar or standard terms to communicate the experience can raise the risk of losing the early warnings the person perceived.

Sensitivity to operations

HROs “are attentive to the front line, . . . Anomalies are noticed while they are still tractable and can still be isolated . . . . People who refuse to speak up out of fear undermine the system, which knows less than it needs to know to work effectively.” (pp. 12-13) “Being sensitive to operations is a unique way to correct failures of foresight.” (p. 97)

In our experience, nuclear plants are generally good in this regard; most include a focus on operations among their critical success factors.

Commitment to resilience

“HROs develop capabilities to detect, contain, and bounce back from those inevitable errors that are part of an indeterminate world.” (p. 14) “. . . environments that HROs face are typically more complex than the HRO systems themselves. Reliability and resilience lie in practices that reduce . . . environmental complexity or increase system complexity.” (p. 113) Because it's difficult or impossible to reduce environmental complexity, the organization needs to makes its systems more complex.*** This requires clear thinking and insightful analysis. Unfortunately, actual organizational response to disturbances can fall short. “. . . systems often respond to a disturbance with new rules and new prohibitions designed to present the same disruption from happening in the future. This response reduces flexibility to deal with subsequent unpredictable changes.” (p. 72)

Deference to expertise.

“Decisions are made on the front line, and authority migrates to the people with the most expertise, regardless of their rank.” (p. 15) Application of expertise “emerges from a collective, cultural belief that the necessary capabilities lie somewhere in the system and that migrating problems [down or up] will find them.” (p. 80) “When tasks are highly interdependent and time is compressed, decisions migrate down . . . Decisions migrate up when events are unique, have potential for very serious consequences, or have political or career ramifications . . .” (p. 100)

This is another ideal that can fail in practice. We've all seen decisions made by the highest ranking person rather than the most qualified one. In other words, “who is right” can trump “what is right.”

Relationship to safety culture

Much of the chapter on culture is based on the ideas of Schein and Reason so we'll focus on key points emphasized by Weick and Sutcliffe. In their view, “culture is something an organization has [practices and controls] that eventually becomes something an organization is [beliefs, attitudes, values].” (p. 114, emphasis added)

“Culture consists of characteristic ways of knowing and sensemaking. . . . Culture is about practices—practices of expecting, managing disconfirmations, sensemaking, learning, and recovering.” (pp. 119-120) A single organization can have different types of culture: an integrative culture that everyone shares, differentiated cultures that are particular to sub-groups and fragmented cultures that describe individuals who don't fit into the first two types. Multiple cultures support the development of more varied responses to nascent problems.

A complete culture strives to be mindful, safe and informed with an emphasis on wariness. As HRO principles are ingrained in an organization, they become part of the culture. The goal is a strong SC that reinforces concern about the unexpected, is open to questions and reporting of failures, views close calls as a failure, is fearful of complacency, resists simplifications, values diversity of opinions and focuses on imperfections in operations.

What else is in the book?

One chapter contains a series of audits (presented as survey questions) to assess an organization's mindfulness and appreciation of the five principles. The audits can show an organization's attitudes and capabilities relative to HROs and relative to its own self-image and goals.

The final chapter describes possible “small wins” a change agent (often an individual) can attempt to achieve in an effort to move his organization more in line with HRO practices, viz., mindfulness and the five principles. For example, “take your team to the actual site where an unexpected event was handled either well or poorly, walk everyone through the decision making that was involved, and reflect on how to handle that event more mindfully.” (p. 144)

The book's case studies include an aircraft carrier, a nuclear power plant,**** a pediatric surgery center and wildland firefighting.

Our perspective

Weick and Sutcliffe draw on the work of many other scholars, including Constance Perin, Charles Perrow, James Reason and Diane Vaughan, all of whom we have discussed in this blog. The book makes many good points. For example, the prescription for mindfulness and the five principles can contribute to an effective context for decision making although it does not comprise a complete management system. The authors' recognize that reliability does not mean a complete lack of performance variation, instead reliability follows from practices that recognize and contain emerging problems. Finally, there is evidence of a systems view, which we espouse, when the authors say “It is this network of relationships taken together—not necessarily any one individual or organization in the group—that can also maintain the big picture of operations . . .” (p. 142)

The authors would have us focus on nascent problems in operations, which is obviously necessary. But another important question is what are the faint signals that the SC is developing problems? What are the precursors to the obvious signs, like increasing backlogs of safety-related work? Could that “human error” that recently occurred be a sign of a SC that is more forgiving of growing organizational mindlessness?

Bottom line: Safetymatters says check out Managing the Unexpected and consider adding it to your library.

* K.E. Weick and K.M. Sutcliffe, Managing the Unexpected: Resilient Performance in an Age of Uncertainty, 2d ed. (San Francisco, CA: Jossey-Bass, 2007). Also, Wikipedia has a very readable summary of HRO history and characteristics.

** More on normalization and rationalization: “On the actual day of battle naked truths may be picked up for the asking. But by the following morning they have already begun to get into their uniforms.” E.A. Cohen and J. Gooch, Military Misfortunes: The Anatomy of Failure in War (New York: Vintage Books, 1990), p. 44, quoted in Managing the Unexpected, p. 31.

*** The prescription to increase system complexity to match the environment is based on the system design principle of requisite variety which means “if you want to cope successfully with a wide variety of inputs, you need a wide variety of responses.” (p. 113)

**** I don't think the authors performed any original research on nuclear plants. But the studies they reviewed led them to conclude that “The primary threat to operations in nuclear plants is the engineering culture, which places a higher value on knowledge that is quantitative, measurable, hard, objective, and formal . . . HROs refuse to draw a hard line between knowledge that is quantitative and knowledge that is qualitative.” (p. 60)

Inhibiting Excessive Risk Taking by Executives

The Federal Reserve Board has been doing some arm twisting with U.S. financial services companies to adjust their executive compensation plans - and those plans are in fact being modified to cap bonuses associated with achieving performance goals.  These actions have their genesis in the financial crisis where it appeared that incentives could encourage excessive risk taking by management.  A Wall Street Journal article* notes “regulators are still looking at ways to lower risk in the banking system, even if it means interfering with private pay practices.”  This follows a similar trend in Europe and where some firms are considering increasing salaries to make up for less bonus potential.

These actions merit some thoughtful consideration within the U.S. nuclear industry.  While the Fed’s concern is excessive business risk, the analog in nuclear operations is safety risk.  Both go to ensuring that the “system” (banking or nuclear production) remain within controlled limits.  As we have noted in prior blog posts (July 6 and July 9, 2010), there have been trends for nuclear executive compensation to both escalate and include significant performance based components.  The increased salaries probably reflect competition for the best qualified executives and are indicative of the great responsibilities of nuclear management.  However the trend to include large short-term bonuses (comprising up to 60-70% of total compensation) may be indicative of the evolution of “nuclear generation as a business” and the large profit potential available at high capacity factors.  Whatever the nominal amount of pressure on nuclear executives to achieve operating goals, the presence of very large monetary incentives can only increase that pressure.  In a strong safety culture environment where perception of management’s priorities is central, incentive based compensation plans can easily create presumptions regarding the motivation for management decisions.  At least one nuclear utility has concluded that incentives were not appropriate and taken action to adjust their compensation plans.  We have advocated dialing back incentives in favor of more direct compensation.

It is also rather interesting that the Fed decided to step into the province of private compensation practices.  A similar initiative by the NRC seems unlikely given its reluctance to impinge on management performance in any manner.  As noted in our February 28, 2013 post the NRC has included some nominal but poorly focused language on incentives in its Safety Culture Common Language Path Forward.  This seems to indicate that the NRC believes incentives are or could be relevant.  The best approach may be for the NRC to become more intrusive - to determine if compensation plans have the potential to lead to excessive risk taking.  This would require the NRC to obtain compensation plan information from its licensees, characterize the extent and magnitude of performance based incentives, and consider the effect of such incentives in assessing specific operational issues that arise in its normal regulatory oversight activities.  Only if some relationship appeared would the NRC need to consider whether to take action similar to the Fed or other means to ameliorate risk taking.


 
*  A.Lucchetti and J. Steinberg, "Regulators Get Banks to Rein In Bonus Pay," Wall Street Journal (April 23, 2013).

IAEA on Safety Culture in New Plant Design and Construction

The International Atomic Energy Agency (IAEA) has a 2012 publication* that provides guidance on establishing a strong safety culture (SC) during the design and construction of new nuclear power plants.  The report's premise, with which we agree, is a weak SC during plant design and construction can lead to later problems during plant operations.   

Major issues can arise during plant design and construction.  For example, the numerous organizations involved may have limited direct experience and/or insufficient knowledge of nuclear safety requirements, or projects may be located in countries with no existing nuclear industry or countries may have a nuclear industry but no recent construction experience.

The report attempts to cover the different needs, challenges and circumstances that may face project participants (governments, regulators, owners, designers, builders, manufacturers, etc.) anywhere in the world.  Most of the content addresses generic issues, e.g., understanding SC, the role of leadership, appropriate management systems, or communication and organizational learning.  Each issue is discussed in terms of specific challenges, goals, and recommended approaches and methods.  However, in their effort to attain maximum coverage (scope) IAEA sacrifices depth.  For example, the discussion of leadership covers five pages of the report but scarcely mentions the two most important activities of leaders: decision making and modeling safety-focused behavior.

If we look at the report's specific advice and recommendations, we see uneven coverage of the observable artifacts we consider essential for a strong SC: a decision making process that appropriately values safety, an effective corrective action program and financial incentives that reward safety performance. 

Decision making process
 
One overall challenge facing new projects is “Conflicts between schedule, cost and safety objectives can adversely affect conservative decision making and the maintenance of a questioning attitude, or impair the ability to perceive links between short term actions and their long term consequences.” (p. 2)

That's a good starting point but what are the characteristics of an appropriate decision making process?  It seems that decision making should be “conservative” (pp. 32, 34, 39), “broad” (p. 43) and “risk informed” (pp. 50, 51) but the terms are not defined.  More specificity on how the decision making process should handle competing goals, set priorities and assign resources would be useful.
 
What about the decision makers?  Leaders should be able to “Explain the relationships between time periods/horizons and decision making to help resolve competing priorities.” (p. 41)  That's OK but the need goes beyond time periods.  The manager must be able to explain the rationale for significant decisions related to safety.  What were the considerations, assumptions, priorities, alternatives, decision factors and their relative weights, and the applicable laws, rules and regulations?  How should leaders treat devil’s advocates who raise concerns about possible unfavorable outcomes?  Do leaders get the most qualified people involved in key decisions?
 
In addition, leaders should “Simulate decision making in a fast paced, complex environment to help leaders identify risks in their own approaches.”  (p. 41)  This is an excellent approach and we wholeheartedly support it. 

Corrective action

“Ineffective problem identification, inadequate reporting and inadequate corrective actions” (p. 9) were identified in a 2006 investigation as causal factors of construction problems at a nuclear plant site.  But there is no follow-up to describe the characteristics of an effective corrective action program.  There should be more about the CAP's ability to recognize and diagnose problems, formulate and implement solutions that consistently and appropriately consider safety, and monitor the effects of corrective actions. The importance of robust cause analysis, i.e., analysis that finds the real causes of problems so they do not recur, should be mentioned.  This would not be an unreasonable level of detail for this general report.

Financial incentives

The report correctly notes that “In construction environments, cultural attributes such as schedule awareness, cost focus and urgency of problem resolution are reinforced because they are rewarded by immediate measures of success.” (p. 8)  This becomes a specific challenge when “Contractor incentives are often driven by cost and schedule rather than by safety culture performance.” (p. 26).  A recommended fix is to “Establish a reward and incentive programme [sic] for the overall project, with objectives for safety performance and rewards that are either monetary or in the form of future contracts as a long term partner.” (p. 27).  This will probably result in a focus on industrial safety performance rather than the overall SC but it may be the best practical solution.  Periodic assessment of key contractors' SC should be used to identify any general SC issues. 

Our perspective

In prior posts, we have taken the IAEA to task for their overly bureaucratic approach.  So we're pleased to report this document actually provides some useful, sensible guidance (albeit often in an unprioritized, laundry list style) applicable to both countries initially embarking on the nuclear road and more experienced countries experiencing a nuclear renaissance.

The report makes a few important points.  For example, IAEA proposes a systems approach to thinking about all the project participants and the varied work they must accomplish.  “In the case of NPP projects, the ‘system’ involves human–social systems, work processes, complex technologies and multiple organizations in a global economic, energy, environmental and regulatory context.” (p. 11)  This is a viewpoint we have repeatedly advocated in this blog. 

Overall this report is satisfactory and it does refer the reader to other IAEA publications for additional information on specific subjects.  But in trying to provide relevant material to a plethora of stakeholders, the report gives shorter shrift to factors we consider vital to establishing and maintaining a strong SC. 


*  M. Haage (IAEA), “Safety culture in pre-operational phases of nuclear power plant projects” (Vienna : International Atomic Energy Agency,  2012).

Warning Shot for Chevron

White vapor and black smoke.  From CSB report.

On August 6, 2012 a leaking pipe at the Chevron refinery in Richmond, CA led to a fire that shut down a crude oil distillation unit and caused over 15,000 people to report to local hospitals seeking treatment for respiratory and other health issues.  This was not a Texas City.  About 20 of the 15,000 people were admitted to local hospitals and there were some minor injuries to employees in the area of the fire but no fatalities.  However, it should be a wake-up call for Chevron. 

The proximate cause of the leak was a pipe ("4-sidecut") that had corroded because of the fluids that flowed through it.  But the Chevron and Chemical Safety Board (CSB) investigations showed there was a ten-year trail of missed possibilities to identify and correct the problem, including the following: In 2002, an employee inspector had expressed concern about sulfidation corrosion in the 4-sidecut and recommended upgrading it but his recommendation was never implemented.  In the same year, an incident at another Chevron refinery led the company to recommend 100% inspection of pipes for corrosion but this was not implemented at Richmond.  In 2009 and 2010 Chevron promulgated new warnings about sulfur corrosion and reiterated the recommendation for 100% inspection but Richmond did not implement any remedial actions on the 4-sidecut.*  In 2011, after a fire in another pipe, Richmond employees complained to Cal/OSHA about the company ignoring corrosion dangers but Chevron rationalized their way out of the issue.**

Chevron's incident investigation, including a root cause analysis, resulted in multiple corrective actions that will ring familiar to our readers.  Summarized, they are: look harder for corrosion; upgrade the hardware reliability program and supporting procedures; increase oversight and training; implement new rules for evaluating leaks; and emphasize the importance of process safety in decision making.  In even fewer words, tweak the system and retrain.

There is no mention of safety culture (SC) but the odor of a weak or compromised SC wafts from the report.  In a strong SC, the 2002 inspector would have identified the potential problem, documented it in the corrective action program and monitored progress until the issue was resolved.  The corrective action program would have evaluated, prioritized and resourced the problem's resolution consistent with its safety significance.  Outside experience and directives (from other Chevron entities or elsewhere) would have been regularly integrated into local operating practices, including inspection, maintenance and process procedures.

We are not alone in recognizing the importance of SC.  The local county supervisor, who also chairs the Bay Area Air Quality Management District, said “We need to do a thorough review of the safety culture at the refinery.”***  The CSB's managing director said the company had a “tolerance for allowing piping to run toward failure” and “I think it points to a certain cultural issue.”****  The CSB's interim report says “After reviewing evidence and decisions . . . the CSB has determined that issues relating to safety culture are relevant to this incident. The CSB will examine the Chevron Richmond Refinery’s approach to safety, its safety culture and any organizational deficiencies, to determine how to best prevent future incidents.” (p. 61)

We'll see if Chevron gets the hint.


*  CUSA Richmond Investigation Team, “Richmond Refinery 4 Crude Unit Incident August 6, 2012” (April 12, 2013).  Attachment to letter from S. Wildman (Chevron) to R.L. Sawyer (Contra Costa County Health Services), “Seventh Update to the 30-Day Report for the CWS Level 3 Event of August 6, 2012” (April 12, 2013).  

U.S. Chemical Safety and Hazard Investigation Board, “Interim Investigation Report Chevron Richmond Refinery Fire” (April 15, 2013).  In addition to Chevron, the CSB also criticizes regulatory and other government agencies, particularly Cal/OSHA, for shortcomings in their oversight of refinery activities.

**  J. Van Derbeken, “Chevron ignored risk in '11, workers say” sfgate.com (Oct. 13, 2012).

***  J. Van Derbeken, “Chevron fire report shows troubled history” sfgate.com (April 13, 2013).

****  J. Van Derbeken, “Chevron fire sign of weak oversight” sfgate.com (April 15, 2013).

A New Sheriff Coming to DOE?

On April 9th, the nominee for Secretary of Energy, Dr. Ernest Moniz, appeared before the Senate Committee on Energy and Natural Resources.  Most of the three hour hearing was in a Q&A format, with the committee chairman showing special interest in the major problems at Hanford, viz., leaking waste storage tanks and explosive hydrogen accumulation in same, the Waste Treatment Plant (aka Vit Plant) project and the site safety culture (SC).*

With respect to the SC issue, the nominee said it was “unacceptable” for SC to not be where it needs to be.  In response to a question from the committee chairman, Dr. Moniz said he was willing to meet with Vit Plant whistleblowers.  Depending on the outcome of such a meeting, if it occurs, the new Secretary could send a powerful signal to the Hanford site and beyond about his views on SC, Differing Professional Opinion (and related) practices, a Safety Conscious Work Environment and retaliation against employees who question organizational decisions.


*  The meeting video is available here, Hanford is discussed from about 3:05 to 3:20. 

A letter from the DNFSB chairman provides a good summary of the key issues at Hanford.  See P.S. Winokur (DNFSB) to R.L. Wyden (chairman of the Senate Committee on Energy and Natural Resources), letter providing the DNFSB's perspective on the state of nuclear safety at the Hanford Site (April 1, 2013).

How Do You Actually Implement the NRC’s Safety Culture Policy Statement?

As we all know the NRC issued a policy statement on safety culture almost two years ago, presumably to set expectations for the industry to maintain strong safety cultures.  The policy statement is long on fuzzy things like traits and values but unfortunately short on specific “what do we do’s”.  The biggest hint may be in the invocation to accord safety issues the priority warranted by their significance.  Sounds right in theory but how does this actually get operationalized?

We are going to suggest a specific approach to apply safety culture policy in day-to-day nuclear management decision making.  As we’ve argued many times, focusing on specific decisions moves safety from the realm of theory to practice.  In general we think there may be significant advantages for organizations to do more to highlight, document and measure decisions involving safety considerations.  The benefit will be insight and emphasis on the “how and why” decisions are made and whether they achieve the safety balance envisioned by the NRC - and more importantly by the organization’s leadership.

We start with a numeric scale for what we call “decision quality”.  In our minds quality means how well a decision balances the priority accorded safety in light of the significance of the issue being addressed; in other words how well the decision does what the policy statement asks.  Conceptually this implies that an optimal decision achieves just the right balance for safety and that other decisions could under or over-shoot the optimal balance.  Can there be too much priority for safety?  Sure - remember the goal is to perform the nuclear mission safely, not to just pursue safety itself.  Here is the scale:

The scale sets a 0 value as the measure of an optimal balance of safety - meaning that it meets the expectation of the policy statement to give the priority warranted by safety significance.  Increasing positive values are associated with decisions that accord extra weight to safety; increasing negative values accord too little.  Use of a quantitative scale is the first step in being able to grade, track and provide feedback on decisions on a consistent basis.  When coupled with discussion of how significance was assessed and what the appropriate safety response needs to be, it provides many opportunities for a check and adjust process and organizational learning.

This leads to the next question which is: how should significance be determined?  There is of course NRC guidance via the significance determination process (SDP), including the red, yellow, white and green rainbow of significance levels, and this is our starting point.  The SDP include both qualitative (e.g. significant reduction in safety margin) and quantitative criteria (e.g. values of delta CDF and delta LERF).  While qualitative criteria may seem to some as lacking specificity, we’re fine with their use and in any event they are endemic in safety regulation.  We’re actually not that fond of the quantitative criteria since they are inherently hardware centric and do not encompass the complexity of the overall “system” that ultimately determines safety.  To provide quantification our approach is to again create a scale that correlates numeric values with the qualitative criteria.  Such “anchored scales” are a common and effective tool in decision analysis.  In addition we feel that the significance determinations need to be supplemented with an assessment of their uncertainty. 

It doesn’t take reviewing many event reports to see that judgments about safety significance are not always clear cut or unambiguous.  This variability in the adjudged significance can be the enabling mechanism for safety to not receive the appropriate priority - not because the priority doesn’t match the significance but because the significance has been discounted to justify a lower priority.  The catalysts can be as simple as overly optimistic thinking, normalization of deviation, complacency, or failing to ensure that the burden is on showing that something is safe versus showing that it is unsafe.  Our approach is to explicitly address the uncertainty of safety significance by introducing a second quantitative scale for this purpose.  When used together a judgment regarding significance would include both a nominal value (per SDM) and an uncertainty value.  These scales are illustrated below:

For decision making purposes the three scales would operate together to help arrive at appropriate decisions.  The significance scale would provide a nominal risk value.  If there was a little uncertainty in the assessed significance then the objective would be to make a decision that scores approximately “0” on the balance scale.  If there was greater uncertainty in the assessed significance the objective would be to select a decision option that scored higher on the balance scale; essentially giving safety higher priority to accommodate the potentially greater significance.  Decision options that rated negative balance values would avoided.

We see much of the value in this approach to be the focus on making the decision formation process more explicit, transparent and measurable.  Over time this structure provides greater opportunities for the organization to understand decisions and learn from the process not just the outcomes.  We also believe it may provide the basis for inferring and trending the safety culture within an organization.

In an upcoming post we’ll apply these decision scales to a specific plant situation to see how they might work in practice.

2012 NRC Safety Culture Survey Results

ADAMS ML13087A326

Originally published 4-4-13.  This version updated with data from the associated staff briefing slide presentation.

The 2012 NRC Safety Culture and Climate Survey results are available in an Inspector General report* and a consultants' slide briefing.**  The top-level findings are (1) the 2012 results are not as favorable as the previous 2009 survey results and (2) the NRC compares favorably with national norms but lags when compared to a group of high performing companies (with strong financial results and high employee survey scores).  Let's look at some of the details.  All page references are to the report except where noted otherwise. 

The survey's 132 items were aimed at evaluating employee perceptions in 20 categories.  Many of these categories primarily addressed personnel practices—communication, supervision, diversity, training, development and the like.  However, it should come as no surprise to our regular readers that the categories of interest to us address, at least in part, the key business processes of decision making, priority setting and conflict resolution, i.e., areas where the goal of safety often competes with other goals.

Four categories appear to satisfy our criteria:

DPO/Non-Concurrence (DPO): “. . . employee awareness and perceived effectiveness of the Differing Professional Opinions program and the Non-concurrence process.”  This is one type of conflict resolution.  This category had the lowest number of favorable responses (although still over 50%) in the survey. (p. 13)  Three specific DPO items were among those that showed the most slippage, i.e., had fewer favorable responses, in 2012 compared to 2009. (Slides, p. 16)  Region IV had significantly*** fewer favorable 2012 scores on DPO compared to 2009. (p. 26) 

The consultants' cover letter identified this as an area for NRC management attention, saying the agency was “Losing significant ground on negative reactions when raising views different from senior management, supervisor, and peers.” 

NRC Mission and Strategic Plan: “. . . whether employees believe management decisions are consistent with the mission and strategic plan. . . .”    Compared with the high performing companies, the NRC scored 1 point lower on NRC Mission and Strategic Plan. (p. 17)  The Office of New Reactors and Region IV had significantly fewer favorable 2012 scores on NRC Mission and Strategic Plan compared to 2009. (pp. 25-26) 

The NRC Mission and Strategic Plan was identified as one of three key drivers of employee Engagement, also a survey category**** but treated as a dependent variable in a supporting multiple regression analysis.  In responding to specific questions, employees said they believed they were “sufficiently informed about NRC's performance of its mission” and that “management decisions are consistent with the mission” but both items scored significantly lower than in 2009 and compared to the high performing companies. (Slides, p. 33)

Quality focus: “. . . employee views on . . . the sacrifice of quality work due to the need to meet a deadline or the need to satisfy a personal or political agenda.”  This category had the third lowest number of favorable responses in the survey. (p. 13)  This category was also mentioned in the consultants' cover letter: “Reinforcing a key point raised in the focus groups [but one that did not stand out in the survey results], there is a clear opportunity to impact the perception that people sacrifice quality in order to meet metrics.”

Senior Management: “. . . confidence in management’s decisions.”  Compared with the high performing companies, the NRC scored 7 points lower on Senior Management, in a 3-way tie for second lowest. (p. 17)  This result may have been affected by this item: Only 41 percent of the respondents “. . . feel significant actions have been taken as a result of the previous Safety Culture and Climate survey.” (Slides, p. 23)  This issue was included in the list of conclusions to the consultants' report.  On the other hand, at least 75% favorable responses were recorded for senior management providing a clear sense of direction and employee confidence in senior management decisions. (Slides, p. 23)  That may look good but both items scored significantly lower than in 2009 and compared to the high performing companies.

The Office of New Reactors and Region IV had significantly less favorable 2012 scores on Senior Management compared to 2009. (pp. 25-26)  Region IV also had a significantly less favorable 2012 score on Senior Management than the overall NRC score. (p. 23)

Our Perspective

The report consists of mostly charts and graphs, with a lot of superficial data slicing and dicing and some authoritative-sounding conclusions.  The slide presentation shows additional data to illustrate some problem areas.  Both documents reinforce our belief in the limited usefulness of surveys and the problems associated with over-reliance on outside experts.  My “analysis” above is obviously limited but it's difficult to dig deeply because only a few of the 132 specific items are detailed in the report and slides. 

But the available data suggest that raising views inconsistent with the party line can lead to negative reactions.  NRC employees have some confidence the agency makes decisions consistent with its mission but less confidence in their senior management to take action on survey results.   

NRC senior management has a much more favorable view of the agency's situation than the overall organization.  Senior managers' survey responses were significantly more favorable than the overall NRC response in ALL 20 categories and an average of 18 percent more favorable in the 4 categories included in this post. (Slides, p. 37)  This suggests a possible disconnect between the bosses and everyone else.

And speaking of disconnections, it appears neither the group responsible for the Nuclear Renaissance nor Region IV is fully on the same page as the rest of the agency.

Finally, the documents' omission of safety as a goal or priority is notable.  “Nuclear safety” as a goal is only mentioned in the definition of SC.  Safety is mentioned as “safety concepts” in the Training category and the “NRC’s commitment to public safety” in the Continuous Improvement Commitment category.  One might expect safety to be more front and center in the SC survey. 


*  NRC Office of the Inspector General, “2012 NRC Safety Culture and Climate Survey,” OIG-13-A-15 (March 28, 2013)  ADAMS ML13087A326.  Although this was mostly a survey, the consultants (Towers Watson) did conduct some individual interviews and focus groups to help shape the survey content. Interestingly, the definition of safety culture used in the 2012 survey was not the same as the definition in the current NRC policy statement.  Instead, an earlier definition was used to permit comparisons between current survey results and prior years.

**  Towers Watson, “Nuclear Regulatory Commission 2012 Safety Culture and Climate Survey Briefing for NRC Staff” (Nov. 8, 2012).

***  “Significant” means statistically significant.

****  Engagement “Probes employees’ willingness to recommend the NRC as a good place to work, whether they feel they are a part of the agency, their pride in working for the NRC and their belief in NRC goals, objectives, and values.” (p. 10)