An article in the New York Times* earlier this week caught our attention as part of our contemplation of the causes of safety culture issues and effectiveness. The article itself is about the increasing incidence of misconduct by scientists in their research and publications, particularly in scientific journals. There may in fact be a variety of factors that are responsible, including just the sheer accessibility of journal published research and the increased opportunity that errors will be spotted. But the main thrust of the article is that other more insidious forces may be responsible:
“But other forces are more pernicious. To survive professionally, scientists feel the need to publish as many papers as possible….And sometimes they cut corners or even commit misconduct to get there.”
The article goes on to describe how in the scientific community the ability to publish is key to professional recognition, advancement and award of grant money. There is enormous pressure to publish first and publish often to overcome “cutthroat competition”.
So how do retractions of scientific papers relate to nuclear safety culture? In the most general sense the presence and impact of “pressure” on scientists reminds us of the situation in nuclear generation - now very much a high stakes business - and the consequent pressure on nuclear managers to meet business goals and in some cases, personal compensation goals. Nuclear personnel (engineers, managers, operators, craftsmen, etc.), like the scientists in this article, are highly trained and expected to observe certain cultural norms; a strong safety culture is expected. For scientists there is adherence to the scientific method itself and the standards for integrity of their peer community. Yet both may be compromised when the desire for professional success becomes dominant.
The scientific environment is in most ways much simpler than a nuclear operating organization and this may help shed light on the causes of normative failures. Nuclear organizations are inherently large and complex. The consideration of culture often becomes enmeshed in issues such as leadership, communications, expectations, pronouncements regarding safety priorities, perceptions, SCWE, etc. In the simpler scientific world, scientists are essentially sole proprietors of their careers, even if they work for large entities. They face challenges to their advancement and viability, they make choices, and sometimes they make compromises. Can reality in the nuclear operating environment be similar, or is nuclear somehow unique and different?
The promulgation of the NRC’s safety culture policy statement and industry efforts to remain out in front of regulatory scrutiny have led to increasing attention to identifying safety culture issues and achieving a consistently strong safety culture.
The typical scenario for the identification of safety culture problems starts with performance deficiencies of one sort or another, identified by the NRC through the inspection process or internally through various quality processes. When the circumstances of the deficiencies suggest that safety culture traits, values or behaviors are involved, safety culture may be deemed in need of strengthening and a standard prescription is triggered. This usually includes the inevitable safety culture assessment, retraining, re-iteration of safety priorities, re-training in safety culture principles, etc. The safety culture surveys focus on perceptions of problems and organizational “hot spots” but rarely delve deeply into underlying causes. Safety culture surveys generate anecdotal data based on the perceptions of individuals, primarily focused on whether safety culture traits are well established but generally not focused on asking “why” there are deficiencies.
This approach to safety culture seems to us to suffer from several limitations. One is that the standard prescription does not necessarily yield improved, sustainable results, an indication that symptoms are being treated instead of causes. And therein is the source of the other limitation, a lack of explicit consideration of the possible causes that have led to safety culture being deficient. The standard prescribed fixes include an implicit presumption that safety culture issues are the result of inadequate training, insufficient reinforcement of safety culture values, and sometimes the catchall of “leadership” shortcomings.
We think there are a number of potential causes that are important to ensuring strong safety culture but are not receiving the explicit attention they deserve. Whatever the true causes we believe that there will be multiple causes acting in a systematic manner - i.e., causes that interact and feedback in complex combinations to either reinforce or erode the safety culture state. For now we want to use this post to highlight the need to think more about the reasons for safety culture problems and whether a “causal chain” exists. Nuclear safety relies heavily on the concept of root causes as a means to understand the origin of problems and a belief that “fix-the-root cause” will “fix-the-problem”. But a linear approach may not be effective in understanding or addressing complex organizational dynamics, and concerted efforts in one dimension may lead to emergent issues elsewhere.
In upcoming posts we’ll explore specific causes of safety culture performance and elicit readers’ input on their views and experience.
Things have gone from bad to worse for Fort Calhoun. The plant shut down in April 2011 for refueling, but the shutdown was extended to address various issues, including those associated with Missouri River flooding in summer 2011. The plant’s issues were sufficiently numerous and significant that the NRC issued a CAL specifying actions OPPD had to take before restarting.
In addition to these “normal” issues, a fire occurred in June 2011—an incident that has just gotten them a “Red” finding from the NRC. Currently, it is the only plant in the country under NRC Inspection Manual Chapter 0350, which includes a restart checklist. As part of the restart qualification, the NRC will review OPPD’s third-party safety culture survey and, if they aren’t satisfied with the results, NRC will conduct its own safety culture assessment.*
Focusing a little more on Fort Calhoun’s safety culture, one particular item caught our attention: OPPD’s CNO saying, during an NRC-OPPD meeting, that one of their basic problems was their corrective action program culture. (The following is an unscripted exchange, not prepared testimony.)
“Commissioner Apostolakis: . . . what I would be more interested in is to know, in your opinion, what were the top two or three areas where you feel you went wrong and you ended up in this unhappy situation?
“[OPPD CNO] David Bannister: . . . the one issue is our corrective action program culture, our -- and it’s a culture that evolved over time. We looked at it more of a work driver, more of a -- you know, it’s a way to manage the system rather than . . . finding and correcting our performance deficiency.”**
Note the nexus between the culture and the CAP, with the culture evolving to accept a view of the CAP as a work management system rather than the primary way the plant identifies, analyzes, prioritizes and fixes its issues. Notwithstanding Fort Calhoun’s culture creep, the mechanics and metrics of an effective CAP are well-known to nuclear operators around the world. It is a failure of management if an organization loses track of the ball in this area.
What’s Going to Happen?
I have no special insight into this matter but I will try to read the tea leaves. Recently, the NRC has been showing both its “good cop” and “bad cop” personas. The good cop has approved the construction of multiple new nuclear units, thus showing that the agency does not stand in the way of industry extension and expansion.
Meanwhile, the bad cop has his foot on the necks of a few problem plants, including Fort Calhoun. The plant is an easy target: it is the second-smallest plant in the country and isolated (OPPD has no other nuclear facilities). The NRC will not kill the plant but may leave it twisting in the wind indefinitely, reminding us of Voltaire’s famous observation in Candide:
“. . . in this country, it is wise to kill an admiral from time to time so to encourage the others.”
* Fort Calhoun Station Manual Chapter 0350 Oversight Panel Charter (Jan. 12, 2012) ADAMS ML120120661.
** NRC Public Meeting Transcript, Briefing on Fort Calhoun (Feb. 22, 2012) p. 62 ADAMS ML120541135.
It may seem odd to find a source of fresh air in the context of the Massey coal mine disaster of 2010, a topic on which we have posted before. But the news last week of a former mine supervisor’s guilty plea yielded some very direct observations on the breakdown of safety in the mine. In a Wall Street Journal piece on March 29, 2012, it was reported:
“Booth Goodwin, the U.S. Attorney in Charleston, W.Va., wrote in the plea agreement that "‘laws were routinely violated’ by Massey because of a belief that ‘following those laws would decrease coal production.’"
Sometimes it takes a lawyer’s bluntness to cut through all the contributing circumstances and symptoms of a safety failure and place a finger directly on the cause. How often have you seen such unvarnished truth telling with regard to safety culture issues at nuclear plants?
“[The supervisor] specifically pleaded guilty to tipping off miners underground about inspections, falsifying record books, illegally rewiring a mining machine to operate without a functioning methane monitor and altering the mine's ventilation to trick a federal inspector.”
The above findings are more typical of what one sees in nuclear plant inspection reports and which are attributed to lack of strong safety culture. This in turn triggers the inevitable safety culture assessments, retraining, re-iteration of safety priorities, etc that appear to be the standard prescription for a safety culture “fever”. But what - continuing a not so good medical analogy - is causing the fever? And why would one expect that the one size fits all prescription is the right answer?
To us it gets down to something that isn’t receiving enough attention. What are the root causes of the problems that are typically associated with a finding that safety culture needs to be strengthened? We will share our thoughts, and ask for yours, in an upcoming post.
I was curious to see how they treated the areas I have been harping on as critical for effective SC: decision making, corrective action, management incentives and work backlogs. Following is my very subjective rating of how well the draft elements cover the key subject areas.
Decision making – Mostly Satisfactory. “The licensee makes safety-significant or risk-significant decisions using a systematic process.” (Shoop, p. 18) We agree; in fact, we think ALL significant decisions should be made using a systematic process. Why “systematic”? To evidence transparency and robustness, i.e., to maximize the odds that a different decision-maker, if faced with a similar situation, will reach the same or similar answer. However, one important type of decision, the resolution of goal conflict needs improvement. Goal conflict appears focused on personal or professional disagreements; the big picture potential conflict of safety vs production, cost or schedule gets slight mention.
Corrective action – Satisfactory. There are a lot of words about corrective action and the CAP and they cover the important points. A minor gripe is the term “safety” may be overused when referring to identifying, evaluating or correcting problems. A couple of possible unintended consequences of such overuse are to create the impression that (1) only safety-related problems need such thorough treatment or (2) anything someone wants done needs some relation, no matter how tenuous, to safety.
Management incentives – Minimally acceptable. “Senior management incentive program [sic] reflect a bias toward long-term plant performance and safety.” (Shoop, p. 12) One could say more about this topic (and we have, including here and here) but the statement gets over the bar.
Backlogs – Unsatisfactory. The single mention of backlogs is “Maintaining long term plant safety by . . . ensuring maintenance and engineering backlogs which are low enough [to] support safety” (Shoop, p. 32) and even that was the tail end of a list of contributing factors to plant safety. Backlogs are much more important than that. Excessive backlogs are demoralizing; they tell the workforce that accomplishing work to keep the plant, its procedures and its support processes in good repair or up-to-date is not important. Every “problem plant” we worked on in the late 1990s had backlog issues. This is where the title reference to the broken window comes in.
“. . . if a window in a building is broken and is left unrepaired, all the rest of the windows will soon be broken. . . . one unrepaired broken window is a signal that no one cares, and so breaking more windows costs nothing.”**
Our March 11 post reviewed the IAEA SCART guidelines. We found its safety culture characteristics and attributes comprehensive but its “guiding questions” for evaluators were thin gruel, especially in the areas we consider critical for safety culture: decision making, corrective action, work backlogs and management incentives.
This post reviews another document that combines the SCART guidelines, other IAEA documents and the author’s insights to yield a much more robust guidebook for evaluating a facility’s safety culture. It’s called “Guidelines for Regulatory Assessment of Safety Culture in Licensees’ Organisations.”* It starts with the SCART characteristics and attributes but gives more guidance to an evaluator: recommendations for documents to review, what to look for during the evaluation, additional (and more critical) guiding questions, and warning signs that can indicate safety culture weaknesses or problems.
Specific guidance in the areas we consider critical is generally more complete. For example, in the area of decision making, evaluators are told to look for a documented process applicable to all matters that affect safety, attend meetings to observe the decision-making process, note the formalization of the decision making process and how/if long-term consequences of decisions are considered. Goal conflict is explicitly addressed, including how differing opinions, conflict based on different experiences, and questioning attitudes are dealt with, and the evidence of fair and impartial methods to resolve conflicts. Interestingly, example conflicts are not limited to the usual safety vs. cost or production but include safety vs. safety, e.g., a proposed change that would increase plant safety but cause additional personnel rad exposure to implement. Evidence of unresolved conflicts is a definite warning flag for the evaluator.
Corrective action (CA) also gets more attention, with questions and flags covering CA prioritization based on safety significance, the timely implementation of fixes, lack of CA after procedure violations or regulatory findings, verification that fixes are implemented and effective, and overall support or lack thereof for the CAP.
Additional questions and flags cover backlogs in maintenance, corrective actions, procedure changes, unanalyzed physical or procedural problems, and training.
However, the treatment of management incentives is still weak, basically the same as the SCART guidelines. We recommend a more detailed evaluation of the senior managers’ compensation scheme or, in more direct language, how much do they get paid for production, and how much for safety?
The intended audience for this document is a regulator charged with assessing a licensee’s safety culture. As we have previously discussed, some regulatory agencies are evaluating this approach. For now, that’s a no-go in the U.S. In any case, these guidelines provide a good checklist for self-assessors, internal auditors and external consultants.
* M. Tronea, “Guidelines for Regulatory Oversight of Safety Culture in Licensees’ Organisations” Draft, rev. 8 (Bucharest, Romania: National Commission for Nuclear Activities Control [CNCAN], April 2011). In addition to being on the staff of CNCAN, the nuclear regulatory authority of Romania, Dr. Tronea is the founder/manager of the LinkedIn Nuclear Safety group.
Last March, the NRC approved its safety culture policy statement.* At the time, a majority of commissioners issued supplemental comments expressing their concern that the policy statement could be used as a back door to regulation. The policy was issued in June, 2011. Enough time has lapsed to ask: What, if anything has happened, i.e., how is the NRC treating safety culture as it exercises its authority to regulate licensees?
We examined selected NRC documents for some plants where safety culture has been raised as a possible issue and see a few themes emerging. One is the requirement to examine the causes of specific incidents to ascertain if safety culture was a contributing factor. It appears some (perhaps most or all) special inspection notices to licensees include some language about "an assessment of whether any safety culture component caused or significantly contributed to these findings."
The obvious push is to get the licensee to do the work and explicitly address safety culture in their mea culpa to the agency. Then the agency can say, for example, that "The inspection team confirmed that the licensee established appropriate corrective actions to address safety culture."** A variant on this theme is now occurring at Browns Ferry, where the “NRC is reviewing results from safety culture surveys performed by the plant in 2011.”***
The NRC is also showing the stick, at least at one plant. At Fort Calhoun, the marching orders are: “Assess the licensee’s third party evaluation of their safety culture. . . . If necessary, perform an independent assessment of the licensee’s safety culture using the guidance contained in Inspection Procedure 95003."**** I think that means: If you can't/won't/don't perform an adequate safety culture evaluation, then we will. To back up this threat, it appears the NRC is developing procedures and materials for qualifying its inspectors to evaluate safety culture.
The Alternative Dispute Resolution (ADR) process is another way to get safety culture addressed. For example, Entergy got in 10 CFR 50.7 (employee protection) trouble for lowering a River Bend employee’s rating in part because of questions he raised. One of Entergy's commitments following ADR was to perform a site-wide safety culture survey. It probably didn’t help that, in a separate incident, River Bend operators were found accessing the internet when they were supposed to be watching the control board. Entergy also has to look at safety culture at FitzPatrick and Palisades because of incidents at those locations.*****
What does the recent experience imply?
The NRC’s current perspective on safety culture is summed up in an NRC project manager’s post in an internet Nuclear Safety Culture forum: “You seem to [sic] hung up on how NRC is going to enforce safety culture. We aren't. Safety culture isn't required. It won't be the basis for denying a license application. It won't be the basis for citing a violation during an inspection. However, if an incident investigation identifies safety culture as one of the root causes, we will require corrective action to address it.” (Note this is NOT an official agency statement.)
However, our Bob Cudlin made a more expansive prediction in his January 19, 2011 post: “. . . it appears that the NRC will “expect” licensees to meet the intent and the particulars of its policy statement. It seems safe to assume the NRC staff will apply the policy in its assessments of licensee performance. . . . The greatest difficulty is to square the rhetoric of NRC Commissioners and staff regarding the absolute importance of safety culture to safety, the “nothing else matters” perspective, with the inherently limited and non-binding nature of a policy statement.”
While the record to date may support the NRC PM’s view, I think Bob’s observations are also part of the mix. It’s pretty clear the NRC is turning the screw on licensee safety culture effectiveness, even if it’s not officially “regulating” safety culture.
* NRC Commission Voting Record, SECY-11-005, “Proposed Final Safety Culture Policy Statement” (March 7, 2011). I could not locate this document in ADAMS.
** IR 05000482-11-006, 02/07-03/31/2011, Wolf Creek Generating Station - NRC Inspection Procedure 95002 Supplemental Inspection Report and Assessment Followup Letter (May 20, 2011) ADAMS ML111400351.
*** Public Meeting Summary for Browns Ferry Nuclear Plant, Docket No. 50-259 (Feb. 26, 2012) ADAMS ML12037A092.
**** Fort Calhoun Station Manual Chapter 0350 Oversight Panel Charter (Jan. 12, 2012) ADAMS ML120120661.
***** EN-11-026, Confirmatory Order, Entergy Operations Inc. (Aug. 19, 2011) ADAMS ML11227A133; NRC Press Release-I-12-002: “NRC Confirms Actions to be Taken at FitzPatrick Nuclear Plant to Address Violations Involving Radiation Protection Program” (Jan. 26, 2012) ADAMS ML120270073; NRC Press Release-III-12-003: “NRC Issues Confirmatory Orders to Palisades Plant Owner Entergy and Plant Operator” (Jan. 26, 2012) ADAMS ML120270071.
The International Atomic Energy Agency (IAEA) offers the SCART service to Member States. A SCART’s goal is to assess the safety culture in a nuclear facility and provide recommendations for enhancing safety culture going forward. In this post, we provide an overview of the program and our evaluation of it.
What is SCART?
“SCART is an assessment of safety culture based on IAEA standards and guidelines by a team of international and independent safety culture experts.”* A SCART mission can take up to a year start-to-finish, including a pre-SCART visit to finalize the scope of the assessment. The SCART on-site assessment takes two full weeks, including the review team’s on-site organizational activities. The assessment uses document reviews, interviews and observations to gather data, and a fairly prescribed methodology for drawing inferences from the data.
The review team utilizes an evaluation framework consisting of five key safety culture characteristics, which are assessed using 37 attributes. The attributes describe “specific organizational performance or attitude . . . which, if fulfilled, would characterize this performance or attitude as belonging to a strong safety culture.”**
The review team’s findings describe the current state of safety culture at the host facility; the team’s recommendations and suggestions describe ways the safety culture could be improved.
Is it any good?
At first blush, SCART appears overly bureaucratic and time-consuming. However, the assessment team is likely to be comprised of IAEA staff and outside experts from different countries, and the target facility is likely in yet another country. In addition, much of prescribed methodology is aimed at ensuring the team covers all important topics and reaches robust, i.e., repeatable, conclusions. All this takes time and a detailed game plan.
But what about the game plan itself?
The review team forms opinions on five safety culture characteristics by assessing 37 attributes. The characteristics are non-controversial and the set of attributes appears comprehensive. (SCART Guidelines, pp. 25-26) Supporting them is a set of over 300 suggested “guiding questions” for the interviews. For us, the most important aspects are the attributes and questions that address topics we consider essential for an effective safety culture: a successful corrective action program, acceptable work backlogs, a decision making process that appropriately values safety, and management incentives.
Corrective Action Program
Attribute E.5: Learning is facilitated through the ability to recognize and diagnose deviations, to formulate and implement solutions and to monitor the effects of corrective actions” (Guidelines, p. 47)
The relevant*** guiding questions are: “Can staff members or contractors point to examples of problems they have reported which have been fixed?” “How high is the rate of repeat events or errors?”
There’s a lot more than can be asked about the CAP. For example, Who can initiate an action request? How are requests evaluated and prioritized; does safety receive consistent attention and appropriate priority? What are the backlogs and trends? What items are subject to root cause analysis? Does root cause analysis find the real causes of problems, i.e., do the subject problems cease to occur after they have been fixed?
Work Backlogs
“Attribute A.2: Safety is a primary consideration in the allocation of resources:” (Guidelines, p. 28)
The relevant guiding questions are: “Can staff members and contractors describe examples when the allocation of resources affected the backlog of maintenance tasks and nuclear facility modifications? What was the process to resolve the conflict?”
How about: What are the backlogs and trends in every major department? Are backlogs at an acceptable level? Why or why not? If not, then is there a plan to clear the backlogs? Are resources available to implement the plan?
Decision Making Process
Two attributes refer to decision making. “Attribute A.1: The high priority given to safety is shown in documentation, communications and decision making:” (Guidelines, p. 27) “Attribute A.5.: A proactive and long term approach to safety issues is shown in decision-making:” (Guidelines, p. 29)
The relevant guiding questions for A.1 are: “During periods of heavy work-load, in what way do managers ensure that staff members and contractors are reminded that unnecessary haste and shortcuts are inappropriate? Can staff members and contractors describe situations when the rationale for significant decisions related to safety was communicated to a large group of individuals in the nuclear facility? Can staff members and contractors describe situations when assumptions and conclusions of earlier safety decisions were challenged in the light of new information, operating experience or changes in context?”
The relevant guiding questions for A.5 are: “What is the approach of managers at all levels when they have to cope with an unforeseen event requiring more staff at short notice? What happens if, for any reason, production requirements are permitted to interfere with scheduled training modules? What kind of a system for prioritizing maintenance work along safety requirements is established?
We would add: How does the decision-making process handle competing goals, set priorities, treat devil’s advocates who raise concerns about possible unfavorable outcomes, and assign resources? Are the most qualified people involved in key decisions, regardless of their position or rank? How are safety concerns handled in making real-time decisions?
Management Incentives
Incentives are discussed under Attribute A.5 (see preceding section).
The relevant guiding questions are: “What is the major focus of incentives and priorities for senior management? How are management incentive strategies discussed on the corporate level?”
We would add: How is safety incorporated into management incentives, if at all? If safety is addressed, is it limited to industrial safety? Is there any disincentive, e.g., loss of bonus, if safety-related incidents occur or recur?
With over 300 guiding questions, I may have missed some that address our key issues. But the ones identified above seem a little thin in their treatment of the most important issues related to safety culture. We are not saying the other attributes and questions are not important—but they do not address the core of safety culture’s impact on organizational behavior.
Has SCART Been Applied?
Yes. Since 2006, IAEA has conducted three SCART evaluations, two of which occurred at nuclear power plants. (A request to IAEA asking if additional evaluations have taken place went unanswered.) I think we can safely say it is not wildly popular.
Conclusion
The SCART materials provide a good reference for anyone trying to figure out how to evaluate their facility’s safety culture. The comprehensive, step-by-step approach ensures that all attributes are covered and individual expert opinions are melded into team opinions for each attribute and characteristic. However, we doubt anyone would ever use it as a template for self-assessment. It is too resource-intensive, treats key areas lightly and basically creates a static, as opposed to dynamic, snapshot of safety culture. The overall impression reminds me of the apocryphal tale of the man who wrote a book titled 1000 Ways to Make Love; unfortunately, he didn’t know any women.
* C. Viktorsson, IAEA, “Understanding and Assessing Safety Culture,” Symposium on Nuclear Safety Culture: Fostering Safety Culture in Japan’s Nuclear Industry: How To Make It Robust? (Mar 22-23, 2006) p. 18.
