IAEA OSART in USA

The International Atomic Energy Agency (IAEA) offers an Operational Safety Review Team (OSART) program where international teams of experts conduct in-depth reviews of nuclear power plant operational safety performance.  Performance is reviewed in various areas such as management, training, operations, maintenance, etc.  The reports also include comments about plant safety culture (SC).  IAEA has conducted scores of OSART reviews, including U.S. power plants.  This post covers SC-related findings for the U.S. plants; we are interested in what kinds of SC strengths and weaknesses the teams identify during their approximately two-week visits.

North Anna (2000)

This is the earliest OSART report available in the NRC ADAMS database.  There are two versions of the report, one distributed to NRC Commissioners on Nov. 30, 2000* and another released by IAEA on Feb. 12, 2001.**  Both versions include generally complimentary language related to SC saying management is committed to safety and continuous improvement is nurtured. (p. 7 in 2000 report)  In the Operations area the report notes “Policies and management verbal communication are consistent and emphasize the importance of safety first.  Safety culture is referred to at several hierarchical levels of the plant.” (p. 31 in 2000 report)

There is an interesting discrepancy between the two versions.  The first report says “The emphasis the plant is making on self-assessment and improving human performance to bring about a sustaining safety culture is presently balancing the perception of increased cost drivers and short term budgeting.” (p. 13)  This hints at a cost vs. safety goal conflict, a serious challenge to maintaining a strong SC.  However, the second report does not include that statement, instead noting elsewhere that “In the teams opinion, however, there is presently little resource margin to absorb increased demands and although the highly motivated staff continue to look for ways to improve, the potential for complacency and demotivation from the prospect of long term challenges with aging facilities and equipment and tightening budget poses a risk to maintaining that performance.” (p. 9)  Is this just a more politic way of saying the same thing or does it reflect an actual watering down of the report?

Brunswick (2005)

The report*** covers the initial visit and the follow-up visit 19 months later.  SC is treated more thoroughly than in earlier reviews: “An important element of the OSART review is the identification of those findings that exhibit positive and negative attributes of safety culture.” (p. 7)  At Brunswick positive SC attributes included a questioning attitude of all personnel, a strong self-assessment program, and adherence to established procedures.  Improvement opportunities included the need for increased attention for continuous improvement, greater use of self criticism when observing safety systems and components, and further expanding external operating experience activities. (p. 10) 

The team’s in-depth review of the corrective action program (CAP) provides a look at the SC in practice.  “The corrective action programme is very strong and closely monitored.” (p. 97)  Aging mechanisms, especially corrosion, are a challenge for the corrective action program (p. 69) but the plant’s response led to this issue being closed during the follow-up visit.  Rework is analyzed by a committee to address effective corrective actions (p. 57) but maintenance backlogs require constant attention. (p. 59)  After the team’s initial visit the plant developed a new system for classifying condition reports; at follow-up the team observed “Based on the new graded system, the plant is improving opportunities to focus analyses and resources in accordance with the significance of the events.” (p. 100)  Overall, this seems reflective of a SC that supports both the CAP and efforts to improve it.

Another finding reflecting SC was that the plant Management Succession Planning and Development Program did not include safety performance in talent assessment, ranking or as a specified core skill.  Brunswick claimed these factors were implicitly considered but strengthened the program to explicitly include nuclear safety, radiation safety and industrial safety goals for each plant staff member.  The review team found this action sufficient to close the issue. (pp. 21-22)  This is also indicative of a culture that encourages improvement efforts.

Arkansas Nuclear One (2008) 

The positive SC attributes in this report**** include a strong Human Performance program with a focus on continuous improvement, a rigorous and conservative approach when planning and performing tasks, and a willingness of staff to correct each other’s behavior without waiting to be corrected by management.  In addition, conservative decision making is a core value within the management team.  Problem areas include long standing defects throughout the plant (and first attempts to fix problems that do not always work), policies such as overtime and time pressure on outage schedule that suggest the plant is “cost” driven, and the benchmarking program does not work from a broad international base. (pp. 4-5, 34).  Both an ineffective CAP and a cost-driven plant (i.e, goal conflict between cost and safety) reflect a weak SC.

Seabrook (2011)

In the Seabrook report***** SC strengths include consistent reinforcement of key values by the management team and no evidence of production taking precedence over safety.  Opportunities for improvement include a more aggressive approach to addressing long term issues, thus reducing the potential for staff normalization to less than excellent conditions. (pp. 4, 45)

A positive observation is that plant communications stress safety as the no. 1 priority. (p. 7)  Another is that Seabrook has a problem reporting culture based on a low-threshold and high-volume reporting system. (pp. 2, 56)  Personnel who report problems outside their area of responsibility are recognized with a “Good Catch” award.  Personnel writing lots of condition reports is generally a good thing but some corrective actions are closed prematurely. (pp. 54-56, 59)

More significantly, plant problems include a lack of resolution of long term issues, plant material condition deterioration and degraded equipment conditions. (pp. 2, 44-45)  Significant backlogs exist in the CAP, work orders and procedure change requests. (p. 8)  Work backlogs are an indication of a culture that lives with issues rather than resolving them.

Seabrook SC standards are summarized as “very positive” (p. 4) and maybe they are, but the reality of backlogs and unresolved long-term issues is inconsistent with a strong SC. 

Our Perspective

It’s probably unrealistic to expect a group of foreigners to visit a U.S. plant for two weeks and come up with significant SC insights.^  The summaries of SC positives and negatives appear to be relatively superficial but findings in the functional areas can offer a look at the actual underlying SC.  Some of the OSART observations and findings on key artifacts provide information from which we can infer the strength of SC at a plant being reviewed. 

We would like to see greater attention to how important decisions are made at a plant.  Decision making was only mentioned in passing (North Anna, p. 31) or a simple statement like plant performance indicators are a key input to decision making. (ANO, p. 6)

Finally, compensation and reward systems appear to be outside the OSART scope but we know these are critical to reinforce safety-related behavior and overall SC.


*  J.D. Lee (NRC) to NRC Commissioners, “IAEA Report on OSART Mission at North Anna Power Station, January 22 to February 10, 2000” (Nov. 30, 2000).  ADAMS ML010160525.

**  IAEA, “Report of the OSART Mission to the USA North Anna Nuclear Power Plant 22 January to 10 February 2000” (Released Feb. 12, 2001).  ADAMS ML010470115.

***  IAEA, “Report of the OSART Mission to the Brunswick Nuclear Plant United States of America 9-25 May 2005 and Follow Up Visit 4-8 December 2006” no date.  ADAMS ML071100006.

****  IAEA, “Report of the OSART Mission to the Arkansas Nuclear One Nuclear Power Plant United States of America 15 June – 2 July 2008” no date.  ADAMS ML083440148.

*****  P. Freeman (Seabrook) to NRC, “IAEA Final Report OSART Mission to Seabrook Nuclear Power Plant USA 6-23 June 2011” (Mar. 16, 2012).  ADAMS ML12081A105.

^  A partial view is suggested in the Foreword to the Seabrook report where the authors note that an OSART review represents a “snapshot in time.”  In fairness, it would be equally unrealistic to expect a team of Americans to visit a foreign plant and develop a deep understanding of plant culture in a similar time period.

NRC "National Report" to IAEA

A March 25, 2014 NRC press release* announced that Chairman Macfarlane presented the Sixth National Report for the Convention on Nuclear Safety** to International Atomic Energy Agency (IAEA) member countries.  The report mentions safety culture (SC) several times, as discussed below.  There is no breaking news in a report like this.  We’re posting about it only because it provides an encyclopedic review of NRC activities including a description of how SC fits into their grand scheme of things.  We also tie the report’s contents to related posts on Safetymatters.  The numbers shown below are section numbers in the report.

6.3.11 Public Participation 

This section describes how the NRC engages with stakeholders and the broader public.  As part of such engagement, the NRC says it expects employers to maintain an open environment where workers are free to raise safety concerns. “These expectations are communicated through the NRC’s Safety Culture Policy Statement” and other regulatory directives and tools. (p. 72)  This is pretty straightforward and we have no comment.

8.1.6.2 Human Resources

Section 8 describes the NRC, from its position in the federal government to how it runs its internal activities.  One such activity is the NRC Inspector General’s triennial General Safety Culture and Climate Survey for NRC employees.  Reporting on the most recent (2012) survey, “the NRC scored above both Federal and private sector benchmarks, although in 2012 the agency did not perform as strongly as it had in the past.” (p. 96)  We posted on the internal SC survey back on April 6, 2013; we felt the survey raised a few significant issues.

10.4 Safety Culture

Section 10 covers activities that ensure that safety receives its “due priority” from licensees and the NRC itself.  Sub-section 10.4 provides an in-depth description of the NRC’s SC-related policies and practices so we excerpt from it at length.

The discussion begins with the SC policy statement and the traits of a positive (sic) SC, including Leadership, Problem identification and resolution, Personal accountability, etc.

The most interesting part is 10.4.1 NRC Monitoring of Licensee Safety Culture which covers “the policies, programs, and practices that apply to licensee safety culture.” (p. 118)  It begins with the Reactor Oversight Process (ROP) and its SC-related enhancements.  NRC staff identified 13 components as important to SC, including decision making, resources, work control, etc.  “All 13 safety culture components are applied in selected baseline, event followup, and supplemental IPs [inspection procedures].” (p. 119)

“There are no regulatory requirements for licensees to perform safety culture assessments routinely. However, depending on the extent of deterioration of licensee performance, the NRC has a range of expectations [emphasis added] about regulatory actions and licensee safety culture assessments, . . .” (p. 119)

“In the routine or baseline inspection program, the inspector will develop an inspection finding and then identify whether an aspect of a safety culture component is a significant causal factor of the finding. The NRC communicates the inspection findings to the licensee along with the associated safety culture aspect. 

“When performing the IP that focuses on problem identification and resolution, inspectors have the option to review licensee self-assessments of safety culture. The problem identification and resolution IP also instructs inspectors to be aware of safety culture components when selecting samples.” (p. 119)

“If, over three consecutive assessment periods (i.e., 18 months), a licensee has the same safety culture issue with the same common theme, the NRC may ask [emphasis added] the licensee to conduct a safety culture self-assessment.” (p. 120)

If the licensee performance degrades to Column 3 of the ROP Action Matrix and “the NRC determines that the licensee did not recognize that safety culture components caused or significantly contributed to the risk-significant performance issues, the NRC may request [emphasis added] the licensee to complete an independent assessment of its safety culture.” (p. 120)

For licensees in Column 4 of the ROP “the NRC will expect [emphasis added] the licensee to conduct a third-party independent assessment of its safety culture. The NRC will review the licensee’s assessment and will conduct an independent assessment of the licensee’s safety culture . . .” (p. 120)

ROP SC considerations “provide the NRC staff with (1) better opportunities to consider safety culture weaknesses . . . (2) a process to determine the need to specifically evaluate a licensee’s safety culture . . . and (3) a structured process to evaluate the licensee’s safety culture assessment and to independently conduct a safety culture assessment for a licensee . . . .  By using the existing Reactor Oversight Process framework, the NRC’s safety culture oversight activities are based on a graded approach and remain transparent, understandable, objective, risk-informed, performance-based, and predictable.” (p. 120)

We described this hierarchy of NRC SC-related activities in a post on May 24, 2013.  We called it de facto regulation of SC.  Reading the above only confirms that conclusion.  When the NRC asks, requests or expects the licensee to do something, it’s akin to a military commander’s “wishes,” i.e., they’re the same as orders.

10.4.2 The NRC Safety Culture 

This section covers the NRC’s actions to strengthen its internal SC.  This actions include appointing an SC Program Manager; integrating SC into the NRC’s Strategic Plan; developing training; evaluating the NRC’s problem identification, evaluation and resolution processes; and establishing clear expectations and accountability for maintaining current policies and procedures. 

We would ask how SC affects (and is affected by) the NRC’s decision making and resource allocation processes, work practices, operating experience integration and establishing personal accountability for maintaining the agency’s SC.  What’s good for the goose (licensee) is good for the gander (regulator).

Institute of Nuclear Power Operations (INPO) 

INPO also provided content for the report.  Interestingly, it is a 39-page Part 3 in the body of the report, not an appendix.  Part 3 covers INPO’s mission, organization, etc. and includes a section on SC.

6. Priority to Safety (Safety Culture)

The industry and INPO have their own definition of SC: “An organization’s values and behaviors—modeled by its leaders and internalized by its members—that serve to make nuclear safety the overriding priority.” (p. 230)

“INPO activities reinforce the primary obligation of the operating organizations’ leadership to establish and foster a healthy safety culture, to periodically assess safety culture, to address shortfalls in an open and candid fashion, and to ensure that everyone from the board room to the shop floor understands his or her role in safety culture.” (p. 231)

We believe our view of SC is broader than INPO’s.  As we said in our July 24, 2013 post “We believe culture, including SC, is an emergent organizational property created by the integration of top-down activities with organizational history, long-serving employees, and strongly held beliefs and values, including the organization's “real” priorities.  In other words, SC is a result of the functioning over time of the socio-technical system.  In our view, a CNO can heavily influence, but not unilaterally define, organizational culture including SC.” 

Conclusion

This 341 page report appears to cover every aspect of the NRC’s operations but, as noted in our introduction, it does not present any new information.  It’s a good reference document to cite if someone asks you what the NRC is or what it does.

We found it a bit odd that the definition of SC in the report is not the definition promulgated in the NRC SC Policy Statement.  Specifically, the report says the NRC uses the 1991 INSAG definition of SC: “that assembly of characteristics and attitudes in organizations and individuals which establishes that, as an overriding priority, nuclear safety issues receive the attention warranted by their significance.” (p. 118)

The Policy Statement says “Nuclear safety culture is the core values and behaviors resulting from a collective commitment by leaders and individuals to emphasize safety over competing goals to ensure protection of people and the environment.”***

Of course, both definitions are different from the INPO definition provided above.  We’ll leave it as an exercise for the reader to figure out what this means.


*  NRC Press Release No: 14-021, “NRC Chairman Macfarlane Presents U.S. National Report to IAEA’s Convention on Nuclear Safety” (Mar. 25, 2014).  ADAMS ML14084A303.

**  NRC NUREG-1650 Rev. 5, “The United States of America Sixth National Report for the Convention on Nuclear Safety” (Oct. 2013).  ADAMS ML13303B021. 

***  NUREG/BR 0500 Rev 1, “Safety Culture Policy Statement” (Dec 2012).  ADAMS ML12355A122.  This definition comports with the one published in the Federal Register Vol. 76, No. 114 (June 14, 2011) p. 34777.

Safety Paradigm Shift

We came across a provocative and persuasive presentation by Jean Pariès Dédale, "Why a Paradigm Shift Is Needed" from the IAEA Experts Meeting in May of this year.*  Many of the points resonate with our views on nuclear safety management; in particular complexity, the fallacy of the "predetermination envelope"- making a system more reliable within its design envelope but more susceptible outside that envelope; deterministic and probabilistic rationalization that avoids dealing with complexity of the system; and unknown-unknowns.  We also believe it will take a paradigm shift, however unlikely it may be at least in the U.S. nuclear industry.  Interestingly, Dédale does not appear to have a nuclear power background and develops his paradigm argument across multiple events and industries.

Dédale poses a very fundamental question: since the current safety construct has shown vulnerabilities to actual off-normal events should the response be, do more of the same but better and with more rigor? Or should the safety paradigm itself be challenged?  The key issue underlying the challenge to this construct is how to cope with complexity.  He means complexity in the same terms we have posted about numerous times.

Dédale notes “The uncertainty generated by the complexity of the system itself and by its environment is skirted through deterministic or probabilistic rationality.” (p. 8)  We agree.  Any review of condition reports and Tech Spec variances indicates a wholesale reliance on risk based rationale for deviations from nominal requirements.  And the risk based argument is almost always based on an estimated small probability of an event that would challenge safety, often enhanced by a relatively short exposure time frame.  As we highlighted in a prior post, Nick Taleb has long cautioned against making decisions based on assessments of probabilities, which he asserts we cannot know, versus consequences which are (sometimes uncomfortably) knowable.

How does this relate to safety management issues including culture?

We see a parallel between constructs for nuclear safety and safety culture.  The nuclear safety construct is constrained both in focus and evolution, heavily reliant on the design basis philosophy (what Dédale labels “predetermination fallacy”) dating back to the 1960s.  Little has changed over the succeeding 50 years; even the advent of PRA has been limited to “informing” the implementation of this approach.  Safety culture has emerged over the last 10+ years as an added regulatory emphasis though highly constrained in its manifestation as a policy statement.  (It is in fact still quite difficult to square the NRC’s characterization of safety culture as critical to safety** yet stopping way short of any regulation or requirements.)  The definitional scope of safety culture is expressed in a set of traits and related values and behaviors.  As with nuclear safety it has a limited scope and relies on abstractions emphasizing, in essence, individual morality.  It does not look beyond people to the larger environment and “system” within which people function.  This environment can bring to bear significant influences that can challenge the desired traits and values of safety culture policy and muddle their application to decisions and actions.  The limitations can be seen in the assessments of safety culture (surveys and similar) as well as the investigation of specific events, violations or non-conformances by licensees and the NRC.  We’ve read many of these and rarely have we encountered any probing of the “why” associated with perceived breakdowns in safety culture.

One exception and a very powerful case in point is contained in our post dated July 29, 2010.  The cited reference is an internal root cause analysis performed by FPL to address employee concerns and identified weaknesses in their corrective action program.  They cite production pressures as negatively impacting employee trust and recognition, and perceptions of management and operational decisions.  FPL took steps to change the origin and impact of production pressures, relieving some of the burden on the organization to contain those influences within the boundaries of safe operation.

Perhaps the NRC believes that it does not have the jurisdiction to probe these types of issues or even require licensees to assess their influence.  Yet the NRC routinely refers to “licensee burden” - cost, schedule, production impacts - in accepting deviations from nominal safety standards.****  We wonder if a broader view of safety culture in the context of the socio-technical system might better “inform” both regulatory policy and decisions and enhance safety management.


*  J.P. Dédale, "Why a Paradigm Shift Is Needed," IAEA International Experts’ Meeting on Human and Organizational Factors in Nuclear Safety in the Light of the Accident at the Fukushima Daiichi Nuclear Power Plant, Vienna May 21-24, 2013.

**  The NRC’s Information Notice 2013-15 states that safety culture is “essential to nuclear safety in all phases…”
 
***  "NRC Decision on FPL (Part 2)," Safetymatters (July 29, 2010).  See slide 18, Root Cause 2 and Contributing Causes 2.2 and 2.4. 

****  10 CFR 50.55a(g)(6)(i) states that the Commission may grant such relief and may impose such alternative requirements as it determines is authorized by law and will not endanger life or property or the common defense and security and is otherwise in the public interest, given the consideration of the burden upon the licensee (emphasis added).

Dynamic Interactive Training

The words dynamic and interactive always catch our attention as they are intrinsic to our world view of nuclear safety culture learning.  Carlo Rusconi’s presentation* at the recent IAEA International Experts’ Meeting on Human and Organizational Factors in Nuclear Safety in the Light of the Accident at the Fukushima Daiichi Nuclear Power Plant in Vienna in May 2013 is the source of our interest.

While much of the training described in the presentation appeared to be oriented to the worker level and the identification of workplace type hazards and risks, it clearly has implications for supervisory and management levels as well.

In the first part of the training students are asked to identify and characterize safety risks associated with workplace images.  For each risk they assign an index based on perceived likelihood and severity.  We like the parallel to our proposed approach for scoring decisions according to safety significance and uncertainty.**

“...the second part of the course is focused on developing skills to look in depth at events that highlight the need to have a deeper and wider vision of safety, grasping the explicit and implicit connections among technological, social, human and organizational features. In a nutshell: a systemic vision.” (slide 13, emphasis added)  As part of the training students are exposed to the concepts of complexity, feedback and internal dynamics of a socio-technical system.  As the author notes, “The assessment of culture within an organization requires in-depth knowledge of its internal dynamics”.  (slide 15)

This part of the training is described as a “simulation” as it provides the opportunity for students to simulate the performance of an investigation into the causes of an actual event.  Students are organized into three groups of five persons to gain the benefit of collective analysis within each group followed by sharing of results across groups.  We see this as particularly valuable as it helps build common mental models and facilitates integration across individuals.  Last, the training session takes the student’s results and compares them to the outcomes from a panel of experts.  Again we see a distinct parallel to our concept of having senior management within the nuclear organization pre-analyze safety issues to establish reference values for safety significance, uncertainty and preferred decisions.  This provides the basis to compare trainee outcomes for the same issues and ultimately to foster alignment within the organization.

Thank you Dr. Rusconi.


*  C. Rusconi, “Interactive training: A methodology for improving Safety Culture,” IAEA International Experts’ Meeting on Human and Organizational Factors in Nuclear Safety in the Light of the Accident at the Fukushima Daiichi Nuclear Power Plant, Vienna May 21-24, 2013.

**  See our blog posts dated April 9 and June 6, 2013.  We also remind readers of Taleb’s dictate to decision makers to focus on consequences versus probability in our post dated June 18, 2013.

IAEA on Instituting Regulation of Licensee Safety Culture

The International Atomic Energy Agency (IAEA) has published a how-to report* for regulators who want to regulate their licensees' safety culture (SC).  This publication follows a series of meetings and workshops, some of which we have discussed (here and here).  The report is related to IAEA projects conducted “under the scope of the Regional Excellence Programme on Safe Nuclear Energy–Norwegian Cooperation Programme with Bulgaria and Romania. These projects have been implemented at the Bulgarian and Romanian regulatory bodies” (p. 1)

The report covers SC fundamentals, regulatory oversight features, SC assessment approaches, data collection and analysis.  We'll review the contents, highlighting IAEA's important points, then provide our perspective.

SC fundamentals

The report begins with the fundamentals of SC, starting with Schein's definition of SC and his tri-level model of artifacts, espoused values and basic assumptions.  Detail is added with a SC framework based on IAEA's five SC characteristics:

• Safety is a clearly recognized value
• Leadership for safety is clear
• Accountability for safety is clear
• Safety is integrated into all activities
• Safety is learning driven.

The SC characteristics can be described using specific attributes.

Features of regulatory oversight of SC 

This covers what the regulator should be trying to achieve.  It's the most important part of the report so we excerpt the IAEA's words.

“The objective of the regulatory oversight of safety culture, focused on a dynamic process, is to consider and address latent conditions that could lead to potential safety performance degradation at the licensees’ nuclear installations. . . . Regulatory oversight of safety culture complements compliance-based control [which is limited to looking at artifacts] with proactive control activities. . . . ” (p. 6, emphasis added)

“[R]egulatory oversight of safety culture is based on three pillars:

“Common understanding of safety culture. The nature of safety culture is distinct from, and needs to be dealt with in a different manner than a compliance-based control. . . .

“Dialogue. . . . dialogue is necessary to share information, ideas and knowledge that is often qualitative. . . .

“Continuousness. Safety culture improvement needs continuous engagement of the licensee. Regulatory oversight of safety culture therefore ideally relies on a process during which the regulator continuously influences the engagement of the licensee.” (p. 7)

“With regards to safety culture, the regulatory body should develop general requirements and enforce them in order to ensure the authorized parties have properly considered these requirements. On the other hand, the regulatory body should avoid prescribing detailed level requirements.” (p. 8)  The licensee always has the primary responsibility for safety.

Approaches for assessing SC

Various assessment approaches are currently being used or reviewed by regulatory bodies around the world. These approaches include: self-assessments, independent assessments, interaction with the licensee at a senior level, focused safety culture on-site reviews, oversight of management systems and integration into regulatory activities.  Most of these activities are familiar to our readers but a couple merit further definition.  The “management system” is the practices, procedures and people.**  “Integration into regulatory activities” means SC-related information is also collected during other regulatory actions, e.g., routine or special inspections.

The report includes a table (recreated below) summarizing, for each assessment approach, the accuracy of results and resources required.  Accuracy is judged as realistic, medium or limited and resource requirements as high, medium and low.  The table thus shows the relative strengths and weaknesses of each approach.

		Criteria
Approaches	Accuracy of SC picture	Effort	Management involvement	Human and Organizational Factors & SC skills
Self-assessment	Medium	Low (depending on	Low	Medium
Review		who initiates the		(to understand
(high experience and		self-assessment,		deliverables)
skills of the		regulator or
reviewers are		licensee)
assumed)
Independent	Medium	Low	Low	Medium
assessment Review				(to understand
(high experience and				deliverables)
skills of the
reviewers are
assumed)
Interaction with the	Limited (however	Medium	High	Medium
Licensee at Senior	can support a
Level	shared
	understanding)
Focused Safety	Realistic (gives	High	Medium	High
Culture On-Site	depth in a moment
Review	of time)
Oversight of	Medium (Reduced	Low	Low	Medium
Management System	if only formal
Implementation	aspects are
	considered)
Integration into	Medium (when	Medium (after an	Medium (with an	Medium (specific
Regulatory	properly trended	intensive initial	intensive initial	training
Activities	and analyzed)	introduction)	support)	requirement and
				experience sharing)

Data collection, analysis and presenting findings to the licensee

The report encourages regulators to use multiple assessment approaches and multiple data collection methods and data sources.  Data collection methods include observations; interviews; reviews of events, licensee documents and regulator documents; discussions with management; and other sources such as questionnaires, surveys, third-party documents and focus groups.  The goal is to approach the target from multiple angles.  “The aim of data analysis is to build a safety culture picture based on the inputs collected. . . . It is a set of interpreted data regarding the organizational practices and the priority of safety within these practices. (p. 17)

Robust data analysis “requires iterations [and] multi-disciplinary teams. A variety of expertise (technical, human and organizational factors, regulations) are necessary to build a reliable safety culture picture. . . . [and] protect against bias inherent to the multiple sources of data.” (p. 17)

The regulator's picture of SC is discussed with the licensee during periodic or ad hoc meetings.  The objective is to reach agreement on next steps, including the implementation of possible meeting actions and licensee commitments.

Our perspective

The SC content is pretty basic stuff, with zero new insight.  From our viewpoint, the far more interesting issue is the extension of regulatory authority into an admittedly soft, qualitative area.  This issue highlights the fact that the scope of regulatory authority is established by decisions that have socio-political, as well as technical, components.  SC is important, and certainly regulatable.  If a country wants to regulate nuclear SC, then have at it, but there is no hard science that says it is a necessary or even desirable thing to do.

Our big gripe is with the hypocrisy displayed by the NRC which has a SC policy, not a regulation, but in some cases implements all the steps associated with regulatory oversight discussed in this IAEA report (except evaluation of management personnel).  For evidence, look at how they have been pulling Fort Calhoun and Palisades through the wringer.


*  G. Rolina (IAEA), “Regulatory oversight of safety culture in nuclear installations” IAEA TECDOC 1707 (Vienna: International Atomic Energy Agency, 2013).

**  A management system is a “set of interrelated or interacting elements (system) for establishing policies and objectives and enabling the objectives to be achieved in an efficient and effective way. . . . These elements include the structure, resources and processes. Personnel, equipment and organizational culture as well as the documented policies and processes are parts of the management system.” (p. 30)

IAEA on Safety Culture in New Plant Design and Construction

The International Atomic Energy Agency (IAEA) has a 2012 publication* that provides guidance on establishing a strong safety culture (SC) during the design and construction of new nuclear power plants.  The report's premise, with which we agree, is a weak SC during plant design and construction can lead to later problems during plant operations.   

Major issues can arise during plant design and construction.  For example, the numerous organizations involved may have limited direct experience and/or insufficient knowledge of nuclear safety requirements, or projects may be located in countries with no existing nuclear industry or countries may have a nuclear industry but no recent construction experience.

The report attempts to cover the different needs, challenges and circumstances that may face project participants (governments, regulators, owners, designers, builders, manufacturers, etc.) anywhere in the world.  Most of the content addresses generic issues, e.g., understanding SC, the role of leadership, appropriate management systems, or communication and organizational learning.  Each issue is discussed in terms of specific challenges, goals, and recommended approaches and methods.  However, in their effort to attain maximum coverage (scope) IAEA sacrifices depth.  For example, the discussion of leadership covers five pages of the report but scarcely mentions the two most important activities of leaders: decision making and modeling safety-focused behavior.

If we look at the report's specific advice and recommendations, we see uneven coverage of the observable artifacts we consider essential for a strong SC: a decision making process that appropriately values safety, an effective corrective action program and financial incentives that reward safety performance. 

Decision making process
 
One overall challenge facing new projects is “Conflicts between schedule, cost and safety objectives can adversely affect conservative decision making and the maintenance of a questioning attitude, or impair the ability to perceive links between short term actions and their long term consequences.” (p. 2)

That's a good starting point but what are the characteristics of an appropriate decision making process?  It seems that decision making should be “conservative” (pp. 32, 34, 39), “broad” (p. 43) and “risk informed” (pp. 50, 51) but the terms are not defined.  More specificity on how the decision making process should handle competing goals, set priorities and assign resources would be useful.
 
What about the decision makers?  Leaders should be able to “Explain the relationships between time periods/horizons and decision making to help resolve competing priorities.” (p. 41)  That's OK but the need goes beyond time periods.  The manager must be able to explain the rationale for significant decisions related to safety.  What were the considerations, assumptions, priorities, alternatives, decision factors and their relative weights, and the applicable laws, rules and regulations?  How should leaders treat devil’s advocates who raise concerns about possible unfavorable outcomes?  Do leaders get the most qualified people involved in key decisions?
 
In addition, leaders should “Simulate decision making in a fast paced, complex environment to help leaders identify risks in their own approaches.”  (p. 41)  This is an excellent approach and we wholeheartedly support it. 

Corrective action

“Ineffective problem identification, inadequate reporting and inadequate corrective actions” (p. 9) were identified in a 2006 investigation as causal factors of construction problems at a nuclear plant site.  But there is no follow-up to describe the characteristics of an effective corrective action program.  There should be more about the CAP's ability to recognize and diagnose problems, formulate and implement solutions that consistently and appropriately consider safety, and monitor the effects of corrective actions. The importance of robust cause analysis, i.e., analysis that finds the real causes of problems so they do not recur, should be mentioned.  This would not be an unreasonable level of detail for this general report.

Financial incentives

The report correctly notes that “In construction environments, cultural attributes such as schedule awareness, cost focus and urgency of problem resolution are reinforced because they are rewarded by immediate measures of success.” (p. 8)  This becomes a specific challenge when “Contractor incentives are often driven by cost and schedule rather than by safety culture performance.” (p. 26).  A recommended fix is to “Establish a reward and incentive programme [sic] for the overall project, with objectives for safety performance and rewards that are either monetary or in the form of future contracts as a long term partner.” (p. 27).  This will probably result in a focus on industrial safety performance rather than the overall SC but it may be the best practical solution.  Periodic assessment of key contractors' SC should be used to identify any general SC issues. 

Our perspective

In prior posts, we have taken the IAEA to task for their overly bureaucratic approach.  So we're pleased to report this document actually provides some useful, sensible guidance (albeit often in an unprioritized, laundry list style) applicable to both countries initially embarking on the nuclear road and more experienced countries experiencing a nuclear renaissance.

The report makes a few important points.  For example, IAEA proposes a systems approach to thinking about all the project participants and the varied work they must accomplish.  “In the case of NPP projects, the ‘system’ involves human–social systems, work processes, complex technologies and multiple organizations in a global economic, energy, environmental and regulatory context.” (p. 11)  This is a viewpoint we have repeatedly advocated in this blog. 

Overall this report is satisfactory and it does refer the reader to other IAEA publications for additional information on specific subjects.  But in trying to provide relevant material to a plethora of stakeholders, the report gives shorter shrift to factors we consider vital to establishing and maintaining a strong SC. 


*  M. Haage (IAEA), “Safety culture in pre-operational phases of nuclear power plant projects” (Vienna : International Atomic Energy Agency,  2012).

DOE Nuclear Safety Workshop

The DOE held a Nuclear Safety Workshop on September 19-20, 2012.  Safety culture (SC) was the topic at two of the technical breakout sessions, one with outside (non-DOE) presenters and the other with DOE-related presenters.  Here’s our take on the outsiders’ presentations.

Chemical Safety Board (CSB)

This presentation* introduced the CSB and its mission and methods.  The CSB investigates chemical accidents and makes recommendations to prevent recurrences.  It has no regulatory authority. 

Its investigations focus on improving safety, not assigning blame.  The CSB analyzes systemic factors that may have contributed to an accident and recognizes that “Addressing the immediate cause only prevents that exact accident from occurring again.” (p. 5) 

The agency analyzes how safety systems work in real life and “why conditions or decisions leading to an accident were seen as normal, rational, or acceptable prior to the accident.” (p. 6)  They consider organizational and social causes, including “safety culture, organizational structure, cost pressures, regulatory gaps and ineffective enforcement, and performance agreements or bonus structures.” (ibid.)

The presentation included examples of findings from CSB investigations into the BP Texas City and Deepwater Horizon incidents.  The CSB’s SC model is adapted from the Schein construct.  What’s interesting is their set of artifacts includes many “soft” items such as complacency, normalization of deviance, management commitment to safety, work pressure, and tolerance of inadequate systems.

This is a brief and informative presentation, and well worth a look.  Perhaps because the CSB is unencumbered by regulatory protocol, it seems freer to go where the evidence leads it when investigating incidents.  We are impressed by their approach.
 
NRC

The NRC presentation** reviewed the basics of the Reactor Oversight Process (ROP) and then drilled down into how potential SC issues are identified and addressed.  Within the ROP, “. . . a safety culture aspect is assigned if it is the most significant contributor to an inspection finding.” (p.12)  After such a finding, the NRC may perform an SC assessment (per IP 95002) or request the licensee to perform one, which the NRC then reviews (per IP 95003).

This presentation is bureaucratic but provides a useful road map.  Looking at the overall approach, it is even more disingenuous for the NRC to claim that it doesn’t regulate SC.

IAEA

There was nothing new here.  This infomercial for IAEA*** covered the basic history of SC and reviewed contents of related IAEA documents, including laundry lists of desired organizational attributes.  The three-factor IAEA SC figure presented is basically the Schein model, with different labels.  The components of a correct culture change initiative are equally recognizable: communication, continuous improvement, trust, respect, etc.

The presentation had one repeatable quote: “Culture can be seen as something we can influence, rather than something we can control” (p. 10)

Conclusion

SC conferences and workshops are often worthless but sometimes one does learn things.  In this case, the CSB presentation was refreshingly complete and the NRC presentation was perhaps more revealing than the presenter intended.


*  M.A. Griffon, U.S. Chemical Safety and Hazards Investigation Board, “CSB Investigations and Safety Culture,” presented at the DOE Nuclear Safety Workshop (Sept. 19, 2012). 

**  U. Shoop, NRC, “Safety Culture in the U.S. Nuclear Regulatory Commission’s Reactor Oversight Process,” presented at the DOE Nuclear Safety Workshop (Sept. 19, 2012).

***  M. Haage, IAEA, “What is Safety Culture & Why is it Important?” presented at the DOE Nuclear Safety Workshop (Sept. 19, 2012).

Regulatory Influence on Safety Culture

In September, 2011 the Nuclear Energy Agency (NEA) and the International Atomic Energy (IAEA) held a workshop for regulators and industry on oversight of licensee management.  “The principal aim of the workshop was to share experience and learning about the methods and approaches used by regulators to maintain oversight of, and influence, nuclear licensee leadership and management for safety, including safety culture.”*

Representatives from several countries made presentations.  For example, the U.S. presentation by NRC’s Valerie Barnes and INPO’s Ken Koves discussed work to define safety culture (SC) traits and correlate them to INPO principles and ROP findings (we previously reviewed this effort here).  Most other presentations also covered familiar territory. 

However, we were very impressed by Prof. Richard Taylor’s keynote address.  He is from the University of Bristol and has studied organizational and cultural factors in disasters and near-misses in both nuclear and non-nuclear contexts.  His list of common contributors includes issues with leadership, attitudes, environmental factors, competence, risk assessment, oversight, organizational learning and regulation.  He expounded on each factor with examples and additional detail. 

We found his conclusion most encouraging:  “Given the common precursors, we need to deepen our understanding of the complexity and interconnectedness of the socio-political systems at the root of organisational accidents.”  He suggests using system dynamics modeling to study archetypes including “maintaining visible convincing leadership commitment in the presence of commercial pressures.”  This is totally congruent with the approach we have been advocating for examining the effects of competing business and safety pressures on management. 

Unfortunately, this was the intellectual high point of the proceedings.  Topics that we believe are important to assessing and understanding SC got short shrift thereafter.  In particular, goal conflict, CAP and management compensation were not mentioned by any of the other presenters.

Decision-making was mentioned by a few presenters but there was no substantive discussion of this topic (the U.K. presenter had a motherhood statement that “Decisions at all levels that affect safety should be rational, objective, transparent and prudent”; the Barnes/Kove presentation appeared to focus on operational decision making).  A bright spot was in the meeting summary where better insight into licensees’ decision making process was mentioned as desirable and necessary by regulators.  And one suggestion for future research was “decision making in the face of competing goals.”  Perhaps there is hope after all.

(If this post seems familiar, last Dec 5 we reported on a Feb 2011 IAEA conference for regulators and industry that covered some of the same ground.  Seven months later the bureaucrats had inched the football a bit down the field.)


*  Proceedings of an NEA/IAEA Workshop, Chester, U.K. 26-28 Sept 2011, “Oversight and Influencing of Licensee Leadership and Management for Safety, Including Safety Culture – Regulatory Approaches and Methods,” NEA/CSNI/R(2012)13 (June 2012).

Going Beyond SCART: A More Useful Guidebook for Evaluating Safety Culture

Our March 11 post reviewed the IAEA SCART guidelines.  We found its safety culture characteristics and attributes comprehensive but its “guiding questions” for evaluators were thin gruel, especially in the areas we consider critical for safety culture: decision making, corrective action, work backlogs and management incentives.

This post reviews another document that combines the SCART guidelines, other IAEA documents and the author’s insights to yield a much more robust guidebook for evaluating a facility’s safety culture.  It’s called “Guidelines for Regulatory Assessment of Safety Culture in Licensees’ Organisations.”*  It starts with the SCART characteristics and attributes but gives more guidance to an evaluator: recommendations for documents to review, what to look for during the evaluation, additional (and more critical) guiding questions, and warning signs that can indicate safety culture weaknesses or problems.

Specific guidance in the areas we consider critical is generally more complete.  For example, in the area of decision making, evaluators are told to look for a documented process applicable to all matters that affect safety, attend meetings to observe the decision-making process, note the formalization of the decision making process and how/if long-term consequences of decisions are considered.  Goal conflict is explicitly addressed, including how differing opinions, conflict based on different experiences, and questioning attitudes are dealt with, and the evidence of fair and impartial methods to resolve conflicts.  Interestingly, example conflicts are not limited to the usual safety vs. cost or production but include safety vs. safety, e.g., a proposed change that would increase plant safety but cause additional personnel rad exposure to implement.  Evidence of unresolved conflicts is a definite warning flag for the evaluator. 

Corrective action (CA) also gets more attention, with questions and flags covering CA prioritization based on safety significance, the timely implementation of fixes, lack of CA after procedure violations or regulatory findings, verification that fixes are implemented and effective, and overall support or lack thereof for the CAP. 

Additional questions and flags cover backlogs in maintenance, corrective actions, procedure changes, unanalyzed physical or procedural problems, and training.

However, the treatment of management incentives is still weak, basically the same as the SCART guidelines.  We recommend a more detailed evaluation of the senior managers’ compensation scheme or, in more direct language, how much do they get paid for production, and how much for safety?

The intended audience for this document is a regulator charged with assessing a licensee’s safety culture.  As we have previously discussed, some regulatory agencies are evaluating this approach.  For now, that’s a no-go in the U.S.  In any case, these guidelines provide a good checklist for self-assessors, internal auditors and external consultants.


*  M. Tronea, “Guidelines for Regulatory Oversight of Safety Culture in Licensees’ Organisations” Draft, rev. 8 (Bucharest, Romania:  National Commission for Nuclear Activities Control [CNCAN], April 2011).  In addition to being on the staff of CNCAN, the nuclear regulatory authority of Romania, Dr. Tronea is the founder/manager of the LinkedIn Nuclear Safety group.  

IAEA’s Safety Culture Assessment Review Team (SCART) Program

The International Atomic Energy Agency (IAEA) offers the SCART service to Member States.  A SCART’s goal is to assess the safety culture in a nuclear facility and provide recommendations for enhancing safety culture going forward.  In this post, we provide an overview of the program and our evaluation of it.

What is SCART?

“SCART is an assessment of safety culture based on IAEA standards and guidelines by a team of international and independent safety culture experts.”*  A SCART mission can take up to a year start-to-finish, including a pre-SCART visit to finalize the scope of the  assessment.  The SCART on-site assessment takes two full weeks, including the review team’s on-site organizational activities.  The assessment uses document reviews, interviews and observations to gather data, and a fairly prescribed methodology for drawing inferences from the data. 

The review team utilizes an evaluation framework consisting of five key safety culture characteristics, which are assessed using 37 attributes.  The attributes describe “specific organizational performance or attitude . . . which, if fulfilled, would characterize this performance or attitude as belonging to a strong safety culture.”**

The review team’s findings describe the current state of safety culture at the host facility; the team’s recommendations and suggestions describe ways the safety culture could be improved.

Is it any good?

At first blush, SCART appears overly bureaucratic and time-consuming.  However, the assessment team is likely to be comprised of IAEA staff and outside experts from different countries, and the target facility is likely in yet another country.  In addition, much of prescribed methodology is aimed at ensuring the team covers all important topics and reaches robust, i.e., repeatable, conclusions.  All this takes time and a detailed game plan.

But what about the game plan itself?  

The review team forms opinions on five safety culture characteristics by assessing 37 attributes.  The characteristics are non-controversial and the set of attributes appears comprehensive. (SCART Guidelines, pp. 25-26)  Supporting them is a set of over 300 suggested “guiding questions” for the interviews.  For us, the most important aspects are the attributes and questions that address topics we consider essential for an effective safety culture: a successful corrective action program, acceptable work backlogs, a decision making process that appropriately values safety, and management incentives.

Corrective Action Program

Attribute E.5: Learning is facilitated through the ability to recognize and diagnose deviations, to formulate and implement solutions and to monitor the effects of corrective actions” (Guidelines, p. 47)

The relevant*** guiding questions are: “Can staff members or contractors point to examples of problems they have reported which have been fixed?”  “How high is the rate of repeat events or errors?”

There’s a lot more than can be asked about the CAP.  For example, Who can initiate an action request?  How are requests evaluated and prioritized; does safety receive consistent attention and appropriate priority?  What are the backlogs and trends?  What items are subject to root cause analysis?  Does root cause analysis find the real causes of problems, i.e., do the subject problems cease to occur after they have been fixed?  

Work Backlogs

“Attribute A.2: Safety is a primary consideration in the allocation of resources:” (Guidelines, p. 28)

The relevant guiding questions are: “Can staff members and contractors describe examples when the allocation of resources affected the backlog of maintenance tasks and nuclear facility modifications? What was the process to resolve the conflict?”

How about: What are the backlogs and trends in every major department?  Are backlogs at an acceptable level?  Why or why not?  If not, then is there a plan to clear the backlogs?  Are resources available to implement the plan?

Decision Making Process

Two attributes refer to decision making.  “Attribute A.1: The high priority given to safety is shown in documentation, communications and decision making:” (Guidelines, p. 27)   “Attribute A.5.: A proactive and long term approach to safety issues is shown in decision-making:” (Guidelines, p. 29)

The relevant guiding questions for A.1 are: “During periods of heavy work-load, in what way do managers ensure that staff members and contractors are reminded that unnecessary haste and shortcuts are inappropriate?  Can staff members and contractors describe situations when the rationale for significant decisions related to safety was communicated to a large group of individuals in the nuclear facility?  Can staff members and contractors describe situations when assumptions and conclusions of earlier safety decisions were challenged in the light of new information, operating experience or changes in context?”

The relevant guiding questions for A.5 are: “What is the approach of managers at all levels when they have to cope with an unforeseen event requiring more staff at short notice?  What happens if, for any reason, production requirements are permitted to interfere with scheduled training modules? What kind of a system for prioritizing maintenance work along safety requirements is established?

We would add:   How does the decision-making process handle competing goals, set priorities, treat devil’s advocates who raise concerns about possible unfavorable outcomes, and assign resources?  Are the most qualified people involved in key decisions, regardless of their position or rank?  How are safety concerns handled in making real-time decisions?

Management Incentives

Incentives are discussed under Attribute A.5 (see preceding section).

The relevant guiding questions are: “What is the major focus of incentives and priorities for senior management?  How are management incentive strategies discussed on the corporate level?”

We would add: How is safety incorporated into management incentives, if at all?  If safety is addressed, is it limited to industrial safety?  Is there any disincentive, e.g., loss of bonus, if safety-related incidents occur or recur?   


With over 300 guiding questions, I may have missed some that address our key issues.  But the ones identified above seem a little thin in their treatment of the most important issues related to safety culture.  We are not saying the other attributes and questions are not important—but they do not address the core of safety culture’s impact on organizational behavior. 

Has SCART Been Applied?

Yes.  Since 2006, IAEA has conducted three SCART evaluations, two of which occurred at nuclear power plants.  (A request to IAEA asking if additional evaluations have taken place went unanswered.)  I think we can safely say it is not wildly popular.

Conclusion

The SCART materials provide a good reference for anyone trying to figure out how to evaluate their facility’s safety culture.  The comprehensive, step-by-step approach ensures that all attributes are covered and individual expert opinions are melded into team opinions for each attribute and characteristic.  However, we doubt anyone would ever use it as a template for self-assessment.  It is too resource-intensive, treats key areas  lightly and basically creates a static, as opposed to dynamic, snapshot of safety culture.  The overall impression reminds me of the apocryphal tale of the man who wrote a book titled 1000 Ways to Make Love; unfortunately, he didn’t know any women.


*  C. Viktorsson, IAEA, “Understanding and Assessing Safety Culture,” Symposium on Nuclear Safety Culture: Fostering Safety Culture in Japan’s Nuclear Industry: How To Make It Robust?  (Mar 22-23, 2006) p. 18.

**  SCART Guidelines: Reference Report for IAEA Safety Culture Assessment Review Team (SCART), IAEA, Vienna (July 2008) p. 4.

***  Each attribute is followed by many guiding questions.  I have selected the questions that appear most related to our key topics.