IAEA’s Safety Culture Assessment Review Team (SCART) Program

The International Atomic Energy Agency (IAEA) offers the SCART service to Member States.  A SCART’s goal is to assess the safety culture in a nuclear facility and provide recommendations for enhancing safety culture going forward.  In this post, we provide an overview of the program and our evaluation of it.

What is SCART?

“SCART is an assessment of safety culture based on IAEA standards and guidelines by a team of international and independent safety culture experts.”*  A SCART mission can take up to a year start-to-finish, including a pre-SCART visit to finalize the scope of the  assessment.  The SCART on-site assessment takes two full weeks, including the review team’s on-site organizational activities.  The assessment uses document reviews, interviews and observations to gather data, and a fairly prescribed methodology for drawing inferences from the data. 

The review team utilizes an evaluation framework consisting of five key safety culture characteristics, which are assessed using 37 attributes.  The attributes describe “specific organizational performance or attitude . . . which, if fulfilled, would characterize this performance or attitude as belonging to a strong safety culture.”**

The review team’s findings describe the current state of safety culture at the host facility; the team’s recommendations and suggestions describe ways the safety culture could be improved.

Is it any good?

At first blush, SCART appears overly bureaucratic and time-consuming.  However, the assessment team is likely to be comprised of IAEA staff and outside experts from different countries, and the target facility is likely in yet another country.  In addition, much of prescribed methodology is aimed at ensuring the team covers all important topics and reaches robust, i.e., repeatable, conclusions.  All this takes time and a detailed game plan.

But what about the game plan itself?  

The review team forms opinions on five safety culture characteristics by assessing 37 attributes.  The characteristics are non-controversial and the set of attributes appears comprehensive. (SCART Guidelines, pp. 25-26)  Supporting them is a set of over 300 suggested “guiding questions” for the interviews.  For us, the most important aspects are the attributes and questions that address topics we consider essential for an effective safety culture: a successful corrective action program, acceptable work backlogs, a decision making process that appropriately values safety, and management incentives.

Corrective Action Program

Attribute E.5: Learning is facilitated through the ability to recognize and diagnose deviations, to formulate and implement solutions and to monitor the effects of corrective actions” (Guidelines, p. 47)

The relevant*** guiding questions are: “Can staff members or contractors point to examples of problems they have reported which have been fixed?”  “How high is the rate of repeat events or errors?”

There’s a lot more than can be asked about the CAP.  For example, Who can initiate an action request?  How are requests evaluated and prioritized; does safety receive consistent attention and appropriate priority?  What are the backlogs and trends?  What items are subject to root cause analysis?  Does root cause analysis find the real causes of problems, i.e., do the subject problems cease to occur after they have been fixed?  

Work Backlogs

“Attribute A.2: Safety is a primary consideration in the allocation of resources:” (Guidelines, p. 28)

The relevant guiding questions are: “Can staff members and contractors describe examples when the allocation of resources affected the backlog of maintenance tasks and nuclear facility modifications? What was the process to resolve the conflict?”

How about: What are the backlogs and trends in every major department?  Are backlogs at an acceptable level?  Why or why not?  If not, then is there a plan to clear the backlogs?  Are resources available to implement the plan?

Decision Making Process

Two attributes refer to decision making.  “Attribute A.1: The high priority given to safety is shown in documentation, communications and decision making:” (Guidelines, p. 27)   “Attribute A.5.: A proactive and long term approach to safety issues is shown in decision-making:” (Guidelines, p. 29)

The relevant guiding questions for A.1 are: “During periods of heavy work-load, in what way do managers ensure that staff members and contractors are reminded that unnecessary haste and shortcuts are inappropriate?  Can staff members and contractors describe situations when the rationale for significant decisions related to safety was communicated to a large group of individuals in the nuclear facility?  Can staff members and contractors describe situations when assumptions and conclusions of earlier safety decisions were challenged in the light of new information, operating experience or changes in context?”

The relevant guiding questions for A.5 are: “What is the approach of managers at all levels when they have to cope with an unforeseen event requiring more staff at short notice?  What happens if, for any reason, production requirements are permitted to interfere with scheduled training modules? What kind of a system for prioritizing maintenance work along safety requirements is established?

We would add:   How does the decision-making process handle competing goals, set priorities, treat devil’s advocates who raise concerns about possible unfavorable outcomes, and assign resources?  Are the most qualified people involved in key decisions, regardless of their position or rank?  How are safety concerns handled in making real-time decisions?

Management Incentives

Incentives are discussed under Attribute A.5 (see preceding section).

The relevant guiding questions are: “What is the major focus of incentives and priorities for senior management?  How are management incentive strategies discussed on the corporate level?”

We would add: How is safety incorporated into management incentives, if at all?  If safety is addressed, is it limited to industrial safety?  Is there any disincentive, e.g., loss of bonus, if safety-related incidents occur or recur?   


With over 300 guiding questions, I may have missed some that address our key issues.  But the ones identified above seem a little thin in their treatment of the most important issues related to safety culture.  We are not saying the other attributes and questions are not important—but they do not address the core of safety culture’s impact on organizational behavior. 

Has SCART Been Applied?

Yes.  Since 2006, IAEA has conducted three SCART evaluations, two of which occurred at nuclear power plants.  (A request to IAEA asking if additional evaluations have taken place went unanswered.)  I think we can safely say it is not wildly popular.

Conclusion

The SCART materials provide a good reference for anyone trying to figure out how to evaluate their facility’s safety culture.  The comprehensive, step-by-step approach ensures that all attributes are covered and individual expert opinions are melded into team opinions for each attribute and characteristic.  However, we doubt anyone would ever use it as a template for self-assessment.  It is too resource-intensive, treats key areas  lightly and basically creates a static, as opposed to dynamic, snapshot of safety culture.  The overall impression reminds me of the apocryphal tale of the man who wrote a book titled 1000 Ways to Make Love; unfortunately, he didn’t know any women.


*  C. Viktorsson, IAEA, “Understanding and Assessing Safety Culture,” Symposium on Nuclear Safety Culture: Fostering Safety Culture in Japan’s Nuclear Industry: How To Make It Robust?  (Mar 22-23, 2006) p. 18.

**  SCART Guidelines: Reference Report for IAEA Safety Culture Assessment Review Team (SCART), IAEA, Vienna (July 2008) p. 4.

***  Each attribute is followed by many guiding questions.  I have selected the questions that appear most related to our key topics.