Boeing’s Safety Culture Under the FAA’s Microscope

The Federal Aviation Administration (FAA) recently released its report* on the safety culture (SC) at Boeing.  The FAA Expert Panel was tasked with reviewing SC after two crashes involving the latest models of Boeing’s 737 MAX airplanes.  The January 2024 door plug blowout happened as the report was nearing completion and reinforces the report’s findings.

737 MAXX door plug

The report has been summarized and widely reported in mainstream media and we will not review all its findings and recommendations here.  We want to focus on two parts of the report that address topics we have long promoted as being keys to understanding how strong (or weak) an organization’s SC is, viz., an organization’s decision-making processes and executive compensation.  In addition, we will discuss a topic that’s new to us, how to ensure the independence of employees whose work includes assessing company work products from the regulator’s perspective.

Decision-making

An organization’s decision-making processes create some of the most visible artifacts of the organization’s culture: a string of decisions (guided by policies, procedures, and priorities) and their consequences.

The report begins with a clear FAA description of decision-making’s important role in a Safety Management System (SMS) and an organization’s overall management.  In part, an “SMS is all about decision-making. Thus it has to be a decision-maker's tool, not a traditional safety program separate and distinct from business and operational decision making.” (p. 10)

However, the panel’s finding on Boeing’s SMS is a mixed bag.  “Boeing provided evidence that it is using its SMS to evaluate product safety decisions and some business decisions. The Expert Panel’s review of Boeing’s SMS documentation revealed detailed procedures on how to use SMS to evaluate product safety decisions, but there are no detailed procedures on how to determine which business decisions affect safety or how they should be evaluated under SMS.” (emphasis added) (p. 35)

The associated recommendation is “Develop detailed procedures to determine which business activities should be evaluated under SMS and how to evaluate those decisions.” (ibid.)  We think the recommendation addresses the specific problem identified in the finding.

One of the major inputs to a decision-making system is an organization’s priorities.  The FAA says safety should always be the top priority but Boeing’s commitment to safety has arguably weakened over time.

“Boeing provided the Expert Panel with a copy of the Boeing Safety Management System Policy, dated April 2022, which states, in part, “… we make safety our top priority.” Boeing revised this policy in August 2023 with . . .  a change to the message “we make safety our top priority” to “safety is our foundation.”” (p. 29)

Lowering the bar did not help.  “The [Expert] panel observed documentation, survey responses, and employee interviews that did not provide objective evidence of a foundational commitment to safety that matched Boeing’s descriptions of that objective.” (p. 22)

Boeing also created seeds of confusion for its safety decision makers.  Boeing implemented its SMS to operate alongside (and not replace or integrate with) its existing safety program.

“During interviews, Boeing employees highlighted that SMS implementation was not to disrupt existing safety program or systems.  SMS operating procedure documents spoke of SMS as the overarching safety program but then also provided segregation of SMS-focused activities from legacy safety activities . . .” (p. 24)

Executive compensation

We have long said that if safety performance is important to an organization then their senior managers’ compensation should have a safety performance-related component. 

Boeing has included safety in its executive financial incentive program.  Safety is one of five factors comprising operational performance which, in turn, is combined with financial performance to determine company-level performance.  Because of the weights used in the incentive model, “The Product Safety measure comprised approximately 4% of the overall 2022 Annual Incentive Award.” (p. 28)

Is 4% enough to influence executive behavior?  You be the judge.

Employee independence from undue management influence   

Boeing’s relationship with the FAA has an aspect that we don’t see in other industries. 

Boeing holds an Organization Designation Authorization (ODA) from the FAA. This allows Boeing to “make findings and issue certificates, i.e., perform discretionary functions in engineering, manufacturing, operations, airworthiness, or maintenance on behalf of the [FAA] Administrator.” (p. 12)

Basically, the FAA delegates some of its authority to Boeing employees, the ODA Unit Members (UMs), who then perform certain assessment and certification tasks.  “When acting as a representative of the Administrator, an individual is required to perform in a manner consistent with the policies, guidelines, and directives of the FAA. When performing a delegated function, an individual is legally distinct from, and must act independent of, the ODA holder.” (ibid.)  These employees are supposed to take the FAA’s view of situations and apply the FAA’s rules even if the FAA’s interests are in conflict with Boeing’s business interests. 

This might work in a perfect world but in Boeing’s world, it’s had and has problems, primarily “Boeing’s restructuring of the management of the ODA unit decreased opportunities for interference and retaliation against UMs, and provides effective organizational messaging regarding independence of UMs. However, the restructuring, while better, still allows opportunities for retaliation to occur, particularly with regards to salary and furlough ranking.” (emphasis added) (p. 5)  In addition, “The ability to comply with the ODA’s approved procedures is present; however, the integration of the SMS processes, procedures, and data collection requirements has not been accomplished.” (p. 26)

To an outsider, this looks like bad organizational design and practices. 

The U.S. commercial nuclear industry offers a useful contrast.  The regulator (Nuclear Regulatory Commission) expects its licensees to follow established procedures, perform required tests and inspections, and report any problems to the NRC.  Self-reporting is key to an effective relationship built on a base of trust.  However, it’s “trust but verify.”  The NRC has their own full-time employees in all the power plants, performing inspections, monitoring licensee operations, and interacting with licensee personnel.  The inspectors’ findings can lead, and have led, to increased oversight of licensee activities by the NRC.

Our perspective

It’s obvious that Boeing has emphasized production over safety.  The problems described above are evidence of broad systemic issues which are not amenable to quick fixes.  Integrating SC into everyday decision-making is hard work of the “continuous improvement” variety; it will not happen by management fiat.  Adjusting the compensation plan will require the Board to take safety more seriously.  Reworking the ODA program to eliminate all pressures and goal conflicts may not be possible; this is a big problem because the FAA has effectively deputized 1,000 people to perform FAA functions at Boeing. (p. 25)

The report only covers the most visible SC issues.  Complacency, normalization of deviation, the multitude of biases that can affect decision-making, and other corrosive factors are perennial threats to a strong SC and can affect “the natural drift in organizations.” (p. 40)  Such drift may lead to everything from process inefficiencies to tragic safety failures.

Boeing has taken one step: they fired the head of the 737 MAX program.**  Organizations often toss a high-level executive into a volcano to appease the regulatory gods and buy some time.  Boeing’s next challenge is that the FAA has given Boeing 90 days to fix its quality problems highlighted by the door plug blowout.***

Bottom line: Grab your popcorn, the show is just starting.  Boeing is probably too big to fail but it is definitely going to be pulled through the wringer. 

*  “Section 103Organization Designation Authorizations (ODA) for Transport Airplanes Expert Panel Review Report,” Federal Aviation Administration (Feb. 26, 2024). 

**  N. Robertson, “Boeing fires head of 737 Max program,” The Hill (Feb. 21, 2024).

***  D. Shepardson and V. Insinna, “FAA gives Boeing 90 days to develop plan to address quality issues,” Reuters (Feb. 28, 2024).

Leadership, Decisions, Systems Thinking and Nuclear Safety Culture

AcciMap Excerpt

We recently read a paper* that echoes some of the themes we emphasize on Safetymatters, viz., leadership, decisions and a systems view.  Following is an excerpt from the abstract:

“Leadership is progressively being recognized as a key** factor in supporting successful performance across a range of domains. . . . the decisions and actions that characterize safety leadership thus become important emergent properties in the prevention of incidents, which should be considered within the context of the broader organizational system and not merely constrained to understanding events or conditions that shape performance at the ‘sharp end’.”  [emphasis added]

The authors go on to analyze decisions and actions after a mining incident (landslide) using a combination of three different schemes: Rasmussen’s Risk Management Framework (RMF) and corresponding AcciMap, and the Critical Decision Method (CDM).

The RMF describes work systems as comprised of various levels and argues that safety performance is affected by decisions and actions at all levels from politicians in the external environment down through company executives and managers and finally to individual workers.  Rasmussen’s AcciMap is an expansive causal diagram for an accident or incident that displays the contributions (or omissions) at each level in the RMF and their connections.

CDM uses semi-structured interviews to obtain information about how individuals formulate their decisions, including context such as background knowledge and immediate influencing factors.  Consistent with the RMF, case study interviews were conducted with individuals at different organizational levels.  CDM data were used to construct the AcciMap.

We won’t go into the details of the analysis but it identified over a dozen key decisions made at different organizational levels before and during the incident; most were connected to at least one other key decision.  The AcciMap illustrates decisions and communications across multiple levels and thus provides a useful picture of how an organization anticipates and responds to an unusual situation.

Our Perspective

The authors argue, and we agree, that this type of analysis provides greater detail and insight into the performance of an organization’s safety management system than traditional accident investigations (especially those focused on finding someone to blame).

This article does not specifically discuss culture.  But the body of decisions an organization produces is the strongest evidence and most visible artifact of its culture.  Organizational decisions are far more important than responses to surveys or interviews where people can report what they believe (or hope) the culture is, or what they think their audience wants to hear.

We like that RMF and AcciMap are agnostic: they can be used to analyze either “what went wrong” or “what went right” scenarios.  (The case study was in the latter category because no one was hurt in the incident.)  If an assessor is looking at a sample of decisions to infer a nuclear organization’s culture, most of those decisions will have had positive (or at least no negative) consequences.

The authors are Australian academics but this short (8 pages total) paper is quite readable and a good introduction to CDM and Rasmussen’s constructs.  The references include people whose work we have positively reviewed on Safetymatters, including Dekker, Hollnagel, Leveson and Reason.

Bottom line: There is nothing about culture or nuclear here, but the overall message reinforces our beliefs about how to think about Nuclear Safety Culture.


*  S-L Donovana, P.M. Salmonb and M.G. Lennéa, “The leading edge: A systems thinking methodology for assessing safety leadership,” Procedia Manufacturing 3 (2015), pp. 6644–6651.  Available at sciencedirect.com; retrieved Jan. 19, 2017.

**  Note they do not say “one and only” or even “most important.”

The Criminalization of Safety (Part 3)

Our Perspective

The facts and circumstances of the events described in Table 1 in Part 1 point to a common driver - the collision of business and safety priorities, with safety being compromised.  Culture is inferred as the “cause” in several of the events but with little amplification or specifics.^[1]  The compromises in some cases were intentional, others a product of a more complex rationalization.  The events have been accompanied by increased criminal prosecutions with varied success. 

We think it is fair to say that so far, criminalization of safety performance does not appear to be an effective remedy.  Statutory limitations and proof issues are significant limitations with no easy solution. The reality is that criminalization is at its core a “disincentive”.  To be effective it would have to deter actions or decisions that are not consistent with safety but not create a minefield of culpability.  It is also a blunt instrument requiring rather egregious behavior to rise to the level of criminality.  Its best use is probably as an ultimate boundary, to deter intentional misconduct but not be an unintended trap for bad judgment or inadequate performance.  In another vein, criminalization would also seem incompatible with the concept of a “just culture” other than for situations involving intentional misconduct or gross negligence.

Whether effective or not, criminalization reflects the urgency felt by government authorities to constrain excessive risk taking, intentional or not, and enhance oversight.  It is increasingly clear that current regulatory approaches are missing the mark.  All of the events catalogued in Table 1 occurred in industries that are subject to detailed safety and environmental regulation.  After the fact assessments highlight missed opportunities for more assertive regulatory intervention, and in the Flint cases there are actual criminal charges being applied to regulators.  The Fukushima event precipitated a complete overhaul of the nuclear regulatory structure in Japan, still a work in progress.  Post hoc punishments, no matter how severe, are not a substitute.

Nuclear Regulation Initiatives

Looking specifically at nuclear regulation in the U.S. we believe several specific reforms should be considered. It is always difficult to reform without the impetus of a major safety event, but we could see these actions as ones that could appear obvious in a post-event assessment if there was ever an “O-ring” moment in the nuclear industry.^[2]

1. The NRC should include the safety management system in its regulatory activities.

The NRC has effectively constructed a cordon sanitaire around safety management by decreeing that “management” is beyond the scope of regulation.  The NRC relies on the fact that licensees bear the primary responsibility for safety and the NRC should not intrude into that role.  If one contemplates the trend of recent events scrutinizing the performance of regulators following safety events, this legalistic “defense” may not fare well in a situation where more intrusive regulation could have made the difference.

The NRC does monitor “safety culture” and often requires licensees to address weaknesses in culture following performance issues.  In essence safety culture has become an anodyne for avoiding direct confrontation of safety management issues.  Cynically one could say it is the ultimate conspiracy - where regulators and “stakeholders” come together to accept something that is non-contentious and conveniently abstract to prevent a necessary but unwanted (apparently by both sides) intrusion into safety management.

As readers of this blog know, our unyielding focus has been on the role of the complex socio-technical system that functions within a nuclear organization to operate nuclear plants effectively and safely.  This management system includes many drivers, variables, feedbacks, culture, and time delays in its processes, not all of which are explicit or linear.  The outputs of the system are the actions and decisions that ultimately produce tangible outcomes for production and safety.  Thus it is a safety system and a legitimate and necessary area for regulation.

NRC review of safety management need not focus on traditional management issues which would remain the province of the licensee.  So organizational structure, personnel decisions, etc. need not be considered.^[3]  But here we should heed the view of Daniel Kahneman where he suggests we think of organizations as “factories for producing decisions” and therefore, think of decisions as a product.  (See our Nov. 4,2011 post, A Factory for Producing Decisions.)  Decisions are in fact the key product of the safety management system.  Regulatory focus on how the management system functions and the decisions it produces could be an effective and proactive approach.

We suggest two areas of the management system that could be addressed as a first priority: (1) Increased transparency of how the management system produces specific safety decisions including the capture of objective data on each such decision, and (2) review of management compensation plans to minimize the potential for incentives to promote excessive risk taking in operations.

2. The NRC should require greater transparency in licensee management decisions with potential safety impacts.

Managing nuclear operations involves a continuum of decisions balancing a variety of factors including production and safety.  These decisions may occur with individuals or with larger groups in meetings or other forums.  Some may involve multiple reviews and concurrences.  But in general the details of decision making, i.e., how the sausage is made, are rarely captured in detail during the process or preserved for later assessment.^[4]  Typically only decisions that happen to yield a bad outcome (e.g., prompt the issuance of an LER or similar) become subject to more intensive review and post mortem.  Or actions that require specific, advance regulatory approval and require an SER or equivalent.^[5]  

Transparency is key.  Some say the true test of ethics is what people do when no one is looking.  Well the converse of that may also be true - do people behave better when they know oversight is or could be occurring?  We think a lot of the NRC’s regulatory scheme is already built on this premise, relying as it does on auditing licensee activities and work products.

Thinking back to the Davis Besse example, the criminal prosecutions of both the corporate entity and individuals were limited to providing false or incomplete information to the NRC.  There was no attempt to charge on the basis of the actual decisions to propose, advocate for, and attempt to justify, that the plant could continue to operate beyond the NRC’s specified date for corrective actions.  The case made by First Energy was questionable as presented to the NRC and simply unjustified when accounting for the real facts behind their vessel head inspections.

Transparency would be served by documenting and preserving the decision process on safety significant issues.  These data might include the safety significance and applicable criteria, the potential impact on business performance (plant output, cost, schedule, etc), alternatives considered, and the participants and their inputs to the decision making process, and how a final decision was reached.   These are the specifics that are so hard or impossible to reproduce after the fact.^[6]  The not unexpected result: blaming someone or something but not gaining insight into how the management system failed.

This approach would provide an opportunity for the NRC to audit decisions on a routine basis.  Licensee self assessment would also be served through safety committee review and other oversight including INPO.  Knowing that decisions will be subject to such scrutiny also can promote careful balancing of factors in safety decisions and serve to articulate how those balances are achieved and safety is served.  Having such tangible information shared throughout the organization could be the strongest way to reinforce the desired safety culture.

3. As part of its regulation of the safety management system, the NRC should restrict incentive compensation for nuclear management that is based on meeting business goals.

We started this series of posts focusing on criminalization of safety.  One of the arguments for more aggressive criminalization is essentially to offset the powerful pull of business-based incentives with the fear of criminal sanctions.  This has proved to elusive.  Similarly attempting to balance business incentives with safety incentives also is problematic.  The Transocean experience illustrates that quite vividly.^[7]

Our survey several years ago of nuclear executive compensation indicated (1) the amounts of compensation are very significant for the top nuclear executives, (2) the compensation is heavily dependent on each year’s performance, and (3) business performance measured by EPS is the key to compensation, safety performance is a minor contributor.  A corollary to the third point might be that in no cases that we could identify was safety performance a condition precedent or qualification for earning the business-based incentives. (See our July 9, 2010 post, Nuclear Management Compensation (Part 2)).  With 60-70% of total compensation at risk, executives can see their compensation, and that of the entire management team, impacted by as much as several million dollars in a year.  Can this type of compensation structure impact safety?  Intuition says it creates both risk and a perception problems.  Virtually every significant safety event in Table 1 has reference to the undue influence of production priorities on safety.  The issue was directly raised in at least one nuclear organization^[8] which revised its compensation system to avoid undermining safety culture. 

We believe a more effective approach is to minimize the business pressures in the first place.  We believe there is a need for a regulatory policy that discourages or prohibits licensee organizations from utilizing significant incentives based on financial performance.  Such incentives invariably target production and budget goals as they are fundamental to business success.  To the extent safety goals are included they are a small factor or based on metrics that do not reflect fundamental safety.  Assuring safety is the highest priority is not subject to easily quantifiable and measurable metrics - it is judgmental and implicit in many actions and decisions taken on a day-to-day basis at all levels of the organization.  Organizations should pay nuclear management competitively and generously and make informed judgments about their overall performance.

Others have recognized the problem and taken similar steps to address it.  For example, in the aftermath of the financial crisis of 2008 the Federal Reserve Board has been doing some arm twisting with U.S. financial services companies to adjust their executive compensation plans - and those plans are in fact being modified to cap bonuses associated with achieving performance goals. (See our April 25, 2013 post, Inhibiting Excessive Risk Taking by Executives.)

Nick Taleb (of Black Swan fame) believes that bonuses provide an incentive to take risks.  He states, “The asymmetric nature of the bonus (an incentive for success without a corresponding disincentive for failure) causes hidden risks to accumulate in the financial system and become a catalyst for disaster.”  Now just substitute “nuclear operations” for “the financial system”.

Central to Taleb’s thesis is his belief that management has a large informational advantage over outside regulators and will always know more about risks being taken within their operation. (See our Nov. 9, 2011 post, Ultimate Bonuses.)  Eliminating the force of incentives and providing greater transparency to safety management decisions could reduce risk and improve everybody’s insight into those risks deemed acceptable.

Conclusion

In industries outside the commercial nuclear space, criminal charges have been brought for bad outcomes that resulted, at least in part, from decisions that did not appropriately consider overall system safety (or, in the worst cases, simply ignored it.)  Our suggestions are intended to reduce the probability of such events occurring in the nuclear industry.

---

^[1] It raises the question whether anytime business priorities trump safety it is a case of deficient culture.  We have argued in other blog posts that sufficiently high business or political pressure can compromise even a very strong safety culture.  So reflexive resort to safety culture may be easy but not be very helpful.

^[2] Credit to Adam Steltzner author of The Right Kind of Crazy recounting his and other engineers’ roles in the design of the Mars rovers.  His reference is to the failure of O-ring seals on the space shuttle Challenger.

^[3] We do recognize that there are regulatory criteria for general organizational matters such as for the training and qualification of personnel. 

^[4] In essence this creates a “safe harbor” for most safety judgments and to which the NRC is effectively blind.

^[5] In Davis Besse much of the “proof” that was relied on in the prosecutions of individuals was based on concurrence chains for key documents and NRC staff recollections of what was said in meetings.  There was no contemporaneous documentation of how First Energy made its threshold decision that postponing the outage was acceptable, who participated, and who made the ultimate decision.  Much was made of the fact that management was putting great pressure on maintaining schedule but there was no way to establish how that might have directly affected decision making.

^[6] Kahneman believes there is “hindsight bias”.  Hindsight is 20/20 and it supposedly shows what decision makers could (and should) have known and done instead of their actual decisions that led to an unfavorable outcome, incident, accident or worse.  We now know that when the past was the present, things may not have been so clear-cut.  See our Dec.18, 2013 post, Thinking, Fast and Slow by Daniel Kahneman.

^[7] Transocean, owner of the Deepwater Horizon oil rig, awarded millions of dollars in bonuses to its executives after “the best year in safety performance in our company’s history,” according to an annual report…’Notwithstanding the tragic loss of life in the Gulf of Mexico, we achieved an exemplary statistical safety record as measured by our total recordable incident rate and total potential severity rate.’”  See our April 7, 2011 post for the original citation in Transocean's annual report and further discussion.

^[8] “The reward and recognition system is perceived to be heavily weighted toward production over safety”.  The reward system was revised "to ensure consistent health of NSC”.  See our July 29, 2010 post, NRC Decision on FPL (Part 2).

Training....Yet Again

U.S. Navy SEALS in Training

We have beat the drum on the value of improved and innovative training techniques for improving safety management performance for some time.  Really since the inception of this blog where our paper, “Practicing Nuclear Safety Management,”* was one of the seminal perspectives we wanted to bring to our readers.  We continue to encounter knowledgeable sources that advocate practice-based approaches and so continue to bring them to our readers’ attention.  The latest is an article from the Harvard Business Review that calls attention to, and distinguishes, “training” as an essential dimension of organizational learning.  The article is “How the Navy SEALS Train for Leadership Excellence.”**  The author, Michael Schrage,*** is a research fellow at MIT who reached out to a former SEAL, Brandon Webb, who transformed SEAL training.  The author contends that training, as opposed to just education or knowledge, is necessary to promote deep understanding of a business or market or process.  Training in this sense refers to actually performing and practicing necessary skills.  It is the key to achieving high levels of performance in complex environments. 

One of Webb’s themes that really struck a chord was: “successful training must be dynamic, open and innovative…. ‘It’s every teacher’s job to be rigorous about constantly being open to new ideas and innovation’, Webb asserts.”  It is very hard to think about much of the training in the nuclear industry on safety culture and related issues as meeting these criteria.  Even the auto industry has recently stepped up to require the conduct of decision simulations to verify the effectiveness of corrective actions - in the wake of the ignition switch-related accidents. (see our May 22, 2014 post.)

In particular the reluctance of the nuclear industry and its regulator to address the presence and impact of goal conflicts on safety continues to perplex us and, we hope, many others in the industry.   It was on the mind of Carlo Rusconi more than a year ago when he observed: “Some of these conflicts originate high in the organization and are not really amenable to training per se” (see our Jan. 9, 2014 post.)  However a certain type of training could be very effective in neutralizing such conflicts - practicing making safety decisions against realistic fact-based scenarios.  As we have advocated on many occasions, this process would actualize safety culture principles in the context of real operational situations.  For the reasons cited by Rusconi it builds teamwork and develops shared viewpoints.  If, as we have also advocated, both operational managers and senior managers participated in such training, senior management would be on the record for its assessment of the scenarios including how they weighed, incorporated and assessed conflicting goals in their decisions.  This could have the salutary effect of empowering lower level managers to make tough calls where assuring safety has real impacts on other organizational priorities.  Perhaps senior management would prefer to simply preach goals and principles, and leave the tough balancing that is necessary to implement the goals to their management chain.  If decisions become shaded in the “wrong” direction but there are no bad outcomes, senior management looks good.  But if there is a bad outcome, lower level managers can be blamed, more “training” prescribed, and senior management can reiterate its “safety is the first priority” mantra.


*  In the paper we quote from an article that highlighted the weakness of “Most experts made things worse.  Those managers who did well gathered information before acting, thought in terms of complex-systems interactions instead of simple linear cause and effect, reviewed their progress, looked for unanticipated consequences, and corrected course often. Those who did badly relied on a fixed theoretical approach, did not correct course and blamed others when things went wrong.”  Wall Street Journal, Oct. 22, 2005, p. 10 regarding Dietrich Dörner’s book, The Logic of Failure.  For a comprehensive review of the practice of nuclear safety, see our paper “Practicing Nuclear Safety Management”, March 2008.

**  M. Schrage, "How the Navy SEALS Train for Leadership Excellence," Harvard Business Review (May 28, 2015).

***  Michael Schrage, a research fellow at MIT Sloan School’s Center for Digital Business, is the author of the book Serious Play among others.  Serious Play refers to experiments with models, prototypes, and simulations.

Introducing NuclearSafetySim

We have referred to NuclearSafetySim and the use of simulation tools on a regular basis in this blog.  NuclearSafetySim is our initiative to develop a new approach to safety management training for nuclear professionals.  It utilizes a simulator to provide a realistic nuclear operations environment within which players are challenged by emergent issues - where they must make decisions balancing safety implications and other priorities - over a five year period.  Each player earns an overall score and is provided with analyses and data on his/her decision making and performance against goals.  It is clearly a different approach to safety culture training, one that attempts to operationalize the values and traits espoused by various industry bodies.  In that regard it is exactly what nuclear professionals must do on a day to day basis. 

At this time we are making NuclearSafetySim available to our readers through a web-based demo version.  To get started you need to access the NuclearSafetySim website.  Click on the Introduction tab at the top of the Home page.  Here you will find a link to a narrated slide show that provides important background on the approach used in the simulation.  It runs about 15 minutes.  Then click on the Simulation tab.  Here you will find another video which is a demo of NuclearSafetySim.  While this runs about 45 minutes (apologies) it does provide a comprehensive tutorial on the sim and how to interact with it.  We urge you to view it.  Finally...at the bottom of the Simulation page is a link to the NuclearSafetySim tool.  Clicking on the link brings you directly to the Home screen and you’re ready to play.

As you will see on the website and in the sim itself, there are reminders and links to facilitate providing feedback on NuclearSafetySim and/or requesting additional information.  This is important to us and we hope our readers will take the time to provide thoughtful input, including constructive criticism.  We welcome all comments. 

Timely Safety Culture Research

In this post we highlight the doctoral thesis paper of Antti Piirto, “Safe Operation of Nuclear Power Plants – Is Safety Culture an Adequate Management Method?”*  One reason for our interest is the author’s significant background in nuclear operations.**  Thus his paper has academic weight but is informed by direct management experience.

It would be impossible to credibly summarize all of the material and insights from this paper as it covers a wide swath of safety management and culture and associated research.  The pdf is 164 pages.  In this post we will provide an overview of the material with pointers to some aspects that seem most interesting to us.

Overview

The paper is developed from Piirto’s view that “Today there is universal acceptance of the significant impact that management and organisational factors have over the safety significance of complex industrial installations such as nuclear power plants. Many events with significant economic and public impact had causes that have been traced to management deficiencies.” (p. i)  It provides a comprehensive and useful overview of the development of safety management and safety culture thinking and methods, noting that all too often efforts to enhance safety are reactive.

“For many years it has been considered that managing a nuclear power plant was mostly a matter of high technical competence and basic managerial skills.” (p. 3)  And we would add, in many quarters there is a belief that safety management and culture simply flow from management “leadership”.  While leadership is an important ingredient in any management system, its inherent fuzziness leaves a significant gap in efforts to systematize methods and tools to enhance performance outcomes.  Again citing Piirto, safety culture is “especially vague to those carrying out practical safety work. Those involved...require explanations concerning how safety culture will alter their work” (p. 4)

Piirto also cites the prevalence in the nuclear industry of “unilateral thinking” and the lack of exposure to external criticism of current nuclear management approaches, accompanied by “homogeneous managerial rhetoric”. (p. 4)

“Safety management at nuclear power plants needs to become more transparent in order to enable us to ensure that issues are managed correctly.” (p. 6)  “Documented safety thinking provides the organisation with a common starting point for future development.” (p. 8)  Transparency and the documentation (preservation) of safety thinking resonates with us.  When forensic efforts have been made to dissect safety thinking (e.g., see Perin’s book Shouldering Risks) it is apparent how illuminating and educational such information can be.

Culture as Control Mechanism

Piirto describes organizational culture as…”a socially constructed, unseen, and unobservable force behind organisational activities.” (p. 13)  “It functions as an organisational control mechanism, informally approving or prohibiting behaviours.” (p. 14)

We would say that in terms of a control mechanism, culture’s effect should be clarified as being one of perhaps many mechanisms that ultimately combine to determine actual behavior.  In our conceptual model safety culture specifically can be thought of as a resistance to other non-safety pressures affecting people and their actions.  (See our post dated June 29, 2012.)  Piirto calls culture a “powerful lever” for guiding behavior. (p. 15)  The stronger the resident safety culture is the more leverage it has to keep in check other pressures.  However it is also almost inevitable that there can be some amount of non-safety pressure that compromises the control leverage of safety culture and perhaps leads to undesired outcomes.

Some of Piirto’s most useful insights can be found on p. 14 where he explains that culture at its essence is “a concept rather than a thing” - and a concept created in people’s minds.  We like the term “mental model” as well.  He goes on to caution that we must remember that culture is not just a set of structural elements or constructs - “It also is a dynamic process – a social construction that is undergoing continual reconstruction.”  Perhaps another way of saying this is to realize that culture cannot be understood apart from its application within an organization.  We think this is a significant weakness of culture surveys that tend to ask questions in the abstract, e.g., “Is safety a high priority?”, versus exploring precisely how safety priorities are exercised in specific decisions and actions of the organization.

Piirto reviews various anthropologic and sociologic theories of culture including debate about whether culture is a dependent or independent variable (p.18), the origins of safety culture, and culture surveys. (pp. 23-24)

Some other interesting content can be found starting at Section 2.2.7 (p. 29) where Piirto reviews approaches to the assessment of safety culture, which really amounts to - what is the practical reality associated with a culture.  He notes “the correlation between general preferences and specific behaviour is rather modest.” and “The Situational Approach suggests that the emphasis should be put on collecting data on actual practices, real dilemmas and decisions (what is also called “theories in use”) rather than on social norms.” (p. 29)

Knowledge Management and Training

Starting on p. 39 is a very useful discussion of Knowledge Management including its inherently dynamic nature.  Knowledge Management is seen as being at the heart of decision making and in assessing options for action.

In terms of theories of how people behave, there are two types, “...the espoused theory, or how people say they act, and the theory-in-use, or how people actually act. The espoused theory is easier to understand. It describes what people think and believe and how they say they act. It is often on a conscious level and can be easily changed by new ideas and information. However, it is difficult to be aware of the theory-in-use, and it is difficult to change...” (p. 46)

At this juncture we would like to have seen a closer connection between the discussions of Knowledge Management and safety management.  True, ensuring that individuals have the benefit of preserving, sustaining and increasing knowledge is important, but how exactly does that get reflected in safety management performance?  Piirto does draw an analogy between systematic approaches to training and proposes that a similar approach would benefit safety management, by documenting how safety is related to practical work.  “This would turn safety culture into a concrete tool. Documented safety thinking provides the organisation with a common starting point for future development.” (p. 61)

One way to document safety thinking is through event investigation.  Piirto observes, “Event investigation is generally an efficient starting point for revealing the complex nature of safety management. The context of events reveals the complex interaction between people and technology in an organisational and cultural context. Event investigations should not only focus on events with high consequences; in most complex events a through investigation will reveal basic causes of great interest, particularly at the safety management level. Scientific studies of event investigation techniques and general descriptions of experience feedback processes have had a tendency to regard event investigations as too separated from a broader safety management context.”  (p. 113)

In the last sections of the paper Piirto summarizes the results of several research projects involving training and assessment of training effectiveness, knowledge management and organizational learning.  Generally these involve the development and training of shift personnel.

Take Away

Ultimately I’m not sure that the paper provides a simple answer to the question posed in its title: Is safety culture an adequate management method?  Purists would probably observe that safety culture is not a management method; on the other hand I think it is hard to ignore the reliance being placed by regulatory bodies on safety culture to help assure safety performance.  And much of this reliance is grounded in an “espoused theory” of behavior rather than a systematic, structured and documented understanding of actual behaviors and associated safety thinking.  Such “theory in use” findings would appear to be critical in connecting expectations for values and beliefs to actual outcomes.  Perhaps the best lesson offered in the paper is that there needs to be a much better overall theory of safety management that links cultural, knowledge management and training elements.


*  A. Piirto,  “Safe Operation of Nuclear Power Plants – Is Safety Culture an Adequate Management Method?” thesis for the degree of Doctor of Science in Technology (Tampere, Finland: Tampere Univ. of Technology, 2012).

**  Piirto has a total of 36 years in different management and supervision tasks in a nuclear power plant organization, including twelve years as the Manager of Operation for the Olkiluoto nuclear power plant.

Dynamic Interactive Training

The words dynamic and interactive always catch our attention as they are intrinsic to our world view of nuclear safety culture learning.  Carlo Rusconi’s presentation* at the recent IAEA International Experts’ Meeting on Human and Organizational Factors in Nuclear Safety in the Light of the Accident at the Fukushima Daiichi Nuclear Power Plant in Vienna in May 2013 is the source of our interest.

While much of the training described in the presentation appeared to be oriented to the worker level and the identification of workplace type hazards and risks, it clearly has implications for supervisory and management levels as well.

In the first part of the training students are asked to identify and characterize safety risks associated with workplace images.  For each risk they assign an index based on perceived likelihood and severity.  We like the parallel to our proposed approach for scoring decisions according to safety significance and uncertainty.**

“...the second part of the course is focused on developing skills to look in depth at events that highlight the need to have a deeper and wider vision of safety, grasping the explicit and implicit connections among technological, social, human and organizational features. In a nutshell: a systemic vision.” (slide 13, emphasis added)  As part of the training students are exposed to the concepts of complexity, feedback and internal dynamics of a socio-technical system.  As the author notes, “The assessment of culture within an organization requires in-depth knowledge of its internal dynamics”.  (slide 15)

This part of the training is described as a “simulation” as it provides the opportunity for students to simulate the performance of an investigation into the causes of an actual event.  Students are organized into three groups of five persons to gain the benefit of collective analysis within each group followed by sharing of results across groups.  We see this as particularly valuable as it helps build common mental models and facilitates integration across individuals.  Last, the training session takes the student’s results and compares them to the outcomes from a panel of experts.  Again we see a distinct parallel to our concept of having senior management within the nuclear organization pre-analyze safety issues to establish reference values for safety significance, uncertainty and preferred decisions.  This provides the basis to compare trainee outcomes for the same issues and ultimately to foster alignment within the organization.

Thank you Dr. Rusconi.


*  C. Rusconi, “Interactive training: A methodology for improving Safety Culture,” IAEA International Experts’ Meeting on Human and Organizational Factors in Nuclear Safety in the Light of the Accident at the Fukushima Daiichi Nuclear Power Plant, Vienna May 21-24, 2013.

**  See our blog posts dated April 9 and June 6, 2013.  We also remind readers of Taleb’s dictate to decision makers to focus on consequences versus probability in our post dated June 18, 2013.

Implementing Safety Culture Policy Part 2

This post continues our discussion of the implementation of safety culture policy in day-to-day nuclear management decision making, started in our post dated April 9, 2013.   In that post we introduced several parameters for quantitatively scoring decisions: decision quality, safety significance and significance uncertainty.  At this time we want to update the decision quality label, using instead “decision balance”.

To illustrate the application of the scoring method we used a set of twenty decisions based on issues taken from actual U.S. nuclear operating experience, typically those that were reported in LERs.  As a baseline, we scored each issue for safety significance and uncertainty.  Each issue identified 3 to 4 decision options for addressing the problem - and each option was annotated with the potential impacts of the decision on budgets, generation (e.g. potential outage time) and the corrective action program.   We scored each decision option for its decision balance (how well the decision option balances safety priority) and then identified the preferred decision option for each issue.  This constitutes what we refer to as the “preferred decision set”.  A pdf file of one example issue with decision choices and scoring inputs is available here. 

Our assumption is that the preferred decision set would be established/approved by senior management based on their interpretation of the issues and their expectations for how organizational decisions should reflect safety culture.  The set of issues would then be used in a training environment for appropriate personnel.  For purposes of this example, we incorporated the preferred decision set into our NuclearSafetySim* simulator to illustrate the possible training experience.  The sim provides an overall operational context tracking performance for cost, plant generation and CAP program and incorporating performance goals and policies.

Chart 1

In the sim application a trainee would be tasked with assessing an issue every three months over a 60 month operational period.  The trainee would do this while attempting to manage performance results to achieve specified goals.  For each issue the trainee would review the issue facts, assign values for significance and uncertainty, and select a decision option.  Chart 1 compares the actual decisions (those by the trainee) to those in the preferred set for our prototype session.   Note that approximately 40% of the time the actual decision matched the preferred decision (orange data points).  For the remainder of the issues the trainee’s selected decisions differed.  Determining and understanding why the differences occurred is one way to gain insight into how culture manifests in management actions.

As we indicated in the April 9 post, each decision is evaluated for its safety significance and uncertainty in accordance with quantified scales.  These serve as key inputs to determining the appropriate balance to be achieved in the decision.  In prior work in this area, reported in our posts dated July 15, 2011 and October 14, 2011 we solicited readers to score two issues for safety significance.  The reported scores ranged from 2 to 10 (most scores between 4 to 6) for one issue and ranged 5 to 10 (most scores 6 to 8) for the other issue.  This reflects the reality that perceptions of safety significance are subject to individual differences.  In the current exercise, similar variations in scoring were expected and led to differences between the trainee’s scores and the preferred decision set.  The variation may be due to the inherent subjective nature of assessing these attributes and other factors such as experience, expertise, biases, and interpretations of the issue.  So this could be one source of difference in the trainee decision selections versus the preferred set, as the decision process attempts to match action to significance. 

Another source could be in the decision options themselves.   The decision choice by a trainee could have focused on what the trainee felt was the “best” (i.e., most efficacious) decision versus an explicit consideration of safety priority commensurate with safety significance.  Additionally decision choices may have been influenced by their potential impacts, particularly under conditions where performance was not on track to meet goals. 

Chart 2

Taking this analysis a bit further, we looked at how decision balance varied over the course of the simulation.  As discussed in our April 9 post we use decision balance to create a quantitative measure of how well the goal of safety culture is being incorporated in a specific decision - the extent to which the decision accords the priority for safety commensurate with its safety significance.  In the instant exercise, each decision option for each issue has been assigned a balance value as part of the preferred scoresheet.**  Chart 2 shows a timeline of decision balances - one for the preferred decision set and the other for the actual decisions made by the trainee.  A smoothing function has been applied to the discrete values of balance to provide a continuous track. 

The plots illustrate how decision balance may vary over time, with specific decisions reflecting greater or lesser emphasis on safety.  During the first half of the sim the decision balances are in fairly close agreement, reflecting in part that in 5 of 8 cases the actual decisions matched the preferred decisions.  However in the second half of the sim significant differences emerge, primarily in the direction of weaker balances associated with the trainee decisions.  Again, understanding why these differences emerge could provide insight into how safety culture is actually being practiced within the organization. Chart 3 adds in some additional context.

Chart 3

The yellow line is a plot of “goal pressure” which is simply a sum of the differences in actual performance in the sim to goals for cost, generation and CAP program.  Higher values of pressure are associated with performance lagging the goals.  Inspection of the plot indicates that goal pressure was mostly modest in the first half of the sim before an initial spike up and further increases with time.  The blue line, the decision balance of the trainee, does not show any response to the initial spike, but later in the sim the high goal pressure could be seen as a possible contributor to decisions trending to lower balances.  A final note is that over the course of the entire sim, the average values of preferred and actual balance are fairly close for this player, perhaps suggesting reasonable overall alignment in safety priorities notwithstanding decision to decision variations. 

A variety of training benefits can flow from the decision simulation.  Comparisons of actual to preferred decisions provide a baseline indication of how well expected safety balances are being achieved in realistic decisions.  Consideration of contributing factors such as goal pressure may illustrate challenges for decision makers.  Comparisons of results among and across groups of trainees could provide further insights.  In all cases the results would provide material for discussion, team building and alignment on safety culture.

In our post dated November 4, 2011 we quoted the work of Kahneman, that organizations are “factories for producing decisions”.  In nuclear safety, the decision factory is the mechanism to actualize safety culture into specific priorities and actions.  A critical element of achieving strong safety culture is to be able to identify differences between espoused values for safety (i.e., the traits typically associated with safety culture) and de facto values as revealed in actual decisions. We believe this can be achieved by capturing decision data explicitly, including the judgments on significance and uncertainty, and the operational context of the decisions.

The next step is synthesizing the decision and situational parameters to develop a useful systems-based measure of safety culture.  A quantity that could be tracked in a simulation environment to illustrate safety culture response and provide feedback and/or during nuclear operations to provide a real time pulse of the organization’s culture.


* For more information on using system dynamics to model safety culture, please visit our companion website, nuclearsafetysim.com.

** It is possible for some decision options to have the same value of balance even though they incorporate different responses to the issue and different operational impacts.