Culture de la Sécurité

If you are paying attention you noticed we’re using French words.  And the reason is the current news regarding Air France and its safety practices and safety culture.

The principal finding of an independent study of the airline’s operations was a lack of “strong safety leadership at all levels of management" as reported in a January 26 Wall Street Journal (WSJ) article.*

While Air France has refused to make the report public, a review by the WSJ stated that the study was “sharply critical of broad aspects of the safety culture”.

Over the previous several days there have been articles in the WSJ preceding and attending the completion of the independent study.  The first of the articles** previewed some of the findings and provided favorable commentary based on an understanding that Air France would be making the report public (“Air France to Disclose…”).  This was characterized as a move toward greater openness on the part of the airline and commented, “safety experts said it was unusual for a large company, especially an airline, to give outsiders such latitude publicly to expose gaps in safety systems.”  And quoting Bill Voss, president of the Flight Safety Foundation, a global air-safety advocacy organization based in Alexandria, Va., "It's extraordinary that they are willing to release the results," said Mr. Voss. The process "gives me confidence there will be follow-through."

Well, in a follow-up article the next day, when receipt of the study results was announced by Air France, the reaction was disappointment as the airline indicated it would not release the report or its recommendations.  Nonetheless the airline trumpeted its actions:

“According to an Air France news release, the report said that creating the outside review team ‘in a public manner and providing it a broad charter’ to examine flight safety ‘was a courageous act’ and an example of safety leadership ‘rarely seen in today's international aviation industry.’” ***

In deciding on whether Air France should be awarded the Légion d'honneur, one might also consider that the current safety study was undertaken in the wake of four serious crashes between 1999 and 2009 (the most recent being the flight from Rio to Paris that went down mid-Atlantic).  In addition, it follows a prior safety study,

“Finished in 2006, that report was distributed to more than 4,000 company pilots and was widely praised for its frankness about shortcomings within the carrier. Although Air France has said its executive committee made formal decisions to implement the report's recommendations, critics of the airline continue to maintain management didn't aggressively pursue the changes.” (Jan 24, 2011)

I think we’ve seen this before.  Think about the safety studies after the Challenger crash, but not really implemented and then the disintegration on reentry of the Columbia.  Or BP and the Texas oil refinery fire followed by the Deepwater Horizon last summer.  Obviously safety assessments, no matter how strong and how independent, ultimately require the subject organization to implement changes.  We think the current Air France report correctly fingers safety leadership by management “starting at the top”.  And it never fails in these situations that top management describes as its highest priority…...can anybody guess……that’s right, it’s “safety first”.  (Jan 26, 2011)

Perhaps if it was safety first Air France might be releasing the study and its recommendations.  Wouldn’t that help make real its safety priority and wouldn’t such transparency help ensure that the recommendations are actually implemented?   We have commented in prior posts on transparency and we will continue to emphasize its importance to safety culture across all industries.


*  A. Pasztor, D. Michaels and D. Gauthier-Villars, “Air France Panel Cites Wide Safety Deficiencies,” WSJ.com (Jan 26, 2011).

** A. Pasztor and D. Michaels, “Air France to Disclose Review's Criticisms,” WSJ.com (Jan 24, 2011).

***  A. Pasztor, “Air France Enhances Safety Efforts,” later re-headlined “Air France Withholds Key Report,” WSJ.com (Jan 25, 2011).

A Nuclear Model for Oil and Gas

The President’s Commission has issued its report on the Deepwater Horizon disaster.* The report reviews the history of the tragedy and makes recommendations based on lessons learned.  This post focuses on the report’s use of the nuclear industry, in particular the role played by INPO, as a model for an oil and gas industry safety institute and auditor.

The report provides an in-depth review of INPO’s role and methods and we will not repeat that review in this space.  We want to highlight the differences between the oil and gas and nuclear industries, some recognized in the report, that would challenge a new safety auditor. 

First, “The oil and gas industry is more fragmented and diversified in nature. . . .” (p. 240)  The industry includes vertically integrated giants, specialty niche firms and everything in-between.  Some are global in nature while others are regional firms.  In our view, it appears that oil and gas industry participants cooperate with each other in certain instances and compete with each other in different cases.  (In contrast, most [all?] U.S. nuclear plants are not in direct competition with other plants.)  Obtaining agreement to create a relatively powerful industry auditing entity will not be a simple matter.    

Second, “concerns about potential disclosure to business competitors of proprietary information might make it harder to establish an INPO-like entity in the oil and gas industry.” (p. 240)  Oil and gas firms regard technology as an important source of competitive advantage.  “[A]n INPO-like approach might run into problems if companies perceived the potential for inspections of offshore facilities to reveal ‘technical and proprietary and confidential information that companies may be reluctant to share with one another.’” (p. 241)  Not only will it be difficult to get a firm to share its proprietary technology if it may lose competitive advantage by doing so, but this will make it more difficult for the auditing organization to promote the industry-wide use of the most effective, safest technologies

Third, and this could be a potentially large problem, INPO operates in almost total secrecy.  “[INPO] assessment results are never revealed to anyone other than the utility CEOs and site managers, but INPO formally meets with the NRC four times a year to discuss trends and information of “mutual interest.” And if INPO has discovered serious problems associated with specific plants, it notifies the NRC.”  (p. 236)  INPO claims, probably realistically, that maintaining member confidentiality is key to obtaining full and willing cooperation in evaluations. 

However, this secrecy contributes zero to public understanding of and support for nuclear plant operations and owners.  At this point in its evolution, the oil and gas industry needs more transparency in its auditing and oversight functions, not less.  After all, and forgive the bluntness here, very few people have died at U.S. commercial nuclear power plants (and those were in non-nuclear incidents) while the oil and gas industry has suffered numerous fatalities.  We think a government auditor, whose evaluations of facilities and managements would be made public, is the better answer for the oil and gas industry at this time.


*  National Commission on the BP Deepwater Horizon Oil Spill and Offshore Drilling, “Deep Water: The Gulf Oil Disaster and the Future of Offshore Drilling,” Report to the President (Jan 2011).

NRC Policy Statements

For some time we have been thinking about one of the underlying aspects of the NRC’s current safety culture initiative which is to establish a “policy statement” for nuclear safety culture.  As we know the use of a policy statement in this area dates back to 1989.  A subtext to this approach is whether the NRC should “regulate” nuclear safety culture, presumably through the issuance of regulatory rules and requirements.  We recognize that the “regulate” issue is very much a hot button and we are not addressing it at this time.  Instead we thought it might be worthwhile to consider in some detail just what is a Commission policy statement and what it might or might not accomplish.

On the NRC website we looked at what was available regarding policy statements.  There is a Commission Policy Statements page with a listing of the current set of policy statements, organized by topic.  However, most policy statements relate to the conduct of business by the NRC itself with fewer statements addressing substantive regulatory and safety criteria.  We could not locate on the website information regarding how policy statements are intended to be used in the regulatory process.  For that we turned to recent NRC Issuances,* which are adjudicatory decisions by Atomic Safety and Licensing Boards and the Commission itself, to obtain guidance on the applicability and weight accorded NRC Policy Statements. 

“Policy Statements are neither rules nor orders, and therefore do not establish requirements that bind either the agency or the public.”**

This comes from a Commission decision for a case involving the scope of environmental review for an early site permit.  The Commission references a D.C. Circuit Court case that found:

“A general statement of policy . . . does not establish a ‘‘binding norm.’’ It is not finally determinative of the issues or rights to which it is addressed. The agency cannot apply or rely upon a general statement of policy as law because a general statement of policy only announces what the agency seeks to establish as policy.” (p. 240)

In its decision, the Commission goes on to state:

“For the [Atomic Safety and Licensing] Board to suggest that the strictures of the Policy Statement may be enforced as law, or that it in some way creates a substantive mandate, accords too much weight to the Policy Statement.” (p. 240) 

So far so good.  But after finding that the staff’s review satisfied applicable statutory and regulatory requirements (but did not comport with the letter of the policy statement), the Commission ends its decision on a more confusing note.

“We expect conformance with the Policy Statement, and relevant associated guidance, in future licensing actions of this magnitude.” (p. 248)

In another matter involving a policy statement governing the admissibility of contentions in license renewal proceedings, Commissioners Merrifield and McGaffigan joined in a concurring opinion to observe:

“If we are not willing to enforce our policy statements, the statements become meaningless.”***

Finally, an earlier ASLB decision seems to address policy statements on a practical level, where the Board feels “compelled” by Commission policy: 

Notwithstanding these clear inconsistencies, we find ourselves compelled by Commission rulings and policy statements to accept this approach by the Staff because the Commission has advised that their ‘‘longstanding practice . . . grounded in sound policy’’ is to ‘‘leave [ ] to the expert NRC technical staff prime responsibility for technical fact-finding on uncontested matters.’’****

Based on all of this what is the likely impact of the safety culture policy statement on NRC license holders?  On the one hand it appears that the NRC will “expect” licensees to meet the intent and the particulars of its policy statement.  It seems safe to assume the NRC staff will apply the policy in its assessments of licensee performance.  On the other hand, if a licensee does not meet some aspect of the policy it could find solid footing in a challenge to the enforceability of the policy statement.  The greatest difficulty is to square the rhetoric of NRC Commissioners and staff regarding the absolute importance of safety culture to safety, the “nothing else matters” perspective, with the inherently limited and non-binding nature of a policy statement.


*  NRC Issuances are published as NUREG-0750.  Individual volumes are available here. 

**  66 NRC 215 (2007) at 217, In the Matter of: DOMINION NUCLEAR NORTH ANNA, LLC (Early Site Permit for North Anna ESP Site) CLI-07-27 Nov 20, 2007.

***  65 NRC 1 (2007) at 8, In the Matter of: ENTERGY NUCLEAR VERMONT YANKEE, LLC, and ENTERGY NUCLEAR OPERATIONS, INC. (Vermont Yankee Nuclear Power Station) CLI-07-1 Commissioner Jeffrey S. Merrifield, with Whom Commissioner Edward McGaffigan, Jr. Joins, Concurring Jan 11, 2007.

****  64 NRC 460 (2006) at 492, ATOMIC SAFETY AND LICENSING BOARD In the Matter of EXELON GENERATION COMPANY, LLC (Early Site Permit for Clinton ESP Site) Dec 28, 2006.

ACRS and Safety Culture Policy (cont.)

Our previous post reported on the ACRS letter to the NRC endorsing the agency’s approach to developing a safety culture policy.*  We noted the concern of some ACRS members that the policy might be a back door method to impose regulatory requirements while avoiding the requirements of the regulatory process.

The dissenting ACRS members also raised some other interesting issues about the proposed policy.

First, they questioned whether the proposed traits were the most important ones in terms of their contribution to safety.  Why weren’t organizational and individual integrity, and technical competence included?  Good question.  After all, wasn’t an integrity shortfall at the heart of the misleading of the Vermont senate and the willful violations at San Onofre?

Second, they commented “[T]here is faint evidence that the listed traits (individually or collectively) are assured to produce measureable improvements in safety.” (p. 4)  We raised the same issue in our October 22, 2010 post on the NRC safety culture workshop.  What are the linkages, if any, between the traits and measurable or observable safety-related performance?

Our concern about the lack of demonstrated linkages leads to what may be a bedrock question underlying all of the safety culture policy discussion: If the ROP isn’t providing sufficient information to support the NRC’s confidence in a licensee’s safety culture, then how can the agency develop that information in a defined, disciplined and vetted manner?  Is a safety culture policy going to provide that assurance?


*  Letter dated Dec 15, 2010 from S. Abdel-Khalik (ACRS) to G. Jaczko (NRC), subject "Safety Culture Policy Statement," ADAMS Accession Number ML103410358.

ACRS Weighs In on Safety Culture Policy

In mid-December the Advisory Committee on Reactor Safeguards (ACRS) provided the results of its review of the NRC’s proposed nuclear safety culture policy in a letter to NRC Chairman Jaczko.*  The letter reiterated the approach and general structure of the proposed policy and reached a favorable conclusion.  Perhaps the most interesting comment in the main body of the letter is the following:

“Well-intentioned attempts at improving safety and effectiveness have faltered through efforts to overly prescribe correct behavior and to apply rigid scoring systems. We urge that the staff encourage approaches that emphasize thinking and safety awareness over scorecards of metrics that can induce complacency and rote compliance. Issuance of a policy statement, rather than a regulation, is likely to be a more effective way to appropriately engage all the stakeholders.” (p. 4)

The statement is a bit cryptic and we can only guess what the ACRS has in mind when it refers to “scorecards of metrics” or “overly prescribing behavior”.  Are they referring to the ROP?  Is the ACRS concerned that reliance on the ROP metrics (and their almost uniformly green status) may be lulling the industry and the NRC into complacency?  Equally uncertain is why the ACRS believes that a policy statement will lead to more effective results. 

Apparently we are not the only ones to suffer uncertainty.  The ACRS letter includes “Additional Comments” (read: dissenting comments) by three members** who state:

“It is not entirely clear to us what is meant by implementing a policy statement that lacks the authority of regulation. It appears that implementation of the safety culture policy statement may be an indirect method of imposing requirements on licensees without the discipline of the regulatory process. This, of course, is not acceptable.” (p. 4)

Part of the confusion may lie in the intent and authority associated with NRC policy statements.  It appears that the dissenting members feel that a policy statement would be a back door method to impose “requirements”.  Is that true?  We will follow with a detailed look at policy statements and their effect.


*  Letter dated Dec 15, 2010 from S. Abdel-Khalik (ACRS) to G. Jaczko (NRC), subject "Safety Culture Policy Statement," ADAMS Accession Number ML103410358.

** D.A. Powers, J.S. Armijo and J.L. Rempe.

Nothing Else Matters

In early December NRC Commissioner Ostendorff provided the keynote speech at Nuclear Energy Asia.*  A significant portion of his remarks addressed the importance of safety culture to nuclear safety.  In terms of safety culture insight I think it is fair to say there wasn’t much new here.  However the continued emphasis by the NRC at the Commissioner level may be signaling how they may choose to proceed with regulation of nuclear safety culture.

“Decades of experience in the nuclear field have shown that regulators have to do more than simply establish standards. Rather, I believe it to be more appropriate for a regulator to establish a high-level expectation or policy to help foster the development and maintenance of a strong safety culture.” (p. 2)

This sounds consistent with the current safety culture policy track being followed by the NRC staff.  But just preceding this comment, Ostendorff said: 

“Within the national nuclear safety infrastructure, I believe that a strong safety culture is the key foundation. Without this one essential cornerstone – a strong safety culture – nothing else matters.” (p. 2)

This echoes statements by other Commissioners, and in regulatory actions for specific licensees, that safety culture is essential to nuclear safety.  If that is the case, doesn’t it set the bar very high in terms of regulatory responsibility for ensuring adequate safety culture?   And with the bar set so high, is it sufficient for the NRC to just establish a policy or “expectations” for safety culture, but not rules or regulatory requirements?  We will present a much more detailed look at NRC policy making in an upcoming post.


*  W.C. Ostendorff, "Regulatory Perspectives on Nuclear Safety," International Keynote Address, Nuclear Energy Asia 2010, Hong Kong, China (Dec 7, 2010) ADAMS Accession Number ML103420523.

ESP and Safety Culture

A recent New York Times article* on an extrasensory perception (ESP) study and the statistical methods used therein caught our attention.  The article’s focus is on the controversy surrounding statistical significance testing.  “A finding from any well-designed study — say, a correlation between a personality trait and the risk of depression — is considered “significant” if its probability of occurring by chance is less than 5 percent.”  We have all seen such analyses.

However, critics of classical significance testing say a finding based on such a test “could overstate the significance of the finding by a factor of 10 or more,” a sort of super false positive.  The critics claim a better approach is to apply the methods of Bayesian analysis, which incorporates known probabilities, if available, from outside the study.  Check out the comments on the article, especially the reader recommended ones, for more information on statistical methods and issues.  (You can ignore the ESP-related comments unless you have some special interest in the topic). 

What has this got to do with safety culture?

Recall that last October we reported on an INPO study that, among other things, calculated correlations between safety culture survey factors and various safety-related performance measures.  We expressed reservations about the overall approach and results even though a few correlations supported points we have been making in our blog.

The controversy over the ESP study and its associated statistical methods reminds us that analysts in many fields are under pressure to find something “significant.”  This pressure comes from bosses, funding agencies, editors and tenure committees.  Studies that find no effects, or ones not aligned with higher-level organizational objectives, are less likely to be publicized and their authors rewarded.  In addition, I fear some (many?) social science researchers don’t fully understand the statistical methods they are using, i.e., their built-in biases and limitations.  So, once again, caveat emptor.   

By the way, we are not saying or implying the INPO study was biased in any way; we have no information on it other than what was presented at the NRC meeting referenced in our original blog post. 

*  B. Carey, “You Might Already Know This ...,” New York Times (Jan 11, 2011).