Thursday, January 27, 2011

Culture de la Sécurité

If you are paying attention you noticed we’re using French words.  And the reason is the current news regarding Air France and its safety practices and safety culture.

The principal finding of an independent study of the airline’s operations was a lack of “strong safety leadership at all levels of management" as reported in a January 26 Wall Street Journal (WSJ) article.*

While Air France has refused to make the report public, a review by the WSJ stated that the study was “sharply critical of broad aspects of the safety culture”.

Over the previous several days there have been articles in the WSJ preceding and attending the completion of the independent study.  The first of the articles** previewed some of the findings and provided favorable commentary based on an understanding that Air France would be making the report public (“Air France to Disclose…”).  This was characterized as a move toward greater openness on the part of the airline and commented, “safety experts said it was unusual for a large company, especially an airline, to give outsiders such latitude publicly to expose gaps in safety systems.”  And quoting Bill Voss, president of the Flight Safety Foundation, a global air-safety advocacy organization based in Alexandria, Va., "It's extraordinary that they are willing to release the results," said Mr. Voss. The process "gives me confidence there will be follow-through."

Well, in a follow-up article the next day, when receipt of the study results was announced by Air France, the reaction was disappointment as the airline indicated it would not release the report or its recommendations.  Nonetheless the airline trumpeted its actions:

“According to an Air France news release, the report said that creating the outside review team ‘in a public manner and providing it a broad charter’ to examine flight safety ‘was a courageous act’ and an example of safety leadership ‘rarely seen in today's international aviation industry.’” ***

In deciding on whether Air France should be awarded the Légion d'honneur, one might also consider that the current safety study was undertaken in the wake of four serious crashes between 1999 and 2009 (the most recent being the flight from Rio to Paris that went down mid-Atlantic).  In addition, it follows a prior safety study,

“Finished in 2006, that report was distributed to more than 4,000 company pilots and was widely praised for its frankness about shortcomings within the carrier. Although Air France has said its executive committee made formal decisions to implement the report's recommendations, critics of the airline continue to maintain management didn't aggressively pursue the changes.” (Jan 24, 2011)

I think we’ve seen this before.  Think about the safety studies after the Challenger crash, but not really implemented and then the disintegration on reentry of the Columbia.  Or BP and the Texas oil refinery fire followed by the Deepwater Horizon last summer.  Obviously safety assessments, no matter how strong and how independent, ultimately require the subject organization to implement changes.  We think the current Air France report correctly fingers safety leadership by management “starting at the top”.  And it never fails in these situations that top management describes as its highest priority…...can anybody guess……that’s right, it’s “safety first”.  (Jan 26, 2011)

Perhaps if it was safety first Air France might be releasing the study and its recommendations.  Wouldn’t that help make real its safety priority and wouldn’t such transparency help ensure that the recommendations are actually implemented?   We have commented in prior posts on transparency and we will continue to emphasize its importance to safety culture across all industries.


*  A. Pasztor, D. Michaels and D. Gauthier-Villars, “Air France Panel Cites Wide Safety Deficiencies,” WSJ.com (Jan 26, 2011).

** A. Pasztor and D. Michaels, “Air France to Disclose Review's Criticisms,” WSJ.com (Jan 24, 2011).

***  A. Pasztor, “Air France Enhances Safety Efforts,” later re-headlined “Air France Withholds Key Report,” WSJ.com (Jan 25, 2011).

Tuesday, January 25, 2011

A Nuclear Model for Oil and Gas

The President’s Commission has issued its report on the Deepwater Horizon disaster.* The report reviews the history of the tragedy and makes recommendations based on lessons learned.  This post focuses on the report’s use of the nuclear industry, in particular the role played by INPO, as a model for an oil and gas industry safety institute and auditor.

The report provides an in-depth review of INPO’s role and methods and we will not repeat that review in this space.  We want to highlight the differences between the oil and gas and nuclear industries, some recognized in the report, that would challenge a new safety auditor. 

First, “The oil and gas industry is more fragmented and diversified in nature. . . .” (p. 240)  The industry includes vertically integrated giants, specialty niche firms and everything in-between.  Some are global in nature while others are regional firms.  In our view, it appears that oil and gas industry participants cooperate with each other in certain instances and compete with each other in different cases.  (In contrast, most [all?] U.S. nuclear plants are not in direct competition with other plants.)  Obtaining agreement to create a relatively powerful industry auditing entity will not be a simple matter.    

Second, “concerns about potential disclosure to business competitors of proprietary information might make it harder to establish an INPO-like entity in the oil and gas industry.” (p. 240)  Oil and gas firms regard technology as an important source of competitive advantage.  “[A]n INPO-like approach might run into problems if companies perceived the potential for inspections of offshore facilities to reveal ‘technical and proprietary and confidential information that companies may be reluctant to share with one another.’” (p. 241)  Not only will it be difficult to get a firm to share its proprietary technology if it may lose competitive advantage by doing so, but this will make it more difficult for the auditing organization to promote the industry-wide use of the most effective, safest technologies

Third, and this could be a potentially large problem, INPO operates in almost total secrecy.  “[INPO] assessment results are never revealed to anyone other than the utility CEOs and site managers, but INPO formally meets with the NRC four times a year to discuss trends and information of “mutual interest.” And if INPO has discovered serious problems associated with specific plants, it notifies the NRC.”  (p. 236)  INPO claims, probably realistically, that maintaining member confidentiality is key to obtaining full and willing cooperation in evaluations. 

However, this secrecy contributes zero to public understanding of and support for nuclear plant operations and owners.  At this point in its evolution, the oil and gas industry needs more transparency in its auditing and oversight functions, not less.  After all, and forgive the bluntness here, very few people have died at U.S. commercial nuclear power plants (and those were in non-nuclear incidents) while the oil and gas industry has suffered numerous fatalities.  We think a government auditor, whose evaluations of facilities and managements would be made public, is the better answer for the oil and gas industry at this time.


*  National Commission on the BP Deepwater Horizon Oil Spill and Offshore Drilling, “Deep Water: The Gulf Oil Disaster and the Future of Offshore Drilling,” Report to the President (Jan 2011).

Wednesday, January 19, 2011

NRC Policy Statements

For some time we have been thinking about one of the underlying aspects of the NRC’s current safety culture initiative which is to establish a “policy statement” for nuclear safety culture.  As we know the use of a policy statement in this area dates back to 1989.  A subtext to this approach is whether the NRC should “regulate” nuclear safety culture, presumably through the issuance of regulatory rules and requirements.  We recognize that the “regulate” issue is very much a hot button and we are not addressing it at this time.  Instead we thought it might be worthwhile to consider in some detail just what is a Commission policy statement and what it might or might not accomplish.

On the NRC website we looked at what was available regarding policy statements.  There is a Commission Policy Statements page with a listing of the current set of policy statements, organized by topic.  However, most policy statements relate to the conduct of business by the NRC itself with fewer statements addressing substantive regulatory and safety criteria.  We could not locate on the website information regarding how policy statements are intended to be used in the regulatory process.  For that we turned to recent NRC Issuances,* which are adjudicatory decisions by Atomic Safety and Licensing Boards and the Commission itself, to obtain guidance on the applicability and weight accorded NRC Policy Statements. 

Policy Statements are neither rules nor orders, and therefore do not establish requirements that bind either the agency or the public.”**

This comes from a Commission decision for a case involving the scope of environmental review for an early site permit.  The Commission references a D.C. Circuit Court case that found:

A general statement of policy . . . does not establish a ‘‘binding norm.’’ It is not finally determinative of the issues or rights to which it is addressed. The agency cannot apply or rely upon a general statement of policy as law because a general statement of policy only announces what the agency seeks to establish as policy.” (p. 240)

In its decision, the Commission goes on to state:

For the [Atomic Safety and Licensing] Board to suggest that the strictures of the Policy Statement may be enforced as law, or that it in some way creates a substantive mandate, accords too much weight to the Policy Statement.” (p. 240) 

So far so good.  But after finding that the staff’s review satisfied applicable statutory and regulatory requirements (but did not comport with the letter of the policy statement), the Commission ends its decision on a more confusing note.

We expect conformance with the Policy Statement, and relevant associated guidance, in future licensing actions of this magnitude.” (p. 248)

In another matter involving a policy statement governing the admissibility of contentions in license renewal proceedings, Commissioners Merrifield and McGaffigan joined in a concurring opinion to observe:

If we are not willing to enforce our policy statements, the statements become meaningless.”***

Finally, an earlier ASLB decision seems to address policy statements on a practical level, where the Board feels “compelled” by Commission policy: 

Notwithstanding these clear inconsistencies, we find ourselves compelled by Commission rulings and policy statements to accept this approach by the Staff because the Commission has advised that their ‘‘longstanding practice . . . grounded in sound policy’’ is to ‘‘leave [ ] to the expert NRC technical staff prime responsibility for technical fact-finding on uncontested matters.’’****

Based on all of this what is the likely impact of the safety culture policy statement on NRC license holders?  On the one hand it appears that the NRC will “expect” licensees to meet the intent and the particulars of its policy statement.  It seems safe to assume the NRC staff will apply the policy in its assessments of licensee performance.  On the other hand, if a licensee does not meet some aspect of the policy it could find solid footing in a challenge to the enforceability of the policy statement.  The greatest difficulty is to square the rhetoric of NRC Commissioners and staff regarding the absolute importance of safety culture to safety, the “nothing else matters” perspective, with the inherently limited and non-binding nature of a policy statement.


*  NRC Issuances are published as NUREG-0750.  Individual volumes are available here

**  66 NRC 215 (2007) at 217, In the Matter of: DOMINION NUCLEAR NORTH ANNA, LLC (Early Site Permit for North Anna ESP Site) CLI-07-27 Nov 20, 2007.

***  65 NRC 1 (2007) at 8, In the Matter of: ENTERGY NUCLEAR VERMONT YANKEE, LLC, and ENTERGY NUCLEAR OPERATIONS, INC. (Vermont Yankee Nuclear Power Station) CLI-07-1 Commissioner Jeffrey S. Merrifield, with Whom Commissioner Edward McGaffigan, Jr. Joins, Concurring Jan 11, 2007.

****  64 NRC 460 (2006) at 492, ATOMIC SAFETY AND LICENSING BOARD In the Matter of EXELON GENERATION COMPANY, LLC (Early Site Permit for Clinton ESP Site) Dec 28, 2006.

Tuesday, January 18, 2011

ACRS and Safety Culture Policy (cont.)

Our previous post reported on the ACRS letter to the NRC endorsing the agency’s approach to developing a safety culture policy.*  We noted the concern of some ACRS members that the policy might be a back door method to impose regulatory requirements while avoiding the requirements of the regulatory process.

The dissenting ACRS members also raised some other interesting issues about the proposed policy.

First, they questioned whether the proposed traits were the most important ones in terms of their contribution to safety.  Why weren’t organizational and individual integrity, and technical competence included?  Good question.  After all, wasn’t an integrity shortfall at the heart of the misleading of the Vermont senate and the willful violations at San Onofre?

Second, they commented “[T]here is faint evidence that the listed traits (individually or collectively) are assured to produce measureable improvements in safety.” (p. 4)  We raised the same issue in our October 22, 2010 post on the NRC safety culture workshop.  What are the linkages, if any, between the traits and measurable or observable safety-related performance?

Our concern about the lack of demonstrated linkages leads to what may be a bedrock question underlying all of the safety culture policy discussion: If the ROP isn’t providing sufficient information to support the NRC’s confidence in a licensee’s safety culture, then how can the agency develop that information in a defined, disciplined and vetted manner?  Is a safety culture policy going to provide that assurance?


*  Letter dated Dec 15, 2010 from S. Abdel-Khalik (ACRS) to G. Jaczko (NRC), subject "Safety Culture Policy Statement," ADAMS Accession Number ML103410358.

Friday, January 14, 2011

ACRS Weighs In on Safety Culture Policy

In mid-December the Advisory Committee on Reactor Safeguards (ACRS) provided the results of its review of the NRC’s proposed nuclear safety culture policy in a letter to NRC Chairman Jaczko.*  The letter reiterated the approach and general structure of the proposed policy and reached a favorable conclusion.  Perhaps the most interesting comment in the main body of the letter is the following:

“Well-intentioned attempts at improving safety and effectiveness have faltered through efforts to overly prescribe correct behavior and to apply rigid scoring systems. We urge that the staff encourage approaches that emphasize thinking and safety awareness over scorecards of metrics that can induce complacency and rote compliance. Issuance of a policy statement, rather than a regulation, is likely to be a more effective way to appropriately engage all the stakeholders.” (p. 4)

The statement is a bit cryptic and we can only guess what the ACRS has in mind when it refers to “scorecards of metrics” or “overly prescribing behavior”.  Are they referring to the ROP?  Is the ACRS concerned that reliance on the ROP metrics (and their almost uniformly green status) may be lulling the industry and the NRC into complacency?  Equally uncertain is why the ACRS believes that a policy statement will lead to more effective results. 

Apparently we are not the only ones to suffer uncertainty.  The ACRS letter includes “Additional Comments” (read: dissenting comments) by three members** who state:

“It is not entirely clear to us what is meant by implementing a policy statement that lacks the authority of regulation. It appears that implementation of the safety culture policy statement may be an indirect method of imposing requirements on licensees without the discipline of the regulatory process. This, of course, is not acceptable.” (p. 4)

Part of the confusion may lie in the intent and authority associated with NRC policy statements.  It appears that the dissenting members feel that a policy statement would be a back door method to impose “requirements”.  Is that true?  We will follow with a detailed look at policy statements and their effect.


*  Letter dated Dec 15, 2010 from S. Abdel-Khalik (ACRS) to G. Jaczko (NRC), subject "Safety Culture Policy Statement," ADAMS Accession Number ML103410358.

** D.A. Powers, J.S. Armijo and J.L. Rempe.

Thursday, January 13, 2011

Nothing Else Matters

In early December NRC Commissioner Ostendorff provided the keynote speech at Nuclear Energy Asia.*  A significant portion of his remarks addressed the importance of safety culture to nuclear safety.  In terms of safety culture insight I think it is fair to say there wasn’t much new here.  However the continued emphasis by the NRC at the Commissioner level may be signaling how they may choose to proceed with regulation of nuclear safety culture.

“Decades of experience in the nuclear field have shown that regulators have to do more than simply establish standards. Rather, I believe it to be more appropriate for a regulator to establish a high-level expectation or policy to help foster the development and maintenance of a strong safety culture.” (p. 2)

This sounds consistent with the current safety culture policy track being followed by the NRC staff.  But just preceding this comment, Ostendorff said: 

“Within the national nuclear safety infrastructure, I believe that a strong safety culture is the key foundation. Without this one essential cornerstone – a strong safety culture – nothing else matters.” (p. 2)

This echoes statements by other Commissioners, and in regulatory actions for specific licensees, that safety culture is essential to nuclear safety.  If that is the case, doesn’t it set the bar very high in terms of regulatory responsibility for ensuring adequate safety culture?   And with the bar set so high, is it sufficient for the NRC to just establish a policy or “expectations” for safety culture, but not rules or regulatory requirements?  We will present a much more detailed look at NRC policy making in an upcoming post.


*  W.C. Ostendorff, "Regulatory Perspectives on Nuclear Safety," International Keynote Address, Nuclear Energy Asia 2010, Hong Kong, China (Dec 7, 2010) ADAMS Accession Number ML103420523.

Wednesday, January 12, 2011

ESP and Safety Culture

A recent New York Times article* on an extrasensory perception (ESP) study and the statistical methods used therein caught our attention.  The article’s focus is on the controversy surrounding statistical significance testing.  “A finding from any well-designed study — say, a correlation between a personality trait and the risk of depression — is considered “significant” if its probability of occurring by chance is less than 5 percent.”  We have all seen such analyses.

However, critics of classical significance testing say a finding based on such a test “could overstate the significance of the finding by a factor of 10 or more,” a sort of super false positive.  The critics claim a better approach is to apply the methods of Bayesian analysis, which incorporates known probabilities, if available, from outside the study.  Check out the comments on the article, especially the reader recommended ones, for more information on statistical methods and issues.  (You can ignore the ESP-related comments unless you have some special interest in the topic). 

What has this got to do with safety culture?

Recall that last October we reported on an INPO study that, among other things, calculated correlations between safety culture survey factors and various safety-related performance measures.  We expressed reservations about the overall approach and results even though a few correlations supported points we have been making in our blog.

The controversy over the ESP study and its associated statistical methods reminds us that analysts in many fields are under pressure to find something “significant.”  This pressure comes from bosses, funding agencies, editors and tenure committees.  Studies that find no effects, or ones not aligned with higher-level organizational objectives, are less likely to be publicized and their authors rewarded.  In addition, I fear some (many?) social science researchers don’t fully understand the statistical methods they are using, i.e., their built-in biases and limitations.  So, once again, caveat emptor.   

By the way, we are not saying or implying the INPO study was biased in any way; we have no information on it other than what was presented at the NRC meeting referenced in our original blog post. 

*  B. Carey, “You Might Already Know This ...,” New York Times (Jan 11, 2011).

Monday, January 10, 2011

Pick Any Two

Last week principal findings of the BP Oil Spill Presidential Commission were released.   Not surprisingly it cited root causes that were “systemic”, decisions without adequate consideration of risks, and failures of regulatory oversight.  It also cited a lack of a culture of safety at the companies involved in the Deepwater Horizon.  We came across an interesting entry in a blog tied to an article in the New York Times by John Broder on January 5, 2011, “Blunders Abounded Before Gulf Oil Spill, Panel Says”.  We thought it was worth passing on. 

Comment No. 7 of 66 submitted by:
Jim S.
Cleveland
January 5th, 2011
7:23 pm

“A fundamental law of engineering (or maybe of the world in general) is "Cheaper, Faster, Better: Pick Any Two".  

Clearly those involved, whether deliberately or by culture, chose Cheaper and Faster.”

Thursday, January 6, 2011

Nuclear Safety Culture Assessment Manual

July 9, 2012 update: How to Get the NEI Nuclear Safety Culture Assessment Manual

The manual is available in the NRC ADAMS database, Accession Numbers ML091810801, ML091810803, ML091810805, ML091810807, ML091810808 and ML091810809.

**********************************************************
 
As recently reported at TheDay.com,* NEI has published a “Nuclear Safety Culture Assessment Manual,” a document that provides guidance for conducting a safety culture (SC) assessment at a nuclear power plant.  The industry has issued the manual and conducted some pilot program assessments in an effort to influence and stay ahead of the NRC’s initiative to finalize a SC policy statement this year.  The NRC is formulating a policy (as opposed to a regulatory requirement) in this area because it apparently believes that SC cannot be directly regulated and/or any attempt to assess SC comes too close to evaluating (or interfering with) plant management, a task the agency has sought to avoid. 

Basically, the manual describes an assessment methodology based on the eight INPO principles for creating/maintaining a strong nuclear safety culture.  It is a comprehensive how-to document including assessment team organization, schedules, interview guidance and questions, sample communication memos, and report templates.  The manual has a strongly prescriptive approach, i.e., it seeks to create a standardized approach which should facilitate comparisons between different facilities and the same facility over time. 

The best news from our perspective is that the NEI assessment approach relies heavily on interviews; it uses a site survey instrument only to identify pre-assessment areas of interest.  It’s no secret that we are skeptical about over-inference with respect to the health of a plant’s safety culture from the snapshot a survey provides.  The assessment also uses direct observations of behavior of employees at all levels during scheduled activities, such and meetings and briefings, and ad-hoc observation opportunities.

A big question is: In a week-long self assessment, can a team discern the degree to which an organization satisfies key principles, e.g., the level of trust in the organization or whether leaders demonstrate a commitment to safety?  I think we have to answer that with “Maybe.”  Skilled and experienced interviewers can probably determine the general status of these variables but may not develop a complete picture of all the nuances.  BUT, their evaluation will likely be more useful than any survey.

There is one obvious criticism with the NEI approach which industry critics have quickly identified.  As David Collins puts it in TheDay.com article, “[T]he industry is monitoring itself - this is the fox monitoring the henhouse."  While the manual is proposed for use by anyone performing a safety culture assessment, including a truly independent third party, the reality is the industry expects the primary users to be utilities performing self assessments or “independent” assessments, which include non-utility people on the team. 


*  P. Daddona, “Nuclear group puts methods into use to foster 'a safety culture',” TheDay.com
(Dec 21, 2010).