Upper Big Branch 1

A few days ago the Governor’s Independent Investigation Panel issued its report on the Upper Big Branch coal mine explosion of April 5, 2010.  The report is over 100 pages and contains considerable detail on the events and circumstances leading up to the disaster, coal mining technology and safety issues.  It is well worth reading for anyone in the business of assuring safety in a complex and high risk enterprise.  We anticipate doing several blog posts on material from the report but wanted to start with a brief quote from the forward to the report, summarizing its main conclusions.

“A genuine commitment to safety means not just examining miners’ work practices and behaviors.  It means evaluating management decisions up the chain of command - all the way to the boardroom - about how miners’ work is organized and performed.”*

We believe this conclusion is very much on the mark for safety management and for the safety culture that supports it in a well managed organization.  It highlights what to us has appeared to be an over-emphasis in the nuclear industry on worker practices and behaviors - and “values”.   And it focuses attention on management decisions - decisions that maintain an appropriate weight to safety in a world of competing priorities and interests - as the sine qua non of safety.  As we have discussed in many of our posts, we are concerned with the emphasis by the nuclear industry on safety culture surveys and training in safety culture principles and values as the primary tools of assuring a strong safety culture.  Rarely do culture assessments focus on the decisions that underlie the management of safety to examine the context and influence of factors such as impacts on operations, availability of resources, personnel incentives and advancement, corporate initiatives and goals, and outside factors such as political pressure.  The Upper Big Branch report delves into these issues and builds a compelling basis for the above conclusion, a conclusion that is not limited to the coal industry.

*  Governor’s Independent Investigation Panel, “Report to the Governor: Upper Big Branch,” National Technology Transfer Center, Wheeling Jesuit University (May 2011), p. 4.

Mental Models and Learning

A recent New York Times article on teaching methods* caught our eye.  It reported an experiment by college physics professors to improve their freshmen students’ understanding and retention of introductory material.  The students comprised two large (260+) classes that usually were taught via lectures.  For one week, teaching assistants used a collaborative, team-oriented approach for one of the classes.  Afterward, this group scored higher on the test than the group that received the traditional lecture.  

One of the instructors reported, “. . . this class actively engages students and allows them time to synthesize new information and incorporate it into a mental model . . . . When they can incorporate things into a mental model, we find much better retention.”

We are big believers in mental models, those representations of the world that people create in their minds to make sense of information and experience.  They are a key component of our system dynamics approach to understanding and modeling safety culture.  Our NuclearSafetySim model illustrates how safety culture interacts with other variables in organizational decision-making; a primary purpose for this computer model is to create a realistic mental model in users’ minds.

Because this experiment helped the students form more useful mental models, our reaction to it is generally favorable.  On the other hand, why is the researchers’ “insight” even news?  Why wouldn’t a more engaging approach lead to a better understanding of any subject?  Don’t most of you develop a better understanding when you do the lab work, code your own programs, write the reports you sign, or practice decision-making in a simulated environment?

*  B. Carey, “Less Talk, More Action: Improving Science Learning,” New York Times (May 12, 2011).

Shifting the Burden

Pitot tube

This post emanates from the ongoing investigations of the crash of Air France flight 447 from Rio de Janeiro to Paris.  In some respects it is a follow-up to our January 27, 2011 post on Air France’s safety culture.  An article in the New York Times Sunday Magazine* explores some of the mysteries surrounding the loss of the plane in mid-Atlantic.  One of the possible theories for the crash involves the pitot tubes used on the Airbus plane.  Pitot tubes are instruments used on aircraft to measure air speed.  The pitot tube measures the difference between total (stagnation) and static pressure to determine dynamic pressure and therefore velocity of the air stream.  Care must be taken to assure that the pitot tubes do not become clogged with ice or other foreign matter as it would interrupt or corrupt the airspeed signal provided to the pilots and the auto-pilot system. 

On the flight 447 aircraft, three Thales AA model pitot tubes were in use.  They are produced by a French company and cost approximately $3500 each.  The Times article goes on to explain:

"...by the summer of 2009, the problem of icing on the Thales AA was known to be especially common….Between 2003 and 2008, there were at least 17 cases in which the Thales AA had problems on the Airbus A330 and its sister plane, the A340.  In September 2007, Airbus issued a ‘service bulletin’ suggesting that airlines replace the AA pitots with a newer model, the BA, which was said to work better in ice.”

Air France’s response to the service bulletin established a policy to replace the AA tubes “only when a failure occurred”.  A year later Air France then asked Airbus for “proof” that the model BA tubes worked better in ice.  It took Airbus another 6-7 months to perform tests that demonstrated the superior performance of the BA tubes, following which Air France proceeded with implementing the recommended change for its A330 aircraft.  Unfortunately the new probes had not yet been installed at the time of flight 447.

Much is still unknown about whether in fact the pitot tubes played a role in the crash of flight 447 and of the details of Air France’s consideration of deploying replacements.  But there is a sufficient framework to pose some interesting questions regarding how safety considerations were balanced in the process, and what might be inferred about the Air France safety culture.  Most clearly it highlights how fundamental the decision making process is to safety culture.

What is clear is that Air France’s approach to this problem “shifted the burden” from assuring that something was safe to proving that it was unsafe.  In legal usage this involves transferring the obligation to prove a fact in controversy from one party to another.  Or in systems thinking (which you may have noticed we strongly espouse) it denotes a classic dynamic archetype - a problem arises, it can be ameliorated through either a short term, symptom based response or a fundamental solution that may take additional time and/or resources to implement.  Choosing the short term fix provides relief and reinforces the belief in the efficacy of the response.  Meanwhile the underlying problem goes unaddressed.  For Air France, the service bulletin created a problem.  Air France could have immediately replaced the pitot tubes or undertaken its own assessment of pitot tubes with replacement to follow.  This would have taken time and resources.  Nor did Air France appear to try to address the threshold question of whether the existing AA model instruments were adequate - in nuclear industry terms, were they “operable” and able to perform their safety function?  Air France apparently did not even implement interim measures such as retraining to improve pilot’s recognition and response to pitot tube failures or incorrect readings.  Instead, Air France shifted the burden back to Airbus to “prove” their recommendation.  The difference between showing that something is not safe versus that it is safe is as wide as, well, the Atlantic Ocean.

What we find particularly interesting about shifting the burden is that it is just another side of the complacency coin.  Most people engaged in safety culture science recognize that complacency is a potential contributor to the decay and loss of effectiveness of safety culture.  Everything appears to be going OK so there is less need to pursue issues, particularly those lacking safety impact clarity.  Not pursuing root causes, not verifying corrective action efficacy, loss of questioning attitude and lack of resources could all be telltale signs of complacency.  The interesting thing about shifting the burden is that it yields much the same result - but with the appearance that action is being taken. 

The footnote to the story is the response of Air Caraibes to similar circumstances in this time frame.  The Times article indicates Air Caraibes experienced two “near misses” with Thales AA pitot tubes on A330 aircraft.  They immediately replaced the parts and notified regulators.


*  W.S. Hylton, "What Happened to Air France Flight 447?" New York Times Magazine (May 4, 2011).

On the Other Hand

Our prior post on the award of safety performance bonuses at Transocean may have left you, and us, wondering about the ability of large corporations to walk the talk.  Well, better news today with an article from the Wall Street Journal* recounting the decision by Southwest Airlines to preemptively ground its 737s after a fuselage tear on one of the planes.  

As told in the article, the Southwest management appears to have rapidly responded to the event (over a weekend) with technical assessment including advice from Boeing.  The bottom line on the technical side was uncertainty regarding the cause of the failure and the implications for other similar 737s.  It was also clear that Southwest placed the burden on an affirmative showing that the planes were safe rather than requiring evidence that they weren’t.  With the issue “up in the air” the CEO acted quickly and decisively with the grounding order and the conduct of inspections as recommended by Boeing.  

The decision resulted in the cancellation of over 600 flights and no doubt inconvenienced many Southwest passengers, and will have a substantial cost impact to the airline.  The action by Southwest was described as “unusual” as it did not wait for a directive from the government or Boeing to remove planes from service. 

(Ed. note:  Southwest’s current approach is even more remarkable in light of how recently their practices were not exactly on the side of the angels.  In 2008, the FAA fined Southwest $7.5 million for knowingly flying planes that were overdue for mandatory structural inspections.)

*  T.W. Martin, A. Pasztor and P. Sanders, "Southwest's Solo Flight in Crisis," wsj.com (Apr 8, 2011).

Incredible

“...notwithstanding the tragic loss of life in the Gulf of Mexico, we [Transocean] achieved an exemplary statistical safety record as measured by our total recordable incident rate (‘‘TRIR’’) and total potential severity rate (‘‘TPSR’’).  As measured by these standards, we recorded the best year in safety performance in our Company’s history, which is a reflection on our commitment to achieving an incident free environment, all the time, everywhere.”*

Good grief.  Did Transocean really say this?  Eleven people including nine Transocean employees died in the Deepwater Horizon oil rig explosion.  The quote is from Transocean’s 2010 Annual Report and Proxy recently filed with the SEC.  It provides another illuminating example where the structure and award of management incentives speak much greater volumes than corporate safety rubrics.  (For our report on compensation structures within nuclear power companies and the extent to which such compensation included incentives other than safety, look here and here.)  Or as the saying goes, “Follow the money”.

To fully comprehend how Transocean’s incentive program purports to encourage safety performance we are providing the following additional quotes from its Annual Report.

“Safety Performance.  Our business involves numerous operating hazards and we remain committed to protecting our employees, our property and the environment. Our ultimate goal is expressed in our Safety Vision of ‘‘an incident-free workplace—all the time, everywhere…..

"The [Compensation] Committee measures our safety performance through a combination of our total recordable incident rate (‘‘TRIR’’) and total potential severity rate (‘‘TPSR’’).

•    "TRIR is an industry standard measure of safety performance that is used to measure the frequency of a company’s recordable incidents and comprised 50% of the overall safety metric. TRIR is measured in number of recordable incidents per 200,000 employee hours worked.

•    "TPSR is a proprietary safety measure that we use to monitor the total potential severity of incidents and comprised 50% of the overall safety metric. Each incident is reviewed and assigned a number based on the impact that such incident could have had on our employees and contractors, and the total is then combined to determine the TPSR.

"The occurrence of a fatality may override the safety performance measure.

"….Based on the foregoing safety performance measures, the actual TRIR was 0.74 and the TPSR was 35.4 for 2010. These outcomes together resulted in a calculated payout percentage of 115% for the safety performance measure for 2010. However, due to the fatalities that occurred in 2010, the Committee exercised its discretionary authority to modify the TRIR payout component to zero, which resulted in a modified payout percentage of 67.4% for the safety performance measure." (p. 45)

The treatment of bonuses for Transocean execs was picked up in various media outlets and met with, shall we say, skepticism.  Transocean responded to the blowback with the following:

“We acknowledge that some of the wording in our 2010 proxy statement may have been insensitive in light of the incident that claimed the lives of eleven exceptional men last year and we deeply regret any pain that it may have caused...” **

Note that the apology is directed at the “wording” of the proxy, not to the actual award of bonus compensation for safety performance.  We are tempted here to make some reference to “density” but it is self-evident.

Perhaps realizing that something more would be appropriate, Transocean announced yesterday that members of the senior management team would be donating their bonuses to the Deepwater Horizon Memorial Fund.*** 

Oops, actually they will be donating just the “safety portion” of their bonuses to the fund.  All other bonus amounts and incentive awards are not affected and the Transocean incentive structure for safety performance remains unchanged for 2011.

*  Transocean Proxy Statement and 2010 Annual Report, p. 35.

**  J. McMahon, "Transocean Execs Get Bonuses after ‘Best Year in Safety,’ Despite Gulf Oil Disaster," Forbes.com (Apr 2, 2011 2:23am).

***  Announcement by Transocean Ltd. Senior Management Team, Zug, Switzerland (Apr 5, 2011 MARKETWIRE via COMTEX).

Combustible Gas

As we observed in our prior blog post, the publication by the Union of Concerned Scientists of their new study of nuclear near misses would likely generate a combustible gas that could find some ignition sources, at least among like-minded nuclear critics.  Thus the March 22, 2011 article* in The Nation magazine was predictable, including the comments by Henry Meyers that the UCS study is evidence of a lack of “serious oversight for twenty years” by the NRC.  Evidence of this includes the reduction in NRC violations and fines in the late 1990s and the contention that then-Chairman Dr. Shirley Jackson caved to political pressure.  Disregarded are the facts that many nuclear plants underwent enormous performance improvement programs in that period and the consolidation of nuclear ownership under a small number of advanced nuclear enterprises.**  These nuclear operators had the significant management, technical  and financial resources to ensure operating excellence in their plants, resulting in much better regulatory compliance.

But it would be a mistake to dismiss lightly the direction that UCS and Christian Parenti of The Nation are taking the post-Fukushima discussion of nuclear safety.  Their thesis is that the current risky state of the nuclear industry in the U.S. (“a fleet of old nuclear plants and the 40,000 tons of nuclear waste they have created”) is due to the lack of strong safety culture, and that the NRC has been compromised through political pressure and the corrosive influence of an inadequate industry safety culture.  Thus,

“...it is imperative to overhaul the inadequate, industry-dominated safety culture that has developed over the past twenty years.  This eroded safety culture is a source of serious danger—and it must be fixed.”

Approaching the current state of nuclear safety from this direction has the potential to open a Davis-Besse size hole in the carefully constructed safety record of the nuclear industry.  By its essence safety culture is perhaps the most far ranging indictment of safety; far more extensive than any specific technical issues that have historically been the target of nuclear critics.  It targets an unprotected flank of both the industry and the NRC; including the recent process where consensus and stakeholder involvement has been emphasized by the NRC to the point that the above quote will gain traction.  The product, a safety culture policy statement by the NRC, something that is not even enforceable, will be framed as a continuation of a lack of “serious oversight” and serve well the newly energized anti-nuclear community. 

*  C. Parenti, "After Three Mile Island: The Rise and Fall of Nuclear Safety Culture," The Nation (Mar 22, 2011).

**  Nuclear industry consolidation was predicted and described in a paper I co-authored with NYPA's Bob Schoenberger, "Capturing Stranded Value in Nuclear Plant Assets," The Electricity Journal 9 (June 1996): 59-65.

Never Let a Good Crisis Go To Waste

“You don’t ever want a crisis to go to waste; it’s an opportunity to do important things that you would otherwise avoid.” So said Rahm Emanuel, memorably, several years ago.  Perhaps taking a page from the Emanuel book, the Union of Concerned Scientists took the opportunity last Thursday to release a report chronicling a series of problems it had investigated at U.S. nuclear plants.*  Apparently the events in Japan pumped plenty of fresh oxygen into the UCS war room in time for them to trot out their latest list of concerns regarding nuclear plant safety.

[UCS senior scientist Edwin] “Lyman was speaking in a conference call with reporters on the release of a report examining critical problems — known as “near misses” — at various nuclear facilities in the United States last year, and the N.R.C.’s handling of critical problems”

David Lochbaum, the author of the report and the director of the nuclear safety program for the organization, was quoted as:

[The report] “also suggested that federal regulators needed to do more to investigate why problems existed in the first place — including examining the overall safety culture of companies that operate nuclear power plants — rather than simply order them to be fixed.”

It could be that the UCS is aiming at the heart of the recent discussions surrounding the NRC’s new policy statement on safety culture.  It is clear that the NRC has little appetite to regulate the safety culture of its licensees; instead urging licensees to maintain a strong safety culture and and taking action only if “results” are not acceptable.  UCS would like specific issues, such as the “near misses” in their report, to be broadly interpreted to establish a more fundamental, cultural flaw in the enterprise itself.

Perhaps the larger question raised by the events in Japan is the dominance of natural phenomena in challenging man-made structures, and whether safety culture provides any insulation.  While the earthquake itself seemed fairly well contained at the nuclear plants, the tsunami easily over powered the sea wall at the facility and caused widespread disability of crucial plant systems.  Does this sound familiar?  Does it remind one of a Category 5 hurricane sweeping aside the levees in New Orleans?  Or the overwhelming forces of an oil well blowout brushing aside the isolation capability of a blowout preventer? 

John McPhee’s 1990 book The Control of Nature chronicles a number of instances of man’s struggle against nature - in his view, one that is inevitably bound to fail.  Often the very acts undertaken to “control nature” contribute to future failures of that control.  McPhee cites the leveeing of the Mississippi, leading to faster channel flows, more silting, more leveeing, and ultimately the kind of macro disaster occurring in Katrina.  Or the “debris bins” built in the canyons above Los Angeles communities.  The bins fill over successive storms, eventually leading to failures of the bins themselves and catastrophic mud and debris floods in the downstream valleys.

It is probably inevitable that in the aftermath of Japan there will be calls to up the design criteria of nuclear plants to higher levels of earthquakes and other natural phenomena.  The expectation will be that this will provide the absolute protection desired by the public or groups such as UCS.  Until of course the next storm or earthquake that is incrementally larger, or in a worse location or in combination with some other event, that supersedes the more stringent assumptions.

Safety culture cannot deliver on an expectation that safety is absolute or without limits. It can and should emphasize the priority and unflagging attention to safety that maximizes the capacity of a facility and its staff to withstand unforeseen challenges .   We know that the Japan event proves the former.  It will be equally important to determine if it also showed the latter.   

*  T.Zeller Jr., "Citing Near Misses, Report Faults Both Nuclear Regulators and Operators," New York Times, Green: A Blog About Energy and the Environment (Mar 17, 2011, 1:50 PM)

Safety Culture Performance Indicators

In our recent post on safety culture management in the DOE complex, we concentrated on documents created by the DOE team.  But there was also some good material in the references assembled by the team.  For example, we saw some interesting thoughts on performance indicators in a paper by Andrew Hopkins, a sociology professor at The Australian National University.*  Although the paper was prepared for an oil and gas industry conference, the focus on overall process safety has parallels with nuclear power production.

Contrary to the view of many safety culture pundits, including ourselves, Professor Hopkins is not particularly interested in separating lagging from leading indicators; he says that trying to separate them may not be a useful exercise.  Instead, he is interested in a company’s efforts to develop a set of useful indicators that in total measure or reflect the state of the organization’s risk control system.  In his words, “. . . the important thing is to identify measures of how well the process safety controls are functioning.  Whether we call them lead or lag indicators is a secondary matter.  Companies I have studied that are actively seeking to identify indicators of process safety do not make use of the lead/lag distinction in any systematic way. They use indicators of failure in use, when these are available, as well as indicators arising out their own safety management activities, where appropriate, without thought as to whether they be lead or lag. . . . Improving performance in relation to these indicators must enhance process safety. [emphasis added]” (p. 11)

Are his observations useful for people trying to evaluate the overall health of a nuclear organization’s safety culture?  Possibly.  Organizations use a multitude of safety culture assessment techniques including (but not limited to) interviews; observations; surveys; assessments of the CAP and other administrative processes, and management metrics such as maintenance performance, all believed to be correlated to safety culture.  Maybe it would be OK to dial back our concern with identifying which of them are leading (if any) and which are lagging.  More importantly, perhaps we should be asking how confident we are that an improvement in any one of them implies that the overall safety culture is in better shape. 

*  A. Hopkins, "Thinking About Process Safety Indicators," Working Paper 53, National Research Centre for OHS Regulation, Australian National University (May 2007).  We have referred to Professor Hopkins’ work before (here and here).

Culture Wars

We wanted to bring to our readers attention an article from the McKinsey Quarterly (March 2011) that highlights the ability of management simulators to be powerful business tools.  The context is the use of such “war games” in assisting management teams to accomplish their business goals; but we would allow that their utility extends to other challenges such as managing safety culture.

“Well-designed war games, though not a panacea, can be powerful learning experiences that allow managers to make better decisions.”

“...the company designed a game to answer the more strategic question: how can we win market share given the budget pressures on the Department of Defense and the moves of competitors? The game tested levers such as pricing, contracting, operational improvements, and partnerships.  The outcome wasn’t a tactical playbook—a list of things to execute and monitor—but rather strategic guidance on the industry’s direction, the most promising types of moves, the company’s competitive strengths and weaknesses, and where to focus further analysis.” (p. 3)  We have often used the term “levers” to bring attention to the need for managers to understand when and how to take actions to bring about a desired safety culture result.  Levers connote control and, as with any control system, control must be based on an understanding of the system’s dynamics.  Importantly the above quote distinguishes the outcome of the simulated experience is not a “playbook”, but “guidance” (we would add a deeper understanding and developed skills) that can be applied in the real world.

Interestingly the article mentions the use of games to facilitate or achieve organizational alignment around a strategic decision.  This treads very close to our contention that using a safety culture simulator offers a powerful environment within which managers can interact including developing common mental models and understanding of culture dynamics.  As noted in the article, “This shared experience...has continued to stimulate discussions across the company…” (p. 4)  What could be more valuable for reinforcing safety culture than informed and broad based discussion within the organization?  As Horn says, “It’s often beneficial, however, to repeat a game for the sake of organizational alignment ... usually, the wider group of employees who will implement the decision. Most people learn better by doing, and when they have shared experiences, they are more likely to embrace change.”

Safety Culture in the DOE Complex

This post reviews a Department of Energy (DOE) effort to provide safety culture assessment and improvement tools for its own operations and those of its contractors.

Introduction

The DOE is responsible for a vast array of organizations that work on DOE’s programs.  These organizations range from very small to huge in size and include private contractors, government facilities, specialty shops, niche manufacturers, labs and factories.  Many are engaged in high-hazard activities (including nuclear) so DOE is interested in promoting an effective safety culture across the complex.

To that end, a task team* was established in 2007 “to identify a consensus set of safety culture principles, along with implementation practices that could be used by DOE . . .  and their contractors. . . . The goal of this effort was to achieve an improved safety culture through ISMS [Integrated Safety Management System] continuous improvement, building on operating experience from similar industries, such as the domestic and international commercial nuclear and chemical industries.”  (Final Report**, p. 2)

It appears the team performed most of its research during 2008, conducted a pilot program in 2009 and published its final report in 2010.  Research included reviewing the space shuttle and Texas City disasters, the Davis-Besse incident, works by gurus such as James Reason, and guidance and practices published by NASA, NRC, IAEA, INPO and OSHA.

Major Results

The team developed a definition of safety culture and described a process whereby using organizations could assess their safety culture and, if necessary, take steps to improve it.

The team’s definition of safety culture:

“An organization’s values and behaviors modeled by its leaders and internalized by its members, which serve to make safe performance of work the overriding priority to protect the workers, public, and the environment.” (Final Report, p. 5)

After presenting this definition, the report goes on to say “The Team believes that voluntary, proactive pursuit of excellence is preferable to regulatory approaches to address safety culture because it is difficult to regulate values and behaviors. DOE is not currently considering regulation or requirements relative to safety culture.” (Final Report, pp. 5-6)

The team identified three focus areas that were judged to have the most impact on improving safety and production performance within the DOE complex: Leadership, Employee/Worker Engagement, and Organizational Learning. For each of these three focus areas, the team identified related attributes.

The overall process for a using organization is to review the focus areas and attributes, assess the current safety culture, select and use appropriate improvement tools, and reinforce results. 

The list of tools to assess safety culture includes direct observations, causal factors analysis (CFA), surveys, interviews, review of key processes, performance indicators, Voluntary Protection Program (VPP) assessments, stream analysis and Human Performance Improvement (HPI) assessments.***  The Final Report also mentioned performance metrics and workshops. (Final Report, p. 9)

Tools to improve safety culture include senior management commitment, clear expectations, ISMS training, managers spending time in the field, coaching and mentoring, Behavior Based Safety (BBS), VPP, Six Sigma, the problem identification process, and HPI.****  The Final Report also mentioned High Reliability Organization (HRO), Safety Conscious Work Environment (SCWE) and Differing Professional Opinion (DPO). (Final Report, p. 9)  Whew.

The results of a one-year pilot program at multiple contractors were evaluated and the lessons learned were incorporated in the final report.

Our Assessment

Given the diversity of the DOE complex, it’s obvious that no “one size fits all” approach is likely to be effective.  But it’s not clear that what the team has provided will be all that effective either.  The team’s product is really a collection of concepts and tools culled from the work of outsiders, combined with DOE’s existing management programs, and repackaged as a combination of overall process and laundry lists.  Users are left to determine for themselves exactly which sub-set of tools might be useful in their individual situations.

It’s not that the report is bad.  For example, the general discussion of safety culture improvement emphasizes the importance of creating a learning organization focused on continuous improvement.  In addition, a major point they got right was recognizing that safety can contribute to better mission performance.  “The strong correlation between good safety performance with good mission performance (or productivity or reliability) has been observed in many different contexts, including industrial, chemical, and nuclear operations.” (Final Report, p. 20)

On the other hand, the team has adopted the works of others but does not appear to recognize how, in a systems sense, safety culture is interwoven into the fabric of an organization.  For example, feedback loops from the multitude of possible interventions to overall safety culture are not even mentioned.  And this is not a trivial issue.  An intervention can provide an initial boost to safety culture but then safety culture may start to decay because of saturation effects, especially if the organization is hit with one intervention after another.

In addition, some of the major, omnipresent threats to safety culture do not get the emphasis they deserve.  Goal conflict, normalization of deviance and institutional complacency are included in a list of issues from the Columbia, Davis-Besse and Texas City events (Final Report, p. 13-15) but the authors do not give them the overarching importance they merit.  Goal conflict, often expressed as safety vs mission, should obviously be avoided but its insidiousness is not adequately recognized; the other two factors are treated in a similar manner. 

Two final picky points:  First, the report says it’s difficult to regulate behavior.  That’s true but companies and government do it all the time.  DOE could definitely promulgate a behavior-based safety culture regulatory requirement if it chose to do so.  Second, the final report (p. 9) mentions leading (vs lagging) indicators as part of assessment but the guidelines do not provide any examples.  If someone has some useful leading indicators, we’d definitely like to know about them. 

Bottom line, the DOE effort draws from many sources and probably represents consensus building among stakeholders on an epic scale.  However, the team provides no new insights into safety culture and, in fact, may not be taking advantage of the state of the art in our understanding of how safety culture interacts with other organizational attributes. 


*  Energy Facility Contractors Group (EFCOG)/DOE Integrated Safety Management System (ISMS) Safety Culture Task Team.

**  J. McDonald, P. Worthington, N. Barker, G. Podonsky, “EFCOG/DOE ISMS Safety Culture Task Team Final Report”  (Jun 4, 2010).

***  EFCOG/DOE ISMS Safety Culture Task Team, “Assessing Safety Culture in DOE Facilities,” EFCOG meeting handout (Jan 23, 2009).

****  EFCOG/DOE ISMS Safety Culture Task Team, “Activities to Improve Safety Culture in DOE Facilities,” EFCOG meeting handout (Jan 23, 2009).

BP Exec Quit Over Safety Before Deepwater Disaster

Today’s Wall Street Journal has an interesting news item about a BP Vice President who quit prior to the Deepwater Horizon disaster because he felt BP "was not adequately committed to improving its safety protocols in offshore drilling to the level of its industry peers." The full article is available here.

“what people do, not why they do it…”

Our perseverance through over three hours of the web video of the Commission meeting on the proposed safety culture policy statement was finally rewarded in the very last minute of discussion.  Commissioner Apostolakis reiterated some of his concerns with the direction of the policy statement, observing that the NRC is a performance-based agency and:

“...we really care about what people do and maybe not why they do it….”

Commissioner Apostolakis was amplifying his discomfort with the inclusion of values along with behaviors in the policy as values are inherently fuzzy, not measurable, and may or may not be a prerequisite to the right behaviors.  Perhaps most of all, he believed omitting the reference to core values would not detract from the definition of safety culture. 

Earlier in the meeting Commissioner Apostolakis had tried to draw out the staff on whether the definition of safety culture needed values in addition to behaviors [at time 2:34:58], and would it be a fatal flaw to omit “core values”.  The staff response was illuminating.  The justification offered for retaining values was “stakeholder consensus”, and extensive outreach efforts that supported inclusion.  (But why was it so important to stakeholders?)  The staff went on to clarify: “culture does not lend itself to be inspectable”, but “having values with behaviors is what culture is all about”.   Frankly we’re not sure what that means, but we do know that safety culture behaviors are inspectable because they are observable and measurable.

That much of the staff’s justification for including values in the policy statement seemed to reside in the fact that all the stakeholders had agreed received positive endorsement by Chairman Jaczko when he observed:  “...Commissioner Magwood I think made a profound point that there was value in this process here that may be tremendously more important than the actual policy statement was the fact that people got together and started talking about this and realized that across this wide variety of stakeholders, there was pretty good agreement about the kinds of things that we were talking about.”

Chairman Jaczko also weighed in on the values-behaviors contrast, coming down firmly on the inclusion of values and offering the following justification:

“...not all entities with a good safety culture will have necessarily the right values…”

Respectfully, we believe at a minimum this will further confuse the NRC’s policy on safety culture, and in all likelihood places emphasis in exactly the wrong place.  Is the Chairman agreeing all that matters is what people do?  Or is he suggesting that the NRC would find fault with a licensee that was acting consistent with safety but did not manifest the “right” values.  And how would the NRC reach such a finding?  More fundamentally, isn’t Commissioner Apostolakis correct in his blunt statement - that we [NRC] don’t care why they [licensees] do it?

More Hope

Our prior post highlighted a comment early in the January 24, 2011 Commission meeting to review the proposed policy statement on nuclear safety culture. 

In the context of her advocacy for regulations in addition to a policy statement, attorney Billie Garde stated she “hoped” that proceeding with just a policy statement was the right decision.  We thought her warning of the fallout from a possible future nuclear event would get some attention.  It did, at least with Commissioner Svinicki who sought some clarification of Garde’s concern.  Just prior to this clip, Svinicki had observed that in her mind a policy statement can’t supplant an appropriate regulatory framework in terms of compelling certain behaviors.  No matter what you think about the appropriateness of a policy statement versus other regulatory actions, Garde is certainly correct that the question will be asked in the future: Did the NRC do enough?

“I Hope For All Our Sakes This is Right”

On January 24, 2011 the NRC Commissioners met to review the proposed policy statement on nuclear safety culture developed by the NRC staff. This most recent effort was chartered by the Commission more than 3 years ago and represents the next step in the process to publish the proposed statement for public comment.

“25 years is long enough to build a policy statement…” for nuclear safety culture. This observation by Billie Garde* in her opening remarks to the Commissioners, with her timeline referring to the Chernobyl and space shuttle Challenger accidents in 1986. She also emphasized that the need was to now focus on implementation of the policy statement. She maintained her position that a policy statement alone would not be sufficient and that regulation would be necessary to assure consistent and reliable implementation.

In that regard she lays claim to one of the more disconcerting observations made at the meeting, the gist of which can be summed up as, “I hope for all our sakes this is right…”

Here’s the video clip with the exchange between Garde and Commissioner Apostolakis.

We will be following up with additional posts with highlights from the Commission session.

*  Billie Garde is an attorney in Washington, D.C.  Her NRC website bio is here.

Culture de la Sécurité

If you are paying attention you noticed we’re using French words.  And the reason is the current news regarding Air France and its safety practices and safety culture.

The principal finding of an independent study of the airline’s operations was a lack of “strong safety leadership at all levels of management" as reported in a January 26 Wall Street Journal (WSJ) article.*

While Air France has refused to make the report public, a review by the WSJ stated that the study was “sharply critical of broad aspects of the safety culture”.

Over the previous several days there have been articles in the WSJ preceding and attending the completion of the independent study.  The first of the articles** previewed some of the findings and provided favorable commentary based on an understanding that Air France would be making the report public (“Air France to Disclose…”).  This was characterized as a move toward greater openness on the part of the airline and commented, “safety experts said it was unusual for a large company, especially an airline, to give outsiders such latitude publicly to expose gaps in safety systems.”  And quoting Bill Voss, president of the Flight Safety Foundation, a global air-safety advocacy organization based in Alexandria, Va., "It's extraordinary that they are willing to release the results," said Mr. Voss. The process "gives me confidence there will be follow-through."

Well, in a follow-up article the next day, when receipt of the study results was announced by Air France, the reaction was disappointment as the airline indicated it would not release the report or its recommendations.  Nonetheless the airline trumpeted its actions:

“According to an Air France news release, the report said that creating the outside review team ‘in a public manner and providing it a broad charter’ to examine flight safety ‘was a courageous act’ and an example of safety leadership ‘rarely seen in today's international aviation industry.’” ***

In deciding on whether Air France should be awarded the Légion d'honneur, one might also consider that the current safety study was undertaken in the wake of four serious crashes between 1999 and 2009 (the most recent being the flight from Rio to Paris that went down mid-Atlantic).  In addition, it follows a prior safety study,

“Finished in 2006, that report was distributed to more than 4,000 company pilots and was widely praised for its frankness about shortcomings within the carrier. Although Air France has said its executive committee made formal decisions to implement the report's recommendations, critics of the airline continue to maintain management didn't aggressively pursue the changes.” (Jan 24, 2011)

I think we’ve seen this before.  Think about the safety studies after the Challenger crash, but not really implemented and then the disintegration on reentry of the Columbia.  Or BP and the Texas oil refinery fire followed by the Deepwater Horizon last summer.  Obviously safety assessments, no matter how strong and how independent, ultimately require the subject organization to implement changes.  We think the current Air France report correctly fingers safety leadership by management “starting at the top”.  And it never fails in these situations that top management describes as its highest priority…...can anybody guess……that’s right, it’s “safety first”.  (Jan 26, 2011)

Perhaps if it was safety first Air France might be releasing the study and its recommendations.  Wouldn’t that help make real its safety priority and wouldn’t such transparency help ensure that the recommendations are actually implemented?   We have commented in prior posts on transparency and we will continue to emphasize its importance to safety culture across all industries.


*  A. Pasztor, D. Michaels and D. Gauthier-Villars, “Air France Panel Cites Wide Safety Deficiencies,” WSJ.com (Jan 26, 2011).

** A. Pasztor and D. Michaels, “Air France to Disclose Review's Criticisms,” WSJ.com (Jan 24, 2011).

***  A. Pasztor, “Air France Enhances Safety Efforts,” later re-headlined “Air France Withholds Key Report,” WSJ.com (Jan 25, 2011).

A Nuclear Model for Oil and Gas

The President’s Commission has issued its report on the Deepwater Horizon disaster.* The report reviews the history of the tragedy and makes recommendations based on lessons learned.  This post focuses on the report’s use of the nuclear industry, in particular the role played by INPO, as a model for an oil and gas industry safety institute and auditor.

The report provides an in-depth review of INPO’s role and methods and we will not repeat that review in this space.  We want to highlight the differences between the oil and gas and nuclear industries, some recognized in the report, that would challenge a new safety auditor. 

First, “The oil and gas industry is more fragmented and diversified in nature. . . .” (p. 240)  The industry includes vertically integrated giants, specialty niche firms and everything in-between.  Some are global in nature while others are regional firms.  In our view, it appears that oil and gas industry participants cooperate with each other in certain instances and compete with each other in different cases.  (In contrast, most [all?] U.S. nuclear plants are not in direct competition with other plants.)  Obtaining agreement to create a relatively powerful industry auditing entity will not be a simple matter.    

Second, “concerns about potential disclosure to business competitors of proprietary information might make it harder to establish an INPO-like entity in the oil and gas industry.” (p. 240)  Oil and gas firms regard technology as an important source of competitive advantage.  “[A]n INPO-like approach might run into problems if companies perceived the potential for inspections of offshore facilities to reveal ‘technical and proprietary and confidential information that companies may be reluctant to share with one another.’” (p. 241)  Not only will it be difficult to get a firm to share its proprietary technology if it may lose competitive advantage by doing so, but this will make it more difficult for the auditing organization to promote the industry-wide use of the most effective, safest technologies

Third, and this could be a potentially large problem, INPO operates in almost total secrecy.  “[INPO] assessment results are never revealed to anyone other than the utility CEOs and site managers, but INPO formally meets with the NRC four times a year to discuss trends and information of “mutual interest.” And if INPO has discovered serious problems associated with specific plants, it notifies the NRC.”  (p. 236)  INPO claims, probably realistically, that maintaining member confidentiality is key to obtaining full and willing cooperation in evaluations. 

However, this secrecy contributes zero to public understanding of and support for nuclear plant operations and owners.  At this point in its evolution, the oil and gas industry needs more transparency in its auditing and oversight functions, not less.  After all, and forgive the bluntness here, very few people have died at U.S. commercial nuclear power plants (and those were in non-nuclear incidents) while the oil and gas industry has suffered numerous fatalities.  We think a government auditor, whose evaluations of facilities and managements would be made public, is the better answer for the oil and gas industry at this time.


*  National Commission on the BP Deepwater Horizon Oil Spill and Offshore Drilling, “Deep Water: The Gulf Oil Disaster and the Future of Offshore Drilling,” Report to the President (Jan 2011).

NRC Policy Statements

For some time we have been thinking about one of the underlying aspects of the NRC’s current safety culture initiative which is to establish a “policy statement” for nuclear safety culture.  As we know the use of a policy statement in this area dates back to 1989.  A subtext to this approach is whether the NRC should “regulate” nuclear safety culture, presumably through the issuance of regulatory rules and requirements.  We recognize that the “regulate” issue is very much a hot button and we are not addressing it at this time.  Instead we thought it might be worthwhile to consider in some detail just what is a Commission policy statement and what it might or might not accomplish.

On the NRC website we looked at what was available regarding policy statements.  There is a Commission Policy Statements page with a listing of the current set of policy statements, organized by topic.  However, most policy statements relate to the conduct of business by the NRC itself with fewer statements addressing substantive regulatory and safety criteria.  We could not locate on the website information regarding how policy statements are intended to be used in the regulatory process.  For that we turned to recent NRC Issuances,* which are adjudicatory decisions by Atomic Safety and Licensing Boards and the Commission itself, to obtain guidance on the applicability and weight accorded NRC Policy Statements. 

“Policy Statements are neither rules nor orders, and therefore do not establish requirements that bind either the agency or the public.”**

This comes from a Commission decision for a case involving the scope of environmental review for an early site permit.  The Commission references a D.C. Circuit Court case that found:

“A general statement of policy . . . does not establish a ‘‘binding norm.’’ It is not finally determinative of the issues or rights to which it is addressed. The agency cannot apply or rely upon a general statement of policy as law because a general statement of policy only announces what the agency seeks to establish as policy.” (p. 240)

In its decision, the Commission goes on to state:

“For the [Atomic Safety and Licensing] Board to suggest that the strictures of the Policy Statement may be enforced as law, or that it in some way creates a substantive mandate, accords too much weight to the Policy Statement.” (p. 240) 

So far so good.  But after finding that the staff’s review satisfied applicable statutory and regulatory requirements (but did not comport with the letter of the policy statement), the Commission ends its decision on a more confusing note.

“We expect conformance with the Policy Statement, and relevant associated guidance, in future licensing actions of this magnitude.” (p. 248)

In another matter involving a policy statement governing the admissibility of contentions in license renewal proceedings, Commissioners Merrifield and McGaffigan joined in a concurring opinion to observe:

“If we are not willing to enforce our policy statements, the statements become meaningless.”***

Finally, an earlier ASLB decision seems to address policy statements on a practical level, where the Board feels “compelled” by Commission policy: 

Notwithstanding these clear inconsistencies, we find ourselves compelled by Commission rulings and policy statements to accept this approach by the Staff because the Commission has advised that their ‘‘longstanding practice . . . grounded in sound policy’’ is to ‘‘leave [ ] to the expert NRC technical staff prime responsibility for technical fact-finding on uncontested matters.’’****

Based on all of this what is the likely impact of the safety culture policy statement on NRC license holders?  On the one hand it appears that the NRC will “expect” licensees to meet the intent and the particulars of its policy statement.  It seems safe to assume the NRC staff will apply the policy in its assessments of licensee performance.  On the other hand, if a licensee does not meet some aspect of the policy it could find solid footing in a challenge to the enforceability of the policy statement.  The greatest difficulty is to square the rhetoric of NRC Commissioners and staff regarding the absolute importance of safety culture to safety, the “nothing else matters” perspective, with the inherently limited and non-binding nature of a policy statement.


*  NRC Issuances are published as NUREG-0750.  Individual volumes are available here. 

**  66 NRC 215 (2007) at 217, In the Matter of: DOMINION NUCLEAR NORTH ANNA, LLC (Early Site Permit for North Anna ESP Site) CLI-07-27 Nov 20, 2007.

***  65 NRC 1 (2007) at 8, In the Matter of: ENTERGY NUCLEAR VERMONT YANKEE, LLC, and ENTERGY NUCLEAR OPERATIONS, INC. (Vermont Yankee Nuclear Power Station) CLI-07-1 Commissioner Jeffrey S. Merrifield, with Whom Commissioner Edward McGaffigan, Jr. Joins, Concurring Jan 11, 2007.

****  64 NRC 460 (2006) at 492, ATOMIC SAFETY AND LICENSING BOARD In the Matter of EXELON GENERATION COMPANY, LLC (Early Site Permit for Clinton ESP Site) Dec 28, 2006.

ACRS and Safety Culture Policy (cont.)

Our previous post reported on the ACRS letter to the NRC endorsing the agency’s approach to developing a safety culture policy.*  We noted the concern of some ACRS members that the policy might be a back door method to impose regulatory requirements while avoiding the requirements of the regulatory process.

The dissenting ACRS members also raised some other interesting issues about the proposed policy.

First, they questioned whether the proposed traits were the most important ones in terms of their contribution to safety.  Why weren’t organizational and individual integrity, and technical competence included?  Good question.  After all, wasn’t an integrity shortfall at the heart of the misleading of the Vermont senate and the willful violations at San Onofre?

Second, they commented “[T]here is faint evidence that the listed traits (individually or collectively) are assured to produce measureable improvements in safety.” (p. 4)  We raised the same issue in our October 22, 2010 post on the NRC safety culture workshop.  What are the linkages, if any, between the traits and measurable or observable safety-related performance?

Our concern about the lack of demonstrated linkages leads to what may be a bedrock question underlying all of the safety culture policy discussion: If the ROP isn’t providing sufficient information to support the NRC’s confidence in a licensee’s safety culture, then how can the agency develop that information in a defined, disciplined and vetted manner?  Is a safety culture policy going to provide that assurance?


*  Letter dated Dec 15, 2010 from S. Abdel-Khalik (ACRS) to G. Jaczko (NRC), subject "Safety Culture Policy Statement," ADAMS Accession Number ML103410358.

ACRS Weighs In on Safety Culture Policy

In mid-December the Advisory Committee on Reactor Safeguards (ACRS) provided the results of its review of the NRC’s proposed nuclear safety culture policy in a letter to NRC Chairman Jaczko.*  The letter reiterated the approach and general structure of the proposed policy and reached a favorable conclusion.  Perhaps the most interesting comment in the main body of the letter is the following:

“Well-intentioned attempts at improving safety and effectiveness have faltered through efforts to overly prescribe correct behavior and to apply rigid scoring systems. We urge that the staff encourage approaches that emphasize thinking and safety awareness over scorecards of metrics that can induce complacency and rote compliance. Issuance of a policy statement, rather than a regulation, is likely to be a more effective way to appropriately engage all the stakeholders.” (p. 4)

The statement is a bit cryptic and we can only guess what the ACRS has in mind when it refers to “scorecards of metrics” or “overly prescribing behavior”.  Are they referring to the ROP?  Is the ACRS concerned that reliance on the ROP metrics (and their almost uniformly green status) may be lulling the industry and the NRC into complacency?  Equally uncertain is why the ACRS believes that a policy statement will lead to more effective results. 

Apparently we are not the only ones to suffer uncertainty.  The ACRS letter includes “Additional Comments” (read: dissenting comments) by three members** who state:

“It is not entirely clear to us what is meant by implementing a policy statement that lacks the authority of regulation. It appears that implementation of the safety culture policy statement may be an indirect method of imposing requirements on licensees without the discipline of the regulatory process. This, of course, is not acceptable.” (p. 4)

Part of the confusion may lie in the intent and authority associated with NRC policy statements.  It appears that the dissenting members feel that a policy statement would be a back door method to impose “requirements”.  Is that true?  We will follow with a detailed look at policy statements and their effect.


*  Letter dated Dec 15, 2010 from S. Abdel-Khalik (ACRS) to G. Jaczko (NRC), subject "Safety Culture Policy Statement," ADAMS Accession Number ML103410358.

** D.A. Powers, J.S. Armijo and J.L. Rempe.

Nothing Else Matters

In early December NRC Commissioner Ostendorff provided the keynote speech at Nuclear Energy Asia.*  A significant portion of his remarks addressed the importance of safety culture to nuclear safety.  In terms of safety culture insight I think it is fair to say there wasn’t much new here.  However the continued emphasis by the NRC at the Commissioner level may be signaling how they may choose to proceed with regulation of nuclear safety culture.

“Decades of experience in the nuclear field have shown that regulators have to do more than simply establish standards. Rather, I believe it to be more appropriate for a regulator to establish a high-level expectation or policy to help foster the development and maintenance of a strong safety culture.” (p. 2)

This sounds consistent with the current safety culture policy track being followed by the NRC staff.  But just preceding this comment, Ostendorff said: 

“Within the national nuclear safety infrastructure, I believe that a strong safety culture is the key foundation. Without this one essential cornerstone – a strong safety culture – nothing else matters.” (p. 2)

This echoes statements by other Commissioners, and in regulatory actions for specific licensees, that safety culture is essential to nuclear safety.  If that is the case, doesn’t it set the bar very high in terms of regulatory responsibility for ensuring adequate safety culture?   And with the bar set so high, is it sufficient for the NRC to just establish a policy or “expectations” for safety culture, but not rules or regulatory requirements?  We will present a much more detailed look at NRC policy making in an upcoming post.


*  W.C. Ostendorff, "Regulatory Perspectives on Nuclear Safety," International Keynote Address, Nuclear Energy Asia 2010, Hong Kong, China (Dec 7, 2010) ADAMS Accession Number ML103420523.

ESP and Safety Culture

A recent New York Times article* on an extrasensory perception (ESP) study and the statistical methods used therein caught our attention.  The article’s focus is on the controversy surrounding statistical significance testing.  “A finding from any well-designed study — say, a correlation between a personality trait and the risk of depression — is considered “significant” if its probability of occurring by chance is less than 5 percent.”  We have all seen such analyses.

However, critics of classical significance testing say a finding based on such a test “could overstate the significance of the finding by a factor of 10 or more,” a sort of super false positive.  The critics claim a better approach is to apply the methods of Bayesian analysis, which incorporates known probabilities, if available, from outside the study.  Check out the comments on the article, especially the reader recommended ones, for more information on statistical methods and issues.  (You can ignore the ESP-related comments unless you have some special interest in the topic). 

What has this got to do with safety culture?

Recall that last October we reported on an INPO study that, among other things, calculated correlations between safety culture survey factors and various safety-related performance measures.  We expressed reservations about the overall approach and results even though a few correlations supported points we have been making in our blog.

The controversy over the ESP study and its associated statistical methods reminds us that analysts in many fields are under pressure to find something “significant.”  This pressure comes from bosses, funding agencies, editors and tenure committees.  Studies that find no effects, or ones not aligned with higher-level organizational objectives, are less likely to be publicized and their authors rewarded.  In addition, I fear some (many?) social science researchers don’t fully understand the statistical methods they are using, i.e., their built-in biases and limitations.  So, once again, caveat emptor.   

By the way, we are not saying or implying the INPO study was biased in any way; we have no information on it other than what was presented at the NRC meeting referenced in our original blog post. 

*  B. Carey, “You Might Already Know This ...,” New York Times (Jan 11, 2011).

Pick Any Two

Last week principal findings of the BP Oil Spill Presidential Commission were released.   Not surprisingly it cited root causes that were “systemic”, decisions without adequate consideration of risks, and failures of regulatory oversight.  It also cited a lack of a culture of safety at the companies involved in the Deepwater Horizon.  We came across an interesting entry in a blog tied to an article in the New York Times by John Broder on January 5, 2011, “Blunders Abounded Before Gulf Oil Spill, Panel Says”.  We thought it was worth passing on. 

Comment No. 7 of 66 submitted by:
Jim S.
Cleveland
January 5th, 2011
7:23 pm

“A fundamental law of engineering (or maybe of the world in general) is "Cheaper, Faster, Better: Pick Any Two".  

Clearly those involved, whether deliberately or by culture, chose Cheaper and Faster.”

Nuclear Safety Culture Assessment Manual

July 9, 2012 update: How to Get the NEI Nuclear Safety Culture Assessment Manual

The manual is available in the NRC ADAMS database, Accession Numbers ML091810801, ML091810803, ML091810805, ML091810807, ML091810808 and ML091810809.

**********************************************************

 

As recently reported at TheDay.com,* NEI has published a “Nuclear Safety Culture Assessment Manual,” a document that provides guidance for conducting a safety culture (SC) assessment at a nuclear power plant.  The industry has issued the manual and conducted some pilot program assessments in an effort to influence and stay ahead of the NRC’s initiative to finalize a SC policy statement this year.  The NRC is formulating a policy (as opposed to a regulatory requirement) in this area because it apparently believes that SC cannot be directly regulated and/or any attempt to assess SC comes too close to evaluating (or interfering with) plant management, a task the agency has sought to avoid. 

Basically, the manual describes an assessment methodology based on the eight INPO principles for creating/maintaining a strong nuclear safety culture.  It is a comprehensive how-to document including assessment team organization, schedules, interview guidance and questions, sample communication memos, and report templates.  The manual has a strongly prescriptive approach, i.e., it seeks to create a standardized approach which should facilitate comparisons between different facilities and the same facility over time. 

The best news from our perspective is that the NEI assessment approach relies heavily on interviews; it uses a site survey instrument only to identify pre-assessment areas of interest.  It’s no secret that we are skeptical about over-inference with respect to the health of a plant’s safety culture from the snapshot a survey provides.  The assessment also uses direct observations of behavior of employees at all levels during scheduled activities, such and meetings and briefings, and ad-hoc observation opportunities.

A big question is: In a week-long self assessment, can a team discern the degree to which an organization satisfies key principles, e.g., the level of trust in the organization or whether leaders demonstrate a commitment to safety?  I think we have to answer that with “Maybe.”  Skilled and experienced interviewers can probably determine the general status of these variables but may not develop a complete picture of all the nuances.  BUT, their evaluation will likely be more useful than any survey.

There is one obvious criticism with the NEI approach which industry critics have quickly identified.  As David Collins puts it in TheDay.com article, “[T]he industry is monitoring itself - this is the fox monitoring the henhouse."  While the manual is proposed for use by anyone performing a safety culture assessment, including a truly independent third party, the reality is the industry expects the primary users to be utilities performing self assessments or “independent” assessments, which include non-utility people on the team. 


*  P. Daddona, “Nuclear group puts methods into use to foster 'a safety culture',” TheDay.com
(Dec 21, 2010).