Monday, May 12, 2014

Willful Violations at Indian Point

We report in this post on a situation that developed at Indian Point more than two years ago and was just recently closed out via NRC notices of violation to an individual (a Chemistry Manager for Entergy Nuclear Operations) and to Entergy Nuclear Operations itself. 

What should we make of another willful misconduct episode?  A misguided individual who made some bad choices but where the actual impact on safety (per Entergy and the NRC) was not significant?  The individual resigned (and plead to a felony conviction and probation), corrective actions to reinforce proper behaviors have been taken, and violations issued...what difference does it make?

The Events Surrounding the Misconduct

We are attaching a series of references as they contain more detail than we can recount in a blog post.  In particular Reference 4 provides the most comprehensive rendition of the relevant events.  Very briefly this is what occurred: During 2011 routine testing of diesel fuel oil at Indian Point (IP), as required by Tech Specs, indicated that the limits on particulate concentration were exceeded.  The Chemistry Manager with responsibility for this testing did not report (initiate Condition Reports) the anomalous results which would have resulted in the reserve fuel oil storage tank (RFOST) being declared inoperable.  The LCO is 30 days and if operability was not restored, shutdown of both IP units would have been required. [Ref 2, Cover Letter]  In early 2012 as part of a systems engineering self-assessment, the anomalous results and lack of reporting were identified.  The Chemistry Manager falsely indicated that re-sampling and testing had been performed which were acceptable.  He subsequently made false data entries to support this story.

A short time later employee concerns were filed via the Entergy Ethics Line and the Employee Concerns Program (ECP).  Entergy initiated an investigation using outside attorneys (Morgan Lewis).  At the same time the NRC initiated an Office of Investigations (OI) investigation.  The Chemistry Manager refused to cooperate in the investigation and resigned.  Subsequent testing of the fuel oil indicated limits were being exceeded and compensatory actions were taken.  Pursuant to the investigations the Chemistry Manager admitted willful misconduct.  The US Attorney issued a criminal complaint and ultimately the manager plead to a felony and received probation.  Entergy was cited for a Severity Level III violation, civil penalty waived.

Further Observations

Plowing through the documentation of this issue left us with a few lingering questions.  One is with regard to the sanitized LER that Entergy submitted to the NRC in August 2012.  The LER makes no mention of the filing of employee concerns, investigation by outside attorneys or the NRC OI investigation.  For that matter the LER never mentions that the cause of the event was willful misconduct by a department manager.  Rather it characterizes the situation in the abstract - as a failure to use the corrective action program.  In other words a whole lot was happening in the background which would cast the event in a different light, including its potential significance.*

While the cited violations are linked to the misconduct of the Chemistry Manager, it appears there had been ongoing issues within the Chemistry Department for some time: entering test data diligently, understanding the significance of the data, and initiating CRs.  “The circumstances surrounding the violations are of concern to the NRC because they indicate a lack of consideration for (and/or knowledge of) TS requirements by ENO Chemistry staff.  The NRC also noted that the Chemistry Manager would not have had the opportunity to commit the violations had ENO staff exhibited the proper regard for the site TS.”  [Ref 4, p. 4]  But in its chronology of events, Entergy contends that in March 2102 there was “no reason to question the integrity of former Chemistry Manager…” [Ref 4, Encl 2, slide 15].  Perhaps not the integrity, but what about management effectiveness? 

Further context.  Entergy gives itself credit for how it responded to the evolving situation.  They highlight that a self-assessment team identified the anomalies (true), that employees raised concerns through established programs (true), that Entergy conducted an investigation (true).  [Ref 4, Encl 2, slide 35]  But what is missing is that normal business processes (management oversight, QA audits, or Chemistry Department personnel) did not identify the anomalies prior to the self-assessment; that employees felt the need to use the Ethics Line and the ECP rather than directly raising within the management chain; that upon discovery of the anomalies, it appears that Entergy went to great lengths to avoid declaring that the fuel oil did not meet specs.**  The net result is that the RFOST was able to be maintained as operable for almost three months before definitive action was taken to filter the oil. [Ref 4, Encl 2, slides 17-21]

Why?

The most interesting and relevant question posed by these events is why did the Chemistry Manager take the actions he did?  “The Manager said that he falsified the data because he needed more time to prove his theory [that the IP Chemistry Department’s sampling practices were poor] and incorporate new test methods, and he had not wanted the plant to unnecessarily shut down.”  [Ref 2, Encl 1] That is the extent of what the NRC reports on its investigation of the motive of the Chemistry Manager.  An employee for 29 years undertakes a series of deliberate violations of his professional responsibilities “to prove his theory”.  Perhaps. 

One of the final corrective actions implemented for this event occurred in December 2013 when the General Manager for Plant Operations briefed the Department Managers on deliberate misconduct.  Included was a statement, "If we have to shutdown the plant we will do so". [Ref 4, Encl 2, slide 32] Without reading too much into a single bullet point, one wonders if this is a tacit acknowledgment by Entergy that the Chemistry Manager may have been influenced to do what he did because he did not want to be the cause of a plant shutdown.

We would be very interested to see how much probing was done by the NRC investigators, or Entergy’s attorneys, of this individual’s motive, particularly in terms of any perceived pressure to keep the plant operating.  Such pressure needn’t come from Entergy, it seems self-evident that Indian Point’s licensing situation and the long standing political opposition within New York State poses an existential threat to the plant.  If his motive was just a matter of a revised test “theory”, were these the first out-of-spec fuel oil test results on his watch?  If there had been others, how were they handled?  How long had he been in the position?  Had he initiated any other actions prior to this time to investigate the testing protocol?  As we noted in our post dated September 12, 2013 regarding the NRC’s Information Notice on willful violations, in none of the cited examples did the NRC provide any perspective on the motives of the individuals or the potential effects of the environment within which they were working.

Safety and Safety Culture

How does all of this shed any light on safety and safety culture? 

A key dimension of safety culture is the accurate assessment of safety significance.  The position of Entergy, and adopted by the NRC***, was that the actual impact of the violations on reactor safety was not significant. [Ref 4, Encl 2, slide 36]  Also note that NRC finds that all of this is in the ROP category for “green” significance. The argument is a familiar one.  TS limits are conservative and below what is actually “OK”.  And if particulates are a problem there are filters on the diesel generators, and these can be changed out during operation of the diesels if necessary.  This is a familiar characterization - safety significance is evaluated within the strict boundaries of the NRC’s safety construct of design basis assumptions, almost exclusively hardware based.  As we noted in our September 24, 2013 post, this ignores the larger environment and “system” within which people actually function. 

The Synergy Safety Culture Survey conducted from Feb to April 2012 is cited as finding a “healthy work environment in Chemistry Department” - yet this was at the very time test results were being falsified by the manager and employees were resorting to the ECP to raise issues.  Other assessments by the NRC and INPO also did not identify issues. [Ref 4, Encl 2, slide 29].  There is reference to an “independent investigation” of the employee concerns but the documentation does not reveal who did the investigation or its findings.  The investigation found “no one interviewed” had a reluctance to raise an issue.  Nowhere is the prior use of the Ethics Line and ECP by several individuals on an anonymous basis explained. 

Something that is hard to square is the NRC assertion that there is a strong link between willful violations and safety culture, and the results of these various assessments at Indian Point by Synergy, the NRC and INPO.  So if there is a link, and safety culture assessments don’t reveal its presence, are the assessments valid?  Or if the assessments are valid, is there really a link with willful misconduct? 

Here’s our take.  Willful misconduct is an indication of an issue with the safety culture.  But the issue arises out of a broader and more complex context than the NRC or industry is willing to address.  At Indian Point there is an overriding operating context where the extension of the plants’ operating licenses is being contested by powerful political forces in New York State.  If the licenses are not extended, the plants close and people lose their jobs.  This is not theoretical as the Entergy-owned plant, Vermont Yankee, is doing just that.  If you are an employee at Indian Point, you must feel that pressure every day.  When an issue comes up such as failed diesel fuel tests that could result in temporary shutdown of both units, it is an additional threat to the viability of the plant.  That pressure can create a powerful desire to rationalize the fuel tests are not valid and/or that slightly contaminated fuel isn’t a significant safety concern because…[see Entergy and NRC agreement that it is not a significant safety concern].  So there is a situation where there is an immediate and significant penalty (shutdown of both units) versus a test result that may or may not be valid or of real safety significance.  The result: deliberate misconduct in burying the test results but also very possibly (I am speculating) the individual and others in the organization can still believe that safety is not impacted.  As actions are consistent with “real” safety significance, it preserves the myth that safety culture is still healthy.


*  As stated in the NRC Enforcement Policy (on page 9, section 2.2.1.d): “Willful violations are of particular concern because the NRC’s regulatory program is based on licensees and their contractors, employees, and agents acting with integrity and communicating with candor. The Commission cannot tolerate willful violations. Therefore, a violation may be considered more significant than the underlying noncompliance if it includes indications of willfulness.” [NRC Information Notice 2013-15]

**  The sequence of events starting in March 2012 in response to RFOST sample (by off-site testing lab) being out of spec: the RFOST is declared inoperable but a supervisor declares that the sample test method was not appropriate, the department procedure is revised to allow on-site testing of a new sample (what was site review process? procedure revision appears to have occurred and become effective in one day), and the test results are now found acceptable.  This allows the RFOST to be declared operable. Without telling anyone, the former Chem Mgr sends a split sample for off-site testing and it comes back over spec.  Why wouldn’t plant management have required a split sample in the first place to verify on-site test?  Two employee concerns are filed, the ML investigation is initiated and the Chemistry Manager resigns.  At the next sampling in mid-April, once again the on-site analysis finds the sample to be within spec but management now requires outside testing in light of the resignation of the Chemistry Manager.  Outside test indicates out-of-spec but an “evaluation” concludes that the in-house results are valid and  RFOST remains “operable”.  Another month goes by and sample is taken in late May.  Sample sent outside, late June results indicate out-of-spec.  This time the RFOST is declared inoperable.  Not clear if late May sample was tested on-site (or why not) and why this time the outside test result is deemed valid.  A final footnote, one of the corrective actions for this event was to discontinue on-site oil analysis but no discussion of why, or why it had been approved in the first place.

***  “the underlying technical findings would have been evaluated as having very low safety significance (i.e. green) under the Reactor Oversight Process (ROP) because the higher fuel oil particulate concentration would not have impacted the ability of the EDGs to fulfill their safety function.” [Ref 4, p. 3]

References

1 - J.A. Ventosa (Entergy) to NRC, Licensee Event Report # 2012-007-00 (Aug. 20, 2012).  ADAMS ML12235A541.

2 - NRC to J. Ventosa, NRC Inspection Report Nos. 05000247/2013011 & 05000286/2013011 and NRC Office of Investigation Reports No. 1-2012-036 (Dec. 18, 2013)  ADAMS ML13354B806.

3 - NRC to D. Wilson (former Chemistry Mgr.), Notice of Violation and Order Prohibiting Involvement in NRC-Licensed Activities (April 29, 2014).  ADAMS ML14118A337.

4 - NRC to J. Ventosa, Notice of Violation (April 29, 2014).  ADAMS ML14118A124.

1 comment:

  1. Bob, you inquire:

    "Something that is hard to square is the NRC assertion that there is a strong link between willful violations and safety culture, and the results of these various assessments at Indian Point by Synergy, the NRC and INPO. So if there is a link, and safety culture assessments don’t reveal its presence, are the assessments valid? Or if the assessments are valid, is there really a link with willful misconduct?"

    It certainly seems that the connections in this instance reflect a good deal of ambiguity concerning motive and what the actual internal culture seascape is at the site. I think your observation that "rectitude" NRC style tends to be driven by the framing of safety significance around the state of design basis configurations points to a standing concept of regulation gulf between the ROP "cornerstones" with their quantitative substance and the "cross-cutting areas" to which the whole Nuclear Safety Culture edifice has become attached. NSC at described in the recent NUREG is about as far from the crispness of the metrics as can be imagined.

    Luckily for NRC (perhaps at least for a while) there are only a few persons with the knowledge and experience to question this scheme and seemingly those have no open channel to NRC management through which to post such questions.

    I would point out that the description of the Chemistry Manager's decision as "willful" doesn't seem at all helpful as a discouragement to similar conduct if there is no way of knowing "willful in relation to what self-serving outcome." Serious misjudgment no doubt, but hardly on a par with deliberate fraudulent conduct - and it begs those questions about why this guy ended up out on such a distant branch all by himself.

    ReplyDelete

Thanks for your comment. We read them all. We would like to display them under their respective posts on our main page but that is not how Blogger works.