Wednesday, April 22, 2015

More Evidence of Weak Safety Culture in DOE

DNFSB Headquarters
We have posted many times about safety culture (SC) issues in the Department of Energy (DOE) empire.  Many of those issues have been raised by the Defense Nuclear Facilities Safety Board (DNFSB), an overseer of DOE activities.  Following is a recent example based on a DNFSB staff report.*

The Radcalc Imbroglio

Radcalc is a computer program used across the DOE complex (and beyond) to determine the transportation package classification for radioactive materials, including radioactive waste, based on the isotopic content.  Radcalc errors could lead to serious consequences, e.g., exposure to radiation or explosions, in the event of a transportation accident.  DOE classified Radcalc as safety software and assigned it the second highest level of rigor in DOE’s software quality assurance (SQA) procedures.

A DNFSB audit found multiple deficiencies with respect to Radcalc, most prominently DOE’s inability to provide any evidence of federal oversight of Radcalc during the software's lifetime (which dates back to the mid-1990s).  In addition, there was no evidence DOE contractors had any Radcalc-related QA plans or programs, or maintained software configuration management.  Neither DOE nor the contractors effectively used their corrective action program to identify and correct software problems.  DNFSB identified other problems but you get the idea.

DNFSB Analysis

As part of its analysis of problems and causes, the DNFSB identified multiple contributing factors including the following related to organization.  “There is an apparent lack of a systematic, structured, and documented approach to determine which organization within DOE is responsible to perform QA audits of contractor organizations.  During the review, different organizations within DOE stated that they thought another organization was responsible for performing Radcalc contractor QA audits.  DOE procedures do not clearly delineate which organization is responsible for QA/SQA audits and assessments.” (Report, p. 4)

Later, the report says “In addition, this review identified potentially significant systemic [emphasis added] concerns that could affect other safety software. These are: inadequate QA/SQA requirement specification in DOE contracts and the lack of policy identifying the DOE organizations in charge of performing QA assessments to ensure compliance; unqualified and/or inadequate numbers of qualified federal personnel to oversee contract work; . . . and additional instances of inadequate oversight of computer work within DOE (e.g., Radtran).” (Report, p. 5)

Our Perspective

Even without the DNFSB pointing out “systemic” concerns, this report practically shouts the question “What kind of SC would let this happen?”  We are talking about a large group of organizations where a significant, safety-related activity failed to take place and the primary reason (excuse) is “Not my group’s job.”  And no one took on the task to determine whose job it was.  This underlying cultural attitude could be as significant as the highly publicized SC problems at individual DOE facilities, e.g., the Hanford Waste Treatment Plant or the Waste Isolation Pilot Plant.

The DNFSB asked DOE to respond to the report within 90 days.  What will such a report say?  Let’s go out on a limb here and predict the report will call for “improved procedures, training and oversight.”  The probability of anyone facing discipline over this lapse: zero.  The probability of DOE investigating its own and/or contractor cultures for a possible systemic weakness: also zero.  Why?  Because there’s no money in it for DOE or the contractors and the DNFSB doesn’t have the organizational or moral authority to force it to happen.

We’ve always championed the DNFSB as the good guys, trying to do the right thing with few resources.  But the sad reality is they are a largely invisible backroom bureaucracy.  When a refinery catches fire, the Chemical Safety Board is front and center explaining what happened and what they’ll recommend to keep it from happening again.  When was the last time you saw the DNFSB on the news or testifying before Congress?  Their former chairman retired suddenly late last year, with zero fanfare; we think it’s highly likely the SC initiative he championed and attempted to promulgate throughout DOE went out the door with him.


*  J.H. Roberson (DNFSB) to D.M. Klaus (DOE), letter (Mar. 16, 2015) with enclosed Staff Issue Report “Review of Federal Oversight of Software Quality Assurance for Radcalc” (Dec. 17, 2014).  Thanks to Bill Mullins for bringing this document to our attention.

5 comments:

  1. What happens in Las Vegas stays in Las Vegas?

    What happens at DOE is a fair sample of the entire United States Government.
    DVA
    DEA
    Secret Service
    ACA Web Site Rollout
    Nuclear Intrusion by Old Nun

    Can you connect the dots?

    Competence?
    Integrity?
    Compliance?
    Transparency?

    How do we start a grass roots movement to make government operational excellence part of the 2016 presidential conversation?

    ReplyDelete
  2. The larger fraction of the deficit in "operational excellence" is to be found with large corporations and their lobbying staffs who since the time of President Reagan have relentlessly propagated the myth that "Government is the problem".

    The Chemical Safety Board, the NTSB and the GAO are examples of other independent investigative bodies, but it seems (on the record) that forceful investigations and reports do not receive a resonant hearing down at those grass roots of organization to which appeals are endless to create needed reforms.

    The 25 year history of the DNFSB is a master use case for the proposition that since the end of the Cold War, the political structure of this country has become more and more dominated by the view that Shareholder Value, and promptly returned at that, is the only measure of effectiveness that counts.

    In 1774 the Continental Congress assembled for the first time as described here: http://www.history.com/topics/american-revolution/the-continental-congress -

    "From 1774 to 1789, the Continental Congress served as the government of the 13 American colonies and later the United States. The First Continental Congress, which was comprised of delegates from the colonies, met in 1774 in reaction to the Coercive Acts, a series of measures imposed by the British government on the colonies in response to their resistance to new taxes. In 1775, the Second Continental Congress convened after the American Revolutionary War (1775-83) had already begun. In 1776, it took the momentous step of declaring America’s independence from Britain. Five years later, the Congress ratified the first national constitution, the Articles of Confederation, under which the country would be governed until 1789, when it was replaced by the current U.S. Constitution."

    This was not a "grassroots movement" these were the principal stakeholders in the emerging American possibility deciding to act upon an agreed upon dissatisfaction - economic tyranny of George III and the English Parliament. Some were quite prosperous, others were what today we would describe as "entrepreneurs." Each had enough skin in the game to make the hard choice to "take a different course."

    They tried the "more excellence in governance" tack and within two years the Declaration of Independence emerged. Not every signer saw that coming in 1774; but enough of those vested principals signed on to move the Revolution to a decisive conflict stage. Of course it took eight years to "win" and then until 1812 before the English gave up their determination to "clawback" what they lost at Yorktown.

    1774-1812: about the lifetime of the DNFSB; also the duration of the different schools of thought (one hesitates to say action) about Nuclear Safety Culture. It would be a great thing to be past the "times of great trial" with the Global and US Nuclear Energy Enterprise, but the reality is that we are not. Dr. Bill's dots are the virtues institutions might aspire to once they have secured legitimate standing in the larger world. Nuclear energy has failed to gain such standing.

    ReplyDelete
  3. With a plutocracy buying its way in the Congress and hoping to do so with the White House - on the very platform of "enduring government incompetence" (aka "starve the Beast") - where does one imagine that "excellence" of any sort fits?

    The claw-back that counts in my life time is the one of plutocratic commerce (aka Predatory Capitalism) to demolish the Middle Class stakeholdings that emerged after WWII. The rush to big LWR's (i.e. ready or not, one for every utility soup pot) by the NSSS vendors and their AEC compatriots was one of the opening gambits in the claw-back movement. Interestingly, by 1975, the financiers had already concluded that building NPP was a bad risk, but here we are 40 years later still milking that first generation of (sterile) cash cows. Amazing!

    Meanwhile, the Nuclear Weapons Labs are proudly displaying their equanimity in coming up with clever ideas to reuse the Iranian separations plant - that can't get waste reliably in the ground at WIPP, but hey the trash was never the Innovator's strong suit anyway.

    Of course if the AEC's who hold the big waste management and national laboratory operating contracts would take some responsibility to connect those dots of Dr. Bill's the WIPP fiasco could have been avoided. But we might ask; who in senior management actually experienced any significant economic consequences from this deficit in attention to QA Plan commitments?

    It's 1774 Redux my friends; the abused stakeholders today are more numerous and varied than 240 years ago, but we face the same systems problem. There is no silver bullet agency or report to be had - the powers that be laugh at such. If there is to be a movement, it will have to come "out of the middle" just as was the case with the first Continental Congress. As Rickover noted, we must "Develop the Capacity to Learn from Experience" - it always was a Call to (hard) Action under the "Concept of Total Responsibility" by "Technically Self-Sufficient" leaders.
    Continuing:

    The one thing the DNFSB has failed to do (much like NRC) is to get engaged with the political component of their mission. We don't lack for knowledge of the challenges to be faced (e.g. eliminating the ALARA Principle and adopting an integrated ALARP mechanism such as in the UK); we definitely lack for leaders to call and come to that first Congress.

    ReplyDelete
  4. Hello Double Bills,

    Thanks for your comments. In the future, you don't have to be so reserved. Please don't hesitate to tell our visitors what you REALLY think.

    ReplyDelete
  5. Grumpy old men and all that...

    ReplyDelete

Thanks for your comment. We read them all. We'd like to display them under their respective posts on our main page but that's not how Blogger works.