DNFSB Headquarters |
The Radcalc Imbroglio
Radcalc is a computer program used across the DOE complex (and beyond) to determine the transportation package classification for radioactive materials, including radioactive waste, based on the isotopic content. Radcalc errors could lead to serious consequences, e.g., exposure to radiation or explosions, in the event of a transportation accident. DOE classified Radcalc as safety software and assigned it the second highest level of rigor in DOE’s software quality assurance (SQA) procedures.
A DNFSB audit found multiple deficiencies with respect to Radcalc, most prominently DOE’s inability to provide any evidence of federal oversight of Radcalc during the software's lifetime (which dates back to the mid-1990s). In addition, there was no evidence DOE contractors had any Radcalc-related QA plans or programs, or maintained software configuration management. Neither DOE nor the contractors effectively used their corrective action program to identify and correct software problems. DNFSB identified other problems but you get the idea.
DNFSB Analysis
As part of its analysis of problems and causes, the DNFSB identified multiple contributing factors including the following related to organization. “There is an apparent lack of a systematic, structured, and documented approach to determine which organization within DOE is responsible to perform QA audits of contractor organizations. During the review, different organizations within DOE stated that they thought another organization was responsible for performing Radcalc contractor QA audits. DOE procedures do not clearly delineate which organization is responsible for QA/SQA audits and assessments.” (Report, p. 4)
Later, the report says “In addition, this review identified potentially significant systemic [emphasis added] concerns that could affect other safety software. These are: inadequate QA/SQA requirement specification in DOE contracts and the lack of policy identifying the DOE organizations in charge of performing QA assessments to ensure compliance; unqualified and/or inadequate numbers of qualified federal personnel to oversee contract work; . . . and additional instances of inadequate oversight of computer work within DOE (e.g., Radtran).” (Report, p. 5)
Our Perspective
Even without the DNFSB pointing out “systemic” concerns, this report practically shouts the question “What kind of SC would let this happen?” We are talking about a large group of organizations where a significant, safety-related activity failed to take place and the primary reason (excuse) is “Not my group’s job.” And no one took on the task to determine whose job it was. This underlying cultural attitude could be as significant as the highly publicized SC problems at individual DOE facilities, e.g., the Hanford Waste Treatment Plant or the Waste Isolation Pilot Plant.
The DNFSB asked DOE to respond to the report within 90 days. What will such a report say? Let’s go out on a limb here and predict the report will call for “improved procedures, training and oversight.” The probability of anyone facing discipline over this lapse: zero. The probability of DOE investigating its own and/or contractor cultures for a possible systemic weakness: also zero. Why? Because there’s no money in it for DOE or the contractors and the DNFSB doesn’t have the organizational or moral authority to force it to happen.
We’ve always championed the DNFSB as the good guys, trying to do the right thing with few resources. But the sad reality is they are a largely invisible backroom bureaucracy. When a refinery catches fire, the Chemical Safety Board is front and center explaining what happened and what they’ll recommend to keep it from happening again. When was the last time you saw the DNFSB on the news or testifying before Congress? Their former chairman retired suddenly late last year, with zero fanfare; we think it’s highly likely the SC initiative he championed and attempted to promulgate throughout DOE went out the door with him.
* J.H. Roberson (DNFSB) to D.M. Klaus (DOE), letter (Mar. 16, 2015) with enclosed Staff Issue Report “Review of Federal Oversight of Software Quality Assurance for Radcalc” (Dec. 17, 2014). Thanks to Bill Mullins for bringing this document to our attention.