Saturday, April 6, 2013

2012 NRC Safety Culture Survey Results

ADAMS ML13087A326
Originally published 4-4-13.  This version updated with data from the associated staff briefing slide presentation.

The 2012 NRC Safety Culture and Climate Survey results are available in an Inspector General report* and a consultants' slide briefing.**  The top-level findings are (1) the 2012 results are not as favorable as the previous 2009 survey results and (2) the NRC compares favorably with national norms but lags when compared to a group of high performing companies (with strong financial results and high employee survey scores).  Let's look at some of the details.  All page references are to the report except where noted otherwise. 

The survey's 132 items were aimed at evaluating employee perceptions in 20 categories.  Many of these categories primarily addressed personnel practices—communication, supervision, diversity, training, development and the like.  However, it should come as no surprise to our regular readers that the categories of interest to us address, at least in part, the key business processes of decision making, priority setting and conflict resolution, i.e., areas where the goal of safety often competes with other goals.

Four categories appear to satisfy our criteria:

DPO/Non-Concurrence (DPO): “. . . employee awareness and perceived effectiveness of the Differing Professional Opinions program and the Non-concurrence process.”  This is one type of conflict resolution.  This category had the lowest number of favorable responses (although still over 50%) in the survey. (p. 13)  Three specific DPO items were among those that showed the most slippage, i.e., had fewer favorable responses, in 2012 compared to 2009. (Slides, p. 16)  Region IV had significantly*** fewer favorable 2012 scores on DPO compared to 2009. (p. 26) 

The consultants' cover letter identified this as an area for NRC management attention, saying the agency was “Losing significant ground on negative reactions when raising views different from senior management, supervisor, and peers.” 

NRC Mission and Strategic Plan: “. . . whether employees believe management decisions are consistent with the mission and strategic plan. . . .”    Compared with the high performing companies, the NRC scored 1 point lower on NRC Mission and Strategic Plan. (p. 17)  The Office of New Reactors and Region IV had significantly fewer favorable 2012 scores on NRC Mission and Strategic Plan compared to 2009. (pp. 25-26) 

The NRC Mission and Strategic Plan was identified as one of three key drivers of employee Engagement, also a survey category**** but treated as a dependent variable in a supporting multiple regression analysis.  In responding to specific questions, employees said they believed they were “sufficiently informed about NRC's performance of its mission” and that “management decisions are consistent with the mission” but both items scored significantly lower than in 2009 and compared to the high performing companies. (Slides, p. 33)

Quality focus: “. . . employee views on . . . the sacrifice of quality work due to the need to meet a deadline or the need to satisfy a personal or political agenda.”  This category had the third lowest number of favorable responses in the survey. (p. 13)  This category was also mentioned in the consultants' cover letter: “Reinforcing a key point raised in the focus groups [but one that did not stand out in the survey results], there is a clear opportunity to impact the perception that people sacrifice quality in order to meet metrics.”

Senior Management: “. . . confidence in management’s decisions.”  Compared with the high performing companies, the NRC scored 7 points lower on Senior Management, in a 3-way tie for second lowest. (p. 17)  This result may have been affected by this item: Only 41 percent of the respondents “. . . feel significant actions have been taken as a result of the previous Safety Culture and Climate survey.” (Slides, p. 23)  This issue was included in the list of conclusions to the consultants' report.  On the other hand, at least 75% favorable responses were recorded for senior management providing a clear sense of direction and employee confidence in senior management decisions. (Slides, p. 23)  That may look good but both items scored significantly lower than in 2009 and compared to the high performing companies.

The Office of New Reactors and Region IV had significantly less favorable 2012 scores on Senior Management compared to 2009. (pp. 25-26)  Region IV also had a significantly less favorable 2012 score on Senior Management than the overall NRC score. (p. 23)

Our Perspective

The report consists of mostly charts and graphs, with a lot of superficial data slicing and dicing and some authoritative-sounding conclusions.  The slide presentation shows additional data to illustrate some problem areas.  Both documents reinforce our belief in the limited usefulness of surveys and the problems associated with over-reliance on outside experts.  My “analysis” above is obviously limited but it's difficult to dig deeply because only a few of the 132 specific items are detailed in the report and slides. 

But the available data suggest that raising views inconsistent with the party line can lead to negative reactions.  NRC employees have some confidence the agency makes decisions consistent with its mission but less confidence in their senior management to take action on survey results.   

NRC senior management has a much more favorable view of the agency's situation than the overall organization.  Senior managers' survey responses were significantly more favorable than the overall NRC response in ALL 20 categories and an average of 18 percent more favorable in the 4 categories included in this post. (Slides, p. 37)  This suggests a possible disconnect between the bosses and everyone else.

And speaking of disconnections, it appears neither the group responsible for the Nuclear Renaissance nor Region IV is fully on the same page as the rest of the agency.

Finally, the documents' omission of safety as a goal or priority is notable.  “Nuclear safety” as a goal is only mentioned in the definition of SC.  Safety is mentioned as “safety concepts” in the Training category and the “NRC’s commitment to public safety” in the Continuous Improvement Commitment category.  One might expect safety to be more front and center in the SC survey. 

*  NRC Office of the Inspector General, “2012 NRC Safety Culture and Climate Survey,” OIG-13-A-15 (March 28, 2013)  ADAMS ML13087A326.  Although this was mostly a survey, the consultants (Towers Watson) did conduct some individual interviews and focus groups to help shape the survey content. Interestingly, the definition of safety culture used in the 2012 survey was not the same as the definition in the current NRC policy statement.  Instead, an earlier definition was used to permit comparisons between current survey results and prior years.

**  Towers Watson, “Nuclear Regulatory Commission 2012 Safety Culture and Climate Survey Briefing for NRC Staff” (Nov. 8, 2012).

***  “Significant” means statistically significant.

****  Engagement “Probes employees’ willingness to recommend the NRC as a good place to work, whether they feel they are a part of the agency, their pride in working for the NRC and their belief in NRC goals, objectives, and values.” (p. 10)

1 comment:

  1. Goodhart's Law initially stated by Marilyn Strathern: "When a measure becomes a target, it ceases to be a good measure." We see this at work in Colorado under the CSAP testing in K-12 education. The purpose of CSAP was to measure teacher effectiveness. Instead we get teachers spending significant time preparing children for the test.

    My paraphrase is: When you want it bad you get it bad. When you want it worse you get it worse.

    The 2009 measures became targets; at least to some extent. The 2012 survey measured something but not necessarily what it was intended to measure. Its the old Soviet nail factory dilemma. If you seek a production goal of number of nails, then you get a lot of small nails. If you set the production goal in terms of weight, then you get a small number of very big nails.

    The direct approach to improving the new targets did not result in improving the measures. When you want it bad you get it bad. The 2012 measures indicate that the direct approach to achieving a positive safety culture does not work. Apparently, NRC spent three years making that determination for us, but without reaching that conclusion.

    So, isn't the next step to try to figure out what does work? The alternative seems to be to try what the did for three more years. If they try three more years of what didn't work, then we have solid data that the NRC is not a learning organization.


Thanks for your comment. We read them all. We'd like to display them under their respective posts on our main page but that's not how Blogger works.