Perception and Reality

In our October 18, 2010 post on how perception and reality may factor into safety culture surveys we ended with a question about the limits of the usefulness of surveys without a separate assessment to confirm the actual conditions within the organization.  Specifically, it makes us wonder, can a survey reliably distinguish between the following three situations:

-    an organization with strong safety culture with positive survey perceptions;
-    an organization with compromised safety culture but still reporting positive survey perceptions due to imperfect knowledge or other motivations;
-    an organization with compromised safety culture but still reporting positive survey perceptions due to complacency or normalization of lesser standards.

In our August 23, 2010 post we had raised a similar issue as follows:

“the overwhelming majority of nuclear power plant employees have never experienced a significant incident (we’re excluding ordinary personnel mishaps).  Thus, their work experience is of limited use in helping them assess just how strong their safety culture actually is.”

With what we know today it appears to us that safety culture survey results alone should not be used to reach conclusions about the state of safety culture in the organization or as a predictor of future safety performance.  Even comparisons across plants and the industry seem open to question due to the potential for significant and perhaps unknowable variation of perceptions of those surveyed. 

How would we see surveys contributing to knowledge of the safety culture in an organization?  In general we would say that certain survey questions can provide useful information where the objective is to elicit the perceptions of employees (versus a factual determination) on certain issues.  There is still the impediment that some employees’ perceptions will be colored, e.g., they will discern the “right” answer or will be motivated by other factors to bias their answers. 

What kind of questions might be perception-based?  We would say in areas where the perceptions of the organization are as important or of as much interest as the actual reality.  For example, whether the organization perceives that there is a bias for production goals over safety goals.  The existence of such a perception could have wide ranging impacts on individuals including their willingness to raise concerns or rigorously pursue their causes.  Even if the perceptions derived from the survey are not consistent with reality, it is important to understand that the perception exists and take steps to correct it.  Questions that go to ascertaining trust in management would also be useful as trust is largely a matter of perception.  It is not enough for management to be trustworthy.  Management must also be perceived as trustworthy to realize its benefit.   

The complication is that perception and reality are pulling in different directions. This signifies that although reality is certainly always present, perception is pulling at it and in many instances shaping reality. The impact of this relationship is that if not properly managed, perception will take over and will lessen if not eliminate the other attributes, especially reality.

It would suggest that a useful goal or trait of safety culture is to bring perception as close to reality as possible.  Perceptions that are inflated or unduly negative only distort the dynamics of safety management.  As with most complex systems, perceptions generally exist with some degree of time delay relative to actual reality.  Things improve, but perceptions lag as it takes time for information to flow, attitudes to adjust to new information, and new perceptions take hold.  Using perception data from surveys combined with the forensics of assessments can provide the necessary calibration to bring perception and reality into alignment.

Perception Is/Is Not Reality?

This post will continue our thoughts re the use of safety culture surveys.  The Oxford Dictionary says reality is the state of things as they actually exist, rather than as they may appear or may be thought to be.  Another theory of reality is that there is no objective reality.  Such belief is that there simply and literally is no reality beyond the perceptions, beliefs and attitudes we each have about reality.  In other words, “perception is reality”.  So, when a safety culture surveys is conducted, what reality is it measuring?  Is the purpose of the survey to determine an “objective” reality based on what an informed and knowledgeable person would say?  Or is the purpose simply to catalog the range of perceptions of reality held by those surveyed, whether accurate or not?  Why does it matter?

In our August 11, 2010 post we noted that UK researcher Dr. Kathryn Mearns referred to safety culture surveys as “perception surveys”, since they focus on people’s perceptions of attitudes, values and behaviors.  In a followup post on August 27, 2010 reporting some followup communications with Dr. Mearns we quoted her as follows:

“I see the survey results as a ‘temperature check’ but it needs a more detailed diagnosis to find out what really ails the safety culture.”

If one agrees that surveys are perception-based, it creates something of a dilemma as to which reality is of interest.  If “things as they actually exist” is important, then surveys alone may be of limited value, even misleading, without thorough diagnostic assessments, which is Dr. Mearns' point.  On the other hand, if perception itself is important, then surveys offer a window into that reality.  We think both realities have their place.

We find some empirical support for these ideas from the results of a recent safety culture assessment at Nuclear Fuel Services.*  The report is quite lengthy (over 300 pages) and exhaustive in its detail.  The assessment was done as part of a commitment by the owners of Nuclear Fuel Services (NFS) to the NRC and in response to ongoing safety performance issues at its facilities.  The assessment was performed by an independent team and included a safety culture survey.  It is the survey results that we focus on.

In reporting the results of the survey, the team identified a number of cautions as to the interpretation of NFS workforce perceptions.  The team found that survey numerical ratings were inflated due to the lack of an accurate frame of reference or adequate understanding of a particular cultural attribute.  This conclusion was based on the findings of the overall assessment project.  The team found the workforce perceptions to be “generally (and in some cases significantly) more positive than warranted” (p. 40) or justified by actual performance.

We found these results to be interesting in several respects.  First there is the acknowledgment that surveys simply compile the perceptions of individuals in the organization.  In the NFS case the assessment team concluded that the reported perceptions were inaccurate based on the team’s own detailed analysis of the organization.

Perhaps more interesting was that this inherent subjectivity of perceptions was attributed in this project to the lack of knowledge and frame of reference of the NFS staff, specifically related to standards of excellence associated with commercial nuclear sites.  This resonates with an observation from our August 23 post that “workers who had been through an accident recognized a relatively safer (riskier) environment better than workers who had not.”  In other words, people’s perceptions are influenced by the limits of their own experiences and context.  Makes sense.

The NFS assessment team goes on to indicate that the results of a prior safety culture survey a year earlier also are compromised based on the very time frame in which it was administered.  “It is reasonable to assume that the survey numerical ratings would have been lower if the survey had been administered after the workforce had become aware of the facts associated with the series of operational events that occurred” [prior to the survey].  (p. 41)  We would add there are probably numerous other factors that could easily bias perceptions, e.g., people being sensitive to what the “right answer” is and responding on that basis; complacency; the effect of externalities such as a significant corporate initiative dependent on the performance of the nuclear business; normalization of deviation; job-related incentives, etc.

We think it is very likely that the assessment team was correct in discounting the NFS survey results.  The question is, can any other survey results be relied on absent independent calibration by detailed organizational assessments?  We will take this up in a forthcoming post.

*  "Information to Fulfill Confirmatory Order, Section V, Paragraph 3.e" (Jun 29,2010)  ADAMS Accession Number ML101820096.

Survival of the Safest

One of our goals with SafetyMatters is bringing thought provoking materials to our readers, particularly materials they might not otherwise come across.  This post is an example from the greater business world and the current state of the U.S. economy.  Once again it is based on some interesting research from professors at Yale University* and described in an article in the New York Times.**

“Corporate managers struggling to preserve their companies and protect their core employees have inadvertently contributed to a vicious cycle of rising unemployment and plummeting national morale. If we are to break out of this downward spiral, we first need to understand the problem…professional managers throughout the business world see it as their job to keep work-force morale high. But, paradoxically, the actions they take for their own workplaces often make the overall crisis more severe.”

These issues have been the subject of research by Yale economics professor Truman Bewley.  While his specific focus is on labor markets and how wages respond (or don’t respond) to periods of reduced demand, some of the insights channel directly into the current issues of safety culture at nuclear plants. 

Bewley’s approach was to interview hundreds of corporate managers at length about the driving forces for their actions.  The article goes on to describe how corporate managers respond to recessions by protecting their most important staff, but paradoxically these actions tend to produce unforeseen and often counter-productive results. 

The description of how actions result in unintended consequences is emblematic of the complexity of business systems, where dynamics and interdependencies are not always seen or understood by the managers tasked with achieving results.  Nuclear safety culture exists in such a complex socio-technical system and requires more than just “leadership” to assure long term sustainability. 

This brings us to the first part of Dr. Bewley’s approach - his focus on identifying and understanding the driving forces for managers’ actions.  We see this as precisely the right prescription for improving our understanding of nuclear safety culture dynamics, particularly in cases where safety culture weaknesses have been observed.  A careful and penetrating look at why people don’t act in accordance with safety culture principles would do much to identify the types of factors, such as performance incentives, cost and schedule pressures, etc. that may be at work in an organization.  Driving forces are not necessarily different from root causes - a term more familiar in the nuclear industry - but I tend to prefer it because it explicitly reminds us that safety culture is dynamic, and results from the interaction of many moving parts.  Currently the focus of the industry, and the NRC for that matter, is on safety culture “traits”.  Traits are really the results or manifestations of safety culture and thus build out the picture of what is desired.  But they do not get at what factors actually produce strong safety culture in the first place.

As an example we refer you to a comment we posted on a Nuclear Safety Culture group thread on LinkedIn.com.  Dr. Bill Corcoran initiated a thread asking for proposals of safety culture traits that were at least as important as those in the NRC strawman.  Our response proposed:

 “The compensation structure in the corporation is aligned with its safety priorities and does not create real or perceived conflicts in decisions affecting nuclear safety.” ***

While this was proposed as a “trait” in response to Bill’s request, it is clearly a driving force that will enable and support strong safety culture behaviors and decisions.

* To read about other interesting work at Yale, check out our August 30, 2010 post.

** Robert J. Shiller, "The Survival of the Safest," New York Times (Oct 2, 2010).

*** The link to the thread (including Bob's comment) is here.  This may be difficult for readers who are not LinkedIn members to access.  We are not promoting LinkedIn but the Nuclear Safety Culture group has some interesting commentary.

BP's New Safety Division

It looks like oil company BP believes that creating a new, “global” safety division is part of the answer to their ongoing safety performance issues including most recently the explosion of Deepwater Horizon oil rig in the Gulf of Mexico.  An article in the September 29, 2010 New York Times* quotes BP’s new CEO as stating “safety and risk management [are] our most urgent priority” but does not provide many details of how the initiative will accomplish its goal.  Without seeming to jump to conclusions, it is hard for us to see how a separate safety organization is the answer although BP asserts it will be “powerful”. 

Of more interest was a lesser headline in the article with the following quote from BP’s new CEO:

“Mr. Dudley said he also plans a review of how BP creates incentives for business performance, to find out how it can encourage staff to improve safety and risk management.”

We see this as one of the factors that is a lot closer to the mark for changing behaviors and priorities.  It parallels recent findings by FPL in its nuclear program (see our July 29, 2010 post) and warning flags that we had raised in our July 6 and July 9, 2010 posts regarding trends in U.S. nuclear industry compensation.  Let’s see which speaks the loudest to the organization: CEO pronouncements about safety priority or the large financial incentives that executives can realize by achieving performance goals.  If they are not aligned, the new “division of safety” will simply mean business as usual.

*  The original article is available via the iCyte below.  An updated version is available on the NY Times website.

Games Theory

In the September 15, 2010 New York Times there is an interesting article* about the increasing recognition within school environments that game-base learning has great potential.  We cite this article as further food for thought about our initiatives to bring simulation-based games to training for nuclear safety management.

The benefits of using games as learning spaces is based on the insight that games are systems, and systems thinking is really the curriculum, bringing a nuanced and rich way of looking at real world situations. 

“Games are just one form of learning from experience. They give learners well-designed experiences that they cannot have in the real world (like being an electron or solving the crisis in the Middle East). The future for learning games, in my view, is in games that prepare people to solve problems in the world.” **

“A game….is really just a “designed experience,” in which a participant is motivated to achieve a goal while operating inside a prescribed system of boundaries and rules.” ***  The analogy in nuclear safety management is to have the game participants manage a nuclear operation - with defined budgets and performance goals - in a manner that achieves certain safety culture attributes even as achievement of those attributes comes into conflict with other business needs.  The game context brings an experiential dimension that is far more participatory and immersive than traditional training environments.  In the NuclearSafetySim simulation, the players’ actions and decisions also feedback into the system, impacting other factors such as  organizational trust and the willingness of personnel to identify deviations.  Experiencing the loss of trust in the simulation is likely to be a much more powerful lesson than simply the admonition to “walk the talk” burned into a Powerpoint slide.

* Sara Corbett, "Learning by Playing: Video Games in the Classroom," New York Times (Sep 15, 2010).

** J.P. Gee, "Part I: Answers to Questions About Video Games and Learning," New York Times (Sep 20, 2010).

*** "Learning by Playing," p. 3 of retrieved article.

Missing the Mark at SONGS

In our September 13, 2010 post on the current situation at SONGS we commented on the (in our opinion) undue focus on “leadership” as the sine qua non of safety culture.  Delving into the details of the most recent NRC inspection report* we came across another perplexing organizational response.  This time the issue was deliberate non-compliance.  While deliberate violations do not often get a lot of visibility, we find them potentially useful for illustrating safety culture dynamics.

First the SONGS experience.  Recall that it was a series of deliberate violations by fire watch personnel in the 2001-2006 time frame that started to crystallize safety culture concerns.  To address the problem, SCE committed to providing Corporate Ethics training to managers, supervisors and other specified employees.  The training was completed in 2008.  In 2009 additional ethics training was given to all employees including a SONGS-specific case study.  In addition monitoring programs were enhanced to better detect deliberate violations.

How effective was the training?  As reported in the NRC inspection report, between January 2008 and mid-2010, nine additional instances of deliberate non-compliances were identified.  The inspection report went on to say: “In response to these nine deliberate non-compliances, the licensee performed an Apparent Cause Evaluation….This evaluation identified the need to continue the training and monitoring programs which were developed in response to the Confirmatory Order.”

Did the NRC agree?  “The inspectors determined that this large number of deliberate non-compliances indicated that training on ethics and the disciplinary policy had not been fully effective in eliminating deliberate non-compliances.”  But in a bewildering twist, the NRC goes on to sign off on the issue because actions taken to detect and address deliberate violations have been effective...and the licensee intended to continue taking actions to prevent further instances from occurring. 

Perhaps both SCE and the NRC might have found our recent posts on current academic thinking on the subject of teaching ethics to be of value.  The Yale School of Management’s authors [see our August 30,2010 post] indicated: the concern arises when values are taught in the abstract and reliance is placed on commitments to high ethics without the contextual conflicts that will arise in the real world.  And the MIT article cited in our September 1, 2010 post bluntly reminds us “a decision necessarily involves an implicit or explicit trade-off of values.”  and that companies typically drill employees on values statements and codes of conduct, which have a more “symbolic than instrumental effect”.

We don’t have access to the ethics training provided by SCE but our suspicion is that it probably misses the target in the manner described in these management papers.  In the case of deliberate violations can there be any question that a trade-off of values is occurring?  And if trade-offs are occurring, then one has to ask, Why?  If situational forces are driving behavior, and training was not effective the first time, will repeating the training produce a different result?  

* Letter dated Aug 30, 2010 from R.E. Lantz (NRC) to R.T. Ridenoure (SCE), subject "SAN ONOFRE NUCLEAR GENERATING STATION – NRC FOCUSED BASELINE INSPECTION OF SUBSTANTIVE CROSS-CUTTING ISSUES INSPECTION REPORT 05000361/2010010 and 05000362/2010010," ADAMS Accession Number ML102420696.

Here We Go Again

Back on March 22, 2010 we posted about the challenge of addressing safety culture issues through one-dimensional approaches such as focusing on leadership or reiterating training materials.  We observed that the conventional wisdom that culture is simply leadership driven does not address the underlying complexity of culture dynamics.  San Onofre may be the most recent case in point.  In 2008 new leadership was brought in to the station in response to ongoing culture issues.  Safety culture improved somewhat, at least according to surveys, then it resumed its decline. Last week leadership was changed again following continued pressure by the NRC on cross cutting issues.  Perhaps ironically, one of the more recent actions taken at the station in response to continuing allegations of a “chilled environment” was….leadership training.*

The evolution of events at San Onofre also reinforces another observation we have made about the reliance on safety culture surveys.  As with just about all similar situations, the prescription for weaknesses in “cornerstone” issues by both licensees and the NRC is: conduct a survey.  Looking back in the San Onofre case, the following was determined in its October 2009 survey:

Overall, the Independent Safety Culture Assessment determined that “the safety culture at SONGS is sufficient to support plant operations”.

SCE also reported to the NRC that the survey showed:

Site management is communicating strong and consistent safety messages, including:

-    Safety is the first priority

-    Site personnel are encouraged and expected to identify and report potential safety concerns**

The NRC then conducted additional inspections in early 2010.  “The inspection team determined that the safety culture at SONGS was adequate; however, several areas were identified that needed improvement .... All of the individuals interviewed expressed a willingness to raise safety concerns and were able to provide multiple examples of avenues available, such as their supervisor, writing a notification, other supervisors/managers, or the Nuclear Safety Concerns Program; however, approximately 25% of those interviewed indicated that they perceived that individuals would be retaliated against if they went to the NRC with a safety concern if they were not satisfied with their management’s response.”***

“When asked about the 2009 nuclear safety culture assessment, all of the individuals interviewed remembered having attended a briefing session on the results. However, only the general result of "safety culture was adequate” was recalled by those interviewed.”***

* "SONGS Hit with Stern NRC Rebuke," San Clemente Times (March 2, 2010).

** Slides presented at Nov 5, 2009 SCE-NRC meeting, attached to NRC Meeting Summary dated Nov 20,2009, ADAMS Accession Number ML093240212.

*** Letter dated Mar 2, 2010 from E. Collins (NRC) to R.T. Ridenoure (SCE), subject "Work Environment Issues at San Onofre Nuclear Generating Station—Chilling Effect," ADAMS Accession Number ML100601272.

Making Values Count

In our August 30, 2010 Experiencing Decisions post we highlighted a Wall Street Journal article with some interesting insights into how teaching values must have a strong “experiential” component.  In the real world experiential means that day-to-day decision making is a contact sport, where values collide with business priorities.  In that article reference was made to another paper from the MIT Sloan Management Review, “How to Make Values Count in Everyday Decisions.”*  This work provides a useful and practical resource for follow up reading and implementation.

The Sloan paper authors come right to the point, stating “...a decision necessarily involves an implicit or explicit trade-off of values.” [p.75].  And “decision making is a trade-off between values….[for example] choosing customer safety over short-term financial performance”, referring to the decision by Johnson & Johnson to pull Tylenol off store shelves in 1982.  This is an important perspective but not necessarily one that is very often part of the dialogue about nuclear safety culture.

“The typical approach of many companies is to drill employees on values statements and codes of conduct, but by themselves such sets of principles do not easily permeate everyday decisions.  Recent research suggests that they usually have a more symbolic than instrumental effect.” [p.76]

The authors suggest the use of “decision maps” which is a device to create a picture of the decision process including choices, consequences, outcomes and values.  Note that a distinction is drawn between short term results of a decision (consequences) and longer term impacts (outcomes).  Identifying the longer term consequences of a decision requires thinking through the dynamics of the whole business “system” over multiple time periods.  Think of a pinball machine, perhaps even a pinball machine where you can’t see inside.

One of the problems cited in the paper is that values articulated at the top of the organization can be subverted by the everyday decisions made by staff members, in effect creating a default alternative value structure.  As a practical matter it is the sum of actual decisions that defines the value structure more than the abstract and idealized statements of values.

What is one to do?  Are decision maps the answer?  We’ll leave that to our readers to decide after reading this paper and examining the example provided.  What we can endorse is the authors’ prescription in the last section of the article.  It is based on a belief that leaders should be teachers, and teaching means explaining how decisions are made and how they reflect the values espoused by the leaders.  Basically, decisions need to be explained, used in training and communicated widely within the organization.  We like the idea of identifying a number of recent decisions and examining how the decisions were made, particularly how choices, consequences, and outcomes were linked to values and how values were balanced and traded-off.  This might not always be comfortable but the willingness to use such a process may say more about an organization’s values than any other action.

* “How to Make Values Count in Everyday Decisions”, J.E. Urbany, T.J. Reynolds and J.M. Phillips, MIT Sloan Management Review, Summer 2008, pp. 75-80.