Citizen (Dis)satisfaction: An Experimental Equivalence Framing Study

Asmus Leth Olsen
University of Copenhagen, Denmark

Citizen (Dis)satisfaction: An Experimental
Equivalence Framing Study

Abstract: This article introduces the importance of equivalence framing for understanding how satisfaction measures
affect citizens’ evaluation of public services. Does a 90 percent satisfaction rate have a different effect than a logically
equivalent 10 percent dissatisfaction rate? Two experiments were conducted on citizens’ evaluations of hospital services
in a large, nationally representative sample of Danish citizens. Both experiments found that exposing citizens to a
patient dissatisfaction measure led to more negative views of public service than exposing them to a logically equivalent
satisfaction metric. There is some support for part of the shift in evaluations being caused by a negativity bias: dissatisfaction has a larger negative impact than satisfaction has a positive impact. Both professional experience at a hospital
and prior exposure to satisfaction rates reduced the negative response to dissatisfaction rates. The results call for further
study of equivalence framing of performance information.

Asmus Leth Olsen is assistant professor in the Department of Political Science at
the University of Copenhagen. His research
focuses on the effects of performance
information, political and administrative
psychology, behavioral public administration, and the application of experimental
methods in public administration. His

work has appeared in journals such as
Political Behavior, Public Choice, and
Judgment and Decision Making.
E-mail: ajlo@ifs.ku.dk

Practitioner Points
• The valence (positive/negative) of performance information can have substantial effects on citizens’
perception of public services—even if the underlying performance is exactly the same.
• Presenting citizens with a dissatisfaction rate of 10 percent induces a much more negative evaluation of public
services than presenting them with a logically equivalent satisfaction rate of 90 percent.
• Policy makers must carefully consider how minor equivalent changes in the presentation of performance
information can induce large shifts in citizens’ perceptions of public service performance.

C

itizen and user satisfaction surveys have
become a widespread performance information metric across countries, services, and
levels of government (Bouckaert, Van de Walle, and
Kampen 2005; Stipak 1980). Citizen satisfaction is
seen as a key way of overcoming the many difficulties of measuring actual outcomes in the public sector

by applying more subjective user-centered measures
rather than objective quality or output metrics
(Bouckaert and Van de Walle 2003; Folz 1996;
Holzer and Yang 2004). At the same time, there has
been a focus on potential biases in citizen satisfaction
as an indicator of service outcomes (Kravitz 1998;
Stipak 1979; Van de Walle and Van Ryzin 2011;
Van Ryzin 2013; Van Ryzin et al. 2004; Williams
1994). However, there has been little to no focus on
the potential biases that publicly available satisfaction measures induce in the attitudes and behaviors
of citizens and policy makers. This article turns its
focus to how malleable citizens’ evaluations of public
services are when citizens are confronted with satisfaction measures of different valence but with equivalent
information content. That is, how does a positive
versus a negative framing of the exact same level of
satisfaction affect citizens’ perceptions about a service?

The article advances this research agenda along two
lines. First, it introduces the importance of equivalence framing for our understanding of how satisfaction measures affect citizens’ evaluations of public
services (Druckman 2004; Levin, Schneider, and

Gaeth 1998). Specifically, it points out that logically equivalent changes to the valence of satisfaction
measures can have large effects on citizens’ subsequent
evaluations of public services. The trivial substitution of a 90 percent satisfaction rate for a 10 percent
dissatisfaction rate may transform positive associations
and memories into negative ones for the exact same
underlying information, which, in turn, shifts citizens’
evaluations of the data. The article tests this hypothesis by employing two experimental studies using a
large-scale, nationally representative sample of Danish
citizens (n = 3,443). It will highlight just how sensitive the effect of performance measures on citizens’
perceptions of public services is to minor changes in
the descriptive valence.
Second, the article offers an explanation of the
potency of equivalency frames from the basis of
a negativity bias. The negativity bias implies that
“negative events are more salient, potent, dominant

Public Administration Review,
Vol. 75, Iss. 3, pp. 469–478. © 2015 by
The American Society for Public Administration.
DOI: 10.1111/puar.12337.

Citizen (Dis)satisfaction: An Experimental Equivalence Framing Study 469

in combinations, and generally efficacious
information as an attempt to assign numThe negativity bias implies that
than positive events” (Rozin and Royzman
bers to the inputs, outputs, and outcomes of
2001, 297). In the study of performance
public services (Behn 2003; James 2011b;
citizens are asymmetrical in
information, the negativity bias implies that
their responses to good and bad Moynihan 2008). Many such measures can
citizens are asymmetrical in their responses
performance, reacting mostly to be presented with varying valence of the label
to good and bad performance, reacting
describing the underlying numerical attribute.
the latter.
mostly to the latter. The bias has found some
Examples of this would be “death rates” versus
support in observational studies (James and

“survival rates,” “unemployment rates” versus
John 2007; Boyne et al. 2009) and in experimental studies (James
“employment rates,” or, for our focus here, “satisfaction rates” versus
2011a; Olsen 2013a). As James has argued, “More investigation
“dissatisfaction rates.” With the help of numbers, these labels can
of possible difference in magnitude of effect between informabe shifted while holding the numerical value of the performance
tion about good and bad performance is merited” (2011b, 414).
information logically equivalent. For instance, we can choose to
However, the bias is difficult to detect if there is no neutral point
report a 90 percent satisfaction rate or a 10 percent dissatisfaction rate.
of comparison for the asymmetrical effects of “good” versus “bad”
These are two objectively equivalent ways of presenting the exact
performance.
same information while varying the valence of the wording of the
performance measure.
Another challenge is the extent to which “good” and “bad” performance are in fact qualitatively different. Maybe being asymmetrical
However, changing the valence of performance information is
in our response is sometimes warranted if bad performance is inher- important and not trivial because information is often encoded
ently different. The experiments applied here overcome the latter
according to its descriptive valence (Levin and Gaeth 1988;

problem by comparing logically equivalent pieces of negative- and
Quattrone and Tversky 1988). The encoding evokes our associative
positive-valence information. The former challenge is approached
memory by making associations of similar valence more accessible: a
by testing how the framing effect is moderated by individual difpositive-valence description of an attribute leads to positive associaferences in alternative sources of information about performance
tions in our memory, while a negative valence directs our memory
and prior (experimental) exposure to information of opposite
toward negative associations. For example, a dissatisfaction rate
valence (Chingos, Henderson, and West 2012; Druckman 2004;
makes negative associations about a public service more accessible.
Johnsen 2012).
We can think of this as associations with a valence reflecting dissatisfaction moving to the foreground: displeasing experiences, long
In summary, the article adds to the existing body of research in
waiting times, low-quality service, unresponsive employees, and
two major ways: (1) to show how simple variations in equivalent
so on. On the other hand, when we are presented with a satisfacpresentations of performance information affect citizens’ perception rate, associations consistent with being satisfied become more
tion of public services and (2) to test how part of this effect is
accessible. The result is a “valence-consistent shift” in our judgment
driven by a negativity bias in citizens’ responses. In addition,
of the attribute (Druckman 2004; Levin, Schneider, and Gaeth

the more general implications of the study are threefold. First,
1998). That is, positive-valence information will lead to more posithe potency of equivalence frames for citizen satisfaction rates
tive evaluations and negative-valence information to more negative
in particular—and performance information in general—poses
evaluations—even if the underlying numerical information is logifundamental questions about the fragility of these measures.
cally equivalent.
Second, the article highlights the importance of framing research
to our understanding of how performance information can serve
This leads to the first expectation of the study: logically equivalent
an external accountability role. Third, the article informs our
ways of framing citizen satisfaction can shift citizens’ evaluations of
understanding of how reactions to performance information are
public services in either a negative or positive direction. Specifically,
affected by a negativity bias and what this implies for blameexposure to a satisfaction rate should induce more positive evaluavoiding behavior among policy makers. It also has implications
ations of public services, whereas exposure to a dissatisfaction rate
for our understanding of the potential for manipulating perforshould result in more negative evaluations.
mance measures by those responsible for collecting and reporting
them. These implications are further addressed in the concluding
Equivalence Framing Effects and the Negativity Bias
section.

At this point, we have stated the simple expectation that attribute
equivalence frames can cause valence-consistent shifts. Here we
Equivalence Framing and the Valence of Citizen (Dis)
introduce the idea that valence-consistent shifts can be asymmetrisatisfaction
cal. Specifically, the negativity bias implies that negative information
An equivalence framing effect occurs when “two logically equivalent has a stronger impact than positive information of the same magni(but not transparently equivalent) statements of a problem lead
tude. Metareviews of the negativity bias in psychology have found
decision makers to choose different options” (Rabin 1998, 36).
consistent support for this asymmetry across human perception,
With the notion of equivalence framing in mind, we can easily see
memory, decision making, and behavior (Baumeister et al. 2001;
that many performance information metrics can be presented as
Rozin and Royzman 2001). We tend to pay more attention to,
multiple variations of different “objectively equivalent descriptions
and direct more cognitive capacities toward, negative information
of the same problem” (Levin, Schneider, and Gaeth 1998, 150).
than positive. Stronger reactions to negative information are likely
For the study of performance information, equivalence frames
to exist for evolutionary reasons: negative events (illness, combat,
have a very immediate relevance. Most definitions see performance

hunger, etc.) could be life threatening, while positive events usually
470 Public Administration Review • May | June 2015

did not have the same immediacy (Hibbing, Smith, and Alford
2014, 303).

asymmetry of a negativity bias in this case would employ a neutral
reference point in the middle of satisfaction and dissatisfaction.
Here we confront this challenge in two ways.

This idea has also found its way into political psychology, which,
for a relatively long time, has focused on the asymmetrical effects of
“positive” or “negative” information (Lau 1982, 1985). In politics,
the negativity bias means that blame for bad performance is assigned
to a much greater extent than is credit for good performance of a
similar magnitude. Retrospective voting studies have found that
a worsening economy damages the incumbent to a greater extent
than an improving economy helps (Bloom and Price 1975; Kinder
and Kiewiet 1979; Mueller 1973). In recent years, this research has
spread beyond the traditional measures of economic performance

indicators. Boyne et al. (2009) found evidence of a negativity bias in
the effect of municipal performance information on electoral support among English local governments. James and John (2007), also
in an English local government setting, found that voters primarily
punished poor performance and did not reward good performance
to the same extent. Soroka (2006) found that negative economic
performance was covered more intensely in the media than positive
economic performance of a similar magnitude.

The first approach to unlock the dilemma is to consider how
individual differences in alternative sources of information affect
responses to different-valence information. Here we draw on
Johnsen, who speculated whether the negativity bias works differently for “public services where people in general have less direct
experience” (2012, 139). Along the same lines, James has argued
that the negativity bias may depend on how consistent the performance information is with “personal experience or word of mouth”
(2011b, 414). Consistent with this, Chingos, Henderson, and West
(2012) found that mostly citizens with few alternative sources of
information about school performance had the strongest response
to accountability ratings of school performance. Citizens can draw
on multiple informal sources for performance information about
the public sector. These include media reports, personal experience,

advice from family and friends, political debates, or inference from
visible traits of a particular organization, such as its current users,
facilities, staff, or manager. As laid out by James, “Citizens may not
have much of an idea about the overall performance of a local public body only interacting with it on a case-by-case basis for a subset
of services” (2011b, 402).

Recently, there has also been a set of experimental studies focusing
on a negativity bias in citizens’ responses to good and bad performance information. James (2011a) found both experimental and
observational support for a negativity bias.
These alternative sources may alter how
Using a survey, he found that poor prior
performance information is encoded. Direct
If citizens generally respond
performance was punished more than excelexperience can be an important source of
more strongly to negative
lent prior performance was praised. Further,
information about services, which can affect
information, we can expect that how formal performance information is used.
he also found a negativity bias in an experiment on citizens’ service expectations, as
If citizens generally respond more strongly
negatively framed performance
poor performance affected expectations more
to negative information, we can expect that
information will affect mostly
than excellent performance. However, James
negatively framed performance information
citizens with limited prior
(2011b) found no support for a negativity
will affect mostly citizens with limited prior
information.
bias in a laboratory setting in which particiinformation. Therefore, we should expect that
pants were exposed to “good” and “bad” peralternative information sources will diminformance information, although a possible explanation for this may ish the negativity bias of performance information. This would be
be a discrepancy between the information provided and the actual
consistent with the more general finding that individuals with high
performance of the jurisdiction in question. Another recent experi- personal involvement or strongly held attitudes are less suscepment on voters’ prospective performance preferences found support tible to framing effects (Druckman 2011; Levin, Schneider, and
for a negativity bias in relative performance evaluations, as voters
Gaeth 1998).
showed a strong preference for “not falling behind” other countries
while showing very little interest in “getting in front” (Hansen,
The second approach states that the asymmetrical effects of a
Olsen, and Bech 2014). James and Moseley (2014) found reducsatisfaction rate and a dissatisfaction rate can be teased out by
tions in satisfaction for citizens exposed to low performance and no exposing citizens to a sequence of both. The idea is that we can
detectable increase in satisfaction for cases of high performance.
detect asymmetrical responses to positive and negative information by looking at how evaluations change when citizens become
This leads to our second expectation: that the valence-consistent
aware of the equivalency. This can be done by comparing the
shifts will be overly influenced by the negative valence. That is, the
evaluations of citizens exposed to different sequences of conflictdifference we may observe between a citizen satisfaction rate and
ing valence (e.g., negative to positive or positive to negative) with
a citizen dissatisfaction rate is attributable to the larger negative
those of citizens exposed to performance information of the same
impact of the latter and less so to the positive impact of the former.
valence (e.g., positive/positive or negative/negative). Generally,
exposing individuals to mixed types of valence frames should
Negativity Bias or Positivity Bias?
reduce the framing effect as they become aware of the equivalence
A major challenge to the foregoing hypothesis is the lack of a proper (Druckman 2004, 2011). For a negativity bias to be present, we
counterfactual for comparison. If we are able to show that a satisfac- expect that prior exposure to a satisfaction frame should reduce the
tion rate affects citizens’ evaluations of public services differently
valence-consistent shift in subsequent evaluations when exposed to
from a dissatisfaction rate, how can we know which one of the two
a dissatisfaction frame—but this should not be the case the other
frames has a greater impact? Ideally, identifying the underlying
way around.
Citizen (Dis)satisfaction: An Experimental Equivalence Framing Study 471

Design: Two Experimental Studies in a Large
Representative Sample
The expectations formulated earlier will be examined through two
separate experiments conducted in a large, nationally representative sample of Danish citizens. The context for both experiments is
Danish hospital services. The Danish health care system is a cornerstone of the modern Danish welfare state. In the Danish health care
system, 85 percent of the costs are financed through taxes. Health
care services are administered at three different political levels,
namely, the central government, five regions, and 98 municipalities. Here the focus is on hospitals, which are under the political
authority of the five regions. Reporting various measures of patient
satisfaction with hospital care is common practice today across most
developed countries (Kravitz 1998; Mannion, Davies, and Marshall
2005; Pope 2009; Williams 1994). There are already some studies on how performance reporting affects health professionals and
patients (Hibbard, Stockard, and Tusler 2003, 2005).
Study 1: Experiment on Hospital Satisfaction/
Dissatisfaction
The purpose of the first experimental study was twofold: (1) to test
whether valence-consistent shifts in evaluations of hospital services
happened in response to exposing citizens to either satisfaction
rates or dissatisfaction rates, and (2) to test whether the effect was
moderated by citizens’ alternative sources of information about
hospital services in order to identify a negativity bias as an underlying mechanism.
Participants

The study relied on a large, nationally representative sample
recruited through YouGov’s Danish online panel (n = 3,443).
Through the panel, YouGov made contact with 8,204 respondents, which means the response rate was 42 percent. The data were
Table 1 Descriptive Statistics
Variable

Sample

Population

Gender (male)
Age
18–29
30–44
45–59
60–74
Geographic region
Capital area
Zealand
Southern Denmark
Middle Jutland
Northern Jutland

49.8%

49.1%

12.3%
22.1%
29.3%
37.6%

16.6%
27.8%
26.6%
29.3%

24.7%
15.3%
23.9%
23.8%
12.3%

28.3%
14.5%
23.3%
22.4%
11.7%

Note: CAWI (computer-assisted web interviewing) survey in the Danish YouGov
panel.
N = 3,443.

collected during October 15–22, 2012. For participating in the
survey, participants received a number of points, which they could
use in a shop administered by YouGov. The study was restricted
to citizens between the ages of 18 and 74 and was prestratified on
gender, region, age, and political party choice in order to achieve a
near-representative sample of the Danish population. Table 1 shows
descriptive statistics for the sample and for the Danish population
in the specified age range. It shows a sample that, overall, reflects the
general population on these general sociodemographic characteristics. Furthermore, the sample is highly diverse and largely representative in terms of political party choice and education. In summary,
the sample provides strong external validity of the findings across
citizens with very different background characteristics.
Experimental Design and Procedure

The experiment was a between-subjects design, as outlined in
table 2. The premise of the experiment was that respondents were
asked to evaluate a hospital given a single piece of performance
information. The experiment took the following form: All respondents were provided a short factual note about the Danish Health
and Medicines Authority. Following this, they were asked to evaluate the performance of an unnamed hospital. Here the experiment
randomized the performance information provided to respondents
at two levels. Overall, the two levels of treatment constituted an
equivalence framing experiment in which respondents were assigned
logically equivalent pieces of information (Levin, Schneider, and
Gaeth 1998; Tversky and Kahneman 1981). At the first level of
randomization, respondents were randomly assigned to two different conditions. The two conditions differed in how the hospital’s
performance was framed in terms of either patient satisfaction or
patient dissatisfaction. Clearly, the satisfaction frame constituted
the positive-valence framing of performance information, while
the dissatisfaction frame stressed the negative-valence aspects of the
performance information.
In order for the treatments to be logically equivalent, a second
level of random assignment was introduced in which respondents
were randomly assigned various percentages of satisfaction or dissatisfaction. For the two different frames, the numerical content
was drawn from two different uniform distributions. In a uniform
distribution, the values within an interval have an equal probability
of being drawn. In the satisfaction frame, the uniform distribution
ranged from 75.0 percent to 95.0 percent satisfied patients (presented with one decimal). In the dissatisfied frame, this interval was
inverted, providing a range of treatment values from 25.0 percent
to 5.0 percent dissatisfied patients. For instance, some respondents
were given the following treatment: “At the hospital, 10 percent
of the patients are dissatisfied with their treatment.” At the same
time, other respondents were provided with the logical equivalent:

Table 2 Experimental Design in Study 1
Baseline Question (all participants)
Danish Health and Medicines Authority consistently
records how patients experienced their treatment
at Danish hospitals. How do you think the following hospital is doing?

Treatment Frame
A: Satisfaction
N = 1,716
B: Dissatisfaction
N = 1,727

Treatment Wording (randomly assigned)
At the hospital, X% of the patients are
satisfied with their treatment.
At the hospital, X% of the patients are
dissatisfied with their treatment.

Numerical Treatment (randomly assigned)
X ∈ U (75.0, 95.0)
X ∈ U (5.0, 25.0)

Notes: Outline of experiment conducted with YouGov’s Danish online panel (n = 3,443). Participants were randomly assigned to one of two conditions (satisfaction/
dissatisfaction) and randomly assigned a percentage level of satisfaction/dissatisfaction.

472 Public Administration Review • May | June 2015

The assignment of a large range of different values allows for testing
whether the framing of performance is dependent on the numerical
content of the frame. Numerical content has been found to affect
both citizens’ attitudes and the behaviors of policy makers in unexpected ways (Ansolabehere, Meredith, and Snowberg 2013; Olsen
2013b, 2013c). By randomizing the numerical content, the results
cannot be driven by idiosyncratic artifacts of the numerical values
that respondents were given. Therefore, it enhances the robustness
of the findings if they hold for a large range of numerical values. For
the outcome measure, the respondents were asked to provide their
evaluation of the hospital. Their response was given on a 101-point
sliding scale ranging from “very bad” (0) to “very good” (100).
Respondents could not choose to not respond or to provide a “don’t
know” response. Across all treatments, the average response was
55.6 (SD = 24.7) and the median response 57.
It is important to note that the experiment did not allow us to
directly assess the question of a negativity bias. As outlined in the
theory section, if we found a difference in hospital evaluations
between the two frames, we would not be able to directly attribute
this difference solely to one of them specifically. Ideally, we would
like to have some neutral point of satisfaction or dissatisfaction
to which we could compare the two frames in order to measure
which one of them has an asymmetrical impact on evaluations.
Unfortunately, no such neutral category is easy to come up with for
satisfaction measures.
Instead, as argued in the theory section, we used alternative
information sources as a moderator that could shed some light on
which frame was driving a potential effect. In terms of alternative
information sources, we relied on two different indicators. The first
was a dummy variable indicating whether the respondent had been
to a hospital within the last year (17.8 percent). This indicated a
personal experience with hospital services, which can be seen as
an important alternative source of performance information. The
second indicator was a dummy variable indicating whether the
respondent either currently worked or previously had worked at a
hospital (10.6 percent). This indicator captured any type of professional work-related experience in a hospital setting. In summary,
both indicators captured the extent to which respondents had
alternative sources of hospital performance other than the performance information they were provided in the experiment. We also
included some controls in some specifications in order to make sure
that these two indicators did not simply capture some other factor
correlated with alternative information sources and evaluations.
The controls included age, gender, and region. We also included a
dummy for respondents with private sector employment (32.6 percent) to capture differences between them and those in the public
sector or out of work. Finally, a dummy captured respondents with
an intent to vote for one of the four parties that were either in government or that supported the government at the time of the study
(36.5 percent).

0.03
Satisfied frame

0.02
Dissatisfied frame

Density

“At the hospital, 90 percent of the patients are satisfied with their
treatment.” That is, for each set of respondents asked to evaluate a
given rate of hospital satisfaction, the data contain a similar set of
respondents evaluating logically identical metrics of hospital dissatisfaction. The random assignment of percentages provided respondents with around 100 different treatment values under each frame.

0.01

0
0

25

50

75

100

Citizens’ Evaluation of Hospital Performance

Notes: The x-axis represents the dependent variable of citizens’ evaluations of
hospital performance. Higher scores indicate a better evaluation.
N = 3,443.

Figure 1 Density Plot of the Distribution of Responses under
the Two Frames
Empirical Results

The distributions of responses under the satisfied and dissatisfied
frames are reported in the density plot in figure 1. Under the satisfied frame, citizens gave the unnamed hospital an average score of
65.9 (SD = 20.7). However, for the dissatisfied frame, the average
score was only 45.4 (SD = 24.1). On average, citizens evaluated
hospitals under the satisfied frame as significantly better, with an
average difference of 20.5 points (p < .01). The effect is substantially
similar or even larger if medians or trimmed means are calculated.1
The effect is substantial: the mean difference corresponds to about
a one-standard-deviation change in the dependent variable. This
strongly indicates a valence-consistent shift in evaluations induced
by framing performance information as either satisfaction or
dissatisfaction.
In figure 2, the framing effect is shown across the numerical treatments. Table 3 reports further tests of the framing effect. From the
figure and table, we can see that citizens responded positively to both
higher satisfaction and lower dissatisfaction rates. On average, evaluation improved significantly, by around 0.9 point for each percentage point improvement in satisfaction/dissatisfaction (see model B).
We can now compare the frame treatment and numerical treatment
in magnitude. Doing so tells us that an approximately 22-point
improvement in the percentage of satisfied/dissatisfied has the same
effect as changing the overall framing from dissatisfaction to satisfaction. This strongly indicates the potency of the valence framing effect.
We can compare differences in responses to changes in satisfaction and dissatisfaction rates. In table 3, model C, this is done by
interacting the treatment frame with the numerical frame. The
interaction term is positive, which indicates a stronger response to
numerical changes in the dissatisfaction frame. However, the effect
is not significant. This implies that a difference in the numerical
magnitude of positive- and negative-valence information does not
assert any influence. The only thing that matters is the valence of
the information, not its magnitude.
The next step was to understand the underlying mechanisms
for the large difference in evaluations for the satisfaction and

Citizen (Dis)satisfaction: An Experimental Equivalence Framing Study 473

Treatment B: Pct. Dissatisfied with Hospital Services

Citizens’ Evaluation of Hospital Performance

25%

20%

15%

10%

5%

100

Satisfied

75

Dissatisfied

50

25

0
75%

80%

85%

90%

95%

Treatment A: Pct. Satisfied with Hospital Services
Notes: Ordinary least squares estimated slopes for the satisfied (black line and dots) and dissatisfied (gray line and dots) frames. The lower horizontal axis shows the
treatment percent received under the satisfied frame. The upper horizontal axis shows the treatment percent assigned under the dissatisfied frame.
N = 3,443.

Figure 2 Effect of Satisfied Percentage and the Dissatisfied Percentage
Table 3 Ordinary Least Squares Results from Experiment in Study 1

Treatment frame (1 = dissatisfaction)

Model A

Model B

Model C

Model D

Model E

Model F

−20.51**
(.77)

−20.68**
(.74)
.94**
(.06)

−30.25**
(10.89)
0.88**
(.10)
.11
(.13)

−21.46**
(.78)
.93**
(.06)

−21.27**
(.82)
.94**
(.06)

−22.06**
(.86)
.94**
(.06)

Treatment percent
Frame * Treatment percent
Hospital work experience

−1.68
(1.72)

Patient experience

.97
(1.37)

Work experience * Treatment frame

7.39**
(2.41)

Patient * Treatment frame
Intercept
Adjusted R2
F-statistic
N

65.87**
(.54)
.17
717.2
3,443

−13.73**
(5.45)
.22
488.6
3,443

−9.00
(7.66)
.22
325.9
3,443

−13.26**
(5.44)
.22
248.1
3,443

3.32
(1.94)
−14.35**
(5.45)
.22
247.2
3,443

−1.81
(1.74)
0.79
(1.38)
7.31**
(2.41)
3.16
(1.94)
−14.28**
(5.74)
.24
59.2
3,413

Notes: Ordinary least squares estimates with standard errors in parentheses. Significance levels denote *p < .05 and **p < .01. Model F includes the following control
variables: gender dummy, age, government party supporter dummy, dummy for private sector employment, five dummies for educational level, and five regional
dummies. Sample size in model F is reduced due to missing values on the education measure.

dissatisfaction frame. Here we studied heterogeneous responses for
respondents with alternative information sources about hospital
services. Table 3, models D–F report these results. In figure 3,
coefficients with 95 percent confidence intervals are shown for (1)
whether the respondent had been hospitalized within the last year
or (2) whether the respondent currently worked or had worked at
a hospital. The mean differences are based on the coefficients in
table 1, model F.
For respondents with hospital work, the interaction effect is
positive and significant at 7.4 points. That is, in the dissatisfied
frame, respondents with current or prior work experience at a
hospital gave higher evaluations than those with no experience.
474 Public Administration Review • May | June 2015

Interestingly, the main effect of work experience is negative and
insignificant, which suggests that there is no difference under the
satisfied frame. In other words, alternative sources of information matter for the effect of negative-valence information but do
not alter the effect of positive-valence information. This result
is substantially the same if we add a set of control variables (cf.
model F). The finding is also substantially the same if we interact all control variables with the treatment frame in the same
model.2
In figure 3, we can directly compare the effects for the two frames.
The mean difference between the two frames amounts to a statistically significant effect of 9.1 points (p < .01) between those with and

Mean Difference in Evaluation

12
10
8
6
4
2
0
−2
−4

Satisfaction
Dissatisfaction
Hospital work experience vs. none

Satisfaction
Dissatisfaction
Patient experience vs. none

Note: Mean differences in hospital evaluations with 95% confidence intervals.
N = 3,443.

Figure 3 Alternative Information Sources and Frame Effect

without hospital work experience. The other indicator of alternative
sources of information compared respondents that had been to a
hospital in the past year with everybody else. The interaction effect
is also positive but not significant. We do not observe the same
effect for this group. The mean difference-in-difference between the
two frames is 2.4 (p = .31).

Table 4 Experimental Design in Study 2
Information Source

In summary, the distance between evaluations under the positive
and negative frame is reduced by having alternative sources of information, and this effect is driven solely by a less negative response
under the dissatisfied frame. Importantly, if the results were driven
by a generally higher degree of satisfaction with hospital services
among these groups, then we would expect them to respond more
positively under both frames. However, this is not what the findings
indicate. The fact that alternative information sources affect only
evaluations under the dissatisfaction frame indicates that alternative information sources are likely to play a role in diminishing the
negativity bias.3
Study 2: Experiment with Multiple Performance Cues
The second study served two purposes: (1) we were able to assess
whether the results of study 1 would hold up if the performance
information on satisfaction and dissatisfaction was presented in a
context of alternative sources of information, and (2) the sequence
of valence stimuli between study 1 and study 2 allowed us to obtain
a better understanding of the underlying mechanism. Specifically,
the experiment in study 2 occurred after study 1, which allowed
us to analyze the effects in study 2 conditional on the treatment
received in study 1. Study 2 relied on the exact same participants as
study 1.
Experimental Design and Procedure

Study 2 had a between-subjects design. All respondents were confronted with the following hypothetical scenario: “Imagine that you
have become sick and that you therefore need to have a nonemergency operation made at a hospital. About the hospital, you know

Experimental Variation

Media coverage of the hospital is:
Positive/negative
The proportion of former patients who were 90% satisfied/50% satisfied/
satisfied/dissatisfied with their treatment:
10% dissatisfied/50% dissatisfied
Your neighbors view of the hospital is:
Positive/negative
Notes: Respondents were assigned to one of 16 possible combinations of values
for the three stimuli variables. The experiment was conducted with YouGov’s
Danish online panel (n = 3,443).

that . . .” Below the introduction was a box with three types of
information about the hospital that varied in terms of the valence
of the information provided. The three sources of information were
shown simultaneously. Information sources and the experimental
variation are reported in table 4. The main source of information
echoed that of study 1: participants were informed about satisfaction/dissatisfaction performance measures for former patients of the
hospital.
As in study 1, the percentage of satisfied/dissatisfied was varied, here with only four equivalent quantities. The two alternative sources of information were media coverage and neighbors’
opinions. These could vary only in terms of being either positive
or negative. Importantly, these were not equivalents of different
valence but stated actual substantive differences. Combining treatment variations across the treatment types of information sources
generated 16 different treatment combinations (2 x 2 x 4 full factorial design). This gave us approximately 215 respondents in each
condition.
Below the information box, the respondents were asked to report
how likely they were to undergo the operation at the aforementioned hospital. The response was provided on a scale similar to that
used in study 1, a 101-point sliding scale ranging from “not at all
likely” (0) to “very likely” (100). Across all treatments, the average
response was 49.0 (SD = 26.9), with a median of 51.

Citizen (Dis)satisfaction: An Experimental Equivalence Framing Study 475

Table 5 Ordinary Least Squares Results from Study 2
Satisfaction/dissatisfaction (ref. = 10% dissatisfied)

(ref.)

90% satisfied

15.62**
(1.04)
−11.67**
(1.03)
−19.24**
(1.04)
12.97**
(0.73)
13.28**
(0.73)
0.36
386.3
3443

50% satisfied
50% dissatisfied
Media coverage (1 = positive)
Neighbor opinion (1 = positive)
Adjusted R2
F-statistic
N

Notes: Ordinary least squares estimates with standard errors in parentheses.
Dependent variable: Likelihood of undergoing the operation at that hospital,
“not at all likely” (0) to “very likely” (100). Significance levels denote *p < .05
and **p < .01.
Table 6 Mean Responses in Study 2 Conditional on Treatment in Study 1
Study 2
Study 1
Satisfaction frame

Dissatisfaction frame

Satisfaction Frame

Dissatisfaction Frame

54.7
[52.9–56.5]
(n = 868)
55.0
[53.2–56.8]
(n = 855)

45.6
[43.9–47.3]
(n = 848)
40.9
[39.2–42.6]
(n = 872)

Notes: Dependent variable: Likelihood of undergoing the operation at that
hospital, “not at all likely” (0) to “very likely” (100). Mean responses with 95%
confidence intervals in brackets and sample sizes in parentheses.

Empirical Results

However, if respondents received a dissatisfaction frame in study
2, their response depended in part on their treatment frame in
study 1. Specifically, those who received a satisfaction frame in
study 1 evaluated the dissatisfaction frame significantly better, by
4.7 points, in study 2 than those who had dissatisfaction frames in
both studies (p < .01). This did not, however, work in reverse. Those
first exposed to a dissatisfaction frame did not rate the satisfaction
frame any differently from those first exposed to a satisfaction frame
(p = .83). The results are substantially the same or stronger if median
values are calculated.4 In other words, exposure to positive-valence
information diminished the (negative) response to subsequent
negative-valence information—but not the other way around. One
interpretation is that respondents first exposed to the satisfaction
frame became aware of the equivalence and therefore viewed the dissatisfaction frame more mildly (Druckman 2004). This “correction”
then did not happen the other way around, which could indicate
that the satisfaction frame was viewed as the baseline or “regular”
metric. This interpretation supports the negativity bias in the sense
that exposure to both forms leads to a calibration of judgment concerning the negative information and not the positive information.
Conclusion
Citizens’ immediate responses to satisfaction metrics can be highly
contingent on an arbitrary choice of a positive or negative label
description. This is not a trivial fact in a world in which citizens’ satisfaction rates play a key role in informing the public about public
sector performance. This article has shown how equivalence framing
of citizen satisfaction measures can assert a huge impact on how citizens evaluate public services. Two experiments in a large, nationally
representative sample of Danish citizens highlighted the following
major findings on the potency of equivalence framing.

The main results are reported in table 5. First, we can note that
respondents valued positive media coverage and neighbors’ opinions
at the same order of magnitude. Changing
Study 1 found that framing hospital performedia coverage or neighbors’ opinions from
mance in terms of dissatisfaction instead of
Citizens’ immediate responses
negative to positive caused about a 13-point
satisfaction had a substantial negative impact
to satisfaction metrics can be
increase in the likelihood of choosing the hoson citizens’ evaluations of hospital services.
highly contingent on an arbipital for a nonemergency operation. Turning to
The effect was constant along a large intertrary choice of a positive or
the satisfaction/dissatisfaction frames, we find a
val of different numerical treatments for the
negative label description.
similarly large and significant effect as in study
percentage of patients being either satisfied
1. Shifting from a hospital with 90 percent
or dissatisfied. Study 2 replicated the frambeing satisfied to one with 10 percent being dising effect in an experiment that also offered
satisfied changes the likelihood of participants choosing that hospital
alternative sources of information with both negative and positive
by around 15 percentage points. For the lower level of 50 percent
valence. Importantly, the change in effect attributable to the equivasatisfaction/dissatisfaction, the difference is around 7.5 points ( p <
lence frame was comparable to valence changes for media coverage
.01). This highlights that study 1 may have induced an awareness of
and neighbor opinions, which both were nonequivalence frames.
the equivalence between satisfaction and dissatisfaction that reduced
the overall framing effect in study 2 (Druckman 2011).
Theoretically, we also aimed to interpret part of this effect to reflect
a negativity bias. That is, differences in responses to satisfaction and
This being said, it speaks to the power of the equivalency frame of
dissatisfaction measures are driven by a disproportionate impact
satisfaction/dissatisfaction that the effects are similar for those found of negative-valence performance information—in this instance, a
for logically equivalent changes in valence compared with logically dis- dissatisfaction rate. Some additional findings support this intersimilar changes in valence (e.g., positive versus negative media cover- pretation. First, respondents with current or prior work experiage). Finally, we combined the treatments in study 1 and study 2
ence at a hospital were less affected by the dissatisfaction frame
in order to test whether switching frame types changes the effect of
but responded like everyone else to the satisfaction frame. We take
exposure to satisfaction and dissatisfaction percentages. The results
this as an indication that alternative sources of information about
are reported in table 6.
performance diminish the negativity bias. Second, by combining
treatments across studies, we were able to show that those exposed
For those who received the satisfaction frame in study 2, the
to satisfaction in study 1 and dissatisfaction in study 2 were less
frame received in study 1 did not affect their mean evaluations.
negative in their assessment than those exposed to other treatment
476 Public Administration Review • May | June 2015

combinations. We again take this to support a negativity bias
interpretation of the main results because it highlights that prior
exposure to positive-valence information weakens the response to
negative-valence information—but not the other way around.

either negative or positive response among citizens and, from there
on, the rest of the political-administrative system.

A minor cautionary note on the limits of the findings should be
made as a result of the hypothetical, low-incentive setting of the
experimental design. Performance information is often presented in
a more data-rich context of news reports, government Web pages,
or official publications. This being said, the analysis points to the
importance of equivalence framing and, in part, the negativity bias
for citizens’ response to satisfaction measures, and the findings were
very substantial and identified in a large representative sample. In
addition, we could argue that the identified framing effect and negativity bias could be even larger in a real-world setting, where citizens
have to make sense of a greater number of cues about performance.
This could leave them even more vulnerable to the effects found
across the two studies.
The study offers three broader implications for our understanding
of citizens’ satisfaction measures and performance information more
generally: The first implication relates to our use and understanding
of the effects of citizen satisfaction measures. Today, both governments and researchers are focused on citizen satisfaction. We now
know that changing the discussion to dissatisfaction has the potential
to fundamentally affect citizens’ evaluations. This should remind
both practitioners and researchers about the fragile nature of the
performance measures we apply and how they affect the greater
public. Small, arbitrary changes in the reporting of performance
information have the potential to significantly shift the perception
of public services among those receiving the information in both
negative and positive directions.
The second implication is a need to integrate our study of the
effects of performance information into the broader framing
research agenda (James 2011b; Moynihan 2008). There are endless ways to present performance information to citizens, and
shifting the valence of a metric’s descriptive label is just a very
basic example. Other examples include logically equivalent ways
to present performance information on different scales and with
the aid of various reference points (Olsen 2013a). This opens up
a broader research agenda on the informational cues that affect
citizens’ attitudinal and behavioral response to performance
information.
The final implications of an equivalence framing effect and the
underlying mechanism of a negativity bias point at downstream
effects for the broader political-administrative system. The importance of performance information valence adds another example
to the list of ways in which individuals’ perceptions of blame are
potentially malleable (Marvel 2014). Maybe the apparent widespread use of positive-valence performance information is indicative
of policy makers’ fear of backlash. Some have linked the negativity
bias to the general blame avoidance observed at the political level
(Carpenter and Krause 2012; Mortensen 2013; Weaver 1986). As
Hood notes, a “key test of political power can be said to be the
ability to overcome or counteract negativity bias” (2007, 199). It
is therefore highly important that future studies aim to understand
how the informational valence of satisfaction measures can evoke an

Acknowledgments
The research was made possible by Grant No. 0602-02577B from the
Danish Council for Independent Research. An earlier version of the
paper was presented at the 11th Public Management Research Conference, Madison, Wisconsin, June 20–23, 2013. I would like to thank
the conference participants for valuable comments. The manuscript
has also benefited greatly from the feedback from three anonymous
reviewers and the editors of PAR. Any remaining errors are my own.
Notes
1.

The median responses are 72 for the satisfaction frame and 45 for the dissatisfaction frame. This gives a median difference of 27 points. If we