1
When Bigger is Better (and When it is Not): Implicit Bias in Numeric Judgments
ELLIE J. KYUNG
MANOJ THOMAS
ARADHNA KRISHNA*
Forthcoming in the Journal of Consumer Research
2
* Ellie J. Kyung ([email protected]) is Associate Professor of Business
Administration at the Tuck School of Business, Dartmouth College, 100 Tuck Hall, Hanover,
NH 03755. Manoj Thomas ([email protected]) is Associate Professor of Marketing at
the Johnson Graduate School of Management, Cornell University, 353 Sage Hall, Ithaca, NY
14853. Aradhna Krishna ([email protected]) is the Dwight F. Benton Professor of Marketing
at the Ross School of Business, University of Michigan, 701 Tappan Street, Ann Arbor, MI
48109. The authors would like to thank Linda Hagen for her help with German translation, Geoff
Gunning, Kimberlee Hayward, and Matthew Paronto for their feedback and editorial assistance,
Melissa Li for her research support, and Anne Fries for bringing the Stiftung Warentest rating
system to their attention. They would also like to thank Mrinal Thomas, Maxine Park, and
Roxane Park for their assistance in determining the title of this paper through a unanimous vote.
Finally, the authors would also like to thank the reviewers, associate editor, and editor for their
invaluable comments that greatly enhanced the paper. Supplemental materials, including
experiment stimuli, additional graphs, and supplementary analyses are available in the Web
Appendix online.
3
Abstract
Numeric ratings for products can be presented using a bigger-is-better format (1=bad, 5=good)
or a smaller-is-better format with reversed rating poles (1=good, 5=bad). Seven experiments
document how implicit memory for the bigger-is-better format—where larger numbers typically
connote something is better—can systematically bias consumers’ judgments without their
awareness. This rating polarity effect is the result of proactive interference from culturally
determined numerical associations in implicit memory and results in consumer judgments that
are less sensitive to differences in numeric ratings. This is an implicit bias that manifests even
when people are mindful and focused on the task and across a range of judgment types (auction
bids, visual perception, purchase intent, willingness-to-pay). Implicating the role of reliance on
implicit memory in this interference effect, the rating polarity effect is moderated by (i) cultural
norms that define the implicit numerical association, (ii) construal mindsets that encourage
reliance on implicit memory, and (iii) individual propensity to rely on implicit memory. This
research identifies a new form of proactive interference for numerical associations, demonstrates
how reliance on implicit memory can interfere with explicit memory, and how to attenuate such
interference.
Keywords: implicit memory, interference, numerical cognition, rating format, mindset, crosscultural marketing
4
Cause bigger is better
And big is just the best
If you take my advice
You’ll outshine the rest
- Princess Amber’s advice on party planning to Sofia in Disney’s Sofia the First
Whenever people are asked to evaluate things they encounter in life—products, job
candidates, employee performance, research proposals—numeric ratings are inevitably involved.
These numeric ratings can use a format where larger numbers indicate that something is better
(1=bad, 5=good) or have reversed rating poles where smaller numbers indicate that something is
better (1=good, 5=bad). The format that people are accustomed to encountering can be
culturally determined. For example in the United States, product ratings and student grade point
averages are typically presented with bigger-is-better polarity, and there is a strong association
between larger numbers and more positive evaluative judgments, not to mention a strong
association between “bigger” and “better” that permeates adverting for product size, popular
culture, and even children’s songs. However, in Germany, it is the opposite—lower numbers
indicate higher product quality and better student grade point averages, and people have a
culturally determined association between “smaller” and “better.” How then, can varying the
polarity of ratings used to evaluate products, students, or job applications influence judgments?
People might find themselves making evaluations based on a rating polarity format they
are less used to in cross-cultural contexts or simply when an evaluation system uses an opposite
rating polarity. For example, not only does the German equivalent of Consumer Reports, Stiftung
Warentest, rate products using a smaller-is-better system where 0.5 = very good and 5.5 =
unsatisfactory, but also grant applications for the National Institutes of Health in the U.S. are
scored by reviewers on a system ranging from 1 = exceptional to 9 = poor. These ratings are
used in peer review meetings to make funding decisions. We posit that when people attempt to
make evaluations using a rating polarity format opposite to the one they have grown up with, the
5
numerical association they are accustomed to using can bias their judgments because they
experience a form of memory interference.
In the memory literature, interference where information from the past inhibits people’s
ability to use information learned in the future is referred to as proactive interference (Jonides
and Nee 2006; Keppel and Underwood 1962; Wickens, Born, and Allen 1963). For example, you
might have difficulty remembering a new phone number after a move because the old phone
number that you have had for years interferes with your ability to remember the new one.
Similarly, after encountering an advertisement for a brand that includes information such as
price, attribute, or tagline information, consumers’ ability to learn the same information for new
brands encounter later can be impaired due to interference (Blackenship and Whiteley 1941;
Burke and Srull 1988; Keller 1987; Keller, Heckler, and Houston 1998; Unnava and
Sirdeshmukh 1994).
In our research, we introduce a new form of proactive interference for numerical
associations that can systematically bias consumer evaluations. The culturally determined
numerical association that people learn over time becomes part of their implicit memory—the
type of memory that influences judgment without conscious awareness (Graff and Schacter
1987; Roediger 1990; Schacter 1987). This numerical association in implicit memory can then
interfere with people’s ability to make evaluations using a newly learned format with opposite
rating polarity, resulting in judgments that are less sensitive to numeric differences in quality
level. We refer to this new form of proactive interference for numerical associations as the
“rating polarity effect.”
Over a series of seven experiments, we demonstrate that consumers’ decisions are
repeatedly and persistently affected by the rating polarity effect, even when they are well aware
6
that they are using a system with opposite rating polarity. The numeric association in implicit
memory surreptitiously influences consumers’ evaluations without their awareness. When asked
in an online forum whether the polarity of rating format should influence product evaluations,
79% of participants indicated they thought it should have no effect on their judgments (92
participants, 47% female, Mage: 38.7 years). Why, then, is it that people fall prey to this rating
polarity effect, yet believe they are immune to it? We hypothesize that this effect stems from
interference from implicit memory. We designed experiments to delineate the role of implicit
memory in this effect. In doing so, we show that the interference effects between implicit and
explicit memory can be attenuated by reducing reliance on implicit memory. Thus, together the
culturally determined numerical association in implicit memory and the rating polarity of the
evaluation system determine when “bigger-is-better,” and when it is not.
THEORETICAL BACKGROUND
Implicit Numerical Associations
Decades of research on memory and judgments suggest that two different types of
memory processes influence our everyday judgments: 1) information stored in explicit memory,
and 2) associations stored in implicit memory (Graf and Schacter 1987; Schacter 1987). Explicit
memory is characterized by intentional, conscious recollection of episodic information. In
contrast, implicit memory influences a task or judgment without conscious awareness or intent—
it encompasses the influence of past exposures and experiences, and can spontaneously, even
surreptitiously, influence judgments (Graff and Schacter 1987; Roediger 1990; Schacter 1987).
7
Previous research suggests that implicit associations can play an important role in
numerical evaluations (Adaval and Monroe 2002; Bagchi and Davis 2012; King and Janiszewski
2011; Mishra, Mishra, and Nayakankuppam 2006; Monroe and Lee 1999; Monga and Bagchi
2012; Raghubir and Srivastava 2009; Thomas and Morwitz 2009). These implicit associations
can also form based on cultural context. For example, in some cultures people learn to associate
numbers with bad luck—13 in the U.S., 4 in China, Korea, and Japan, and 7 in Ghana, Kenya,
and Singapore (Jahoda 1969; Yates 2007). In others, people with left-to-right reading habits
associate larger numbers with a right-orientation and smaller numbers with a left-orientation, and
this spatial-numerical association is weaker for people in cultures, such as Iran, where people
read from right to left (Dehaene, Bossini, and Giraux 1993).
In this research, we characterize a new type of implicit numerical association: that
between numeric ratings of a particular magnitude and an evaluate judgment. For example,
people in the U.S. tend to have an implicit association in memory that bigger-is-better, naturally
associating higher numbers with higher quality, while those in countries such as Germany tend to
have an implicit association that smaller-is-better, naturally associating lower numbers with
higher quality. This implicit numerical association influences judgments even in situations where
they should not, leading to proactive interference.
Proactive Interference for Numerical Associations
Classic work in proactive interference typically examines how previously learned content
in explicit memory interferes with memory for new learned information. For example, in the
classic Keppel and Underwood (1962) paradigm, participants were shown a series of nonsense
8
consonant syllables one at a time (KQF, MHZ, CXJ) and asked to repeat them back to the
experimenter after retrieval times of various lengths. Their research showed that the successive
recall accuracy for each syllable decreased as a result of proactive interference. Similarly, from
work in consumer behavior, when people were exposed to an initial advertisement with
information about price or other attributes, and then shown a second advertisement with similar
information (versus not), their recall of subsequent information was less accurate (Blackenship
and Whiteley 1941; Burke and Srull 1988; Keller 1991).
We examine a new type of proactive interference that can occur between a numerical
association in implicit memory and a numerical association explicitly provided in a rating
format. Specifically, when using a rating format with polarity opposite to that of the numerical
association stored in implicit memory, people’s evaluations will be unconsciously shifted in the
direction of their implicit numerical association. Thus, when an American consumer who is used
to bigger-is-better rating polarity comes across a rating format with smaller-is-better rating
polarity, her final evaluation will be unconsciously anchored in the direction of a spontaneously,
self-generated implicit numerical association in memory. This anchoring will bias the final
evaluation in the opposite direction towards the bigger-is-better rating polarity.
H1: In cultures where people hold the bigger-is-better numerical association in implicit
memory, product evaluations will be less responsive to differences in numeric
ratings when products are rated using smaller-is-better (vs. bigger-is-better) rating
polarity.
When products are rated at multiple quality levels, we expect that the slope of the relationship
between quality rating and subjective evaluations will be less steep when the products are rated
using smaller- versus bigger-is-better rating polarity.
9
Note that unlike the conscious anchoring-and-adjustment process studied by researchers
such Tversky and Kahneman (1974) and other scholars, the anchoring effect in the rating
polarity effect is unconscious. Because of this, we posit that the underlying process is more
consistent with the literature on subliminal numeric priming (Adaval and Monroe 2002;
Mussweiler and Englich 2005) and unconscious stereotyping (Gilbert and Hixon 1991).
Specifically, because the anchoring process is unconscious and stems from an implicit
association in memory in the rating polarity effect, people do not adjust from an anchor they are
not aware they are using (see Mussweiler and Englich 2005 for a more detailed exposition of
anchoring effects that are not caused by insufficient adjustments).
The Role of Implicit Memory
Although proactive interference is a complex phenomenon that can be studied from
several theoretical perspectives such as memory, learning, automaticity, cognitive control, and
metacognitive monitoring, we restrict our focus to unearthing the role of implicit memory
because proactive interference for numerical associations is based on an underlying implicit
association in memory. If interference from the implicit numerical association, which occurs
automatically without awareness, underlies the rating polarity effect, then proactive interference
for numerical associations should be moderated by factors that activate or inhibit reliance on
implicit memory. We identify three such factors: cultural norms, construal mindset, and
individual propensity to rely on implicit memory.
Cultural Norms. Our theorizing suggests that cultural norms are an important moderator
of proactive interference in numerical cognition. Thus, in a country such as Germany where
10
people hold an opposite numerical association in implicit memory, we hypothesize that the effect
of using a rating with bigger-is-better versus smaller-is-better rating polarity will reverse:
H2: In cultures where people hold the smaller-is-better numerical association in implicit
memory, their product evaluations will be less responsive to differences in numeric
ratings when products are rated using bigger-is-better (vs. smaller-is-better) rating
polarity.
Construal Mindset. Recent research suggests that construal mindset can influence
reliance on implicit memory. People make judgments and decisions along a continuum from
‘abstract’ to ‘concrete’ (Freitas, Gollwitzer, and Trope 2004; Trope and Liberman 2003;
Vallacher and Wegner 1989). In a more abstract mindset, people focus more on higher-level, gist
representations in memory while in a more concrete mindset, people focus more on lower-level,
verbatim representations. As it relates to memory, reliance on gist memory increases the use of
implicit associations in everyday judgments, whereas reliance on verbatim memory reduces it
(Fukakara, Ferguson, and Fujita 2013; Smith and Trope 2006; Rim, Uleman, and Trope 2009).
Based on this premise, Fukakara, Ferguson, and Fujita (2013) show that when comparing multiattribute stimuli, people in an abstract mindset are more likely to rely on gist memory, while
those in a concrete mindset rely more on verbatim memory. Using a false recognition paradigm,
Smith and Trope (2006; see experiment 4) demonstrate that an abstract mindset increases
reliance on implicit associations in gist memory, which in turn increases false recognition. Their
results also show that the effect of abstraction on false recognition judgments can be independent
of changes in effort or motivation. Similarly, research has shown that people in an abstract
mindset are more likely to make spontaneous trait inferences (Rim, Uleman, and Trope 2009)
and rely on stereotypes when making judgments (McCrea, Weiber, and Myers 2012), both of
11
which rely on implicit associations. Thus, prior research suggests that an abstract mindset
increases reliance on implicit memory. Therefore, people should be more susceptible to
spontaneous proactive interference from an implicit numerical association under an abstract
mindset. A concrete mindset, which reduces reliance on implicit associations in memory, should
attenuate this interference.
H3: The rating polarity effect is more likely under conditions of an abstract construal
mindset than under conditions of a concrete construal mindset.
Direct Measure of Implicit (vs. Explicit) Memory. People differ in their propensity to rely
on implicit associations versus explicit rules in everyday judgments as can be seen in their
performance on dual task-tests that require people to allocate their attention between tasks that
require both implicit and explicit memory (de Neys 2006). If the rating polarity effect stems from
spontaneous interference from implicit memory when using a particular rating format, it should
be more pronounced for people who tend to rely more on their implicit memory, but not
necessarily for those who rely more on explicit memory. Based on this premise, we predict:
H4: The rating polarity effect will be stronger for people who chronically rely to a
greater extent on implicit memory.
OVERVIEW OF EXPERIMENTAL PARADIGM
Examining the effects of proactive interference requires a specific experimental
paradigm. As described by Jacoby (1991), measuring the extent of interference between implicit
associations and explicit rules requires comparing outcomes where the implicit association and
explicit rule are consistent versus when they are inconsistent. In our experiments we compare
two conditions—one condition where the numerical association of the rating format and the
12
numerical association in implicit memory are consistent (consistent rating polarity), and another
where the numerical association of the rating format and the numerical association in implicit
memory are inconsistent (inconsistent rating polarity). If there is no interference between the
implicit numerical association in memory and the numerical association used in the rating
format, then judgments in the consistent and inconsistent rating polarity conditions should be
identical. However, if the judgments vary across the two conditions, this is evidence for
interference. The difference in evaluations between participants in the consistent versus
inconsistent rating polarity conditions reflects the extent of interference (Jacoby 1991).
Note that, a key objective of this research is to demonstrate that the rating polarity effect
is caused by implicit memory interference and that it manifests without the participants’
intention or awareness. To this end, for each experiment we conduct pre-evaluation and postevaluation comprehension tests. The pre-evaluation comprehension test after exposure to the
rating format, but before exposure to the stimuli, ensures that participants have read and
understood the information on rating polarity. Participants are allowed to proceed to the
experiment only after they pass this test. The post-exposure comprehension test is done after the
experiment and enables us to rule out inattention, miscomprehension, or forgetting as possible
alternative explanations for our results.
PRETEST: WILLINGNESS TO PAY
As a preliminary test of this experimental paradigm, we conducted an experiment
involving a sealed bid auction. Seventy two participants at a U.S. university were given the
opportunity to bid on a stainless steel mug that was described either as having a quality rating of
13
6.1 on a scale where 1 = unsatisfactory and 7 = very good (bigger-is-better) or an equivalent
rating of 1.9 on scale where 1 = very good and 7 = unsatisfactory (smaller-is-better). Those that
read a description using bigger-is-better rating polarity bid significantly more (M = $4.42) than
those that read one using smaller-is-better rating polarity (M = $2.70, F (1, 70) = 4.80, p = .03;
see Web Appendix A for full description). This difference between the two conditions offers
preliminary support for the proposed interference effect.
EXPERIMENT 1: VISUAL PERCEPTION
Experiment 1 examines whether the rating polarity effect in the domain of visual
perception. Can the rating polarity effect distort visual perception and fool our eyes? All
participants were shown the same set of before and after photographs, and we examined whether
the rating polarity effect influenced their visual perception across products of both high and low
quality. Comprehension, mood, and need-for-cognition were also measured to rule out potential
alternative explanations.
Method
Participants. One hundred and seventy-two undergraduates from a U.S. university (63%
female; average age: 21.1 years) participated in this computer study in exchange for course
credit. They were randomly assigned to one of four conditions in a 2 (rating polarity: consistent
vs. inconsistent) x 2 (quality level: low vs. high) between-subjects design.
Procedure. Participants were told they would be presented information about a new
14
peroxide-free, home teeth-whitening product currently available in Europe that the producer is
considering launching in the U.S. Their task was to evaluate how much change they saw in the
before versus after photographs that accompanied the product information. Participants were told
that a quality rating would be provided by a reputable consumer welfare agency in Europe
known for its evaluation of consumer products. Those in the consistent rating polarity condition
were told that 1 = unsatisfactory and 7 = very good while those in the inconsistent rating polarity
condition were told that 1 = very good and 7 = unsatisfactory. Half the participants were given a
high quality rating (consistent condition: 6.1; inconsistent: 1.9) and half were given a low quality
rating (consistent condition: 1.9; inconsistent: 6.1). Participants were asked to confirm that they
understood this rating format [Yes/No] before proceeding.
Pre-evaluation Comprehension Test. Participants were then asked the meaning of “1” and
“7” rating poles for the quality ratings [very good, unsatisfactory]. If they responded incorrectly,
a message asked them to correct their response, and they could proceed only after answering
these questions correctly. This was done to ensure that the results did not stem from inattention
or miscomprehension of the rating poles.
Visual Perception. All participants were then shown the same photographs of teeth before
and after treatment (see figure 1) and descriptive information, which included the quality level
for the product. The “before” and “after” photographs were pretested as showing moderate
improvement. The quality level rating that participants were shown was either 6.1 or 1.9—each
quality rating corresponded with a low versus high quality product depending on the rating
polarity. Participants were then asked: “How much whiter do the teeth look in the 'after' versus
the 'before' photograph?” [not at all whiter / much whiter], “How much cleaner do the teeth look
in the 'after' versus the 'before' photograph?” [not at all cleaner / much cleaner], “How much
15
improvement do you see in how the teeth look after using the whitener?”[no improvement /
significant improvement]. These questions were asked using an unnumbered slider scale, but
coded as 0 to 100 by the program so as to avoid any numerical association (see Web Appendix
B). The three measures were used as an index of visual perception.
______________________________
Insert figure 1 about here
______________________________
Post-Evaluation Comprehension Test. After entering their bid, participants were asked
the meaning of the rating poles 1 and 7 [very good or unsatisfactory]. This was done to confirm
that the participants did not become confused or forget about the meaning of the ratings.
Rating Polarity Typicality. Participants were also asked to indicate which of the two
numerical associations is more typical: “Higher Numbers Indicate Better Quality” or “Lower
Numbers Indicate Better Quality.” They could also choose a third option labeled “not sure.”
Additional Measures. To rule out possible alternative explanations, participants were also
asked questions about their current mood [slider scales anchored at bad / good, unpleasant /
pleasant, negative / positive], given the 18 items from the short form Need For Cognition (NFC)
scale (Cacciopo, Petty, and Kao 1984), and asked questions about their demographics.
Results and Discussion
Post-Evaluation Comprehension Test. After the main task, all participants answered the
two-rating-pole comprehension test questions. Two percent of participants in the inconsistent and
0% of participants in the consistent rating polarity conditions did not answer both of these
questions correctly. Participants who answered either of these questions incorrectly were
16
excluded from the analysis for this and all subsequent studies, although including them did not
change any of the main results. We followed this procedure in each of our experiments to ensure
that the reported results cannot be ascribed to inattention or miscomprehension.
Rating Polarity Typicality Check. A majority of participants indicated that the bigger-isbetter numerical association is more typical (83.7%), 14.0% indicated a smaller-is-better
association is more typical, and 2.3% indicated they were not sure, confirming the bigger-isbetter numerical association is more dominant in implicit memory for U.S. consumers.
Visual Perception. A two-way ANOVA with rating polarity and quality level as
independent measures and the visual perception index as the dependent measure (α = .82)
revealed a marginally significant effect of quality level. Participants saw the higher quality
product as more effective (M = 75.5) than the lower quality product (M = 70.9, F(1, 165) = 3.71,
p = .06). More importantly, we obtained a significant interaction (F(1, 165) = 4.17, p = .04).
Planned comparison tests revealed, as predicted, that in the consistent rating polarity conditions
(bigger-is-better), participants saw the high quality product as demonstrating significantly greater
improvement in the before versus after photographs (Mhigh = 78.3) than the low quality product
(Mlow = 68.8, F(1, 165) = 7.48, p < .01). However, in the inconsistent rating polarity conditions
(smaller-is-better), the effect of quality rating is attenuated (Mhigh = 72.8 versus Mlow = 73.1, F(1,
165) = .007, p = .93, see figure 2). When the rating polarity used a numerical association
inconsistent with the numerical association in implicit memory, participants’ evaluations of
visual improvement were less responsive to differences in quality level, supporting H1. Web
Appendix C summarizes the means and 95% confidence intervals by conditions across all
experiments, and for further data visualization, also plots the means and 95% confidence
intervals for the consistent versus inconsistent rating polarity conditions.
17
______________________________
Insert figure 2 about here
______________________________
Ruling Out Alternative Explanations. Mood: The three measures of mood were averaged
into an index (α = .93), and a two-way ANOVA with rating polarity and quality level as
independent measures revealed no significant main effects (ps > .80) or interaction (p = .13) on
mood. Thus negative mood from a format with inconsistent rating polarity cannot account for the
results. NFC: We conducted a linear regression analysis where the independent measures were
rating polarity (dummy coded: consistent = 0, inconsistent = 1), the mean centered NFC score,
and the interaction of the two with visual perception as the dependent measure. We obtain no
significant main or interaction effects for NFC (ps > .57) indicating that the rating polarity effect
is independent of NFC, suggesting that even participants with high need for cognition are
susceptible to the rating polarity effect and may not be aware of its effects.
Experiment 1 provides supporting evidence for the rating polarity effect: people
perceived the visual change in before-and-after photos as less impressive when the quality rating
was reported using smaller-is-better polarity even though they all viewed the exact same set of
photos. Their judgments were less responsive to differences in product quality when using a
rating system with polarity inconsistent with the numerical association they hold in implicit
memory. Confusion, forgetting, differences in need-for-cognition, and mood cannot account for
the results.
Although this experiment and the pretest provide evidence of the rating polarity effect,
there are several important questions that remain. First, does the rating polarity effect manifest
for only single judgments or will it continue to manifest across repeated judgments? If the rating
polarity effect stems from spontaneous interference from implicit memory without people’s
18
awareness, it should continue to manifest over multiple judgments, even as participants have
more practice using a format with inconsistent rating polarity. Second, does the rating polarity
effect persist for multiple levels of product quality? And third, does the rating polarity effect
occur only with particular types of response alternatives? If it stems from spontaneous
interference from implicit memory, it should be robust to multiple types of response alternatives.
The following two experiments address these questions.
EXPERIMENTS 2A & 2B: REPEATED EVALUATIONS
Experiments 2a and 2b test for the rating polarity effect in a more conservative context
where each participant evaluates 15 products across 5 levels of product quality. If the rating
polarity effect is caused by confusion or inattention, then it should not manifest in a repeated
measures design where each participant has the opportunity to evaluate many products and learn
from experience. However, if it stems from spontaneous proactive interference from implicit
associations, the effect should manifest even with a repeated measures format, further supporting
H1. Furthermore, to further rule out the possibility that the rating polarity effect manifests due to
orientation of the response format, we ran the experiments using two different response formats:
purchase intent as a binary “yes / no” measure (experiment 2a) and willingness-to-pay as a
vertical drop down menu of dollar amounts in 50-cent increments (experiment 2b). Since the two
experiments were identical in procedure and differed only in the response format, we report the
experiments together. Because differences in mood and need-for-cognition did not affect the
results of the first experiment and or any subsequent experiments, these analyses for possible
alternative accounts are detailed for all subsequent experiments in Web Appendix D.
19
Method for Experiments 2a and 2b
Participants. U.S.-based participants on mTurk (verified by IP address) participated in
these two experiments in return for $1.50: 213 participants in experiment 2a (49% female;
average age: 37.1 years) and 221 participants in experiment 2b (50% female; average age: 36.3
years). They were randomly assigned to the consistent or inconsistent rating polarity condition
within each experiment.
Procedure. Participants were told that the study was being conducted by a large retail
store to understand American consumers’ evaluations of European brands that might be
introduced in the U.S. They were shown brands of five quality levels (1, 2, 3, 4, and 5) in each of
the three different product categories (water, margarine, and toothpaste). Presentation order was
randomized for each participant across the 15 brands (see Web Appendix B for example stimuli).
For each brand, participants saw the brand name, the quality level (ostensibly taken from
Consumer Reports), a photo, and a short tag line communicating the brand positioning.
Participants assigned to the consistent rating polarity condition (bigger-is-better) were
informed that 1 = inadequate, 2 = adequate, 3 = fair, 4 = good, 5 = very good. Those in the
inconsistent rating polarity condition (smaller-is-better) were informed that 1= very good, 2 =
good, 3 = fair, 4 =adequate, 5 = inadequate.
Pre-evaluation Comprehension Test. Before proceeding to the main task, participants
were asked five pre-evaluation comprehension test questions: “If a product had a quality rating
of X, what does it mean?” where X was 5, 4, 3, 2, or 1 respectively. They were asked to select
one meaning for each rating value from the options very good, good, fair, inadequate, and
20
adequate. If a response to any question was incorrect, a message appeared informing the
participant which responses were incorrect and asked them to correct these answers. Participants
could proceed to the main task only after correctly answering all five questions.
Key Dependent Measure. Purchase Intent (Experiment 2a): Participants were asked,
“Would you purchase this product” [Yes or No]. Willingness-to-Pay (Experiment 2b):
Participants were asked to indicate their willingness to pay for each brand in U.S. dollars using a
drop-down list with price options increasing in 50-cent increments from $0 to $6.00.
Post-Evaluation Comprehension Test. After evaluating all 15 brands, participants were
again asked to indicate the meaning of the rating poles 1 and 5 using the same response options
as in the pre-evaluation comprehension task.
Rating Polarity Typicality. Participants were asked the same typicality question as in
experiment 1.
The experiment ended with demographic questions—age, gender, marital status,
measures of mood (7-point scales anchored at bad / good, unpleasant / pleasant, negative /
positive) and involvement (5-point scales anchored at not at all / very).
Results and Discussion
Post-Evaluation Comprehension Test. Participants who responded to at least one of the
rating pole comprehension test questions incorrectly were excluded from the analysis. In
experiment 2a, this included 0.5% of the participants in the consistent and 4.7% of those in the
inconsistent rating polarity condition. In experiment 2b, this included 0.9% of the participants in
the consistent and 0.5% of those in the inconsistent rating polarity condition.
21
Rating Polarity Typicality Check. A vast majority of participants indicated that the
bigger-is-better numerical association is more typical (2a: 97%; 2b: 96%) than the smaller-isbetter one (2a: 1.5%; 2b: 3%), while a small percentage were not sure (2a: 1.5%; 2b: 1%),
confirming that the bigger-is-better numerical association is more dominant in implicit memory.
Purchase Intent (Experiment 2a). Our objective was to test whether rating polarity
moderated the effect of quality level on purchase intent (yes = 1, no = 0). To do so, we dummycoded rating polarity (consistent = 0, inconsistent = 1), and coded quality level as a continuous
variable (inadequate = 1 to very good = 5). Quality levels in the inconsistent rating polarity
condition were reverse coded so that higher numeric scores also indicated higher quality level.
Purchase intent was submitted to a repeated measures logistic regression analysis using
PROC GENMOD in SAS with three independent variables: quality level, rating polarity, and
their interaction. This method treats quality level as a repeated, within-subjects continuous
variable and rating polarity as a between-subjects categorical variable. Because category effects
do not affect the main results across our studies, we included category (toothpaste, margarine,
and water) as an additional within-subjects factor but report results aggregated across the three
product categories across all experiments.
The simple effect of rating polarity was marginally significant (β = .76, p < .06) while the
simple effect of quality level was significant (β = 1.47, p < .01). Most importantly, the two-way
rating polarity x quality level interaction was significant (β = –.34, p < .01). The signs of the
coefficients suggest that a greater proportion of participants are willing to purchase brands with
higher quality levels, but the effect of quality level is weaker for participants evaluating brands
rated with a format using inconsistent versus consistent rating polarity, supporting H1. To
corroborate that these results are not an artifact of the regression assumptions, we plotted the
22
percentage of participants who were willing to purchase the products for the two rating polarity
conditions for each level of quality in figure 3. The pattern of means is consistent with our
interpretation of the results.
______________________________
Insert figure 3 about here
______________________________
Note that the coefficient for the rating polarity x quality level interaction term represents the
extent of interference that stems from the rating polarity effect. Statistically, it is the difference in
evaluations when using a rating format that employs rating polarity that is consistent (bigger-isbetter) versus inconsistent (smaller-is-better) with the numerical association in implicit memory.
Willingness-to-Pay (Experiment 2b). We used the same coding for independent variables
and analysis procedures as experiment 2a, but submitted the willingness-to-pay measure to a
repeated measures regression analysis using PROC MIXED in SAS.
The simple effect of rating polarity was significant (β = .35, p < .01), as was the simple
effect of quality level (β = .54, p < .01). The two simple effects were qualified by a significant
two-way rating polarity x quality level interaction (β = –.10, p < .01). Supporting H1, the signs
of the coefficients suggest that willingness-to-pay increases with higher quality levels, but the
effect of quality level is weaker for participants evaluating brands rated with a format using
inconsistent versus consistent rating polarity. When participants used a format with rating
polarity consistent with their implicit numerical association, on average, a one-level change in
quality level changed willingness-to-pay by $0.54. However, when the rating polarity was
inconsistent with their implicit numerical association, participants’ willingness-to-pay changed
by $0.44 points for a one-level change in quality.
Experiments 2a and 2b provide further evidence of the robustness of the rating polarity
23
effect across 15 repeated measures and 5 levels of product quality. Product evaluations,
measured through binary intent to purchase and vertically-oriented willingness-to-pay, are less
responsive to differences in quality level when products are rated using a format with a rating
polarity that is inconsistent (vs. consistent) with the implicit numerical association in memory.
Mood and involvement cannot account for the results (Web Appendix D).
THE ROLE OF IMPLICIT MEMORY IN THE RATING POLARITY EFFECT
The first three experiments provide evidence for the rating polarity effect and its
robustness, even across repeated judgments and a variety of dependent measures with both
horizontal and vertical orientation. But what causes the rating polarity effect? One question that
might come to mind is whether the rating polarity effect is caused by misattribution of the
subjective experience of fluency. Conceivably, judgments based on formats with consistent
rating polarity are more fluent or easier to make than those with inconsistent rating polarity.
Preference fluency is usually associated with more favorable evaluations (Schwarz 2004; see
also Alter and Oppenheimer 2009 and Lee and Labroo 2004). The observed pattern of results
does not support the fluency account because we do not observe only an effect of rating polarity
where using inconsistent rating polarity results in more unfavorable evaluations overall. Instead
we observe an interaction effect such that using inconsistent rating polarity results in less
sensitivity to differences in product quality overall.
Furthermore, task disfluency or judgment difficulty is typically associated with increased
response latency. To test whether the observed effects can be ascribed to difficulty, we
conducted supplemental analyses using reaction time as a covariate (see Web Appendix E). Even
24
when we include reaction time as a covariate, the effect of rating polarity on judgments remains
significant. Thus the observed effects cannot be ascribed to judgment difficulty.
We propose that the rating polarity effect is caused by spontaneous interference from
implicit memory. To support our proposition that the interference from implicit memory is
spontaneous and occurs without the awareness of the influence of this interference on judgments,
note that we include only those participants that correctly identify the rating polarity used for
their judgments at the end of the judgment task. Yet even participants with high need-forcognition are unable to overcome the effect of rating polarity (see Web Appendix D). Similarly,
participants are unable to overcome the rating polarity effect over 15 repeated judgments. We
conducted additional analyses including order as an independent variable for experiments 2a, 2b,
and all of the experiments that follow (see Web Appendix F). In all of these experiments, the
two-way interaction representing the rating polarity effect remains significant while the threeway interactions between rating polarity, quality level, and order is not significant in any of the
experiments, indicating that the rating polarity effect does not dissipate over fifteen judgments.
If the rating polarity effect stems from the automatic reliance on implicit memory when
making judgments, then it should be moderated by factors that influence the implicit numerical
association itself or the tendency to rely on implicit memory when making judgments such as: 1)
cultural norms that influence numerical associations held in implicit memory (experiment 3), 2)
mindsets that increase reliance on implicit versus explicit memory (experiments 4a and 4b), and
3) measuring direct reliance on implicit memory (experiment 5).
EXPERIMENT 3: MODERATION BY CULTURAL NORMS
25
In a country where a smaller-is-better numerical association is more likely to be held in
implicit memory, such as Germany, using a rating format with bigger-is-better rating polarity
should result in product evaluations that are less responsive to differences in quality level,
supporting H2. Experiment 3 tests this proposition with German participants.
Method
Participants. One hundred members of an online panel in Germany (51% female;
average age: 43.0 years) participated in this experiment in exchange for approximately €4.
Procedure. The stimuli were nearly identical to experiments 2a & 2b, with the following
differences. The questionnaire was translated into German, and the cover story was slightly
modified for a German audience. Participants were informed that a retail store is considering
introducing several European brands in the U.S. market and that the quality ratings were
provided by a “reputable consumer welfare protection agency widely respected in the U.S. for its
unbiased evaluation of consumer products.” To test the robustness of the previous results, we
used a 7-point, non-numeric semantic differential scale of purchase intentions (anchored at
unlikely to buy and likely to buy, with the center labeled as neutral) as the main dependent
variable in this study and the studies that follow.
Results and Discussion
Post-Evaluation Comprehension Test. Twelve percent of participants in the inconsistent
and 10% in the consistent condition did not correctly answer both rating-pole comprehension test
26
questions. As in the previous experiments, these participants were excluded from the analysis, so
the results could not be ascribed to confusion or forgetting (but, all effects persist when we
include these participants).
Rating Polarity Typicality. Forty-nine percent of the participants indicated that the
smaller-is-better numerical association was more typical, 31% indicated that they found the
bigger-is-better numerical association more typical, and 20% indicated they were not sure. These
results suggest that although the smaller-is-better format is more typical in Germany, it is not as
ubiquitous as the bigger-is-better format is in the United States.
Purchase Intentions. We used a regression model identical to experiment 2b, except that
in this case, the smaller-is-better rating polarity was the consistent rating polarity condition and
the bigger-is-better rating polarity was the inconsistent rating polarity condition. Purchase
intention was submitted to a repeated measures regression analysis with quality level, rating
polarity, and their interaction as predictors. The simple effect of rating polarity was significant (β
= .86, p < .01) as was the simple effect of quality level (β = .59, p < .01), but most importantly,
the two-way rating polarity x quality level interaction representing interference from the rating
polarity effect was significant (β = -.28, p < .01).
______________________________
Insert figure 4 about here
______________________________
Mirroring the results of experiments 2a and 2b, purchase intention increases with higher
quality levels, and consumer evaluations are less responsive to changes in quality level when the
brands were rated using a format with rating polarity that is inconsistent versus consistent with
the numerical association in implicit memory. However, in this case, it is the bigger-is-better
rating polarity that is inconsistent with the implicit numerical association in memory. When
27
participants evaluate brands using rating polarity consistent with their implicit numerical
association (smaller-is-better), on average, a one-level difference in quality level changes
purchase intention by 0.59 points; however, it only changes purchase intention by 0.31 points
when participants use rating polarity inconsistent with their implicit numerical association
(bigger-is-better), supporting H2. The means of purchase intention are plotted in figure 4.
Moderation by Implicit Numerical Association. Among German participants,
approximately half of the participants held the smaller-is-better numerical association as more
typical (49%). However, the remaining participants (51%) did not consider smaller-is-better
more typical. If the rating polarity effect stems from spontaneous proactive interference from the
implicit numerical association in memory, then it should be moderated by the strength or
flexibility of this implicit numerical association. More specifically, those participants who have
the implicit numerical association that smaller numbers indicate higher quality should continue
to exhibit the rating polarity effect. But those participants who hold a more flexible implicit
numerical association (e.g. are not sure about the association or find a bigger-is-better
association more typical in a country that typically uses smaller-is-better rating polarity) are less
likely to experience as strong of an interference effect from a competing smaller-is-better
implicit numerical association when using a rating system with opposite rating polarity. Thus,
they are less likely to exhibit the rating polarity effect.
To test this prediction, purchase intentions was submitted to a repeated measures
regression analysis used PROC MIXED in SAS with implicit numerical association (bigger-isbetter or not sure = 0, smaller is better = 1) as an independent variable in addition to rating
polarity and quality level and their two and three-way interaction terms as predicting variables
(see table 1). The three-way interaction between rating polarity, quality level, and implicit
28
numerical association was significant (β = .56, p < .01). Follow-up contrasts reveal that for those
participants with a smaller-is-better implicit numerical association, the rating polarity x quality
level interaction is significant (β = -.54, p < .01), but for those participants with either a biggeris-better implicit numerical association or unsure of their numerical association, the rating
polarity effect does not manifest (β = .01, p = .87, see figure 5 for pattern of means).
Experiment 3 provides initial support implicating the role of implicit memory in the
rating polarity effect. Culture moderates the rating polarity effect: In Germany, where the
smaller-is-better numerical association is more common, the rating format with bigger-is-better
polarity results in more muted responses to differences in quality level, supporting H2.
Furthermore, the additional regression analysis with implicit numerical association as a
moderator revealed that the rating polarity effect is eliminated for those participants with a more
flexible implicit numerical association in memory. This suggests that people who have a less
strongly held implicit numerical association in memory do not experience the same spontaneous
interference from a numerical association. Mood and involvement cannot account for these
effects (Web Appendix D).
______________________________
Insert table 1 and figure 5 about here
______________________________
While experiment 3 provides evidence that having different implicit numerical
associations can reverse the effect of using ratings of different polarity and attenuate the rating
polarity effect, it does not specifically implicate the reliance on implicit versus explicit memory.
The remaining experiments directly examine the role of reliance on implicit memory.
EXPERIMENT 4A: MODERATION BY PRIMED CONSTRUAL MINDSET
29
As discussed earlier, previous research has shown that people in an abstract (versus
concrete) mindset rely more on implicit associations in gist memory. Experiment 4a tests H3 that
the rating polarity effect is more likely to manifest for people in an abstract versus concrete
mindset. Participants are induced with a concrete versus abstract mindset using the Frietas,
Gollwitzer, and Trope (2004) construal mindset manipulation before completing the same
product evaluation task as in experiment 3.
Method
Participants. One hundred and seventy-two U.S.-based mTurk participants (33% female;
average age: 31.2 years) participated in this experiment in return for $1.50. They were randomly
assigned to one of four conditions: rating polarity (consistent v. inconsistent) x construal mindset
(abstract v. concrete).
Procedure. The basic procedure was similar to the previous experiment, with the addition
of a construal mindset manipulation after the pre-evaluation comprehension task and before the
key dependent measures (purchase intention, post-evaluation comprehension task, rating polarity
typicality, mood, involvement, demographics). Participants completed a mindset-priming
manipulation similar to that developed by Freitas, Gollwitzer, and Trope (2004) and were told it
was a thought-listing task that would help design future experiments. Those in the abstract
mindset condition were asked to answer a series of increasingly high level questions about
“Why” they would “Improve and maintain one’s general knowledge,” while those in the concrete
mindset condition were asked to answer a series of increasingly low level questions about “How”
30
they would engage in this same behavior.
Results
Post-Evaluation Comprehension Test. Five percent of participants in the inconsistent and
1% in the consistent rating polarity condition responded to at least one of the two rating-pole
comprehension test questions incorrectly. These participants were excluded from the analysis.
Rating Polarity Typicality. The majority of the participants indicated that the bigger-isbetter numerical association is more typical (94%) than the smaller-is-better one (3%), while 3%
were not sure, confirming that the bigger-is-better association is dominant in implicit memory.
Purchase Intentions. We analyzed the data using a regression model, dummy coding
rating polarity (consistent = 0, inconsistent = 1) and construal mindset (concrete = 0, abstract =
1) and coding quality level as a continuous variable. Purchase intentions were submitted to a
repeated measures regression analysis using PROC MIXED in SAS with the three independent
variables (rating polarity, mindset, and quality level) and their two- and three-way interaction
terms as predicting variables (see table 2). The simple effect of quality level was significant (β =
.91, p < .01), as was the two-way rating polarity x mindset interaction (β = .76, p = .03); the twoway rating polarity x quality level interaction was marginally significant (β = -.10, p = .09). Most
importantly, the three-way rating polarity, mindset, and quality level interaction was significant
(β = -.20, p = .01). Other effects were not significant (ps > .34, see table 2).
______________________________
Insert table 2 about here
______________________________
To examine the significant three-way interaction, we probed the two-way quality level x
31
rating polarity interaction for abstract versus concrete mindsets. We find that the coefficient for
the quality level x rating polarity interaction is statistically significant for those in an abstract
mindset who rely more on implicit memory (β = -.31, p < .01). However, the magnitude of the
coefficient was smaller and marginally significant for participants in a concrete mindset, who
were less likely to rely on implicit memory (β = -.10, p = .09, see table 2). These results suggest
that the rating polarity effect is indeed less likely to manifest for those participants in a more
concrete (vs. more abstract) mindset who rely less on implicit memory, supporting H3.
EXPERIMENT 4B: MODERATION BY CHRONIC CONSTRUAL MINDSET
It is possible that the mindset manipulation, which was administered before the main
experiment, might have introduced an inadvertent confound in the previous study. To rule out
any such possibility and provide further evidence that reliance on implicit versus explicit
memory moderates the rating polarity effect, in this experiment construal mindset was measured
using an established scale (Vallacher and Wegner 1989), rather than manipulated.
Method
Participants. Two hundred and nine undergraduates at a college in the northeast U.S.
participated in an online study in exchange for $5 (43% female; average age: 20.1 years).
Procedure. The procedure for this experiment was identical to the previous experiment,
except that the mindset was measured after the product evaluation task using the Vallacher and
Wegner (1989) Behavioral Identification Form (BIF) and before the post-evaluation
32
comprehension task and measures of mood, attention, and demographics. Participants were
presented with a series of 24 different behaviors (making a list), each with a more abstract
response (getting organized) and a more concrete response (writing things down), and asked to
indicate which of the two descriptions of the behavior they preferred.
Results
Post-Evaluation Comprehension Test. Three percent of participants in the inconsistent
and 1% in the consistent rating polarity condition responded to at least one rating-pole
comprehension test question incorrectly. These participants were excluded from the analysis.
Rating Polarity Typicality. The vast majority of participants indicated that the bigger-isbetter numerical association is more typical (96%) than the smaller-is-better association (1%),
and 3% of participants indicated they were not sure. To confirm that chronic mindset did not
influence which rating polarity was seen as more typical, we conducted a logistic regression to
confirm that there was no significant effect of chronic mindset, rating polarity, or the interaction
between the two on perceived typicality (ps > .30).
Effect of Rating Polarity on Mindset. Participants’ responses to the BIF study were coded
as concrete = 0 and abstract = 1, and summed across the 24 responses, resulting in a continuous
BIF score ranging from 0 to 24 (M = 12.12, STDV = 4.57). This variable was mean-centered for
the key analyses described below. A one-way ANOVA with rating polarity as the independent
measure and construal mindset as the dependent measure confirmed that rating polarity did not
have a significant effect on mindset (p > .23).
Purchase Intention. We again analyzed the data using a regression model, dummy coding
33
rating polarity (consistent as 0, inconsistent 1), entering chronic mindset as a continuous variable
(mean-centered BIF score), and coding quality level as a continuous variable. Purchase
intentions were submitted to a repeated measures regression analysis with the three independent
variables (rating polarity, quality level, chronic mindset) and their two- and three-way interaction
terms as predicting variables using PROC MIXED in SAS (see table 2).
The simple effects of quality level (β = .84, p < .01), rating polarity (β = 1.01, p < .01),
and mindset (β = -.06, p = .05) were significant. The rating polarity x quality level (β = -.28, p <
.01) and rating polarity x mindset (β = .14, p < .01) two-way interactions were significant. The
two-way quality level x mindset interaction was not significant (p > .15). Most importantly, the
three-way interaction was significant (β = -.024, p = .01). Follow-up spotlight tests (Aiken and
West 1991) at one standard deviation above and below the mean of chronic mindset revealed that
for those with a more chronically abstract mindset (more likely to rely on implicit memory), the
two-way interaction between rating polarity and quality level was significant (β = -.39, p < .001,
see table 2). For those with a more chronically concrete mindset (less likely to rely on implicit
memory), the two-way rating polarity x quality level interaction was also significant (β = -.17, p
< .01), but the magnitude of the coefficient was lower, indicating an attenuation of the rating
polarity effect, consistent with H3. Follow-up floodlight analysis are included in Web Appendix
G, which provide further evidence that the coefficient representing the rating polarity effect and
its significance decrease for participants with an increasingly chronic concrete mindset.
EXPERIMENT 5: MODERATION BY INDIVIDUAL RELIANCE ON
IMPLICIT MEMORY
34
The previous two experiments indirectly manipulate or measure reliance on implicit
versus explicit memory. In experiment 5, we directly examine the role of reliance on implicit
versus explicit memory in driving the rating polarity effect. Specifically, we test that the rating
polarity effect will be stronger for people who rely to a greater extent on implicit memory (H4).
To test this, we use the well-established dual-task paradigm for measuring individual reliance on
different types of memory (de Neys 2006, Jacoby 1991). In this paradigm, participants are
concurrently given two tasks where one task requires implicit memory and another task requires
explicit memory—performance on each of the two simultaneous tasks determines individuals’
reliance on implicit and explicit memory (de Neys 2006, Jacoby 1991). We created a
computerized dual-task modeled after the prevailing dual-tasks that can be administered online in
which the computer coded the responses (see Web Appendix B). In this dual-task, participants
had to simultaneously complete a true-false task (which measured reliance on implicit memory),
and a recall task (which measured reliance on explicit memory).
More specifically, in the dual-task, participants were shown a series of simple arithmetic
facts and had to submit a true or false judgment (e.g., “7-5 = 1”). The judgment had to be made
in 3 seconds, and while concurrently engaging in a recall task. For the recall task, participants
had to memorize the series of digits on the right hand side of the equations (e.g., 1 in the
equation above) and report them at the end of a series of true or false questions. Thus, when
making the true-false judgments in this dual-task paradigm, participants had neither time nor
working memory resources, and had to rely on pattern recognition stored in their implicit
memory. Prior research has shown that when you look at “7-5 = 1”, you can intuitively sense,
using implicit memory, whether this statement is true or false, without actually subtracting
(Schunn et al. 1997). Success in the true-false task, measured as the number of correct responses
35
to the true-false questions in this dual-task paradigm, is a direct measure of participants’ reliance
on implicit memory. Success in the recall task, measured as the number of correct responses to
the recall questions in this dual-task paradigm, is a measure of participants’ reliance on explicit
memory. We predicted that the rating polarity effect would be stronger for those participants
who receive a higher score in the true-false task (indicating that they rely more on implicit
memory). Because we posit that the rating polarity effect stems from spontaneous interference
from implicit memory, we predict that subjects’ performance on the implicit memory task, but
not the explicit memory task, will moderate the rating polarity effect.
Method
Participants. Two hundred and two U.S.-based mTurk participants (38% female; average
age: 33.4 years) participated in this experiment in return for $1.50.
Procedure. This experiment was identical to experiment 4b, except that the task
measuring implicit and explicit memory replaced the BIF measurement. After the main
experiment, participants responded to an ostensibly unrelated dual-task test where they were told
that they would complete two tasks simultaneously—a “True-False Task” and a “Recall Task.”
They were shown a series of simple arithmetic facts. For the True-False task, they had to make a
determination of whether the arithmetic fact was true or false within three seconds. For example,
in a block with four facts, participants saw the following four arithmetic facts, one at a time, and
they had to indicate whether each of them is true or false: “7-5 = 1”; “2+3 = 5”; “2+5=6”;
“4+5=8” (see Web Appendix B for stimuli examples). In this block the correct responses to the
four equations are false, true, false, and false respectively. While making these judgments, they
36
had to simultaneously complete a memorization task. For the memorization task, they needed to
memorize the number on the right hand side of equation in the order they saw the number and
recall these numbers in order. For example, while completing the true-false task for the first
block, they had to memorize the right-side digits of the equations listed above and recall the
string of digits 1568 at the end of the block. Participants completed eight such blocks of
equations with increasing span of the recall task; there were four, five, six, six, seven, seven,
eight, and eight equations in each succeeding block (see Web Appendix B). Thus, over eight
blocks, participants made a total of 51 true-false judgments and simultaneously responded to 51
explicit recall questions. Before completing the main dual task of these eight blocks, participants
received three practice blocks with feedback, to understand the task.
Results
Post-Evaluation Comprehension Test. Two percent of participants in the inconsistent and
2% in the consistent rating polarity condition responded to at least one rating-pole
comprehension test question incorrectly. As in the previous studies, these participants were
excluded from the analysis. Additionally, one participant who got all 51 true-false judgments
incorrect was excluded from the analyses.
Rating Polarity Typicality. The vast majority of the participants indicated that the biggeris-better numerical association is more typical (84%) than the smaller-is-better one (16%),
confirming that the bigger-is-better association is more dominant in implicit memory.
Effect of Rating Polarity on Implicit Memory Task Performance. Participants’ propensity
to rely on implicit memory was measured by the number of correct responses to the true-false
37
task. Larger numbers indicate a greater propensity to rely on implicit memory. These scores
ranged from 18 to 51 with a mean of 42.65 and standard deviation of 6.80. A one-way ANOVA
with rating polarity as the independent variable and performance on the true-false task as the
dependent variable confirmed that rating polarity did not have a significant effect on
performance on the true-false task (F < 1).
Purchase Intentions. We analyzed purchase intentions using a regression model, dummy
coding rating polarity (consistent = 0, inconsistent = 1), entering performance on the implicit
memory task as a continuous variable (mean-centered), and coding quality level as a continuous
variable (inadequate = 1, adequate = 2, fair = 3, good = 4, very good = 5). Purchase intentions
were submitted to a repeated measures regression analysis with the three independent variables
(rating polarity, quality level, implicit memory task performance) and their two- and three-way
interaction terms as predicting variables, using PROC MIXED in SAS (see table 3).
The simple effects of rating polarity (β = .60, p < .01), quality level (β = 1.06, p < .01), and
propensity to rely on implicit memory (β = -.05, p < .01) were significant. The two-way
interactions of rating polarity x quality level (β = -.14, p < .01) and quality level x implicit
memory reliance (β = .010, p < . 01) were significant. The two-way interaction between rating
polarity x implicit memory reliance was not significant. Most importantly, the three-way
interaction was significant (β = -. 01, p = .05).
______________________________
Insert table 3 about here
______________________________
Follow-up spotlight tests (Aiken and West 1991) at one standard deviation above and below
average performance scores on the implicit memory task revealed that for those who performed
above average on the implicit memory task, the two-way rating polarity x quality level
38
interaction representing the rating polarity effect was significant (β = -.22, p < .001, see table 3).
However, for those participants who performed below average on the implicit memory task, the
two-way rating polarity x quality level interaction was not significant (β = -. 07, p = .18),
indicating an attenuation of the rating polarity effect. These results support H4. We performed a
similar regression analysis using performance on the explicit memory performance task rather
than the implicit memory task as the dependent measure. The three-way interaction between
rating polarity, quality level, and explicit memory reliance was not statistically significant (β =
.002, p = .59), indicating that although a direct measure of reliance on implicit memory
moderates the rating polarity effect, the direct measure of reliance on explicit memory does not.
DISCUSSION OF EXPERIMENTS 4A, 4B, AND 5.
Together, experiments 4a, 4b, and 5 provide evidence that the rating polarity effect is
more likely to occur when people rely more on their implicit memory. Experiments 4a and 4b
show that participants under an abstract (versus concrete) construal mindset, which increases
reliance on implicit memory, are more likely to exhibit the rating polarity effect. The size of the
coefficient for the rating polarity x quality level interaction, which represents the extent of
interference from the implicit numerical association when using a rating format with inconsistent
rating polarity, is smaller for those participants who are either induced with a concrete mindset
(experiment 4a) or chronically adopt a more concrete mindset (experiment 4b).
In experiment 5, we directly measured the use of implicit versus explicit memory using
the dual-task paradigm and found that the rating polarity effect was moderated by propensity to
use implicit memory: the rating polarity effect was stronger for those participants who relied on
39
implicit memory to a greater extent during the dual-task. Furthermore, reliance on explicit
memory does not moderate the rating polarity effect, providing further evidence that it is an
effect driven by spontaneous interference from the numerical association in implicit memory.
The strongest moderating effect comes from directly measuring individuals’ use of implicit
memory (experiment 5). Mood and involvement cannot account for these effects (Web Appendix
D). These experiments support H3 and H4, demonstrating that people are more likely to
experience spontaneous proactive interference from an implicit numerical association in memory
when in a mindset that encourages reliance on implicit memory or if their individual chronic
propensity is to rely more on implicit memory.
GENERAL DISCUSSION
Across seven experiments, we provide evidence of a new form of proactive interference
between numerical associations: the rating polarity effect. The first three experiments
demonstrated the effect, supporting H1: evaluating a product using a rating format with a
numerical association opposite the one that people hold in implicit memory results in evaluations
that are less responsive to differences in quality level. This effect is robust to a consumer
evaluations ranging across bidding behavior (paradigm pretest), visual perception (experiment
1), purchase intent (experiments 2a, 3, 4a, 4b, and 5), and willingness-to-pay (experiment 2b). It
manifests across a variety of types of response alternatives (open ended, horizontal and vertical
orientation, with and without numerical anchors, binary and scale). Confusion, forgetting, mood,
and involvement are ruled out as potential alternative explanations. The effect occurs for single
and repeated evaluations and is not moderated by need-for-cognition or order, demonstrating the
40
spontaneous and unintended effect of the implicit numerical association in memory.
To understand the mechanism behind the rating polarity effect and why this proactive
interference effect persists even when participants are well aware of the rating polarity
employed, we examine the role of implicit memory. If the rating polarity effect is caused by the
spontaneous use of the implicit numerical association in memory, then the rating polarity effect
should be moderated by factors that moderate reliance on implicit memory. Supporting H2,
experiment 3 demonstrates that for participants from a country with an opposite numerical
association as a cultural norm, the effect of bigger-is-better versus smaller-is-better rating
polarity reverses. In addition, for those participants with a more flexible set of implicit numerical
associations in memory, the rating polarity effect does not manifest as they do not experience
proactive interference from a strong implicit numerical association. Further illustrating the role
of implicit numerical associations behind the rating polarity effect, this proactive interference of
numerical associations diminishes when people rely less on implicit memory whether it is based
on a mindset that leads people to rely less on implicit memory (experiment 4a, 4b, supporting
H3) or on individual differences in relying on implicit memory as directly measured through a
dual-task paradigm (experiment 5, supporting H4).
Theoretical and Managerial Contributions
The current research presents new insights into numerical cognition and makes three
theoretical contributions. First, we characterize a new type of implicit numerical association that
differs depending on cultural context—the association between the magnitude of a number and
an evaluative judgment. Culture can influence whether consumers have a bigger-is-better versus
41
smaller-is-better numerical association in implicit memory, and this implicit association can
spontaneously influence their judgments when using a rating polarity with an opposite numerical
association. Second, our research demonstrates the persistence of interference effects that stem
from an implicit association in memory that has been culturally cultivated over time. The rating
polarity effect manifests over different types of evaluations, over multiple consumer evaluations,
and even when taking into account measures of difficulty and need-for-cognition (NFC).
Third, our research also contributes to the memory literature by identifying novel
moderators of proactive interference, based on unintentional reliance on implicit memory. We
show that the strength of the implicit numerical association in memory moderates the effect of
proactive interference. Furthermore, (both manipulated and measured) construal mindset can
moderate proactive interference, and add to the burgeoning literature on how cognitive mindsets
change memory processes. In particular, in contrast to work that demonstrates that abstract
construal mindsets can attenuate interference when explicit memory interferes with implicit
memory (Kyung and Thomas, 2016), this research demonstrates that concrete construal mindsets
can attenuate interference when implicit memory interferes with explicit memory. Thus, whether
a concrete versus abstract construal mindset can attenuate memory interferences depends on
which form of memory is causing the interference effect. In addition, we show that individual
propensity to rely on implicit memory (as measured by performance on the implicit memory
portion of the dual-task exercise) can moderate proactive interference. If one thinks of the
implicit numerical association (bigger-is-better) as a culturally-induced metaphor (see Casasanto
2014), then our research introduces the notion of interference that stems from metaphorical
incongruity. Future research could examine other metaphors for interference.
From a managerial perspective, our research suggests that rating polarity is an important
42
element of market research design in cross-cultural contexts and for the presentation of results.
Presenting information using rating polarity inconsistent with the numerical association in
implicit memory can make consumers less responsive to differences in numeric quality ratings of
products. As such, retailers should translate numeric ratings into a format that is consistent with
the numerical association that their target customers have in memory. Similarly, any decision
maker that encounters numeric information utilizing rating polarity opposite the numerical
association they hold in implicit memory should be aware of the potential impact on their
judgments: admissions officers or human resource managers who use grade point average
information (GPA) across cultures; policymakers making global decisions based upon crosscultural research, such as life satisfaction surveys where some countries use smaller-is-better
rating polarity and some do not (Diener, Kahneman, and Helliwell 2010); reviewers evaluating
the $30 billion in research grants for the National Institutes of Health (NIH) where proposals are
evaluated using a format where 1 = exceptional and 9 = poor.
Finally our work underscores the importance of understanding cultural implicit
associations that might bias people’s judgments and how difficult it can be to overcome these
implicit associations, such as the implicit association between “bigger” and “better” in the United
States. Thus, we encourage managers and researchers to understand what these implicit
associations might be and how they can influence judgments1.
1
And with that, we conclude this paper as we have found that the association between “bigger”
and better” does not apply to the length of academic papers.
43
Data Collection Paragraph
All study designs and analyses were discussed and agreed upon by all three authors. The first
author supervised the data collection for experiments 1a and 3b through Dartmouth College
students, experiment 1b through Cornell University students, study 2 through a Qualtrics panel
of German participants, and studies 2a, 2b, 3a, and 4 through the mTurk online panel. The first
and second authors separately and jointly analyzed the data from these studies, confirming all
results.
44
REFERENCES
Adaval, Rashmi, and Kent B. Monroe (2002), “Automatic Construction and Use of Contextual
Information for Product and Price Evaluations,” Journal of Consumer Research, 28(4), 57288.
Aiken, Leona S. and Stephen G. West (1991), Multiple Regression: Testing and Interpreting
Interactions, Sage Publications.
Ajzen, Icek, and Martin Fishbein (2005), “The Influence of Attitudes on Behavior,” The
Handbook of Attitudes, eds. Dolores Albarracin, Blair T. Johnson, and Mark P. Zanna,
Lawrence Erlbaum Associates, 173-221.
Alter, Adam L., and Daniel M. Oppenheimer (2009), “Uniting the Tribes of Fluency to Form a
Metacognitive Nation,” Personality and Social Psychology Review, 13(3), 219-35.
Bagchi, Rajesh and Derick F. Davis (2012), “$29 for 70 items or 70 items for $29? How
Presentation Order Affects Package Perceptions,” Journal of Consumer Research, 39(1), 6273.
Blankenship, Albert B., and Paul L. Whitely, (1941), “Proactive Inhibition in the Recall of
Advertising Material,” The Journal of Social Psychology, 13(2), 311-22.
Burke, Raymond R., and Thomas K. Srull (1988), “Competitive Interference and Consumer
Memory for Advertising,” Journal of Consumer Research, 15(6), 55-68.
Casasanto, Daniel (2014), “Experiential origins of mental metaphors: Language, Culture, and the
body”, in The Power of Metaphor: Examining Its Influence on Social Life, M.]. Landau, M.
D. Robinson, and B. P. Meier (Editors).
Conner, Mark T., Marco Perugini, Rick O’Gorman, Karen Ayres, and Andrew Prestwich (2007),
“Relations between Implicit and Explicit Measures of Attitudes and Measures of Behavior:
Evidence of Moderation by Individual Difference Variables,” Personality & Social
45
Psychology Bulletin, 33(12): 1727–40.
Dehaene, Stanislas, Serge Bossini, and Pascal Giraux (1993), “The Mental Representation of
Parity and Number Magnitude,” Journal of Experimental Psychology: General, 12 (3), 37196.
De Neys, Wim (2006), “Dual Processing in Reasoning Two Systems but One Reasoner,”
Psychological Science, 17(5), 428-33.
Diener, Ed, Daniel Kahneman, and John Helliwell (2010), International Differences in Wellbeing, Oxford University Press.
Florack, Arnd, Martin Scarabis, and Herbert Bless (2001), “When Do Associations Matter? The
Use of Automatic Associations toward Ethnic Groups in Person Judgments,” Journal of
Experimental Social Psychology, 37(6), 518–24.
Freitas, Antonio L., Peter Gollwitzer, and Yaacov Trope (2004), “The Influence of Abstract and
Concrete Mindsets on Anticipating and Guiding Others' Self-regulatory Efforts,” Journal of
Experimental Social Psychology, 40(6), 739-52.
Fukukura, Jun, Melissa J. Ferguson, and Kentaro Fujita (2013), “Psychological Distance Can
Improve Decision Making Under Information Overload via Gist Memory,” Journal of
Experimental Psychology: General, 142(3), 658-65.
Gilbert, Daniel T., and J. Gregory Hixon (1991), “The Trouble of Thinking: Activation and
Application of Stereotypic Beliefs,” Journal of Personality and Social Psychology, 60(4),
509-17.
Graf, Peter, and Daniel L. Schacter (1987), “Selective Effects of Interference on Implicit and
Explicit Memory for New Associations,” Journal of Experimental Psychology: Learning,
Memory, and Cognition, 13(1), 45-53.
46
Jacoby, Larry L. (1991), “A Process Dissociation Framework: Separating Automatic From
Intentional Uses of Memory,” Journal of Memory and Language, 30(5), 513-41.
Jahoda, Gustav (1969), The Psychology of Superstition, Oxford, England: Penguin Books.
Jonides, John, and Derek E. Nee (2006), “Brain Mechanisms of Proactive Interference in
Working Memory,” Neuroscience, 139 (1), 181-93.
Keller, Kevin Lane, Susan E. Heckler, and Michael J. Houston (1998), “The Effects of Brand
Name Suggestiveness on Advertising Recall,” Journal of Marketing 62(1), 48-57.
Keller, Kevin Lane (1987), “Memory Factors in Advertising: The Effect of Advertising Retrieval
Cues on Brand Evaluations,” Journal of Consumer Research, 14(3), 316-33.
Kevin Lane Keller (1991), “Memory and Evaluations in Competitive Advertising
Environments,” Journal of Consumer Research, 17 (3), 463-76.
Keppel, Geoffrey, and Benton J. Underwood (1962), “Proactive Inhibition in Short-term
Retention of Single Items,” Journal of Verbal Learning and Verbal Behavior, 1(3), 153-61.
King, Dan, and Chris Janiszewski (2011), “The Sources and Consequences of the Fluent
Processing of Numbers,” Journal of Marketing Research, 48(2), 327-41.
Kyung and Thomas (2016), “When Remembering Disrupting Knowing: Blocking Implicit Price
Memory,” Journal of Marketing Research, 53(6), 937-53.
Labroo, Aparna A. and Vanessa M. Patrick (2009), “Psychological Distancing: Why Happiness
Helps You See the Big Picture,” Journal of Consumer Research, 35(5), 800-9.
Lee, Angela Y., and Aparna A. Labroo (2004), “The Effect of Conceptual and Perceptual
Fluency on Brand Evaluation,” Journal of Marketing Research, 41(2), 151-65.
McCrea, Sean M., Frank Wieber, and Andrea L. Myers (2012), “Construal Level Mind-sets
Moderate Self-and Social Stereotyping,” Journal of Personality and Social
47
Psychology, 102(1), 51-68.
Mishra, Himanshu, Arul Mishra, and Dhananjay Nayakankuppam (2006), “Money: A Bias for
the Whole,” Journal of Consumer Research, 32(4), 541-49.
Monga, Ashwani and Rajesh Bagchi (2012), “Years, Months, and Days vs. 1, 12, and 365: The
Influence of Units Versus Numbers,” Journal of Consumer Research, 39(1), 185-98.
Monroe, Kent B., and Angela Y. Lee (1999), “Remembering versus Knowing: Issues in Buyers’
Processing of Price Information,” Journal of the Academy of Marketing Science, 27(2), 20725.
Mussweiler, Thomas, and Birte Englich (2005), “Subliminal Anchoring: Judgmental
Consequences and Underlying Mechanisms,” Organizational Behavior and Human Decision
Processes, 98(2), 133-43.
John T. Cacioppo, Petty, Richard E., and Chuan Feng Kao (1984), “The Efficient Assessment of
Need for Cognition,” Journal of Personality Assessment, 48(3), 306-7.
Raghubir, Priya, and Joydeep Srivastava (2009), “The Denomination Effect,” Journal of
Consumer Research, 36(4), 701-13.
Rim, SoYon, James S. Uleman, and Yaacov Trope (2009), “Spontaneous Trait Inference and
Construal Level Theory: Psychological Distance Increases Nonconscious Trait Thinking,”
Journal of Experimental Social Psychology, 45(5), 1088-97.
Roediger, Henry L. (1990), “Implicit Memory: Retention without Remembering,” American
Psychologist, 45 (9), 1043-56.
Schacter, Daniel L. (1987), “Implicit Memory: History and Current Status,” Journal of
Experimental Psychology: Learning, Memory, and Cognition, 13(3), 501-18.
Schunn, Christian D., Lynne M. Reder, Adisack Nhouyvanisvong, Daniel Richards, and Philip J.
48
Stroffolino (1997), “To Calculate or Not to Calculate: A Source Activation Confusion Model
of Problem Familiarity's Role in Strategy Selection,” Journal of Experimental Psychology:
Learning, Memory, and Cognition, 23 (1), 3-29.
Schwarz, Norbert (2004), “Meta-cognitive Experiences in Consumer Judgment and Decision
Making,” Journal of Consumer Psychology, 14(4), 332-48.
Smith, Pamela K., and Yaacov Trope (2006), “You Focus on the Forest When You're in Charge
of the Trees: Power Priming and Abstract Information Processing,” Journal of Personality
and Social Psychology, 90(4), 578-96.
Spiller, Stephen A., Gavan J. Fitzsimons, John G. Lynch, and Gary H. McClelland (2013),
“Spotlights, Floodlights, and the Magic Number Zero: Simple Effects Tests in Moderated
Regression,” Journal of Marketing Research, 50(2), 277-88.
Thomas, Manoj and Vicki G. Morwitz (2009), “Heuristics in Numerical Cognition: Implications
for Pricing,” in Handbook of Research in Pricing.
Trope, Yaacov, and Nira Liberman (2003), “Temporal Construal,” Psychological Review,
110(3), 403-21.
Tversky, Amos and Daniel Kahneman (1974), “Judgment under Uncertainty: Heuristics and
Biases,” Science, 185 (4157), 1124–31.
Unnava, H. Rao, and Deepak Sirdeshmukh (1994), “Reducing Competitive Ad
Interference,” Journal of Marketing Research, 31(3), 403-11.
Vallacher, Robin R., and Daniel M. Wegner (1989), “Levels of Personal Agency: Individual
Variation in Action Identification,” Journal of Personality and Social Psychology, 57(4),
660-71.
Wickens, Delos D., David G. Born, and Charles K. Allen (1963), “Proactive Inhibition and Item
49
Similarity in Short-term Memory,” Journal of Verbal Learning and Verbal Behavior 2(5),
440-45.
Yates, J. K. (2007), Global Engineering and Construction, Hoboken, NJ: John Wiley & Sons.
50
Table 1
Experiment 3 Regression Results: Moderation by Implicit Numerical Association
β
Intercept
Rating Polarity
Implicit Numerical Association
Rating Polarity x Implicit
Numerical Association
Quality Level
Rating Polarity x Quality Level
Implicit Numerical Association x
Quality Level
3-way Interaction
1.15
1.60
0.93
-1.56
SE
0.27
0.38
0.39
0.55
0.74
-0.54
-0.29
0.56
DF
85
85
85
85
t
4.19
4.26
2.40
-2.85
p
<0.01
<0.01
0.02
0.01
0.06
0.08
0.08
1242
1242
1242
13.23
-7.11
-3.70
<0.01
<0.01
<0.01
0.11
1242
5.00
<0.01
PLANNED CONTRASTS
β
SE
DF
t
Planned Contrast: Implicit Numerical Association is Smaller-is-Better
Rating Polarity X Quality Level
-0.54
0.08
1242
-7.11
Planned Contrast: Implicit Numerical Association is Bigger-is-Better or Not Sure
Rating Polarity X Quality Level
0.01
0.08
1242
0.16
p
<0.01
0.87
51
Table 2
Experiments 4A & 4B Regression Results: Moderation by Construal Mindset
Intercept
Simple Effects
Rating Polarity
Quality Level
Mindset*
2-way Interaction Effects
Rating Polarity x Quality Level
Rating Polarity x Mindset
Quality x Mindset
3-way Interaction
Rating Polarity x Quality Level x Mindset
β
0.76
Experiment 4a
SE
t
p
0.16 4.67 <.01
β
0.95
Experiment 4b
SE
t
p
0.12
7.93
<0.01
0.12
0.91
-0.10
0.25
0.04
0.24
0.50
23.11
-0.44
0.62
<.01
0.66
1.01
0.84
-0.06
0.17
0.03
0.03
5.97
27.93
-1.97
<0.01
<0.01
0.05
-0.10
0.76
0.06
0.06
0.34
0.06
-1.71
2.22
0.96
0.09
0.03
0.34
-0.28
0.14
0.01
0.04
0.04
0.01
-6.49
3.68
1.41
<0.01
<0.01
0.16
-0.20
0.08
-2.45
0.01
-0.02
0.01
-2.56
0.01
PLANNED CONTRASTS Experiment 4a
β
SE
t
p
Greater Reliance on Implicit Memory**
-0.31 0.06 -5.27 <.01
Rating Polarity x Quality Level
Lower Reliance on Implicit Memory***
-0.10 0.06 -1.71 0.09
Rating Polarity x Quality Level
β
Experiment 4b
SE
t
p
-0.39
0.06
-6.42
<.01
-0.17
0.06
-2.73
0.01
* More abstract mindset (positive coefficient) indicates greater reliance on implicit memory
** More abstract mindset.
*** More concrete mindset.
52
Table 3
Experiment 5 Regression Results: Moderation by Reliance on Implicit Memory
Intercept
Simple Effects
Rating Polarity
Quality Level
Implicit Memory Reliance
2-way Interaction Effects
Rating Polarity x Quality Level
β
0.139
SE
0.115
t
1.21
p
0.23
0.596
1.057
-0.047
0.161
0.026
0.017
3.7
40.05
-2.85
<.01
<.01
<.01
-0.144
0.037
-3.89
<.01
Rating Polarity x Implicit Memory Reliance
0.024
0.024
1.01
0.31
Quality x Implicit Memory Reliance
3-way Interaction
Rating Polarity x Quality Level x Implicit Memory
Reliance
0.01
0.004
2.73
0.01
-0.011
0.005
-2.00
0.05
SE
t
p
0.052
-4.17
<.01
0.052
-1.33
0.18
PLANNED CONTRASTS
β
Greater Propensity to Use Implicit Memory*
-0.218
Rating Polarity x Quality Level
Lower Propensity to Use Implicit Memory**
-0.07
Rating Polarity x Quality Level
** One STDV above the mean for each memory type.
*** One STDV below the mean for each memory type.
53
FIGURE 1
EXPERIMENT 1 STIMULI: BEFORE AND AFTER PHOTOGRAPHS
54
FIGURE 2
EXPERIMENT 1: EFFECT OF QUALITY LEVEL ON VISUAL PERCEPTION BY RATING
POLARITY (U.S.)
Visual Improvement
90
Low Quality
85
78.3
80
75
High Quality
73.1
68.8
72.8
70
65
60
Consistent Rating
Polarity
(Bigger-is-better)
Inconsistent Rating
Polarity
(Smaller-is-better)
Note: All errors bars represent standard errors
55
FIGURE 3
EXPERIMENT 2A: EFFECT OF QUALITY LEVEL ON PURCHASE BY RATING
POLARITY (U.S.)
Consistent rating polarity (Bigger-is-better)
Inconsistent rating polarity (Smaller-is-better)
Proportion that would purchase
100%
75%
50%
25%
0%
Poor
Adequate
Fair
Quality Level
Good
Very
Good
56
FIGURE 4
EXPERIMENT 3: EFFECT OF QUALITY LEVEL ON PURCHASE INTENTION BY
RATING POLARITY (GERMANY)
5
Purchase Intention
4
3
2
1
Consistent rating polarity (Smaller-is-better)
Inconsistent rating polarity (Bigger-is-better)
Poor
Adequate
Fair
Good
Quality Level
Very
Good
57
FIGURE 5
EXPERIMENT 3: MODERATION BY IMPLICIT NUMERICAL ASSOCIATION
(GERMANY)
B. Weak Smaller-is-better Implicit Association
5
5
4
4
Purchase Intention
Purchase Intention
A. Strong Smaller-is-better Implicit Association
3
2
3
2
Consistent (Smaller-is-better)
Inconsistent (Bigger-is-better)
1
Poor
Adequate
Fair
Quality Level
Good
Consistent (Smaller-is-better)
Inconsistent (Bigger-is-better)
Very
Good
1
Poor
Adequate
Fair
Good
Very
Good
Quality Level
1
When Bigger is Better (and When it is Not): Implicit Bias in Numeric Judgments
ELLIE J. KYUNG
MANOJ THOMAS
ARADHNA KRISHNA*
Web Appendix Contents
•
•
•
•
•
•
•
Web Appendix A: Pretest—Bidding Behavior in an Auction
Web Appendix B: Experiment Stimuli
Web Appendix C: Experiment Results (means, standard deviation, and confidence
intervals; graphs of means and confidence intervals)
Web Appendix D: Ruling out possible Alternative Explanation (means and standard
deviation; analyses details)
Web Appendix E: Analyses with Reaction Time as Covariate
Web Appendix F: Order Effect Analyses
Web Appendix G: Experiment 4B Floodlight Analysis
2
WEB APPENDIX A:
PRETEST—BIDDING BEHAVIOR IN AN AUCTION
In this pretest experiment, we test the rating polarity effect (testing H1) in a sealed-bid
auction that entailed financial consequences. We examined whether rating polarity influenced
participants’ bids. Comprehension, involvement, mood, and need-for-cognition were also
measured to rule out potential alternative explanations.
Method
Participants. Seventy-two undergraduates from a U.S. university (60% female; average
age: 20.5 years) participated in the computer study in exchange for $4. Participants were
randomly assigned to either the consistent (bigger-is-better) or inconsistent (smaller-is-better)
rating polarity condition.
Procedure. Participants were told that they would be offered the opportunity to bid for a
Japanese insulated travel mug that was not available for purchase locally, and that the participant
who placed the winning bid would receive the travel mug. They could use all or some portion of
their $4 payment to bid on the mug. If the winning bid was less than $4, the winner would
receive the difference between the payment and their bid price. They were also told they could
bid more than $4, but that they would personally have to pay the difference between their bid
price and the $4 payment to receive the mug. Participants were asked to confirm [Yes / No] that
they understood the instructions for the bidding process before proceeding.
Before they were shown the product information, participants were told that a quality
rating for the mug would be provided by a reputable consumer welfare agency outside of the
U.S.—an agency known for its evaluation of consumer products. They were asked to carefully
read the meaning of the ratings provided. Those in the consistent rating polarity condition were
told that 1 = unsatisfactory and 7 = very good, while those in the inconsistent rating polarity
condition were told that the rating had an opposite polarity of 1 = very good and 7 =
unsatisfactory. They were asked to confirm that they understood this rating format [Yes / No].
PRETEST EXPERIMENT STIMULI
3
Pre-evaluation Comprehension Test. Participants were then asked the meaning of “1” and
“7” rating poles for the quality ratings [very good, unsatisfactory] before proceeding. If they
responded incorrectly, a message came up on the screen asking them to correct their response.
Participants could proceed only after they had answered these questions correctly. This was done
to ensure that the results did not stem from inattention or miscomprehension of the rating poles.
Bid. Participants were then shown a photograph and descriptive information about the
insulated mug, which included a quality rating (see Web Appendix B). The quality rating was 6.1
in the consistent rating polarity condition (bigger-is-better) and 1.9 in the inconsistent rating
polarity condition (smaller-is-better). Note that the two ratings are normatively identical.
Participants were then asked, “How much would you like to bid on this mug?” and were required
to make a bid, but could make any bid they wished, including $0.
Post-Evaluation Comprehension Test. After entering their bid, participants were asked
what the exterior of the mug was made of [stainless steel or plastic], the country of origin of the
product [Germany or Japan], and the meaning of the rating poles 1 and 7 [very good or
unsatisfactory]. This was done to confirm that the participants did not become confused or forget
about the meaning of the ratings during the bidding process.
Rating Polarity Typicality. Participants were also asked to indicate which of the two
numerical associations is more typical: “Higher Numbers Indicate Better Quality” or “Lower
Numbers Indicate Better Quality.” They could also choose a third option labeled “not sure.”
Additional Measures. To rule out other possible alternative explanations, participants
were also asked questions about their current mood [slider scales anchored at bad / good,
unpleasant / pleasant, negative / positive], given the 18 items from the short form Need For
Cognition (NFC) scale (Cacciopo, Petty, and Kao 1984), asked how involved / thoughtful /
attentive they were during the study [slider scales anchored at not at all / very], and asked
questions about their demographics.
Results
Post-Evaluation Comprehension Test. All participants answered both of the questions on
the meaning of the rating poles correctly. Thus, they were neither confused about, nor did they
forget, the meaning of the poles.
Monetary Bid. A one-way ANOVA revealed that, as predicted, participants in the
consistent rating polarity condition bid more on the insulated travel mug (M = $4.42) than those
in the inconsistent rating polarity condition (M = $2.70, F (1, 70) = 4.80, p = .03, see figure 1).
Bid on Mug ($) FIGURE 1: BID VALUES BY RATING POLARITY
$6 $4 $4.42 $2.70 $2 $0 Consistent ra2ng polarity Inconsistent ra2ng polarity (Bigger-­‐is-­‐be;er) (Smaller-­‐is-­‐be;er) 4
Web Appendix C summarizes the means and 95% confidence intervals by conditions across all
experiments, and for further data visualization, also plots the means and 95% confidence
intervals for the consistent versus inconsistent rating polarity conditions.
Rating Polarity Typicality. The majority of participants indicated that the bigger-is-better
numerical association is more typical (90.3%) than the smaller-is-better one (9.7%), and thus the
dominant numerical association in implicit memory.
Ruling Out Alternative Explanations. Mood: To rule out the possibility that an
inconsistent rating polarity (smaller-is-better) might have put participants in a more negative
mood and influenced their information processing, we averaged three mood measures into an
index (α = .92) and conducted a one-way ANOVA, which revealed no difference in mood by
condition (p = .42). Involvement: Similarly, to rule out the possibility that participants might
have been more involved when attempting to use a rating format with inconsistent rating
polarity, we averaged the three measures of involvement into an index (α = .87), and a one-way
ANOVA revealed no difference in involvement by condition (p = .60). Need-for-cognition: We
propose that the rating polarity effect manifests because participants are not aware of, and
therefore cannot control for, its effects. Previous research has shown that individuals who are
higher NFC are more systematic in their judgments (Florack and Scarabis 2001; Conner et al.
2007) because they are more thoughtful and reflective (Ajezen and Fishbein 2005)—for
example, they are less likely to make judgments in line with negative stereotypes. However, if
the rating polarity effect operates outside of people’s awareness, then high NFC individuals
should not be able to overcome the effect of interference from implicit numerical associations in
memory.
We conducted a linear regression where the independent measures were rating polarity
(dummy coded: consistent = 0, inconsistent = 1), the mean centered NFC score, and the
interaction of the two with the bid as the dependent measure. The results revealed a significant
effect of rating polarity (β = -1.84, p = .03), but the effects of NFC (p = .29) and the interaction
between NFC and rating polarity (p = .66) were not significant. This suggests that the rating
polarity effect manifests for people with both low and high NFC and provides some evidence
that the spontaneous interference of implicit memory is non-conscious because high NFC
participants cannot adjust for it.
The means and standard deviations for these additional variables (mood, involvement,
NFC) across all experiments are in Web Appendix D. These measures also show that participant
involvement and attention are relatively high through all of the experiments, across experimental
conditions, and low involvement or attention cannot account for the results.
5
WEB APPENDIX B
EXPERIMENT 1 STIMULI
6
EXPERIMENT 2B STIMULI
7
EXPERIMENT 4A STIMULI
‘WHY’ MINDSET MANIPULATION
8
‘HOW’ MINDSET MANIPULATION
9
EXPERIMENT 5 STIMULI
TRUE / FALSE TASK
RECALL TASK
10
WEB APPENDIX C: EXPERIMENT RESULTS
MEANS, STANDARD DEVIATIONS, AND + / - 95% CONFIDENCE INTERVAL
BIB
Quality
Level
Mean
SD
6.10
4.42
4.12
SIB
Confidence
Interval
Quality
Mean
Level
Lower- UpperBound Bound
PRETEST EXPERIMENT
3.04
5.79
1.90
SD
Confidence
Interval
Lower- UpperBound Bound
2.70
2.18
1.95
3.45
EXPERIMENT 1
Low
68.77
16.88
63.86
73.68
1
73.08
13.92
68.55
77.61
High
78.28
13.97
73.48
83.07
2
72.80
17.27
68.12
77.48
EXPERIMENT 2A
1
5.0%
1.2%
2.6%
7.4%
1
8.6%
1.6%
5.4%
11.8%
2
7.2%
1.5%
4.4%
10.1%
2
14.2%
2.0%
10.2%
18.1%
3
39.0%
2.7%
33.6%
44.4%
3
20.5%
2.3%
15.9%
25.0%
4
83.0%
2.1%
78.9%
87.2%
4
72.9%
2.6%
67.9%
78.0%
5
86.8%
1.9%
83.1%
90.5%
5
80.9%
2.3%
76.4%
85.3%
EXPERIMENT 2B
1
0.73
0.82
0.59
0.88
1
1.00
1.33
0.86
1.14
2
0.97
0.79
0.83
1.12
2
1.24
1.24
1.09
1.38
3
1.72
0.94
1.57
1.86
3
1.57
0.85
1.43
1.72
4
2.33
1.08
2.19
2.47
4
2.33
0.92
2.18
2.47
5
2.75
1.34
2.61
2.90
5
2.67
1.05
2.53
2.81
EXPERIMENT 3
1
2.43
1.71
0.15
2.14
1
3.05
1.83
0.16
2.74
2
2.70
1.60
0.14
2.43
2
2.89
1.68
0.14
2.60
3
2.90
1.31
0.11
2.68
3
3.10
1.58
0.14
2.83
4
4.36
1.83
0.16
4.05
4
3.88
1.53
0.13
3.62
5
4.57
1.97
0.17
4.23
5
4.12
1.89
0.16
3.80
11
MEANS, STANDARD DEVIATIONS, AND + / - 95% CONFIDENCE INTERVAL
CONTINUED
BIB
Quality
Level
Mean
SD
SIB
Confidence
Interval
Quality
Level
Lower- UpperBound Bound
EXPERIMENT 4A
Mean
SD
Confidence
Interval
Lower- UpperBound Bound
CONCRETE
1
1.83
1.35
1.60
2.05
1
1.89
1.61
1.58
2.20
2
2.36
1.38
2.12
2.59
2
2.29
1.39
2.02
2.55
3
3.43
1.34
3.21
3.66
3
3.23
1.36
2.97
3.49
4
4.59
1.65
4.32
4.87
4
4.16
1.79
3.82
4.50
5
5.26
1.75
4.97
5.56
5
5.00
1.93
4.63
5.37
ABSTRACT
1
1.82
1.29
1.59
2.05
1
2.35
1.91
2.02
2.68
2
2.25
1.20
2.03
2.46
2
2.74
1.75
2.43
3.04
3
3.52
1.32
3.29
3.76
3
3.43
1.42
3.18
3.67
4
4.84
1.61
4.56
5.13
4
4.18
1.74
3.87
4.48
5
5.35
1.75
5.04
5.66
5
4.93
2.07
4.57
5.29
EXPERIMENT 4B
1
2.06
1.57
1.88
2.24
1
2.60
2.02
2.37
2.83
2
2.24
1.43
2.08
2.40
2
3.02
1.86
2.81
3.23
3
3.33
1.52
3.16
3.51
3
3.51
1.62
3.33
3.69
4
4.60
1.66
4.42
4.79
4
4.27
2.02
4.04
4.50
5
5.11
1.81
4.90
5.31
5
4.83
2.13
4.59
5.08
EXPERIMENT 5
1
1.45
1.12
1.32
1.58
1
1.82
1.50
1.65
1.99
2
1.87
1.30
1.72
2.02
2
2.33
1.38
2.17
2.48
3
3.22
1.36
3.07
3.38
3
3.34
1.23
3.20
3.48
4
4.73
1.75
4.53
4.93
4
4.60
1.90
4.38
4.81
5
5.29
1.99
5.06
5.52
5
5.24
2.03
5.01
5.47
12
MEAN AND CONFIDENCE INTERVAL GRAPHS
Mean + / -­‐ 95% Confidence Interval 7 Pretest 6 5 4 3 2 Consistent Ra2ng Polarity (BIB) Inconsistent Ra2ng Polarity (SIB) 1 Mean + / -­‐ 95% Confidence Interval Each graph plots the mean evaluation + / - the 95% confidence interval for the consistent and
inconsistent rating polarity conditions. (BIB = bigger-is-better; SIB = smaller-is-better)
85 80 75 70 65 60 0 75% 50% 25% 0% -­‐25% 0 2 4 Consistent Ra2ng Polarity (BIB) Inconsistent Ra2ng Polarity (SIB) Quality Level Low Quality High Quality Quality Level 6 Exp 2a Exp 3 Mean + / -­‐ 95% Confidence Interval Mean + / -­‐ 95% Confidence Interval 100% Exp 1 5 4 3 2 Inconsistent Ra2ng Polarity (BIB) Consistent Ra2ng Polarity (SIB) 1 0 2 Quality Level 4 13
6 5 4 3 2 Consistent Ra2ng Polarity (BIB) Inconsistent Ra2ng Polarity (SIB) 1 0 1 3 4 Exp 4a: Concrete 5 4 3 2 Consistent Ra2ng Polarity (BIB) Inconsistent Ra2ng Polarity (SIB) 1 5 0 1 2 3 Quality Level 4 5 6 6 Mean + / -­‐ 95% Confidence Interval 2 Quality Level Mean + / -­‐ 95% Confidence Interval Exp 4a: Abstract Exp 4b Consistent Ra2ng Polarity (BIB) Inconsistent Ra2ng Polarity (SIB) 5 4 3 2 Mean + / -­‐ 95% Confidence Interval Mean + / -­‐ 95% Confidence Interval 6 Exp 5 Consistent Ra2ng Polarity (BIB) Inconsistent Ra2ng Polarity (SIB) 5 4 3 2 1 1 0 1 2 3 4 Quality Level 5 0 1 2 3 4 5 Quality Level 14
WEB APPENDIX D: RULING OUT ALTERNATIVE EXPLANATIONS
MEANS AND STDV OF POSSIBLE ALTERNATIVE EXPLANATIONS
BIB
Measure
Mood1
Involvement1
NFC
Mood1
NFC
Mood1
NFC
Mood1
NFC
Mood2
Involvement
Mood2
Involvement
Mood
Involvement
Mood2
Involvement
Mood2
Involvement
Mood2
Involvement
Mood2
Involvement
Mood2
Involvement
1
Mean
SD
PRETEST
63.86
14.62
66.32
14.06
3.70
0.46
EXPERIMENT 1
70.40
17.09
3.55
0.54
Good Product
72.14
15.52
3.58
0.51
Bad Product
68.57
18.63
3.51
0.57
EXPERIMENT 2A
5.81
1.12
4.61
0.50
EXPERIMENT 2B
5.74
1.40
4.60
0.51
EXPERIMENT 3
3.99
0.98
4.14
0.74
EXPERIMENT 4A
4.01
0.85
4.54
0.62
Concrete Mindset
4.01
0.74
4.45
0.74
Abstract Mindset
4.01
0.98
4.63
0.44
EXPERIMENT 4B
4.62
0.94
3.93
0.60
EXPERIMENT 5
4.33
1.68
4.82
0.38
SIB
Mean
SD
60.72
64.33
3.64
17.80
17.40
0.57
70.95
3.58
16.51
0.51
68.70
3.52
15.29
0.44
73.01
3.63
17.46
0.57
5.43
4.55
1.27
0.66
5.40
4.67
1.38
0.48
4.13
4.32
0.75
0.63
4.06
4.53
0.74
0.49
4.03
4.52
0.72
0.51
4.09
4.54
0.76
0.49
4.75
3.94
0.98
0.69
4.54
4.79
1.81
0.39
100-point scale, 27-point scale, all others 5-point scale
15
ANALYSIS DETAILS RULING OUT POSSIBLE ALTERNATIVE EXPLANATIONS
Experiments 2a & 2b
Mood: We averaged mood measures into an index (2a: α = .97; 2b: α = .97) and
conducted an ANOVA with rating polarity as the dependent measure. This analysis for
experiment 2a revealed a marginally significant effect of rating polarity where those in the
consistent rating polarity reported slightly more positive mood (M = 5.75) than those in the
inconsistent rating polarity condition (M = 5.40, F(1, 205) = 3.26, p = .07). In order to rule out
the possibility that mood could account for the results, we conducted the analysis using mood as
an additional independent variable. The effect of mood was not significant (β = .11, p = .52) and
the rating polarity x quality level interaction remained significant (β = -.46, p = .01), ruling out
the possibility that mood might account for the results. This analysis for experiment 2b revealed
a significant effect of rating polarity where those in the consistent rating polarity reported
slightly more positive mood (M = 5.82) than those in the inconsistent rating polarity condition
(M = 5.43, F(1, 216) = 5.59, p = .02). In order to rule out the possibility that mood could account
for the results, we conducted the willingness-to-pay analysis using mood as an additional
independent variable. The effect of mood was not significant (β = .07, p = .92) and the rating
polarity x quality level interaction remained significant (β = -.24, p < .01), ruling out the
possibility that mood might account for the results. Involvement: Similarly, to rule out the
possibility that participants might have been more involved when attempting to use a rating
format with inconsistent rating polarity, we averaged the measures of involvement into an index
(2a: α = .92; 2b: α = .95), and an ANOVA revealed no difference in involvement by condition
(2a: p = .30; 2b: p = .44).
16
Experiment 3
Mood: We averaged three mood measures into an index (α = .93) and a PROC GLM
analysis that revealed no difference in mood by condition (p = .46). Involvement: Similarly, we
averaged the measures of involvement into an index (α = .90), and the same PROC GLM
analysis revealed no difference in involvement by condition (p = .22).
Experiment 4a
Mood: The three mood measures were averaged together (α = .90). A regression with
rating polarity, mindset, and their interaction as independent measures and the averaged mood
measure as the dependent measure confirmed that there was no significant effect of rating
polarity, mindset, or their interaction on mood (all ps >.68). Involvement: Similarly, the four
measures of participant involvement were averaged into an index (α = .93), and a regression with
the same independent measures and the involvement index as a dependent measure confirmed
there was no significant effect of rating polarity, mindset, or their interaction on involvement and
attention to the evaluation task (all ps > .23).
Experiment 4b
Mood: The mood measures were averaged together (α = .82). A regression with rating
polarity, mindset, and their interaction as independent measures and the averaged mood measure
as the dependent measure confirmed that there was no significant effect of rating polarity,
mindset, or their interaction on mood (all ps >.29). Involvement: Similarly, the four measures of
participant involvement were averaged into an index (α = .90), and a regression with the same
independent measures and the involvement index as the dependent measure confirmed there was
no significant effect of rating polarity, mindset, or their interaction on involvement and attention
17
to the evaluation task (all ps > .79).
Experiment 5
Mood: The three mood measures were averaged together (α = .98). A regression with
rating polarity, memory type, and their interaction as independent measures and the averaged
mood measure as the dependent measure confirmed that there was no significant effect of rating
polarity, memory, or their interaction on mood (all ps > .40). Involvement: Similarly, the
measures of participant involvement were averaged into an index (α = .78), and a regression with
the same independent measures and the involvement index as a dependent measure confirmed
there was no significant effect of rating polarity, memory type, or their interaction on
involvement with the evaluation task (all ps > .35).
18
WEB APPENDIX E: ANALYSIS WITH REACTION TIME AS COVARIATE
Analysis for each experiment was conducted as a regression (paradigm pretest, experiments 1) or
repeated measures regression (remaining experiments) with the evaluation as the dependent measure and
log reaction time as a covariate. Rating polarity was dummy coded (0 = consistent rating polarity, 1 =
inconsistent rating polarity).
Pretest
Experiment 1
Experiment 2a
Experiment 2b
Experiment 3
Experiment 4a
Experiment 4b
Experiment 5
Effect
Rating Polarity
Reaction Time
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Reaction Time
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Reaction Time
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Reaction Time
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Reaction Time
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Construal Mindset
Rating Polarity x Construal Mindset
Quality Level x Construal Mindset
3-way interaction
Reaction Time
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Construal Mindset
Rating Polarity x Construal Mindset
Quality Level x Construal Mindset
3-way interaction
Reaction Time
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Implicit Memory
Rating Polarity x Implicit Memory
Quality Level x Implicit Memory
3-way interaction
Reaction Time
β
-1.77
0.86
4.56
9.70
-10.00
-1.44
0.70
1.47
-0.33
0.04
0.35
0.54
-0.10
0.05
0.87
0.59
-0.28
0.19
0.14
0.91
-0.11
-0.11
0.75
0.06
-0.20
-0.04
1.02
0.84
-0.28
-0.06
0.14
0.01
-0.03
-0.05
0.61
1.06
-0.15
-0.03
0.01
0.01
-0.01
0.02
SE
0.77
0.47
3.42
3.51
4.82
2.79
0.41
0.08
0.12
0.08
0.11
0.02
0.02
0.03
0.27
0.04
0.06
0.08
0.25
0.04
0.06
0.24
0.35
0.06
0.08
0.05
0.17
0.03
0.04
0.03
0.04
0.01
0.01
0.04
0.16
0.03
0.04
0.02
0.03
0.00
0.01
0.04
DF
69
69
164
164
164
164
202
3095
3095
3095
215
3011
3011
3011
87
1243
1243
1243
162
2319
2319
162
162
2319
2319
2319
197
2800
2800
197
197
2800
2800
2800
192
2719
2719
192
192
2719
2719
2719
t/Z
-2.30
1.84
1.33
2.77
-2.07
-0.52
1.69
17.67
-2.70
-.052
3.27
34.24
-4.53
1.69
3.22
14.78
-4.95
2.53
0.55
23.06
-1.76
-0.46
2.17
0.97
-2.40
-0.85
5.97
27.88
-6.49
-1.99
3.71
1.53
-2.66
-1.18
3.76
40.24
-3.94
-1.90
0.59
1.53
-1.59
0.63
p
0.02
0.07
0.19
0.01
0.04
0.61
0.09
<0.01
<0.01
0.61
<0.01
<0.01
<0.01
0.09
<0.01
<0.01
<0.01
0.01
0.58
<.01
0.08
0.64
0.03
0.33
0.02
0.40
<.01
<.01
<.01
0.05
0.00
0.13
0.01
0.24
0.00
<.01
<.01
0.06
0.55
0.13
0.11
0.53
19
WEB APPENDIX F: ORDER EFFECTS ANALYSES
Rating Polarity x Quality Level x Order Effects are highlighted in gray.
Order was coded as 1 to 15 for each of the 15 products in experiments 2a, 2b, 3, 4a, 4b, and 5
and mean centered for the analysis.
Exp 2a
Exp 2b
Exp 3
Exp 4a
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Order
Rating Polarity x Order
Quality Level x Order
3-way Interaction
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Order
Rating Polarity x Order
Quality Level x Order
3-way Interaction
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Order
Rating Polarity x Order
Quality Level x Order
3-way Interaction
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Mindset
Rating Polarity x Mindset
Quality Level x Mindset
Rating Polarity x Quality Level x Mindset
Order
Rating Polarity x Order
Quality Level x Order
Rating Polarity x Quality Level x Order
Mindset x Order
Rating Polarity x Mindset x Order
Quality Level x Mindset x Order
Rating Polarity x Quality Level x Mindset x Order
β
0.74
1.49
-0.34
-0.18
-0.01
0.06
-0.01
0.36
0.54
-0.10
0.0004
-0.02
0.001
0.003
0.84
0.59
-0.28
-0.05
0.02
0.02
-0.01
0.88
0.96
-0.30
0.08
-0.68
-0.05
0.17
-0.03
0.01
0.01
-0.01
0.06
-0.16
-0.02
0.06
SE
0.43
0.09
0.13
0.06
0.09
0.02
0.03
0.11
0.02
0.02
0.01
0.02
0.00
0.01
0.27
0.04
0.06
0.03
0.05
0.01
0.01
0.24
0.04
0.06
0.24
0.35
0.06
0.08
0.03
0.04
0.01
0.01
0.04
0.07
0.01
0.02
DF
204
3095
3095
3095
3095
3095
3095
217
3060
3060
3060
3060
3060
3060
87
1240
1240
1240
1240
1240
1240
163
2326
2326
163
163
2326
2326
2326
2326
2326
2326
2326
2326
2326
2326
t
1.73
16.82
-2.70
-2.95
-0.17
3.29
-0.31
3.40
34.28
-4.48
0.03
-0.94
0.15
0.66
3.06
14.59
-4.88
-1.54
0.47
1.70
-0.76
3.64
23.07
-5.14
0.35
-1.96
-0.87
2.06
-0.86
0.25
1.39
-0.77
1.38
-2.52
-1.28
2.88
p
0.08
<0.01
<0.01
<0.01
0.87
<0.01
0.76
<0.01
<0.01
<0.01
0.97
0.35
0.88
0.51
<0.01
<0.01
<0.01
0.12
0.64
0.09
0.45
<0.01
<0.01
<0.01
0.72
0.05
0.39
0.04
0.39
0.80
0.16
0.44
0.17
0.01
0.20
<0.01
20
WEB APPENDIX F: ORDER EFFECTS ANALYSES
Exp 4b
Exp 5
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Mindset
Rating Polarity x Mindset
Quality Level x Mindset
Rating Polarity x Quality Level x Mindset
Order
Rating Polarity x Order
Quality Level x Order
Rating Polarity x Quality Level x Order
Mindset x Order
Rating Polarity x Mindset x Order
Quality Level x Mindset x Order
Rating Polarity x Quality Level x Mindset x
Order
Rating Polarity
Quality Level
Rating Polarity x Quality Level
Implicit Memory
Rating Polarity x Implicit Memory
Quality Level x Implicit Memory
Rating Polarity x Quality Level x Implicit
Memory
Order
Rating Polarity x Order
Quality Level x Order
Rating Polarity x Quality Level x Order
Implicit Memory x Order
Rating Polarity x Implicit Memory x Order
Quality Level x Implicit Memory x Order
Rating Polarity x Quality Level x Judge x Order
CONTRASTS
High Reliance on Implicit Memory: Rating
Polarity x Quality Level
Low Reliance on
Implicit Memory: Rating Polarity x Quality Level
β
1.00
0.83
-0.27
-0.05
0.14
0.01
-0.03
-0.04
0.03
0.03
-0.01
0.002
-0.01
-0.002
SE
0.17
0.03
0.04
0.03
0.04
0.01
0.01
0.02
0.03
0.01
0.01
0.01
0.01
0.002
DF
197
2802
2802
197
197
2802
2802
2802
2802
2802
2802
2802
2802
2802
t
5.89
27.42
-6.37
-1.92
3.70
1.37
-2.65
-1.78
0.92
3.97
-1.29
0.38
-0.69
-1.12
p
<0.01
<0.01
<0.01
0.06
<0.01
0.17
<0.01
0.08
0.36
<0.01
0.20
0.71
0.49
0.26
0.004
0.002
2802
1.62
0.11
0.53
1.04
-0.12
-0.05
0.02
0.01
0.17
0.03
0.04
0.02
0.02
0.00
193
2549
2549
193
193
2549
3.14
36.48
-2.95
-2.78
0.95
2.40
<0.01
<0.01
<0.01
<0.01
0.35
0.02
-0.01
0.01
2549
-1.71
0.09
-0.01
-0.03
0.01
-0.001
0.004
0.001
-0.001
-0.0005
0.02
0.03
0.01
0.01
0.003
0.005
0.001
0.001
2549
2549
2549
2549
2549
2549
2549
2549
-0.27
-1.01
1.75
-0.14
1.19
0.24
-0.80
-0.32
0.78
0.31
0.08
0.89
0.23
0.81
0.42
0.75
-0.19
0.06
2549
-3.29
<0.01
-0.05
0.06
2549
-0.87
0.39
21
WEB APPENDIX G: EXPERIMENT 4B FLOODLIGHT ANALYSIS
DIFFERENCE IN THE RATING POLARITY x QUALITY LEVEL INTERACTION ACROSS
BIF VALUES
BIF Value
(0=most concrete,
24=most abstract)
Interference from Rating Lower 95% Upper 95%
Polarity Effect
Confidence Confidence
(Rating Polarity x Quality
Interval
Interval
Level Interaction Coefficient)
t(2810)
p
0
0.02
-0.22
0.26
0.13
0.89
2
-0.03
-0.24
0.17
-0.31
0.76
4
6
8
10
-0.08
-0.25
0.09
-0.91
0.36
-0.13
-0.27
0.01
-1.79
0.07
-0.18
-0.29
-0.06
-3.05
0.002
-0.23
-0.32
-0.13
-4.76
<.0001
12
-0.27
-0.36
-0.19
-6.41
<.0001
14
16
18
20
-0.32
-0.41
-0.23
-7.00
<.0001
-0.37
-0.48
-0.26
-6.62
<.0001
-0.42
-0.56
-0.28
-6.01
<.0001
-0.47
-0.64
-0.30
-5.47
<.0001
22
-0.52
-0.72
-0.32
-5.05
<.0001
24
-0.56
-0.80
-0.33
-4.72
<.0001
Floodlight analysis follows procedures outlined in Spiller et al. (2013).