Experimental Methods in Nonmarket
Valuation
Kyrre Rickertsen
Based on Book Chapter with Frode Alfnes
Workshop in National Chung Cheng University in Chia-Yi, Taiwan
March 8, 2011
2111
2005
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Outline Presentation
 Introduction
 Incentive compatible (IC) valuation mechanisms
 Validity of bid and choices
– Laboratory versus field experiments
– Internal validity (design and other issues)
– External validity – validity of results outside the laboratory
 Some empirical results
 Recommendations
2
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Introduction 1
 When (high-quality) market data are available, they are usually
preferred to experimental data
 Market data are frequently unavailable
– New product or product with new characteristics
• In peak year 1995, 16,900 food and beverage products
introduced in the US (Nestle 2002)
• Controversial technologies used in production
– Growth hormones, GMO, irradiation, and cloning
– New labeling or information
• Nutritional, organic, and sustainable labeling
3
– Maybe little variability in market data (e.g., prices)
– Effects of socioeconomic variables?
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Introduction 2
 Alternatives are:
– Stated preference data
• Can ask about nonexistent products BUT
– Lack economic incentives
– Hypothetical bias (e.g., List and Gallet 2001)
– Experimental data (hypothetical market data)
 Experiments also used to develop economic theory, for example:
– WTP – WTA disparity (e.g., Shogren et al. 1994 - AER)
– Preference reversals (e.g., List 2002 - AER)
4
– Coherent arbitrariness (Ariely et al. 2003 – QJE)
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Introduction 3
 In experimental valuation studies, participants make bids or
choices with real products and real money
 Experiments used for a long time in marketing:
– As a method of research, controlled experimentation is
universally used among scientists. They conduct experiments on
a small scale to discover facts from which overall conclusions
may be reached. In the gradual evolution of marketing research
from an art to a science, some practitioners have endeavored to
employ experimental methods where possible. William
Applebaum and Richard F. Spears (1950: 505)
– The first scientific article describing a test marketing
experiment is Ginzberg (1936) in AER.
5
– Test marketing was the preferred method
• New products or marketing strategies tested in a few stores
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Introduction 4
 Applied economists started to use experimental methods around
1990 to study the value of quality attributes of foods (e.g.,
Menkhaus et al. 1992 – JARE; Buhr et al. 1993 – JARE)
– Little reference to previous marketing literature
– Focus on WTP while previous marketing literature focused on
market shares
– This line of research is in the focus of this presentation
6
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
IC Valuation Mechanisms 1
 A mechanism is incentive compatible (IC) given that the utility of
the participant cannot be increased by submitting a different bid or
making a different choice
 IC mechanisms include (complete listing see table 1):
– Sealed bid auctions
• 2nd price, nth price, random nth price
– Becker-DeGroot-Marschak (BDM) mechanism
– Price list experiments
– Real choice (RC) experiments
 Non-IC mechanisms include:
7
– Dutch and first-price auction
– The price paid is not independent of the winner’s bid
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
IC Valuation Mechanisms 2
Sealed-Bid Auctions with Endogenous Market Prices
 2nd price (Vickrey 1961)
– Participants submit bids and one unit of the good is sold to the
highest bidder for a price equal to second-highest bid
– Two problems
• May not engage low-value bidders (especially in a multiple
trial setting with posted prices)
• Some may see auction as competition with one winner
 A random nth price auction where n-1 units of the good is sold for
the nth highest bid
– Engage high- and low-value bidders
8
– n -1 winners reduce any auction winning utilities
– Lottery effect?
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
IC Valuation Mechanisms 3
Sealed-Bid Auctions with Endogenous Market Prices
 In the endowment approach, the participants are endowed with a
product and bid for an upgrade (Hayes et al. 1995)
– The bid reflects the difference in value between the two alternatives
– The bid (and thereby the value difference) may be affected by all sorts
of anchors
 In full bidding approach, the participants bid on two (or more)
products and one is randomly chosen for sale (Hoffmann et al.
1993)
– The difference in bids reflects the difference in value between the two
alternatives
– Bid differences less affected by anchors
9
 See Lusk and Shogren (2007: 95-112) and Alfnes (2009) for
detailed discussions
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
IC Valuation Mechanisms 4
Becker-DeGroot-Marschak (BDM) Mechanism
 No auction but strategically equivalent to IC auctions
 Each participant submits a sealed bid
 Sales price is randomly drawn from a distribution from zero to a
price that is higher than the anticipated max bid
 If bid higher than price, then the participant purchases one unit for
the drawn price
 Can be conducted with only one participant
– Easy to do at a grocery store
 Lottery element?
10
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
IC Valuation Mechanisms 5
Price List Experiments
 a) Real dichotomous choice
– Offer the participant to buy a commodity for a price. If yes, sale
is completed
 b) Multiple price list (payment card method)
– A list of questions on the form
• “At a price of $8.75 I will buy _____ I will not buy ____”
• One row is implemented as the final sale
 c) Open-ended choice experiment
– How many units do you want to purchase at different prices?
– Can calculate demand curves
11
– But people tend to stock up on storable products at low prices
and the interpretation of the demand curve becomes difficult
www.umb.no
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Department of Economics and Resource Management
IC Valuation Mechanisms 6
Real Choice Experiments
12
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
IC Valuation Mechanisms 7
Real Choice Experiments
 Extension of stated choice experiment
– But your choices have real economic consequences
– Need to have the product
 Participants make choices in a series of scenarios
 Product attributes (including prices) vary between scenarios
 One scenario randomly drawn as binding
 Advantage: The choice task is similar to choices made in a food store
 Weakness: Individual WTP is not observable
– WTP has to be estimated based on choices of all participants
– Estimated WTP for each participants affected by the choices of other
participants
13
– Estimated WTP for each participants affected by model specification
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
IC Valuation Mechanisms 8
Comparisons of Mechanisms
 In theory, sealed bid auctions and BDM should result in identical
bids and bid differences
– In practice, not always the case
 Induced value experiments
– Sealed-bid auctions produce more accurate bids than BDM
(Noussair et al. 2004; Lusk and Rousu 2006)
 Homegrown value experiments
– Impossible to test whether bids equal homegrown values
(private and unknown to experimenter)
– Many comparisons with mixed results (see paper)
14
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
IC Valuation Mechanisms 9
Choice of Mechanism
 Choice of mechanism a trade-off
– Auctions perform better than BDM in induced value
experiments
– 2nd price auction perform well for high-value bidders while
random nth price performs better for low-value bidders
• Recently most studies use n > 2
– BDM easy to implement on individual basis and in the field
– RC experiments resembles choices in grocery stores but do not
give individual WTP values
15
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Validity of Bid and Choices
Laboratory versus Field Experiments
 In marketing literature
– Laboratory experiments are conducted in a laboratory setting
with a high degree of control of external factors
– Field experiments are conducted in an actual market place with
less control but a familiar context for the specific choice
 Harrison and List (2004) have a refined categorization depending
on
– Students versus general population
– Field goods versus induced values
– Field context versus laboratory
16
– Subjects know or do not know that they participate in an
experiment
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Validity of Bid and Choices
Internal Validity 1
 Internal validity:
– The ability to demonstrate that observed correlations are causal (Roe
and Just 2009)
– Does the experiment control for alternative explanations?
 Some factors affecting the internal validity
– Training is important (Plott and Zeiler 2005)
• Most IC mechanisms are unfamiliar and small differences may
affect the bids as discussed above (2nd vs. nth price)
– Presence of field substitutes (Harrison et al. 2004)
• Bids are truncated by prices of field substitutes
• But is cod the only field substitute to cod?
17
– Or is fish in general or maybe meat the field substitute?
• Bid differences may be more robust than bid levels
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Validity of Bid and Choices
Internal Validity 2
 Bid affiliation and posted prices in multiple-round auctions
 In induced value experiments, it has been shown that it takes several
rounds to reach the theoretical WTP
 What about homegrown value experiments? Increasing prices typically
observed
– We have bid affiliation when bids in later rounds are affected by
posted prices in earlier rounds
• Some participants may try to buy at low prices and discover that it
does not work, i.e., “market discovery”
• Some participants may take high posted prices as a signal of
quality and revise their bids, i.e., “preference learning”
18
• Some participants may consider the auction as one competition,
i.e., “auction winning utilities”
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Validity of Bid and Choices
Internal Validity 3
 No easy solution to bid affiliation
 Problem is the interpretation of the bids (and the internal validity)
 One-shot auctions with training with a different product is
recommended by Harrison et al. (2004)
– Do we find the “correct” WTP for unfamiliar food products in
the first shot?
– Does it solve the problem to use a different training product?
• Anchors
• Incidental prices
• Coherent arbitrariness
19
 Recent papers seem to use few but more than one round
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Validity of Bid and Choices
Internal Validity 4
 Participants use numbers that are presented to them as anchors
 Examples on anchors
– Reference price of field substitute
– Distribution of prices in price list experiments
– The limits of the price distribution in BDM
– Bids in training sessions
– Posted prices
20
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Validity of Bid and Choices
Internal Validity 5
 Arbitrary numbers can also serve as anchors
– Nunes and Boatwright (2004) found that exposing the participants to
prices that were completely unrelated to the goods in an auction
(incidental prices) had substantial effects on the bids (e.g., sweatshirt
price affected the bids for CDs)
– Ariely et al. (2003) found that absolute bids on ordinary products (like
computer keyboards) were affected by random numbers such as the
participants’ social security numbers
• Participants were asked to write down their social security
number before bidding. High numbers resulted in high bids in the
following auction (arbitrariness).
• The relative valuations were not affected (coherent)
21
 Given random anchors, we recommended to use the full bidding
approach and relative WTP (or alternatively bid differences when many
zero bids)
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Validity of Bid and Choices
External Validity 1
 External validity
– The ability to generalize the relationships found in a study to other
contexts (Roe and Just 2009). For example, from lab to market
– Mixed results; see paper
 Several factors may reduce the external validity
 Unfamiliar environment:
– People may behave different in a laboratory than a retail setting (Levitt
and List 2007)
– People are not in laboratory because they need to buy food
– The time and day of the experiment are decided by researcher
22
• Corrigan and Rousu (2008) found that people that intended to buy
bananas on the day of the experiment bid more, i.e., regular
buyers may behave as non-buyers in an experiment
– Experiments in a store setting may be preferred but lack of control and
limited possibilities to trainwww.umb.no
participants
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Validity of Bid and Choices
External Validity 2
 Nature of decision task different in store and experiment
– Only a small number of products
– A lot of focus on the (few) products
– Can typically only buy one unit of one product
– May have to bid for products instead of choosing among them
– Random draws may resemble lotteries
 Representative samples
– Self-selected students may behave differently than representative
consumers
• Low age, low income, more education, strong beliefs
– Nonstudent samples may be unrepresentative outside sample area
23
– Users may have different preferences than the general population
• For many food products, 10% of users buy almost everything
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Validity of Bid and Choices
External Validity 3
 Information
– In experiments there is usually a focus on a few products and,
consequently, few prices. Information is typically provided (e.g.,
PowerPoint etc).
• Participants encouraged to make “rational” decisions using the
information
• Small samples encourage the use of within- rather than betweensample tests
– Within-sample tests draw attention to the treatment and gives
larger effects than between-sample tests (Johansson-Stenman
and Svedsäter 2008)
24
– In food store many products, less emphasis on information about
the product and many prices
• Many purchases based on habits
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Validity of Bid and Choices
External Validity 4
 Presence of researchers
– People may change behavior when they are scrutinized
• Behave like they believe the experimenter want
• Give a socially desirable impression
– Especially for product attributes with dimensions such
as animal friendly, “fair trade”, GMO, etc
 Repeated purchase
– People may bid high because they want to try the product
(preference learning)
25
– A high premium in the laboratory does not necessarily imply
repeated purchase of the good
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Examples on Empirical Results
 Food marketing issues: Packing technologies, use of insecticides, use of
different types of feed, grading for tenderness, labels etc
– Segments of consumers with different preferences
– Typically some increased mean WTP for new product
• Do not investigate the profitability of the new product
 Controversial technologies: GMO, hormone treatment, irradiation
– Segments of consumers with different preferences
– US participants more positive than European
 Effects of information
– Moderate effects of scientifically balanced information
– Stronger effects of information from environmental groups, etc
26
– Effects of negative information dominate effects of positive
information
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Our Recommendations 1
No Universal Agreement
 Do not deceive participants or lie to them
– You do not need to tell participants everything but what you tell them
should be true
 Use representative consumers
– Students differ from other consumers
• Likely to have other preferences concerning sensory quality of
foods, attitudes towards animal welfare or controversial
technologies
• Different sensitivity for prices
– Student sample are OK for testing designs and theoretical studies
– No purpose in including vegetarians in a study of beef attributes
27
• Consumption constraint
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Our Recommendations 2
 Make sure the participants understand the mechanism
– Many procedures are unfamiliar and have lottery elements
• Inform and train participants by using the same mechanism and
number of goods as in the experiment
 Calculate relative WTP values
– According to micro theory relative prices are important
– Systematic anchoring effects will cancel out
– WTP differences can be used with many zero bids
 Use a context as free of scrutiny as possible
28
– Let participants feel relaxed and not scrutinized. Fill in any forms in a
room allowing for some privacy and let the participant identify
themselves only through numbers that are unknown for the other
participants
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Our Recommendations 3
 Use mechanisms with strong real economic incentives
– If you have 15 participants there are stronger incentives in a
median price than a second price auction
 Collect background information
– Socioeconomic, attitudinal, and knowledge variables can be
used to define segments
– Segments of interest for policy makers and industry
 Let participants taste unfamiliar products
– Tasting reduces sensory uncertainty, which potentially either
can reduce bids (due to risk aversion) or increase bids
(participant wants to taste a new product)
29
– Tasting results in increased external validity
www.umb.no
School of Economics and Business
NORWEGIAN UNIVERSITY OF LIFE SCIENCES
Our Recommendations 4
 Treat all products equally
– Rotate order of presentation between sessions
– Taste all or none of the products
– Use full bidding format in experimental auctions
 Delete participants with a nonresponse to all alternatives, i.e., always
bid zero or choose “none of the alternatives”
– May reflect that participant does not like any of products
– May reflect inconvenience, lack of trust in the experiment, or pure
laziness
 Let participants evaluate multiple items of heterogeneous products
30
– To avoid deception by letting the participants bid on as many items as
there are participants and, next, let each of the participants draw one
item as their binding product
www.umb.no
Thank You for Your Attention!
2111
2005