Clayton P. Michaud
Thomas W. Sproul
Calum G. Turvey
Introduction

In 2013, Turvey et al. conducted a study looking at the potential to price insurance in
regions that lacked adequate historical yield data using self-reported historic and future
yield outcomes/expectations.
Upon further analysis, he made an interesting finding- that the farmers he surveyed were
systematically over optimistic about their future yield forecasts relative to their historic
outcomes.
 On average, farmers anticipate yields in the coming year come from a distribution
with a higher mean, and lower variance than historical experience justifies.
This is an important result for two reasons:
1.
It might explain the historic finding that farmers require much higher subsidies to
induce crop insurance uptake than the traditional expected utility model would
suggest
2.
The leading model for the discrepancy between objective and subjective probability
weightings, Kahneman and Tversky’s Cumulative Prospect Theory/Probability
Weighting cannot explain this result because their model is unable to change the
support of a probability distribution
Introduction

In this research, we develop a Subjective Probability Transformation Model
that parameterizes this overconfidence bias.
We then use the parameters of this model to examine the population level
distribution of this bias using expectation maximization (EM) to fit a finite
Gaussian mixture model based on the scale and shift parameters from our
model.
We find that the bias can best be described as coming from 3 distinct classes.
 An optimistic group that makes up roughly 67% of our two samples.
 An unbiased group that represents roughly 21%.
 And a pessimistic group that comprises roughly 12%.
We go on to explore how this three classes might further sub-divide and
conclude by examining potential causes of this bias.
Yield Elicitation Survey
Turvey et al., 2013 used a survey method to directly elicit risk
perceptions and historic yield outcomes from rural Chinese farmers.

In November 2011, 780 Chinese farmers were surveyed about their
expectations for next year’s yield, as well as their historical yields.
 Survey took place in 3 counties (25 villages) in Shaanxi Province, China in
October 2011.
 The survey had 9 sections with 117 questions in total. Only a portion of these
were dedicated to crop insurance and the identification of crop yield risks.
 It was administered by 20 Chinese graduate students of the Northwest
Agriculture and Forestry University supervised by faculty researchers.
 About 55% of respondents were male, with an average age of 48.72 years,
and at least high school completion. On average respondents had farmed for
about 27 years but this ranged from first year farmers to about 60 years.
 After eliminating incomplete questionnaires and famers who do not grow
corn or wheat, we have 437 wheat and 442 corn observations, 394 of whom
grew both.
The Survey Question

Note: To avoid possible bias from representativeness, farmers were asked about their
future expectations prior to being asked about their historic yields.
Comparison of Historic Outcomes
and Future Predictions

1400
140
120
1200
y = 0.2361x + 26.894
R² = 0.1048
100
800
y = 0.8099x + 199.17
R² = 0.5497
Subjective Range
Subjective Mode
1000
80
60
600
40
400
20
200
0
200
400
600
800
1000
Objective Mode
1200
1400
0
50
100
150
Objective Range
200
250
Subjective Probability
Transformation Model

We model subjective risk as a function of
objective/historic risk as follows:
F = αr + βH
where F and H represent the future historic yield
respectively, α represents a shift in the location from a
given reference point, and β represents a scaling of the
variance/range of the objective distribution.
Parameterizing the Model

After a number of tests we conclude that the maximum
value serves as the most stable location parameter.
Defining β as
b=
max f - min f
maxh - minh
and solving for α, we get
a=
max f
maxh
-b
Expectation Maximization (EM)

Individual Mixing Parameters: t = ( t ,¼,t )
i
1i
Ci
In each step of the algorithm, we will update the class mixture
probabilities: p c Î( p 1 ,p 2 ,p 3 )
The first step is to calculate:
t =
t
ic
p ct × f( xi | mc ,s c )
C
åp
j =1
t
j
× f ( xi | m j ,s j )
which gives the ''membership probabilities'' for each individual.
The updated mixture probabilities are then given by the average
of these over all individuals, i:
I
1
p ct +1 = å t ict .
I i =1
Expectation Maximization

The next step is the maximization step, in which we
maximize with respect to the parameters, holding
probabilities fixed:
{b
t +1
c
,a
t +1
c
,d
t +1
c
I
} = arg max LL ( q ) = arg maxlnå
N
C
å åt
i =1 n =1 c =1
t
ic
åC t
å
= arg max å å lnåå t ic åf ( xin ;q )å.
å
å
I
N
i =1 n =1
c =1
åf ( xin ; bc ,a c ,d c )
Model Selection

3 Component Gaussian Mixture

6 Component Gaussian Mixture

3 Component Model

‘Unbiased’ 12%
Pessimistic 21%
Optimistic 67%
When our Subjective Probability Transformation Model is applied to a symmetric
Beta Distribution (a, b = 5)
Majority Group Under
Alternative Distributional Shapes

Left-skewed Beta Distribution
Right-skewed Beta Distribution
• Historic mean from Beta Dist.: 1100
• Guarantee under 90% coverage: 990
• Guarantee under 80% coverage: 880
• Historic mean from Beta Dist: 1000
• Guarantee under 90% coverage: 900
• Guarantee under 80% coverage: 800
Factors Influencing Overconfidence

Summary of Results
1. Overconfidence is significantly effected by historic
losses.
•
•
For each year since a historic min, farmers reduce
the scale of their future forecasts by 1%.
Farmers who suffered the worst lost in the previous
year increase the scale of their forecasts by more
than 13%.
2. On average, females are more optimistic, scaling
down the range of their future forecasts by an
average of 12%.
3. Our observed overconfidence cannot be attributed to
increased yields resulting from technological
advancements.
Notes: *,**,*** signify significance at the 90, 95, and 99
percentile, respectively. Standard errors are in
parentheses. Regression run on pooled data.
The mean of the scale coefficient is 0.8 and the mean of
Years Since Historic Min is 8.65.
Conclusion

 We find evidence that heterogeneity in overconfidence can best be
described as coming from a 3-component finite Gaussian mixture
model.
 An optimistic group that makes up roughly 67%.
 An unbiased group that represents roughly 21%.
 And a pessimistic group that comprises roughly 12%.
 Our optimistic majority class cleanly separates into 2 different classes
of optimists- mild and extreme.
 Overconfident future yield forecasts seem to be effected by historical
outcomes in a manner consistent with the ‘hot hand fallacy’ where the
absent of recent losses causes increased overconfidence.