458
More on Model Building and
Selection
(Observation and process error; simulation
testing and diagnostics)
Fish 458, Lecture 15
Observation and Process Error
458

Reminder:


Process uncertainty impacts the dynamics of the
population (e.g. recruitment variability, natural
mortality variability, birth-death processes).
Observation uncertainty impacts how we observe the
population (e.g. CVs for abundance estimates).
Observation and Process Error
(the Dynamic Schaefer model)
458

Let us generalize the Schaefer model to allow for
both observation and process error:



Bt 1  [ Bt  r Bt (1  Bt / K )  Ct ]e wt ;
wt ~ N (0;  p2 )
I t  q Bt evt ;
vt ~ N (0;  o2 )
 p determines the extent of process error, and
 o determines the extent of observation error.
Often we assume that one of the two types of error
dominate and hence assume the other to be zero.
Process error only-I
458

We assume here that  p   o and continue under the
assume that v=0. Under this assumption:
Bt  I t / q so
I t 1 / q  [ I t / q  r ( I t / q){1  I t /( qK )}  Ct ]e wt ; i.e.
I  [ I  rI {1  I /(qK )}  qC ]e wt  Iˆ (r , q, K )e wt
t 1

t
t
t
t
t
If we assume that w is normally distributed, the
likelihood function becomes:
L (I | r , K , q ,  p )  
t 1
1
p
 [ nI t  nIˆt (r , q, K )]2 
exp  

2

2

2
p


Process error only-II
458

Issues to consider:




A process error estimator can only estimate
biomass for years for which index information is
available.
The choice of where to place the process error
term in the dynamics equation is arbitrary (what
“process” is really being modeled?)
You need a continuous time-series of data to
compute all the residuals.
What do you do if there are two series of
abundance estimates!
Observation error only - I
458

We assume here that  o   p and continue under the
assumption that w=0. Under this assumption:
 Bˆ1
Bˆt  
 Bˆt 1  rBˆt 1 (1  Bˆt 1 / K )  Ct
I  q Bˆ evt  Iˆ evt
t

t
t
If we assume that v is normally distributed, the
likelihood function becomes:
L(I | r , B1 , K , q,  o )  
t
1
o
 [ nI t  nIˆt (r , q, K , B1 )]2 
exp  

2
2

2
o


Observation error only - II
458

Issues to consider:




An observation error estimator can estimate
biomass for all years.
The choice of where to place the observation error
term is fairly easy.
There is no need for a continuous time-series of
data and multiple series of abundance estimates
can be handled straightforwardly!
There is a need to estimate an additional
parameter (the initial biomass - often we assume
that B1  K ).
458
Comparing approaches
(Cape Hake)
Note that we can’t compare these models
because the likelihood functions are different
Observation error only
Exploitable biomass
So what can we
say about these
two analyses
4000
Process error only
3000
2000
1000
0
1950
1960
1970
Year
1980
1990
458
Comparing approaches
(Residuals)
The residuals about the fit of the process error estimator seem
more correlated. Formally, a runs test could be conducted.
0.5
0.4
Process error only
0.3
0.2
Residual
Perhaps plot the
residuals against the
predicted values;
look for a lack of
normality
Observation error only
0.1
0
-0.11950
1960
1970
-0.2
-0.3
-0.4
-0.5
Year
1980
1990
Comparing approaches
(Retrospective analyses)
458
We re-run the analysis leaving the last few CPUE data points
out of the analysis – seems to be a pattern here!
3000
All data
Leave out 5 yrs
Exploitable biomass
Analyses along
these lines could
have saved
northern cod!
Leave out 10 years
Leave out 15 years
2000
1000
0
1915
1935
1955
Year
1975
1995
Simulation testing
458
1.
2.
3.
4.
5.
6.
Fit the model to the data.
Run the model forward with process error.
Add some observation noise to the
predicted CPUE
Fit the observation and process error
models.
Compare the estimates from the
observation and the process error
estimators with the true values.
Repeat steps 2-5 many times.
458
Simulation testing (Cape hake)



The simulation testing framework was applied
assuming a process error variation of 0.1 and an
observation error variation of 0.2.
The results were summarized by the distribution for
the difference between the true and estimated
current depletion (the ratio of current biomass to K).
For this scenario, the observation error estimator is
both more precise and less biased. Unless there is
good evidence for high process error variability (there
isn’t for Cape hake), we would therefore prefer the
observation error estimator.
50
Observation error only
Process error only
Frequency
458
Simulation testing (Cape hake)
40
Bias - average error
isn’t zero
30
20
10
0
-100
-50
0
Relative error
50
100
Simulation Testing - Recap
458

The reasons for using simulation testing
include:



we know the correct answer for each generated
data set – this is not the case in the real world;
and
there is no restriction on the types of models that
can be compared (e.g. they need not use the
same data).
The results of simulation testing depend, of
course, on the model assumed for the true
situation.
Readings
458



Haddon (2001); Chapter 10.
Hilborn and Mangel (1997), Chapter 7.
Polacheck et al. (1993).