A GUIDE TO WILDLIFE RESOURCE
VALUE
EFFECTIVENESS EVALUATIONS
DRAFT
VERSION 1.0
10 December 2013
Prepared by
Laura Darling and Kathy Paige, Ministry of Environment
and
Darcy Pickard, ESSA Technologies
Forest and Range Evaluation Program
Wildlife Resource Value
Ministry of Forests, Lands and Natural Resource Operations and
Ministry of Environment
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
-1-
ACKNOWLEDGEMENTS
This document is based on the learning of many individuals working on monitoring and
effectiveness evaluation projects. Many thanks to Scott Barrett, James Dalby, Todd
Manning, Melissa Todd, Richard Thompson for providing review comments.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
-2-
PREFACE
The Forest and Range Practices Act (FRPA) and regulations introduce a transition to a
results-based forest practices framework in British Columbia1. Under this new approach
to forest management, the forest industry is responsible for developing results and
strategies, or using specified defaults, for the sustainable management of resources. The
role of government is to ensure compliance with established results and strategies and
other practice requirements, and evaluate the effectiveness of forest and range practices
in achieving management objectives.
Forest and Range Evaluation Program (FREP)2 is a multi-agency program established to
evaluate whether practices under FRPA are meeting not only the intent of current FRPA
objectives, but also to determine whether the practices and the legislation itself are
meeting government’s broader intent for the sustainable use of resources.
The Minister of Environment is responsible for designation of important mechanisms
described in FRPA to address conservation of wildlife habitat – including Wildlife
Habitat Areas, Ungulate Winter Range and others. The Wildlife Resource Value Team
(WRVT) of FREP, a joint initiative between the Ministry of Environment, and the
Ministry of Forests, Lands and Natural Resource Operations, is responsible for designing
a program for monitoring and evaluating wildlife objectives established under FRPA for
these mechanisms. The WRVT 3 has prepared a strategic framework for monitoring
effectiveness of wildlife resource values, along with various guidance documents and
reports describing work completed under the Wildlife Resource Value component of
FREP.
This Guide to Wildlife Resource Value Effectiveness Evaluations is one of the key
guidance documents available at the FREP WRVT website. It is designed to provide
guidance for establishment of projects to evaluate effectiveness of tools or mechanisms
available in FRPA that are intended to conserve wildlife resource values, particularly
Wildlife Habitat Areas and Ungulate Winter Range. It has been developed with FREP
wildlife resource value objectives in mind and the intended audience is FREP wildlife
resource value project leads.
1
For more information on FRPA and its regulations, resource values, objectives, etc., see
http://www.for.gov.bc.ca/code/ .
2
For more about FREP see: http://www.for.gov.bc.ca/hfp/frep/index.htm
3
For more about the WRVT see the WRVT website: http://www.for.gov.bc.ca/hfp/frep/values/wildlife.htm
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
-3-
TABLE OF CONTENTS
ACKNOWLEDGEMENTS ................................................................................................ 2
PREFACE ........................................................................................................................... 3
TABLE OF CONTENTS .................................................................................................... 4
1 INTRODUCTION ...................................................................................................... 6
1.1 Purpose of Guidance Document ........................................................................... 7
1.2 How to Use this Guidance Document .................................................................. 8
2 PLANNING ................................................................................................................ 9
2.1 Framing the Monitoring Objectives and Questions ........................................... 10
2.2 Prioritizing the Monitoring Objectives and Questions ....................................... 13
2.3 Compiling Knowledge Base .............................................................................. 15
2.3.1
Ecological Knowledge (or Data) ................................................................ 15
2.3.2
Management Activity.................................................................................. 16
2.3.3
Threats and Risks ........................................................................................ 16
2.4 Demonstrating Relationships – Conceptual Models .......................................... 17
2.5 Determining Indicators (Measures) .................................................................... 19
2.5.1
Indicators to Monitor .................................................................................. 20
2.5.2
Thresholds and Expected Values ................................................................ 23
2.5.3
Developing an Evaluation Tool for Effectiveness Indicators ..................... 24
2.6 Pilot testing a protocol........................................................................................ 28
2.7 Tracking Project and Program Efficiency .......................................................... 29
3 DESIGNING A STUDY........................................................................................... 30
3.1 Designing the Approach ..................................................................................... 30
3.1.1
Study design ................................................................................................ 31
3.1.2
Delivery model............................................................................................ 32
3.1.3
Spatial scale ................................................................................................ 32
3.1.4
Temporal Scale ........................................................................................... 33
3.1.5
Intensity....................................................................................................... 33
3.1.6
Location ...................................................................................................... 34
3.1.7
Methods....................................................................................................... 34
3.1.8
Overlap with other monitoring – FREP and other programs ...................... 34
3.2 Sampling Design ................................................................................................ 35
3.2.1
Life History Characteristics ........................................................................ 36
3.2.2
Target Population ........................................................................................ 36
3.2.3
Sampling unit .............................................................................................. 37
3.2.4
Positioning of sampling units...................................................................... 39
3.2.5
Sample size and statistical power ............................................................... 42
3.2.6
Sampling frequency .................................................................................... 43
3.2.7
Field methods .............................................................................................. 46
3.2.8
Planning the data analysis ........................................................................... 47
4 IMPLEMENTING A PROJECT .............................................................................. 49
4.1 Confirm Agency Support ................................................................................... 49
4.2 Prepare a Monitoring Plan.................................................................................. 49
4.3 Data Management and Quality Assurance ......................................................... 50
5 REPORTING ............................................................................................................ 50
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
-4-
5.1 Report Templates ............................................................................................... 50
5.2 Input to Decision-Making and Resource Management ...................................... 51
6 LITERATURE CITED ............................................................................................. 51
Appendix 1. Example study designs ................................................................................ 54
Appendix 2. Examples of sampling designs .................................................................... 58
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
-5-
1 INTRODUCTION
Effectiveness evaluation is an assessment of the results of investing resources and
conducting activities to (make progress toward or) achieve some stated program or
project goal or objective. The investments and activities are internal to the program (or
project), controlled by program decisions and can be described in terms of their
efficiency. The results of the program, the direct or indirect short- or long-term outputs
and outcomes of the program, are described in terms of their effectiveness.
There are many reasons to implement effectiveness evaluations:
 To maximize the impact of limited resources;
 To gain insight into strengths and weaknesses of a particular management
approach;
 To clarify the role of the project in the complex ecological (and social) systems
 To build support by documenting success;
 To improve practices in the face of unavoidable uncertainties and inevitable
change;
 To ensure conservation of target species and ecosystems; and
 To show stakeholders that efforts and policies are justified, dollars well spent.
Effectiveness evaluation fits into a process of checking, self-assessment and reflection
that any program should incorporate into its business to determine if the program is
proceeding as desired and expected. Questions that need to be asked in the evaluation
process are: (1) what is the management objective and where do we want to go? (2) what
practices will be implemented and how will we meet the objective? (3) are we going in
the right direction? and (4) how will we provide feedback so we can change direction and
improve if we need to? Effectiveness evaluation addresses the third question.
Monitoring is a component of an effectiveness evaluation. It is the process and rules for
collecting and analyzing information. The key components of an effectiveness
monitoring and evaluation program include clearly stating the goals, developing a
conceptual model linking relevant ecosystem function components (i.e, species and
habitats) and stressors/disturbances (risks), selection of suitable indicators to monitor,
estimating their status and trend, determining values that trigger a management response,
and linking monitoring results to management (Lint et al 1999; Noon 2003).
There have been several data and program assessment processes developed for a variety
of types of projects and proponents, all with the goals of (a) efficient and effective use of
resources, (b) collecting the right type of data in sufficient quality and quantity to support
the goals of the study, and (c) linking results of the monitoring with decisions for
improved resource management. The Environmental Protection Agency (2006) Data
Quality Objectives Process (DQO) for evaluation monitoring is a series of logical steps
that guide planning for the resource-effective evaluation of environmental data that
measures performance (compliance) and/or acceptance criteria. The National Park
Service (NPS) “Vital Signs Monitoring” (National Park Service 2012) determines the
status and trend in the ecological condition of selected park resources. The
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
-6-
Environmental Monitoring Initiative from the University of Michigan (Schueller et al.
2006) provides a user-friendly step-by-step process based on the Logic Model for
developing and implementing evaluation plans for ecosystem and community-based
projects. And the Nature Conservancy of Canada (NCC) has developed a structured
process for monitoring the effectiveness of NCC conservation strategies to meet program
objectives in the lands they administer or manage (Gordon et al. 2005).
The Wildlife Resource Value Team (WRVT) of the Forest and Range Evaluation
Program (FREP 4), a joint initiative between the Ministry of Environment, and the
Ministry of Forests, Lands and Natural Resource Operations, is responsible for designing
a program for monitoring and evaluating wildlife objectives and management
mechanisms established under the Forest and Range Practices Act (FRPA).
In the context of wildlife resource values and implementation of the FRPA, effectiveness
monitoring and evaluation are conducted to answer broad questions such as: Are we
conserving what we say we are? Are we achieving what we say we are? What can we do
to enhance our success? And more specifically, we might ask: How is the species of
concern (or biodiversity in general) doing? Are our actions having the intended impact?
To answer these questions and assess status and condition of a species, habitat or threats,
a suite of biological and/or threat indicators is tracked over time. When a conservation
intervention or management practice is initiated or underway, comparisons are made of
the current and desired condition, the relative impacts of different treatments, or effects
of pre-and post-treatment condition.
None of the four assessment processes described above (DQO, NPS, EMI and NCC) is
fully compatible with the kind of evaluations needed for FREP, but the strengths of each
(quality assurance concepts from DQO, ecological principles from NPS, the stepwise,
cyclic model from EMI, and the application to land use planning and stewardship
objectives of NCC) have been drawn on to develop this effectiveness evaluation process
for the wildlife resource values of FREP – the Guide to Wildlife Resource Value
Effectiveness Evaluations. The reader is encouraged to review these other assessment
processes to gain a deeper insight into the rationale for and development of sound
effectiveness evaluation projects.
1.1 Purpose of Guidance Document
This document is designed to provide guidance for establishment of projects to evaluate
effectiveness of tools or mechanisms available in FRPA that are intended to conserve
wildlife resource values, particularly Wildlife Habitat Areas (WHA) and Ungulate Winter
Range (UWR). It has been developed with FREP wildlife resource value objectives in
mind and the intended audience is FREP wildlife resource value project leads.
4
For more about FREP see the FREP website: http://www.for.gov.bc.ca/hfp/frep/index.htm .
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
-7-
There are four parts to this document, based on four major steps involved in effectiveness
evaluation, as outlined in Table 1. Each step involves several tasks and output products,
which together will lead to achieving the evaluation objectives. Not all projects will
proceed to the end step – though all tasks are thought to be necessary components of the
compete process, progress will vary for different situations and for some species there
will be value in achieving only some initial tasks (e.g., determine need and current state).
Table 1. Tasks and outputs of the four steps in the development and implementation of
effectiveness evaluations in FREP wildlife resource value projects.
STEP
1. Planning: bound
the problem
2. Designing: the
Approach &
Sampling Design
3. Implementing a
project
4. Evaluation &
Reporting
TASK
Frame monitoring questions & objectives
Consider priority of monitoring question
Determine current state
Build conceptual model of relationships
Select indicators
Establish indicator thresholds or triggers and
how condition or effectiveness will be established
Establish study and sampling design
Determine data collection methods
Test and refine methods (pilot testing)
Determine data analysis methods
Undergo statistical review
Project planning
Secure resources
Training
Outline quality assurance
Collect data
Analyse data
Establish baselines for indicators
Report study results
Summaries and advice to decision makers
OUTPUT PRODUCT
Draft project charter
Conceptual model
Indicators
Standard protocol
Monitoring plan (includes
study and sample design)
Project Charter
Monitoring plan
Field forms
Reports, Extension Notes,
scientific papers,
management
recommendations
Figure 1. The main steps in an effectiveness evaluation.
1.2 How to Use this Guidance Document
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
-8-
Where does my project fit?
Though the target audience for this guidance document has a common responsibility as
project leads, different users will have different needs and therefore different ways they
might use this document. Some investigators may be studying an animal for which there
is very little known so protocol development must begin with developing a basic
understanding of the species and ways to monitor it. Other projects may focus on species
for which biological knowledge and field methods are well developed, or where a
preliminary effectiveness protocol already exists: in these situations there may be no need
to undertake Steps 1 or 2, in Table 1. They may ask: what pieces do I have; where in this
process do I jump in; if steps 1 and 2 are done, what do I do?
Users of this guidance document should determine just where they are in the process of
development and implementation of an effectiveness protocol by reviewing the “action
checklist” (yellow text boxes) at the end of key sections, and insert their project into the
appropriate step to match their need.
How is the document organized?
Sections 2 – 5 describe the four steps for effectiveness evaluation outlined in Table 1.
The reader (project leader) can enter at any appropriate step in the process.
Blue-shaded text boxes at the beginning of a section clarify the intent of the steps or
actions described in the subsequent section.
Yellow-shaded text boxes at the end of a section summarize the activities or steps to be
taken to complete the actions outlined in the section. These steps may be used as an
“action checklist”.
Purple-shaded side-bars provide supporting information or more detailed explanations of
concepts presented in the document.
2 PLANNING
In monitoring projects it is important to use an established or standard protocol and, if
one does not exist, to develop one. A protocol is a procedure, practice or set of rules. In
the realm of environmental surveys, a detailed step-by-step process for collecting field
data might be labelled as a protocol, or a broader definition of protocol might be used that
encompasses steps to develop the study design, collect and analyse the data, and report
the results. The latter, all-encompassing definition of protocol is the one used in this
document. Standardized step-by-step data collection procedures and techniques are
referred to as methods in this document. The goal of a monitoring protocol is to
establish consistency, transparency and minimize or balance measurement error and
sampling error. Measurement error is that which results from differences in collection
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
-9-
methods, rather than actual changes in the environment (Oakley et al. 2003). Sampling
error results from taking measurements from a subset of a population and is minimized
through sampling design.
Before deciding to embark on a project or develop a protocol, it is important to do some
background work to determine if a project is a priority and assess the state of our current
knowledge. Section 2 is arranged to answer the following questions:
2.1
2.2
2.3
2.4
2.5
2.6
2.7
What are the objectives of the monitoring project?
What are the key questions we wish to answer through monitoring?
What are the highest priority objectives and questions for monitoring,
and how does this project rank relative to other objectives and
questions?
What do we already know about the species, its habitat and
management practices?
What are the relationships between the many variables that describe
the species, its environment and threats to its survival?
What are the best species, population, habitat or threat variables (i.e.,
indicators) to measure to assess achievement of project objectives and
provide answers to the priority monitoring questions? What indicator
values will serve as thresholds that will trigger a management
response? Are there standard protocols for measuring indicators?
What potential pilot projects are necessary to test the approach,
indicators and methods?
What performance measures should be tracked to document project
efficiency of the project?
2.1 Framing the Monitoring Objectives and Questions
Section 2.1: In this step, the key management issues and questions are clarified: What is the expected or desired outcome of the management practice? What is
the objective of the FRPA mechanism?
 What key management issues may be hindering the desired outcome of the
management practice?
 Who is the audience for the results of the study – the land manager, a statutory
decision-maker, government policy, the public, or all of these?
 What questions need to be answered in order to address the management issues
and improve the management outcome?
 What is the focus of the issue (and monitoring) – threat reduction, habitat
quality or population status?
 At what scale does the issue (and monitoring) apply – local, watershed, or
regional?
Development of a wildlife resource value effectiveness evaluation project begins with
clarification of the key management issues or questions (the uncertainties) regarding the
FRPA mechanism (Ungulate Winter Range [UWR] or Wildlife Habitat Area [WHA],
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 10 -
etc.). Usually, there is uncertainty about whether the expected outcomes of the current
management scenario (the objectives of the FRPA mechanism) are being achieved.
Pertinent key management issues (or unknowns) are the source of the very specific
questions and hypotheses examined in an effectiveness evaluation project. By zeroing in
on recognised knowledge gaps and relevant management issues, an effectiveness
monitoring project will be directed to the area of greatest need and will produce a sound
evaluation of effectiveness.
Broad scale priority questions that need to be addressed for UWRs and WHAs (see
SIDEBAR #1) have been confirmed by the FRPA Resource Evaluation Working Group5.
Although these questions are both too complex and too general to serve as a focus for a
single evaluation project, they point to several more specific topics that could be targeted
for evaluation, including amount, quality and distribution of UWRs and WHAs, survival
and fitness of the species they are designed for, or trends in threats to the habitat or
species.
SIDEBAR #1 The key question being asked about ungulate winter ranges and wildlife habitat
areas [See http://www.for.gov.bc.ca/hfp/frep/about/questions.htm#questions ] are:
1. “Do ungulate winter ranges maintain the habitats, structures and functions necessary to meet
the species winter habitat requirements, and is the amount, quality and distribution of UWRs
contributing effectively with the surrounding land base (including protected areas and
managed land base) to ensure the winter survival of the species [Project Charter -within the
landscape] now and over time?”.
2. “Do WHAs and the associated GWMs and objectives combine to maintain the habitats,
structures and functions necessary to meet the goal(s) of the WHAs, and is the amount,
quality and distribution of WHAs contributing effectively with the surrounding landbase
(including protected areas and managed landbase) to ensure the survival of the species now
and over time?”
Specific monitoring questions investigated for WHAs and UWRs might be at small or
large spatial scales and/or addressing single or multiple environmental parameters. For
example, specific questions might be:
 Does this WHA provide sufficient habitat to provide for survival of the speciest?
 Is habitat attribute X above a standard habitat quality threshold within the UWR?
 Do WHAs in this region provide better habitat for the species at risk than similar
areas that are not designated as WHAs?
 Is the species at risk population persistent in this UWR?
Selection and refinement of monitoring questions requires consideration of other factors
beyond the management issue itself (Gordon et al. 2005, Schueller et al. 2006). Questions
should be framed considering who regulates the management practice, our ability to
5
The FRPA Resource Evaluation Working Group (FREWG), representing the Ministry of Forests and
Range (Field Services Division, Research Branch and Forest Practices Branch), as well the Ministry of
Environment (MoE), and the Ministry of Agriculture and Lands (MAL) developed the FREP structure and
framework and initiated development of resource value indicators and monitoring /evaluation protocols.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 11 -
influence the management practice, and whether appropriate legal changes are possible.
Opportunities for partnerships, cooperation and cost efficiencies need to be considered as
well as whether the target species or habitat is suited to cost-effective monitoring.
Limitations and potential barriers such as budget constraints, logistics and access issues
also need to be considered. Some of these factors may be taken into account in the
prioritizing process (see section 2.2), however considering them in a broad sense right
from the start will serve as a coarse filter.
In some cases identifying the unknowns and framing management questions will be
straightforward. But some issues will be less clear and determining specific management
questions will be more complex. The concerns about effectiveness of WHAs or other
FRPA mechanisms will need to be defined in detail – each issue should be defined
separately and completely in an uncomplicated format (see 2.4 Conceptual Models).
To be valuable for improved future management, effectiveness evaluation projects must
be based on answerable questions and hypotheses that are specific and precise. The
questions must be SMART (specific; measurable; attainable, achievable and appropriate;
realistic, relevant and results oriented; and time-bound; Doran 1981, Meyer 2003) and
phrased in a way that ensures that the project leader and co-operators agree on the most
important information needed. Questions and hypotheses should be framed to provide the
practitioner with a clear-cut answer and direction for management: not asking for a “yes”
or “no” answer but rather “how much? or “to what extent?” Key questions and the null
hypotheses must be formulated and confirmed prior to initiating monitoring so
appropriate sampling design and data analysis can be applied and the chance of reaching
erroneous conclusions is reduced (Shaffer and Johnson 2008).
Monitoring questions may address issues of population abundance, habitat suitability, use
of the area by wildlife, or mitigation of threats. Questions may address the generic or
specific goals of WHAs and UWRs (see SIDEBAR #2). They may deal with assessing
status of a species or habitat, or effectiveness of an action, and be focused at a local,
landscape or regional level. Or questions may address progress toward meeting
management goals, threat reduction, improving condition or successful completion of
activities. Though focusing on one type of question (say, for example, species and
population questions at a local level) may be believed to address the major management
question, successful effectiveness evaluations examine questions from a variety of types
of questions. Conceptual models (section 2.4) and monitoring indicators (section 2.5)
reflect the same categories.
SIDEBAR #2:
Examples of generic goals that might apply to all WHAs include:
1. Maintain the habitat suitability and key habitat attributes of the area that contribute to survival
of the species in the area (e.g., microclimate, prey species, nest trees)
2. Maintain the use of the area by the species
3. Maintain the ecological processes (ecosystem functioning) that maintains the species
4. Abate stresses from forest and range management practices
Examples of specific goals that might apply to individual WHAs include:
1. Reduce road mortality
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 12 -
2. Minimize soil disturbance
3. Maintain stand structure
4. Maintain security cover
Section 2.1 Key Questions – in practice
 Describe in detail the concerns and issues about the effectiveness of the FRPA
mechanism. Define each problem separately and completely, in an uncomplicated
format.
 Frame the concerns in specific focused questions.
 Ensure that answers to these questions will provide direction for improved
management.
 The next step is to determine if these are priority issues that warrant effectiveness
evaluation monitoring.
 And after that, the questions can be further refined and specific, measurable
hypotheses developed for a monitoring project.
2.2 Prioritizing the Monitoring Objectives and Questions
Section 2.2 : In this step, priorities are determined: Which FRPA mechanisms and which of the species they serve are the highest
priorities for effectiveness monitoring?
 Which management issues (for those species or mechanisms) are the highest
priorities for monitoring?
 Does the proposed project meet Ministry and regional mandates, goals and
priorities?
 Is the proposed project a high priority based on FREP’s Wildlife priority criteria?
 Is the project one of the highest priorities according to the MOE Conservation
Framework priority setting process?
When considering whether to pursue an effectiveness evaluation of a WHA, UWR or
other management tool for wildlife resource values, it is necessary to determine and
confirm the priority of doing the project. Provincial and regional goals and priorities for
the species and the management tool and potential benefits of the evaluation project at the
regional and provincial level must be considered.
Management tools for wildlife resource values under FRPA are established cooperatively
by Ministry of Environment (MOE) and Ministry of Forests, Lands and Natural Resource
Operations (MFLNRO), though led by MOE. Therefore, all evaluation projects should be
aligned with both ministries’ mandates and priorities. High-level goals such as MOE’s
“… to provide for healthy and diverse native species and ecosystems …”, and
MFLNRO’s goal of “ensuring environmental standards are upheld and environmental
sustainability is achieved with resource use activities in British Columbia …” provide the
context for the management tools discussed here. Direction is also provided by explicit
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 13 -
legal goals for wildlife outlined in FRPA regulations: “… without unduly reducing the
supply of timber from British Columbia’s forests, to conserve sufficient wildlife habitat
in terms of amount of area, distribution of areas and attributes of those areas for the
winter survival of species at risk of specified ungulate species” (Section 7 of the Forest
Planning and Practices Regulation and the Woodlot Licence Planning and Practices
Regulation).
Regional goals and objectives must also be considered when determining whether to
initiate an evaluation project – these are the desired outcomes of implementation of the
management tool. Some WHA and UWR goals and objectives, particularly from the
earliest designations, may be unclear, unspecific or difficult to evaluate, and as a result,
they may need to be refined to be clear, measurable, and outcome-based. Documented
objectives of each established WHA or UWR should be reviewed to define the key issues
and questions that need to be investigated and to determine whether effectiveness
evaluation is appropriate. There may also be associated goals related to other FRPA
resource values that may be evaluated in other FREP projects, such as biodiversity,
timber or livestock forage, that the UWR or WHA help to meet.
Therefore prior to undertaking a project, consult provincial and regional goals and
priorities and consider the following criteria:
(i)
Does the project address a priority question (i.e., prioritized questions put
forward by the Wildlife Resource Value Team)?;
(ii)
What is the species conservation priority based on MOE’s Conservation
Framework?
(iii)
What is the province’s investment in management actions (e.g., the number
and total area of UWRs, WHAs, etc)?;
(iv)
What is the relative importance of the WHA, UWR (etc.) to conservation of
the species?;
(v)
What is the uncertainty of the effectiveness of the management action(s) and
the risk to the value (e.g., species, habitat); and
(vi)
What is the likelihood of project success (i.e., project feasibility, benefits,
support).
As an example, the procedures for the Environmental Mitigation Policy recommends
consideration of the uncertainty of the mitigation activity in combination with the risk to
a value to determine whether implementation or effectiveness monitoring is appropriate
(http://www.env.gov.bc.ca/emop/).
The FREP Wildlife program posts provincial priorities determined through a priortysetting process (see http://www.for.gov.bc.ca/hfp/frep/values/wildlife.htm). Projects
determined to be the highest priority should be initiated before lower priority ones.
Section 2.2 Priorities – in practice
 Priority evaluation questions provide the context for developing specific indicators
for effectiveness monitoring.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 14 -


Examine where the species ranks in higher level priorities – ministry mandates,
regional priorities, etc – and whether monitoring is indicated as a priority action in the
Conservation Framework, Monitoring Strategies, etc.
Submit the issue (questions) for consideration to the FREP WRVT – run through the
expert-based Prioritizing Process.
2.3 Compiling Knowledge Base
Section 2.3 : In this step we collate current knowledge and data about:
 Species biology
 Species habitat use and habitat requirements
 Known or potential stressors, threats and disturbances
 Past, current and expected habitat suitability and capability
 Past, current and expected management practices and adjacent activities
Once key issues and questions have been identified and an effectiveness evaluation has
been confirmed as a priority project, the next step is to gather existing biological data and
information about management activities and threats (risks) in the area. This collated
“current state” information will provide a solid baseline or reference point for the
evaluation, and it may confirm the validity of the proposed evaluation objectives or lead
to refinement of them. As well, it provides crucial input to conceptual models that will be
used to refine key questions into monitoring questions and hypotheses.
2.3.1 Ecological Knowledge (or Data)
The importance of collecting pertinent existing information related to the species, its
natural history and ecological role, the particular location, and the management tool
(WHA or UWR) in use, cannot be understated. It is imperative to gather and incorporate
current knowledge, to ensure the project design addresses advances in research and
improvements to indicators. Data and conclusions from baseline, pilot and on-going
studies can be included as collected, and added to the knowledge base over time to refine
understanding.
Potential sources of information should be investigated – published journals, published
and unpublished consultant reports to government or industry, internet publications,
results of PEM (Predictive Ecosystem Mapping) and/or habitat modelling, recognised
species specialists, MOE species and habitat specialists, and local experts, etc. This
information may be text-based or spatial (i.e., maps). Review of this information should
confirm the validity of the need for the proposed evaluation project. As well, this review
of knowledge will confirm existing information gaps and needs, and aid in development
of the study objective and design.
Some general information sources include:
 Conservation Data Centre http://www.env.gov.bc.ca/cdc/
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 15 -







BC Species and Ecosystems Explorer
http://www.env.gov.bc.ca/atrisk/toolintro.html
Identified Wildlife Management Strategy Species Accounts
http://www.env.gov.bc.ca/wld/frpa/iwms/index.html
Wildlife Species Inventory – Species Inventory Database
http://www.env.gov.bc.ca/wildlife/wsi/index.htm
eFlora http://www.geog.ubc.ca/biodiversity/eflora/
eFauna http://www.geog.ubc.ca/biodiversity/efauna/
Wildlife Resource Value Team effectiveness evaluation reports
http://www.for.gov.bc.ca/hfp/frep/values/wildlife.htm
Resources Information Standards Committee http://www.ilmb.gov.bc.ca/risc/
2.3.2 Management Activity
The primary management activity of concern in WRVT effectiveness evaluations is the
FRPA mechanism or tool being used to protect the species of interest (i.e., a WHA or an
UWR) and the management practices associated with that tool (i.e., general wildlife
measures). Parameters of the tool and practices must be documented – for instance size
and shape of a WHA, number and condition of important habitat features within the
WHA, and specifics of required management activities such as no-harvest zones or
minimum grazing stubble heights. As in section 2.3.1, this information may be text-based
or spatial. Documentation of these parameters will serve not only to determine whether
the FRPA tool was implemented as intended, but also to evaluate effectiveness and
validate relationships proposed in the conceptual model (see section 2.4).
It is also important to document past and present land or wildlife management activities
in and around the study area, because these activities may have impacted (or may impact
in the future) the habitat or the target species in some way that might confound the
proposed evaluation. Impacts of past or current practices may require refinement of study
objectives, and have to be considered in development of study design or in analysis of
data.
2.3.3 Threats and Risks
Activities or conditions that exist in or near the study area may upset the natural balance
of the ecosystem. Stressors may exist at a low level with negligible impact; they may
never or quickly or over a long time period develop into major threats to ecological
integrity. One-time or on-going disturbances may be or become substantial threats.
Consideration of threats in a spatial context may provide a different perspective than if
simply a written description. The magnitude of any threat is manifest in the level of risk it
poses to the integrity and well-being of a population or habitat. Some high risks are
riskier than others, for example if there are ineffective management techniques to reduce
a threat. In addition, one must consider cumulative impacts of on-going or of several
concurrent disturbances and threats. Consideration of existing and potential stressors and
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 16 -
threats to the ecosystem under investigation, and their cumulative impacts, may indicate
the need to refine study objectives and/or to consider these factors in study design and
data analysis.
Section 2.3 Current State – in practice
 Collate data and reports.
 Include spatial (GIS, mapping) data
 Summarize knowledge, activities and stressors that impact the species or the habitat
 Prepare for conceptual models!!
2.4 Demonstrating Relationships – Conceptual Models
Section 2.4: In this step we ask:
 What do we know or believe about the relationships between populations,
habitats and stressors (threats)?
 Can the relationships be quantified, or are we limited to qualitative
descriptions?
 What are the interactions and influences in the ecological system?
 What factors and interactions are most important?
Following collation of the current knowledge from previous studies, the next step is to
use conceptual models to examine the functional interactions and relationships between
species and ecological system components, and to link species and habitats (functions) to
threats (risks) and their impacts. A guidance paper by Nyberg (2010) discusses various
types of conceptual models, their strengths and recommendations for use in the wildlife
resource effectiveness evaluation process; the following summarizes some key aspects of that
report.
Usually, a conceptual model describes the effects of various stressors (risks) on the focal
population. This general model is then broken down further to show pathways through which
particular stressors (risks) could affect habitat attributes and population processes
(functions). In the management context of effectiveness evaluation, a conceptual model
illustrates the proposed relationship between a management activity and a species response.
Developing sound models is crucial in the effectiveness evaluation process because from the
relationships demonstrated in the model, a list of candidate indicators is then drafted
(section 2.5) for each potential interaction; indicator variables are selected for monitoring
from that list.
Models are an important component of monitoring and evaluation projects. They can:
 document functional relationships between system components, clarify the
linkages between stressors (risks) and outputs or outcomes, and document the
understanding of the system being managed and evaluated
 highlight important knowledge and data gaps and identify areas of uncertainty
that should be addressed through monitoring or research
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 17 -



identify the most important monitoring questions
identify measurable indicators that will answer the questions
form the basis for more informative and powerful quantitative models
The interaction of biological factors in an ecosystem is complex, and modelling can
improve our ability to understand it. Usually ecosystem parameters are interactive and
models based on separate relationships between a species and its environment are not
meaningful. Developing an explicit, all-encompassing model of important relationships
allows us to compare the importance of changes in different parameters, determine the
net effect of those changes, characterize and quantify cause and effect relationships, and
set priorities for reducing uncertainty. Subsequently, key relationships will be highlighted
to illustrate the focus of an effectiveness evaluation.
Models of relationships can be developed at various levels of detail and scale, from
simple graphs and diagrams to complex mathematical simulations. To develop sound
effectiveness evaluations, it is necessary to build conceptual models to document
relationships between resource values, threats (risks), opportunities and stated goals,
objectives and strategies. Input to conceptual models may come from experts and/or
output of more complex mathematical modeling procedures.
Conceptual models can be built in table, graph or diagram format. General concept maps
(see, for example, “situation maps” in Scheuller et al. 2006), often presented in tabular
format, can provide the logical structure to guide thinking about important questions and
linkages between goals, ecosystems and population traits. Influence diagrams (see
example in Nyberg 2010), using arrows to show direction (and sometimes relative
magnitude) of influence, illustrate relationships among different ecosystem parameters
and provide a clear understanding of how each contributes to the end result. Baysian
belief models (Marcot et al 2001, Nyberg 2010) integrate biological data presented in an
influence diagram, with expert opinion and an estimate of certainty (belief weighting) of
life history parameter values and relationships among parameters. Unlike concept maps
and influence diagrams, Baysian models can be used to predict a range of possible
outcomes and their probabilities for populations, habitats, threats, prey, etc., and therefore
Nyberg (2010) recommends Baysian models as the preferred format for conceptual
models in wildlife effectiveness evaluations, though simpler models may be appropriate
when data are lacking.
To create a model of relationships, it is critical that the model be developed fully and
correctly, to the level of current knowledge. Important interactions and elements need to
be included, and the importance and probability of different parameter values need to be
defined. Models should be built collaboratively with species and habitat experts as well
as land managers. Simpler types of conceptual models can be built in a day’s workshop,
whereas complex mathematical models may require longer preparation time and review.
The complexity of a model may reflect the complexity and scale of the objectives of the
study and the questions being asked.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 18 -
An examination of the finalized conceptual model will reveal important knowledge gaps,
uncertainties around important parameters, and interactions and threats that may
influence achievement of the management objective (i.e., its effectiveness). The variables
that describe the known conditions and relationships and the unknowns, uncertainties and
threats are candidate monitoring indicators in effectiveness evaluation. A list of
measurable variables to consider as monitoring indicators is one output product of the
conceptual model.
Section 2.4 Demonstrating Relationships – in practice
 Use gathered information of the current state.
 Convene workshop of species and habitat experts.
 Develop a conceptual model that describes functional relationships, influences and
interactions among species, habitats and threats (risks) in the broad ecological system.
 List all the measurable variables that are associated with the model relationships.
 Prepare to select indicator variables for monitoring.
2.5 Determining Indicators (Measures)
With development of a sound conceptual model describing species, habitat and threat
(functions and risks) interactions and relationships, and a list of pertinent candidate
measurable variables for monitoring, the next step is to evaluate which variables are the
best indicators of whether a management objective is being achieved, and to select a
logistically feasible combination of these indicators.
Of key importance to identifying indicators for effectiveness evaluation, is that
effectiveness indicators can only be measures of effectiveness if they can be related back
to implementation of the tool or practice – that is, how the management activity was
undertaken.
Section 2.5: In this step we examine the conceptual model and ask:
 What variables can (or should) we measure to determine whether the particular
FRPA mechanism is achieving its objective?
 What variables have the strongest link to directly answering the key monitoring
questions?
 What attributes of the population or the habitat, if measured over time, will
indicate the effectiveness of the mechanism or the associated practice?
 What factors contribute to the success of the practice?
 What balance of population, habitat and threat indicator variables provides the
best opportunity for successfully answering the key monitoring questions?
 What factors (parameters) are unsuitable for monitoring (perhaps because
appropriate methods of evaluation do not exist, or there are difficult logistic
constraints, or high cost) and must be excluded from consideration as a indicator
variable?
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 19 -



What protocols are available? And what few, high priority, practical, measurable
variables should be selected for implementing a pilot study?
What are the predicted values of the selected indicator variables as a result of the
management practice (e.g., implementation of the FRPA mechanism)?
What values of the indicator variables (such as an upper or lower threshold) will
trigger a management response?
2.5.1 Indicators to Monitor
Ecosystem elements, risks and the relationships demonstrated among them in conceptual
models can be monitored directly or by using indicator variables. Indicators are
measurable parameters (attributes) that describe specific characteristics of the species of
interest or the target ecosystem; they represent the key features of the desired outcome
(i.e., the management objective); they give information on the state or condition of a
system element or the relationship between elements in a system.
Some indicators are direct measures of ecosystem or biological parameters of concern,
while some are measures of factors that are known or believed to represent parameters of
concern (i.e., surrogate or proxy indicators). Surrogate indicators may be used when the
parameter of interest cannot be easily or economically measured. For example, habitat
attributes may be used as surrogates for species, or one species for another, and indicators
at one scale may be used to monitor biological diversity at another scale. Though the
relationships are not always certain, a surrogate is expected to reflect the condition of the
unmeasured parameter. And though surrogacy is commonly used in biological
monitoring, validation of surrogacy assumptions is needed in most situations.
Indicators may be qualitative or quantitative. Quantitative indicators reflect a numeric
relationship between certain habitat variables and species response; they are stronger and
often of more value than qualitative indicators. Qualitative indicators, although they
sometimes provide a more complete understanding of a relationship, are subject to
observer subjectivity and are hindered by artificial categorization of the values.
Indicators may be selected to make comparisons over time (i.e., trends), among places
(e.g., different ecological sites or different treatments), or with an ideal (e.g., comparison
with the desired condition or a control) or a target (e.g., ecological threshold or
management trigger).
Field data collection may be used to assess some indicators but some spatial indicators
are assessed using routine evaluations (see section 3.1.5) such as geographic information
systems (GIS) analyses or mapping exercises. In most cases GIS-based assessment is
thought to be cheaper to complete, but field data, which may be used to validate GIS
assessments, is often relatively high value and crucial to a sound evaluation.
Indicators are characterized by:
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 20 -
1) the realm that is measured (habitat, species/population, threat);
2) the part of the ecosystem that is measured (ecosystem function, structure or
species);
3) the intent of the measurement (status assessment or effectiveness of action); and
4) the project “element” for which progress will be measured (objective – progress
toward goal; threat – progress to reduce threats; assets – progress toward
improving assets; and strategies – progress toward completing successful
activities).
In addition, wildlife utilize landscapes at a variety of spatial scales, and what indicators
are measured in effectiveness evaluations and what is inferred from them need to reflect
those spatial scales.
(a) Some indicators will be selected to address large scale management concerns:
maintenance of regional or subregional populations through a network of
management tools in the region, distribution and abundance of wide-ranging or
widely dispersed species, quality of habitat in a large area.
(b) Others will be selected to address concerns that occur at a medium scale: that is,
including a multitude of spatial management tools (e.g., WHAs and UWRs) by
watershed or management unit, such as assessing distribution and abundance of
habitat, or barriers that prevent movement or use of an area.
(c) And finally, others will be selected to address concerns at a small scale: that is,
spatial management tools at a stand level or smaller, addressing, for example,
forest structure and condition, or habitat elements (features) required for longterm survival.
Selecting suitable and meaningful indicators is one of the most important aspects of a
monitoring project. Good indicators are:
Focused
Clearly related
Easy to interpret
Practical, feasible
Fine-scaled
Relevant
Focused on answering a specific evaluation question;
directly reflects and covers key aspects of the management
goal
Clearly related (correlated) to the true parameter they
represent
Easy to interpret; useful in describing the elements or
relationships they represent; and easily communicated to the
target audience and stakeholders
Practical, easy to measure, feasible, reliable, robust and
cost-effective, maximizing information gain while
minimizing time, effort and expenditure; the data be
obtained at an acceptable cost and in a reasonable timeframe
Only as fine-scaled as the question requires; addresses the
scope of the question
Operationally relevant, addressing an issue of concern and
the forest /range practice under study with the appropriate
level of accuracy, precision and scale to provide results that
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 21 -
Responsive & Reliable
Precise and unambiguous
Supported by science
Integrated
will be used in decision making; can be related back to
implementation of the tool or practice; measuring changes
that can be averted by management actions
Quickly responsive to change in a predictable way; sensitive
to changes in management or stressors, reflecting well
understood causal relationships; consistently responsive to
actions we are concerned about; shows differences between
units; shows trends over time
Precise and unambiguous, not clouded by background
processes such as weather, climate, stochastic events or
natural variation (low naturally occurring variability)
Supported by science and widely used (rationale,
methodology and analysis well documented; measurable in
a standard scientifically credible manner); results have
acceptable variability; peer reviewed
Integrated (provides information about multiple levels or
aspects of the system, relevant at site, feature and landscape
scales)
The list of key questions and potential indicators for a study is usually longer than
feasible to evaluate, and a selection of the most valuable, cost-effective indicators will
have to be made. Often some flexibility is required, as the indicators initially selected
may prove to be less valuable as knowledge gaps are identified and new knowledge is
gained, or when selected indictors fail to provide sufficient accuracy. After choosing (or
developing) an indicator to answer a question, it is important to test the indicator in a
pilot study to determine whether it meets the criteria of a good indicator (listed above)
and provides a sound answer to the management question. Pilot studies will also indicate
whether the proposed study is feasible and the data from the selected indicators will make
a valuable contribution to knowledge. Subsequently, indicators should be adjusted or
replaced, and/or other indicators added to ensure effective monitoring.
Most effectiveness evaluations will involve monitoring more than one indicator. In the
realm of wildlife resource values, most projects will evaluate a combination of forest or
range structure indicators and species indicators. Though measuring forest or grassland
structure may appear more cost-effective than measuring population parameters (for
example, resource constraints or logistics issues make it easier to sample elements that do
not move or hide, that are not uncommon or vulnerable), measuring only habitat may fail
to indicate whether providing habitat structure is sufficient to maintain productive animal
populations over time. Measuring species population indicators can provide an early
warning system for critical declines and identify where finer scale population monitoring
is required. Measuring species or population indicators allows comparisons to habitat
benchmarks and helps refine model relationships over longer time periods and larger
areas. However, selection of the correct species indicator is critical – monitoring
indicators for all species is usually not possible, so indicators for the appropriate
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 22 -
informative species (perhaps some focal species or a guild of species) must be chosen to
answer the important monitoring questions.
Sample evaluation questions and indicators for a plethora of objectives for ecosystem
monitoring projects have been developed by the University of Michigan’s Ecosystem
Management Initiative (Schueller et al. 2006), Nature Conservancy Canada (Gordon et al
2005) the US National park Service (2012), and Bunnell (2005). The reader is urged to
review good indicators presented in these documents and the “questions and indicators”
reports on the WRVT webpage6.
2.5.2 Thresholds and Expected Values
In response to a management activity, or as its status is monitored over time, an indicator
value may rise above, fall below or fluctuate around some level that is significant to
management. Often there are thresholds determined from previous studies or values
derived for the study, above or below which management decisions and actions are
triggered to ensure the survival of a species or conservation of an ecosystem. Other times
the management goal is for the indicator to reach (or be stable at) a target value. In other
situations, an indicator is predicted to reach some level in a certain timeframe, based on
knowledge from previous studies or from benchmark or control sites. Threshold, trigger,
predicted and target values should be pre-determined, based on current scientific
knowledge, prior to initiating monitoring. Thresholds may be initially estimated from “a
syntheses of available data, model projections of known relationships, or reasoned
guesses” (Houde et al. 2005).
SIDEBAR #3 Baselines, benchmarks, controls, targets, thresholds and triggers
Several types of data, in various combinations, can be used for comparison in effectiveness evaluations:
baseline, benchmark, control, target, threshold and triggers. It is necessary to determine which type of data
is available and most valuable for a proposed effectiveness evaluation.
Baseline data reflect the condition of the habitat or species at a specified time, either at some time before
establishment of the UWR or WHA, at time of establishment, or at some fixed time when the evaluation
study begins. This information is incomplete for UWRs, ungulates, WHAs, WHFs and species at risk.
Therefore, pilot studies and studies to collect baseline data are recommended prior to or as a first step when
implementing effectiveness evaluation projects. Baseline data from a pilot study can be used to determine
necessary sample sizes and improve field and analytical methods, and later, baseline data, if collected using
the same or a comparable methodology, can be used in comparisons of subsequent measures of habitat or
species biology and estimation of trends over time.
Benchmark data include measures of condition of the habitat or species in a reference condition other than
the baseline value. Benchmarks can be standards or targets you are aiming for or a meaningful point of
reference in time, space or condition. Comparison of effectiveness evaluation data to benchmark data can
provide insight into level of risk and the need for changes in management practices.
6
http://www.for.gov.bc.ca/hfp/frep/values/wildlife.htm
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 23 -
Sometimes a benchmark is a biological threshold value, a target or a management trigger point, that, when
reached would signal a need for a change in management practice. Targets are set either by regulation (or
policy), by certification standards or sometimes by scientific evidence. For example, the density of wildlife
trees in a management zone or the number of bats that use a roost tree in a WHA. Thresholds are usually
upper or lower limits that serve as triggers or early warning signals for responsive action and might be
estimated and then refined through Adaptive Management.
Control data describe a group of subjects (e.g., wildlife species) or conditions (e.g., habitat parameters) that
is not exposed to any treatment (either experimental or operational) and is matched as closely as possible
(i.e., all else being equal) with an experimental group of subjects or conditions. The untreated control group
is used as a standard or yardstick, to detect and measure changes that may occur in the treated group due to
the treatment.
Evaluators will want to compare observed conditions or trends of indicators with
expected values, thresholds, triggers and targets and make predictions of indicator values
expected as a result of the management practice. These predictions should be reflected in
explicit hypotheses regarding expected future conditions and trends tested in the
effectiveness evaluation. Forecasts of indicator values should be as explicit as possible,
ideally being quantitative estimates with confidence limits.
Quantitative models provide valuable input to forecasts of future conditions of indicator
values; simpler models may need to be adjusted, for instance to include alternate
management scenarios. Models and predictions for indicator values can be updated with
new information gathered during monitoring.
Sections 2.5.1 and 2.5.2: Selecting indicators – in practice
 Indicators are measurable attributes or characteristics of a resource value that can
provide reliable information on the sustainability status or state of the resource.
 With expert input, and inconsideration of existing current state information, examine
the conceptual model of relationships to determine potential parameters and variables
that could be measured over time to indicate achievement of FRPA-wildlife resource
value objectives.
 Select highest priority, feasible indicators for effectiveness evaluation study.
 Determine critical threshold, trigger or target values of the selected indicators.
 Predict indicator values as a result of management practice.
2.5.3 Developing an Evaluation Tool for Effectiveness Indicators
Section 2.5.3: In this step, we examine potential outcomes of all indicators combined to
create an evaluation tool, by asking:
 What levels of the population or habitat condition indicators indicate the population
or habitat condition is secure, at risk, or inadequate?
 What levels of the threat indicators indicate extreme, unacceptable, moderate or low
levels of risk to the population, its habitat or the integrity of the WHA or UWR?
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 24 -

What combination of condition and risk indicator values will lead to the conclusion
that the WHA or UWR is effective, at risk or ineffective at meeting management
objectives?
With a number of indicators selected for monitoring and evaluation, the project team
must consider how the indicators will be evaluated to determine effectiveness of the
WHA or UWR. That is, how will the results of the monitoring of population and habitat
condition indicators and threat (risk) indicators be combined for an overall conclusion
about effectiveness?
This step requires:
1) pre-defined categories of condition for each indicator of population and habitat
condition (e.g., poor, satisfactory or good condition; declining, stable or
increasing trend);
2) pre-defined categories of threat for each indicator of risk (e.g., low, moderate,
extreme; long-term or immediate);
3) decision-making processes or protocols (Fig. 1) for assigning overall rating of
condition to the combined suite of indicators (e.g., highly, moderately or not
functional) and overall rating of risk to the combined suite of risk indicators (e.g.,
low, long-term or immediate); and
4) a tool, such as the matrix of Table 2, for assigning effectiveness ratings to the
study area based on overall ratings of condition and risk – that is, a pre-defined
effectiveness rating scale that reflects all potential combinations of condition
rating and risk rating.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 25 -
Figure 1. Example decision-making process: Assessment process for assigning badger
WHA functionality ratings (from Kinley 2009, quoting Newhouse et al. [2007:6]):
“Densities of recent (occupied during year of survey) badger or ground squirrel burrows
considered to be ‘high’ are to be determined regionally based on the results of baseline
WHA assessments. Ground squirrel densities are to be considered only in regions where
ground squirrels are the predominant prey. The density of recent badger burrows must
decline by at least 30% annually (50% biennially) to be considered ‘negative’.”
Table 2. The standard wildlife resource value effectiveness ratings matrix 7 with
typical functionality and risk categories and conventional management
interpretation.
7
see the FREP wildlife resource value monitoring and evaluating framework (2009) at
http://www.for.gov.bc.ca/ftp/hfp/external/!publish/frep/values/Wildlife_Framework_Paper.pdf
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 26 -
Condition
Risk
Low
Moderate
High
High
Effective
Effective
At risk
Moderate
Effective
At risk
At risk
Low
unknown
Not
effective
Not
effective
Management interpretation: Not Effective and At Risk effectiveness ratings
indicate an immediate need for action to prevent or reverse loss of viability of
the WHA or UWR. Not Effective, At Risk and Unknown effectiveness ratings
indicate a need for management changes. An Effective rating does not require
a change in management regime or new actions aside from continued
monitoring.
Assigning condition (functionality) and threat (risk) rating categories to a WHA or
UWR (Figure 1 and Table 2, above) requires a priori recognition of indicator critical
values that have biological or management relevance such as thresholds, targets or
trigger values (see 2.5.2). Critical values (or values relative to them, such as 50% of
the critical value) are often the cut-off values for condition or threat rating categories.
Categories may also reflect direction of trends (favourable versus unfavourable) in
condition or risk. The rating categories are defined from expert opinion or previous
scientific evidence before the effectiveness monitoring begins. They are often key
factors represented in preceding conceptual models.
Assigning overall rating of condition or risk to the combined suite of condition or risk
indicators (step 3, above) can be complex and requires consideration by species
experts. For example, an indicator may have greater importance for species survival
or feasibility of related management action and therefore be weighted more heavily
than other indicators. In another case, a high risk rating for one indicator may
override any lesser value of other risk indicators. All possible combinations of risk
and condition ratings and any situation specific interrelationships among the ratings
should be considered and assigned to overall effectiveness ratings prior to
implementing monitoring.
The effectiveness evaluation matrix (Table 2) that considers condition ratings in
context of risk ratings is a tool that can be used in all evaluations of effectiveness of
wildlife resource value management practices. The categories of functionality and
risk may be adapted (the wording changed) to reflect different projects and
hypotheses (with sufficient knowledge for more categories), but the 3x3 format is
standard for the wildlife resource value effectiveness monitoring and evaluation
program.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 27 -
It is important to consider this final step of effectiveness evaluation before beginning
data collection (a priori) in order that the study approach and sampling design
provide the appropriate data to make the assessment. However, the project team
should re-examine the evaluation tool and revise it (or interpretation of it) as
necessary following the outcome of a pilot study or several years of monitoring (post
hoc).
Section 2.5.3: The evaluation tool – in practice
To prepare for the final step in an effectiveness evaluation, the project team must:
1. Develop a flowchart or decision-tree protocol that will be used to determine
whether the combined condition (functionality) indicators indicate the WHA or
UWR is highly functional, moderately functional, or not functional (other
categories could be good condition, moderate condition, and unacceptable
condition).
2. Develop a flowchart or decision-tree protocol that will be used to determine
whether the combined threat (risk) indicators indicate the level of risk to the
WHA or UWR is nil-to-low, long-term or immediate (other categories could be
low, moderate or extreme risk).
3. Determine the overall effectiveness rating from a matrix (the standard wildlife
resource value program “effectiveness rating matrix”) that cross-references
condition (functionality) and threat (risk) ratings to conclude the WHA or UWR is
effective, at risk or not effective.
2.6 Pilot testing a protocol
During the development of an effectiveness monitoring protocol, pilot studies should
be undertaken to test assumptions, field and analytical methods, and data quality, and
ultimately to improve the protocol. The project leader needs to determine whether a
protocol that is to be implemented is in its final form or requires pilot testing.
Pilot studies require development of a preliminary study design and sampling design
(Section 3) before being implemented (Section 4); the outcome of the pilot may result
in changes to the study design, the sampling design, or the data collection or
analytical methods. The process of feedback for improvement is the crucial product
of pilot studies.
Most projects will require one (or possibly two) season for testing study design, field
methods, data forms and analyses. A pilot season will provide important feedback to
improve methods and outcome of the project and could provide new or updated
baseline data.
Pilot might evaluate …
feasibility of methods and delivery
validity of indicators
focus of questions
Possible feedback:
Adjust methods
Drop or add indicators
Refine questions
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 28 -
utility of field forms
soundness of sampling design
rigor of data, power of statistical tests
rigor of effectiveness evaluation matrix
Revise field forms
Adjust design
Amend rigor
Amend matrix
2.7 Tracking Project and Program Efficiency
Section 2.7: In this section we ask ...
 Is the project cost-effective?
 Are the project (or program) resources (funds and people) well spent?
 Are we getting value for the dollars and effort spent?
Alongside the development and implementation of wildlife resource value
effectiveness evaluation projects, it is important to consider how to assess and track
project or program efficiency. For instance is the money and effort well spent, are we
getting value for the dollars spent, are we getting the best bang-for-the-buck?
Program efficiency evaluation is different from effectiveness evaluation which
measures outcomes of management practice to protect the species or habitat. Program
efficiency measures the success of wildlife resource value program administration,
coordination and leadership in terms of costs and benefits. Information collected to
track efficiency of each project contributes to the assessment of efficiency of the
program and ensures continuing support for the monitoring project.
Performance measures for efficiency of effectiveness evaluation projects are similar
to those of program efficiency and fall into three categories: examples of measures in
each category include  Cost effectiveness
o expenditures (dollars, staff time) – totals and trends
o feasibility of field logistics
 Project (or program) efficacy
o numbers and types of partnerships and co-operators
o level of funding from other sources (partners)
o amount of co-operative data collection (sharing data or data collection
with other FREP or other monitoring projects or programs)
o volume, distribution and timeliness of extension deliverables
 Contribution to improved management
o (trends in) public awareness of management issue and of management
efforts
o changes in management expenditures with gained knowledge
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 29 -
Measures of project efficiency will differ depending on the stage of development of
the effectiveness evaluation: that is, whether the protocol is in development, a pilot
study is underway to test a protocol, or a final protocol is being implemented.
Efforts to track project efficiency should be included in project work plans and
reports.
3 DESIGNING A STUDY
There are two key steps to designing a statistically rigorous study: (1) designing the
overall approach referred to as the study design and (2) establishing the rules that
govern what is measured, referred to as the sampling design. Careful consideration of
both is crucial to successfully addressing the management questions and monitoring
objectives.
The need for careful consideration of the study design cannot be overstated.
Sometimes an intensive experimental design (with treatments or manipulations) may
be undertaken but in many cases we must rely on simpler observational studies.
However, with observational studies one is less certain that the presumed treatment or
activity actually caused the observed response or result (Shaffer and Johnson 2008).
It is strongly recommended that a statistician with bio-statistical experience be
included in discussion and review of the proposed study design, as well as the
sampling design and data analysis plan. Statistical review early in project
development will ensure the right parameters are measured, enough measurements
are taken, effort is not wasted on extraneous sampling, the data are sound and
credible, estimates of reliability can be calculated, the sampling scheme is
appropriately rigorous for robust inferences, and the analyses will provide an answer
to the management issue or question.
Decisions made during the design stage need to be transparent so the monitoring
project may continue for many years even if the project leader has moved on. A clear
documentation of why, what, when, who and how will sustain the program well into
the future.
3.1 Designing the Approach
Section 3.1: During the Design phase, the following questions will be answered:
 What is the key question(s) and the hypothesis being evaluated?
 What is the general delivery model; who will be conducting the various aspects
(planning, field work, analysis, reporting) of the project – HQ, region, district,
contract, MOE/MFR, industry?
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 30 -






At what geographic scale is the project to be conducted – stand, landscape,
region?
At what temporal scale is the project to be conducted – at what time of year and
at what interval?
At what intensity is the project to be conducted – routine, extensive, intensive?
How will sites be selected for monitoring?
What field methods will be used – existing standards or innovative procedures?
What overlap or commonalities exist between this and other monitoring projects?
The first step of designing the study is deciding on the overall approach. What are the
studies objectives? Where will it be conducted? In general, when and how will data be
collected? The components of the study design are addressed in the following sections.
The second step is sampling design. Specific sampling design topics are covered in
Section 3.2.
3.1.1 Study design
There are many possible ways in which to approach an environmental field study (see
Eberhardt and Thomas 1991 for an overview). Choosing the right approach requires
careful consideration of the study objectives, the degree of control required, the desired
level of inference, the effect size of interest, and the tradeoffs surrounding issues of cost
and feasibility of the various approaches. Cochran (1977) describes two broad types of
survey: descriptive and analytical. The objective of descriptive surveys is to obtain
information about general categories of objects (e.g., the frequency of large woody debris
pieces in a watershed); whereas, analytical surveys are used to make comparisons among
groups within the population in order to test hypotheses (e.g., are there fewer large
woody debris pieces in FSWs than in undesignated watersheds?). Hurlbert (1984)
categorizes studies as either: manipulative experiments or mensurative experiments,
where manipulative studies are those where the investigator has control over the factors
in the study and mensurative studies are those where only passive observations are used.
Eberhardt and Thomas (1991) include replication as a key requirement for improving the
strength of inference and describe eight categories of environmental studies that range
from the preferred approach of a controlled experiment with replication to a simple
descriptive sampling approach. Schwarz (2006) provides an excellent summary of the
tradeoffs between different study approaches ranging from descriptive surveys to
designed experiments. Figure 2 illustrates the relationship between the degree of control
and the strength of inference possible for an array of study designs. This section on study
design was adapted from Wieckowski et al. 2008. Examples using WHA monitoring are
described in Appendix 1.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 31 -
Figure 2.
Relationship between degree of control, strength of inference (and ability to determine
causation), and type of study design (from Schwarz 2006).
As a first step, describe generally the overall approach and objective. Houde et al.
(2005) discuss four general approaches for effectiveness monitoring: operational,
experimental, design-based and model-based. Operational approaches use real-life
and readily available treatments or practices applied at an operational scale under
operational conditions. An experimental approach creates and compares different
treatments. Design-based approaches are based on a statistical sampling design
compared to model-based approaches that intensively collect information at a few
representative sites (sometimes called sentinel sites) to create an ecological model
which is applied to similar sites. The most appropriate approach or combination of
approaches depends on the objective of the study (or monitoring question).
Also consider the kinds of comparisons, if any, that will be made: before and after
treatment comparisons (such as before and after harvesting or before and after
implementing a mitigative management practice), comparisons to control sites (e.g.,
no treatment sites), benchmark (desired condition) sites, or to important threshold or
management trigger values.
3.1.2 Delivery model
The overall delivery model is another component of the overall approach. How the
project will be implemented is described in a general level, such as “led from
Headquarters with assistance from District staff”, or “conducted by consultant under
contract administered from Coast Region”. The overall delivery model also describes
inter-agency, academic and/or industry partnerships and proposed or confirmed
funding arrangements.
3.1.3 Spatial scale
The spatial scale of a project is another component of the approach to an effectiveness
monitoring project. As a general rule of thumb, the spatial scale of effectiveness
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 32 -
monitoring should match the scale of the project objectives and management
strategies and actions.
Setting the spatial scale determines how widely the results of the monitoring project
will apply. Extrapolation of study results beyond the area of inference is a common
problem in biological studies. We want to be sure the inference drawn from the
sampling population applies to the broader target population (see section 3.2).
Some projects are at a site-specific scale, focusing on a specific individual UWR or
WHA – for example, a study examines whether desired structural elements and
habitat conditions exist or are being maintained within one WHA. If a species
targeted for conservation occurs entirely within a WHA – does not migrate outside its
boundary – or conservation actions and objectives are restricted to the WHA, then a
site-specific scale is appropriate.
Projects that are at a landscape unit or watershed scale are designed to answer
questions about multiple sites, such as whether current UWR boundaries fit patterns
of observed animal use, whether habitat distribution and abundance within a group of
WHAs is sufficient for species survival or whether additional protection is needed for
other life requisites. If the distribution of a target species is not limited to a WHA or
there are multiple reserves in an area, a landscape scale assessment would be best.
At a regional or sub-regional scale, or even at a provincial scale, projects are designed
to determine whether UWRs, for example, are in sufficient numbers and adequately
distributed to maintain healthy regional populations. If, for example, the target
species is a regionally ranging species and regional scale conservation actions are to
be undertaken, then a regional scale evaluation is appropriate.
3.1.4 Temporal Scale
The temporal scale (timing and duration) of the monitoring and evaluation project is a
valuable descriptive component of the overall approach. This is just a general
statement – the details are more fully developed in the sampling design (see Section
3.2). Temporal scale addresses: What time of year? What re-sampling interval –
weekly, monthly, every three years? Duration of monitoring? Expected completion
date?
3.1.5 Intensity
The overall approach should include a description of evaluation intensity, of which
FREP recognizes two levels8, routine and extensive. The levels reflect the level of
effort, costs and complexity, which depend on the monitoring questions, indicators
and methods, resources available and circumstances. Routine evaluations generally
involve relatively inexpensive and rapid but statistically valid measurements and
visual assessments, including office-based computing exercises. Extensive
8
See more details of the categories of monitoring adopted by FREP at:
http://www.for.gov.bc.ca/hfp/frep/rsm/index.htm
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 33 -
evaluations are usually more focused and rigorous, with more detailed on-the-ground
monitoring involving the collection of categorical data using visual estimates or
relatively simple measurement; data are more quantitative and data analyses involve
detailed comparisons.
Routine and extensive evaluations provide valuable information on current status,
trends and implementation issues and can be applied to different types of wildlife
resource value monitoring. Evaluations may be extremely detailed long-term
commitments or quick assessments designed to identify key areas of concern. Levels
of intensity are not sequential – projects must be initiated at the level that reflects
program priorities and information need.
3.1.6 Location
Location is an important component of the overall project approach. This is a
description of the geographic location (this can be broad or specific; for example:
Northeastern BC, Coast Forest Region, North Island District, ICHdw BEC subzone,
or WHA-999 and its specific location); and the rationale and factors considered in
selecting that location (describing what makes it special or the best site for this
evaluation).
3.1.7 Methods
A brief description of field and office procedures, methods, and techniques is also
part of a description of the overall project approach. This is not the details of the
sampling design (see section 3.2) or the details provided to the field staff – it is a
overview statement. For example, the statement might be a brief description of
existing standard techniques that will be used or adapted, or innovative methods that
will be developed. It might include intentions to ensure data consistency, quality
assurance, or data storage and analysis standards. Similarities and differences in
methods selected for this project from other monitoring projects (especially FREP
monitoring projects) should be noted and briefly rationalized.
3.1.8 Overlap with other monitoring – FREP and other programs
In describing the overall project approach it is important to note any overlap and
commonalities of methods, data collection, sites, etc., or cooperative efforts with
other projects under FREP or any other program.
Section 3.1 Project Design – in practice
Determine the overall project approach in terms of :
 Overall study design and delivery model
 Spatial and temporal scale
 Level of monitoring intensity
 Geographic location
 General description of methods
 Program overlap
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 34 -
3.2 Sampling Design
Section 3.2 A monitoring project will likely involve monitoring several variables. In
this Sampling Design step, given the study design, we must examine the following
questions about all the variables to develop a sound sampling design prior to project
implementation  What is the target population?
 What variable(s) will be measured?
 What is the sampling unit for the measured variable(s)?
 Will samples be stratified or random?
 What sample size is required? Consider effect size.
 How many times a year will you sample a point? In how many years would you
like to detect a change or trend?
 What analytical methods will be applied, and at what levels of significance (alpha
level)?
 What data, if any, is available for measured variables to provide guidance on the
natural variability in measured variables?
 How much money and manpower is available to project?
The purpose of this section is to guide the user through the necessary steps to create a
sampling design. Thompson (2004) describes two necessary components to an
effective sampling design: the spatial and temporal selection of sampling units, and
the measurement protocol within a given sampling unit. The first component
addresses the fact that in most ecological studies we can’t sample all possible areas in
all possible time periods. This sampling variability is the subject of most traditional
statistical sampling books. The second component addresses the fact that in many
ecological studies, especially with rare or elusive species, that the probability of
detection is less than one. In other words, even if an animal or plant is present in a
sampling unit, it may not be detected.
It is also necessary to consider the data analysis during the planning stage; ask
yourself “What would you do with the data, if you had it?” Both spatial and temporal
selection as well as detectability should be incorporated into a sampling design
(Thompson 2004; Yoccoz et al. 2001; Thompson and Seber 1996) along with a
discussion of the intended data analysis (Thompson and Seber 1996). The ability to
create an effective sampling design depends on clearly articulated study objectives as
described in Section 2.1 and 2.2. and knowing the key questions and hypotheses
being evaluated. Given clear objectives, a pilot study, past research, or expert
knowledge should be able to provide the information needed to work through these
questions and design an effective study.
This section is organized into eight steps of developing a sampling design, adapted
from recommendations by Elzinga et al. 2001 and Vesely et al. 2006.
Section
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 35 -
3.2.1
3.2.2
3.2.3
3.2.4
3.2.5
3.2.6
3.2.7
3.2.8
Describe important life history characteristics
Define the target population
Choose an appropriate sampling unit
Determine how should sampling units be positioned
Determine an appropriate sample size at each step of the design
Determine an appropriate sampling frequency and time frame; should
sampling units be permanent or temporary?
Describe the field measurement protocol
Describe how you intend to analyze the data
3.2.1 Life History Characteristics
The life history characteristics of the subject species drive the decisions made at each
step of the sampling design development. A description of life history should
include: what is known or unknown about the spatial and temporal distribution of the
species, some indication of abundance, reproductive strategy, detectability, habitat
use, and anything else related to the life history of the organism which might affect
the sampling design. This step should follow easily from the conceptual model and
indicator development in Section 2.4 and 2.5.
3.2.2 Target Population
The target population has been described as the complete collection of individuals we
wish to study (Lohr 1999); the population about which information is wanted
(Cochrane 1977); or the complete set of units about which we want to make
inferences (Elzinga et al 2001). In order to make inferences about the entire target
population, all individuals within the target population must have some chance of
being selected in the sample. The sampling population is the collection of all
possible sampling units that might have been chosen in a sample; the population from
which the sample was taken (Lohr 1999). The sampling population should be the
same as the target population, although often due to time or budget constraints the
sampling population will be smaller than the target population. If this is the case then
any inferences drawn from the sample only apply to the sampling population
(Cochrane 1977). Extending inference beyond the sample population requires
additional information and expert judgment. If the sampling population doesn’t cover
the entire target population, then collecting supplemental information to describe how
the two populations differ may help to understand the limitations of the inference.
For example: if the target population is all fish-bearing streams in British Columbia,
but the sampling population only included streams up to a certain elevation, then we
shouldn’t extend inferences based on lower elevation streams to those at higher
elevation. Similarly, if deep pools couldn’t be sampled due to some logistical
constraint, then we shouldn’t make inference to deep pools, as they would effectively
be removed from the target population.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 36 -
Figure 3. Three different conceptual relationships among the target population of
inference, the sampling frame, and the sample of units drawn for characterization (
excerpted from Skalski et al. 2005, pg. 364)
3.2.3 Sampling unit
The sampling unit is the actual unit of measurement. The sampling population is
divided into many sampling units. The list of all possible sampling units is called the
sampling frame (Lohr 1999). Individual animals or plants generally represent the
sampling units for measured organismal characteristics such as: height, weight, and
health. Plots or similar area delineations are generally the sampling units for
population measures such as: abundance, density, or biomass. However, it may be
necessary to have multiple levels of sampling units. For example, if interested in
characteristics such as height or weight, individual animals seem the obvious
sampling unit, but without knowing how many animals exist or their locations, you
can’t actually create a sampling frame. Consequently you may first have to define
sampling units as plots or transects in space, and then measure characteristics of
individual animals observed/captured within the plot or transect. This is an example
of a multi-stage sampling design, where the plot or transect is the primary sampling
unit and the individual organism is the secondary sampling unit. Multi-stage
sampling designs are very common and are discussed in more detail later in this
section.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 37 -
If the sampling unit is some measure of area, such as a plot or a quadrat, the next
thing to consider is the size and shape of the sampling unit. Generally the aim is to
choose the sampling unit that provides the most precision for the least effort. A good
strategy to accomplish this is to choose the size and shape of sampling unit that will
minimize the between-unit variability without adding too much sampling effort. A
good idea, if feasible, is to choose a plot size large enough to ensure some
observations or captures within the sample unit (i.e., avoid too many zeros). An
excessive number of plots with zero individuals of interest will likely complicate
associated analysis.. Elzinga et al. 2001 provide a useful summary of key
considerations when choosing the size and shape of the sampling unit for ecological
studies, a condensed list is provided here:
 Travel and setup time vs. searching and measuring time: Evaluate the tradeoff
in effort and precision between measuring fewer large plots vs. more small
plots. How hard is it to find and measure the individuals of interest? Are
individuals large and conspicuous like trees, or are they small and difficult to
find?
 Spatial distribution of the individuals in the population: Many species of
plants and animals have clustered distributions. If the distribution is clustered
then small plots will tend to result in frequent zeros, whereas larger plots will
be more likely to intersect a cluster. The orientation of the sampling units is
also important. For example, if there is a possible gradient in the
measurement of interest related to some environmental covariate such as sun
exposure, slope, or moisture content, then the sampling units should be
oriented to cross the gradients. For example, if moisture content was an issue,
between-plot variability could be reduced by laying out plots so that each plot
had a similar range of moisture content. An alternative would be to stratify
plots based on the environmental differences (see discussion on stratified
sampling).
 Edge effects: It can be difficult to determine whether an individual lies within
or outside a plot boundary. Some observers may tend to include all boundary
sightings, while others may ignore them all. Either way can lead to biased
estimates and the sampling protocol should explicitly define a consistent rule
for how to handle edge observations. Rectangular plots have the greatest edge
per unit area, while circles have the least.
 Abundance of target population: An initial sense of abundance is useful in
choosing an appropriate sized sampling unit. The goal is to avoid either
creating a sample unit so large that you will have to count thousands of
individuals or so small that you end up with many empty plots.
 Ease in sampling: Long, narrow quadrats are generally easiest to search
(Krebs 1989, Elzinga et al. 2001). An observer can move in one direction
through the entire plot with little likelihood of losing track of individuals
already counted. As an example, in a recent pilot study (Lisa Tedesco pers.
comm.), a long rectangular plot was found preferable to a circular plot of a
similar area for measuring grass height. The rectangle plot allowed clear
sighting of all of plants and measurement with little extraneous movement.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 38 -

Conversely in the circular plot, it was difficult to keep track of which plant
had been measured and it was far more time consuming as the observer had to
shift position frequently.
Disturbance effects: It is important to consider the impact the sampling may
have directly on the species of interest and on the interpretation of monitoring
results. This is important for two reasons: first and obviously, it would be
counterproductive to impact the species of interest (which in the case of the
FREP program will often be a species at risk); secondly, sampling disturbance
to sites (especially around permanent sampling sites) could over time affect
the behaviour of the monitored species.
Results from pilot studies, information from other published research, or expert
opinion based on biological characteristics can be used to choose the appropriate size
and shape of the sampling unit according to the above considerations.
3.2.4 Positioning of sampling units
Now that the target population, sampling frame, and sampling unit are defined, how
should the actual sample be chosen? Probability sampling theory requires that each
sampling unit has a known probability of being chosen and that they are selected at
random (Cochrane 1977).
Standard probabilistic designs:
There are three basic probabilistic sampling designs that are most commonly used:
simple random sampling (SRS), systematic random sampling (SysRS), and stratified
random sampling (StrRS). Simple random sampling is when a random sample of all
sampling units within the sampling frame is selected, for example: drawing numbers
from a hat. Systematic random sampling is when sampling units are selected at
regular intervals using a randomly selected starting point, for example: reading every
tenth name from the phone book, or taking a sample every ten meters. Stratified
random sampling is when the sampling frame is divided into non-overlapping groups
(strata) based on some characteristic such as sex or habitat type, and then a random
sample is chosen from each of the strata. SRS can always be used, but this may not
always be the most efficient choice. In other words, it may require a larger sample
size to obtain comparable information about the population using a SRS as opposed
to another more efficient sampling design. If the population of interest is randomly
distributed then the SysRS approximates the SRS (Lohr 1999). If the target
population changes proportional to position (e.g., samples taken upstream vs.
downstream) a SysRS may be an appropriate way to ensure spatial coverage. If the
target population displays regular or cyclical characteristics then a SysRS is a poor
choice. A StrRS is more efficient when there is less variability within strata than
between strata (Cochrane 1977; Lohr 1999). It may also be useful if estimates for
individual strata are desired as well as for the entire population.
Generalized Random-Tessellation Stratified (GRTS) designs:
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 39 -
GRTS is a spatially-balanced probabilistic survey design developed by the US
Environmental Protection Agency (EPA) under their Environmental Monitoring and
Assessment Program. GRTS overcomes some of the shortcomings of both simple
random sampling (which tends to “clump” sampling sites) and systematic sampling
by generating an ordered, spatially balanced and unbiased set of sites that represent
the population from which the sample sites will be drawn. Software required to
create a GRTS design include: psurvey.design, R statistical package (both available
from the US EPA’s Software Downloads for Spatial Survey Design and Analysis
webpage), and ArcGis. The EPA’s Monitoring and Design team
([email protected]) will also supply customized GRTS
generated sample points for selected areas upon request, as long as the client
indicates or provides the necessary GIS layers for the intended spatial sample frame
(e.g. stream networks, polygon boundaries etc.). The example GRTS design in
Figure 4 shows a random, spatially balanced draw of a pre-selected number of
sampling points within each of 5 user-defined stream order strata within a watershed
hydrology network.
Stream
Order
1st
2nd
3rd
4th
5th
% of sample
sites
10
10
40
30
10
Figure 4. Hypothetical EPA GRTS draw of random, spatially balanced sampling points within stream
order classifications for the Methow Basin in Oregon (source: Phil Larsen, EPA)
Judgment or non-probabilistic designs:
Judgment samples are selected subjectively. The scientist may choose sites according
to some prior belief about where individuals should be found, they may be chosen
arbitrarily to be representative of the target population, or they may be chosen just for
convenience. So called ‘representative reaches’ in stream surveys are an example of
a judgment sample. However, without a census of the target population (e.g.,
stream), it is impossible to be sure that you have chosen a representative sample (e.g.,
reach). McDonald (2004) describes many examples where allocating some effort
outside supposed core areas provided considerable improvement in their
understanding of distribution and estimates of abundance for rare and elusive
populations. For example, if core habitat had relatively consistent densities, while
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 40 -
marginal habitat density fluctuated with the size of the population, then ignoring the
marginal habitat would mask the true health of the population. Sometimes sites that
are thought to be particularly sensitive ‘sentinel sites’ are selected for monitoring, but
this too is a controversial issue (Edwards 1998). Making inference from judgment
samples only has lead, for example, to some famous miscalculations in predicting
election results (Edwards 1998).
Non-standard probabilistic designs:
There are many other variations on these basic designs that have been developed to
address particular situations including: cluster sampling, adaptive sampling, and
distance sampling (see Appendix 2 for other sampling designs). Any combination of
these designs can be combined in a multi-stage sampling design. For example, a
simple random sample of transects could be chosen from each stratum within the
target population, and then a systematic random sample of plots within each transect
could be selected, followed by a census of individuals within each plot. Calculating
an estimate of the mean from a multi-stage sample is fairly intuitive, but the variance
calculations are more complicated. The typical mistake that is made is to treat all of
the observations as though they were drawn at random from the target population. In
reality the secondary sampling units (plots in this case) are a sample from the
transect, not the population. Increasing the number of plots within transects only
helps improve the precision of the estimate of the single transect. This is one form of
pseudo-replication as discussed by Hurlbert (1984).
Strategy:
There is no set prescription for deciding how to position sampling units, but
stratification can be a very powerful tool for improving the efficiency of a design, and
so assessing whether or not stratification is appropriate is a good place to start. A
series of questions combined with an understanding of the strengths and weaknesses
of each of the different sampling designs should enable an appropriate allocation of
sampling units within the population or strata.
a. Is a stratified design appropriate?
 Are there obvious groupings to the target population, such as: ownership
boundaries, habitat types, differences in density? If so, is it reasonable to
think that the variability between groups would be greater than the variability
within these groups?
 Would it be useful to know how precise the estimates are for particular groups
within the populations?
b. How should sampling units be positioned within the population (or strata)?
 How big is the total target population?
 How much time does it take to move between sampling locations?
 Are the objects distributed at random, uniformly (typical of territorial
animals), or in clusters?
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 41 -



Are there any regular features in the landscape, such as: ridges, fence lines,
roads, riffle/pool sequences... that might be related to the response variable of
interest?
Are there any known gradients in the target population, such as: upstream vs.
downstream, low moisture to high moisture, elevation etc…?
How time consuming is the actual measurement process within each sampling
unit?
3.2.5 Sample size and statistical power
The next step is to determine the appropriate number of samples to take at each level
of the sampling design to ensure sufficient precision for the study objective and
optimize the allocation of monitoring effort and cost (too many samples waste dollars
and effort, too few samples can’t answer the questions of interest).
Sample size calculators use preliminary estimates of variability to help you assess the
trade-off between effort and precision. There are many free programs (e.g., Program
TRENDS or MONITOR) available to help determine the best sample size for your
project and they all function in essentially the same way.
In general sample size calculations require:
 estimates of variability within and between sampling units at each stage of the
design (preliminary estimates of variability may be obtained through pilot
studies, similar studies conducted elsewhere, or theoretical behaviour of the
system);
 the desired level of precision (alpha and beta levels – see below re: power);
 cost of sampling at each stage of the design;
 the significance test of interest (i.e. the difference between two groups or a
significant trend over time); and
 knowledge about the distribution of the data of interest.
There is a danger in using a generic sample size calculator without understanding the
underlying assumptions of that calculator. Sample size calculations depend on the
study design (e.g. simple, stratified, multi-stage etc…), the distribution of the data,
the study objective (e.g. comparing two groups vs. assessing trends). The most useful
calculators include some measure of cost/effort in a sample size calculator in order to
optimize the precision against cost trade-offs. Since the calculations are generally not
difficult, it is probably a good idea to have a statistician develop a sample size
calculator for your project, especially if the study/sampling design is unusual.
In statistics, power refers to the ability to detect an effect (i.e., a significance test of
interest such as difference in distribution, shift in location, trend over time etc…).
Power analyses are based on the same concepts as sample size calculations, but are
used to determine the size of that effect. The terms “alpha level” and “beta level” are
often used to describe the two possible errors made in analysis of data: alpha level
refers to the probability of not detecting an effect given that it exists (false negative);
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 42 -
and beta level refers to the probability of detecting an effect when it doesn’t exist
(false positive). Power is defined as 1-beta, or the probability of detecting an effect
given that an effect does exist. In order to calculate the power, the kind of statistical
test that will be performed (e.g., one- vs. two-tailed t-test) must be known. Typically
statistical power is calculated for a range of sample sizes where increasing sample
size results in increasing precision and therefore power. It is useful to display power
as it changes with sample size and cost/effort.
3.2.6 Sampling frequency
Should the samples be collected daily, monthly, annually, every 10 years…? The
appropriate sampling frequency will differ by study objective and species. Is the
study interested in a one-time estimate of status, or is the objective to monitor the
species or community over time? Often in ecological studies the temporal scale of
interest is annual (i.e. annual estimates of abundance, health, survival etc…). These
estimates can also be combined into multi-year studies that allow for estimates of
trends over time and comparisons between years. There are a number of interesting
and challenging statistical design questions that arise depending on the temporal scale
of interest.
Deciding when to sample within a year:
When and how often should sampling be done within a year in order to obtain annual
estimates? This depends on the life history of the species of interest and the target
population. Generally speaking the best time to sample the population is when the
probability of detection is the greatest, either due to behaviour, colour (i.e. mating
colouration or vegetation colouration), time of day, or season.
Questions to consider:
 Does the species migrate or hibernate?
 Does the behaviour change with season? For example, do they become
more or less conspicuous during mating season?
 Is the species nocturnal or diurnal?
 Does the species only live for part of a year (plant or animal)?
 Are there any major events within the year that could change the response
variable of interest?
o Hunting season
o Significant overwinter mortality
o Grazing, if interested in grass height
 If measuring an important habitat feature like cover availability that might
change for a species with season, is your question of interest cover
availability during the best conditions, worst conditions, both, or only
when the species of interest is present?
 What period of time within the year are you interested in making inference
about?
Long-term monitoring programs:
Sampling designs spanning multiple years are important when trying to consider
changes to individuals or populations over time. There are many conflicting opinions
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 43 -
about how sampling designs over time should be implemented. A major discussion
point is whether permanent (long-term) or temporary sites should be selected. The
typical response is: “use permanent sites for trend detection and temporary sites for
status assessment”. In reality it is rarely that simple. There are advantages and
disadvantages to each approach and researchers differ in their opinions of which
approach to take. A brief summary of the advantages and disadvantages to each of
these approaches is described here to provide some insight into the debate.
Temporary sites:
Advantages:
 good procedure for locating rare items (McDonald 2003)
 can use to estimate ‘net change’ (McDonald 2003)
 better strategy for estimating the average over all occasions (Cochrane 1977)
 removes the cost of permanently marking sites (Elzinga et al. 2001)
 can automatically account for changes in population composition (McDonald
2003)
 no conditioning impacts
Disadvantages:
 lower efficiency method for net change (McDonald 2003)
 does not allow estimation of individual change (McDonald 2003)
Permanent sites:
Advantages:
 useful when there is a high degree of correlation between sampling units from
one period to the next, such as might occur with long lived plants or long lived
and sedentary animals (Elzinga et al. 2001).
 planning and survey design work are minimized (McDonald 2003)
 well suited to estimate gross change and other components of individual
change (McDonald 2003)
 most powerful for detecting linear trend (Urquhart and Kincaid 1999)
Disadvantages:
 risk that simply due to chance, the sites selected will be the ones that change.
A new sample each year would have a better chance of picking up a true
change, but the sample size needed in each time period would be greater
(Vesely et al. 2006)
 only effective if the sampling effort is sufficient to represent the range of
conditions across the area of inference (Vesely et al. 2006); this approach
can’t account for changes in population composition over time (McDonald
2003)
 can be costly to permanently mark sites (Elzinga et al.)
 serious potential for impacting the site in such a way that affects the measured
variable through repeated years of sampling (Elzinga et al. 2001; Buckland et
al. 1993; McDonald 2003)
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 44 -



permanent markers may provide a perch for raptors or songbirds
vegetation may be trampled
habitat or individuals may be disturbed
An interesting finding by McDonald (2003) was that the term ‘trend’ is used in different
ways by different researchers; perhaps this is the source of some of the controversy
around ‘permanent vs. temporary’ sites. For example, what scale is the change of
interest: reach, stream, watershed, province…? McDonald (2003) provides several useful
definitions that may be helpful in clarifying the study objective and hence determining
the best design:
Net change: measurement of total change in a parameter arising from all sources
 change in mean or total response
 individual change can happen without causing net change (as fish move from
one stream segment to another), so individual stream segments could
experience a trend while the overall population of the watershed does not.
Individual change: change experienced by an individual or particular member of the
population, this can be further divided into three categories:
 gross change: change in response of a particular population unit (e.g. change
in pH of a particular lake)
 average gross change: if all rivers in a collection of rivers have higher levels
of suspended sediment (so the same change occurs to many individual units)
 instability: variance of responses from individual population units
Another option which has been popularized recently, although the idea has been
discussed by many including: Jessen (1942) and Cochrane (1977), is to use a design with
some combination of repeated visits and new sites. These designs have been referred to
by many names: revisit, repeated, rotating, split-panel, replacement, etc. McDonald
(2003) provides an excellent review of these designs and develops consistent
terminology. These designs provide a compromise between temporary and permanent
sites that allow the estimation of both status and trends. McDonald (2003) found that
“consensus opinion among the reviewed articles appeared to be that some sort of splitpanel design had the best chance of satisfying the sometimes competing objectives
inherent in many environmental monitoring projects”. The following definitions from
McDonald (2003) are helpful:
Panel: A group of population units that are always all sampled during the same
sampling occasion or time period. Note that a population unit may be a member of
more than one panel under this definition.
Revisit design: The rotation of sampling effort among survey panels; the pattern of
visits to the panel; the plan by which population units are visited and sampled through
time.
Membership design: The method used to populate the survey panels; the way in
which units of the population become members of a panel.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 45 -
Implementing and analyzing these designs can be difficult. The analysis is complicated
and consistent funding is required to avoid missing data. While these designs in general
are recommended, the specific frequency of return visits and the sample sizes required
depend on the specific study and the life history of the subject. Generally it is
recommended that if trend is the primary objective then >50% of sampling effort should
be allocated to the revisited sites and if both status and trends are important then the
sampling effort should be allocated equally to the revisited and new panels (McDonald
2003). More research is needed to address the optimal allocation of sampling effort
among panels (McDonald 2003).
3.2.7 Field methods
Field methods are the actual methods used to collect data within a given sampling unit.
Existing established methods or protocols should be used first where possible (Vesely et
al. 2006) – a number of peer-reviewed and tested field protocols are available at the BC
Resources Inventory Standards Committee (RISC) website 9.
Pollack et al. (2004) recommend that field methods be designed to minimize detection
error. Questions that arise about detection include: Are field methods appropriate for the
species? Are individuals hard to detect because they are rare or because the method isn’t
efficient? Is the animal of interest ‘available’ to the sampling method? Thompson (2004)
describes diverse examples where much of the population may not be available to be
counted:
 Aerial surveys of marine mammals where some individuals will be underwater;
 Surface counts of ants or other insects where only the foraging component of the
population is available;
 Salamanders or frogs where a portion of the population is underground,
underwater or otherwise not visible;
 Bird point counts where not all birds may sing;
 Terrestrial mammals in areas with dense vegetation (e.g., elk, moose).
Where detectability is not perfect (or reasonably close to perfect) it must be accounted
for. There are a number of techniques available to address this issue including: capturerecapture, where probability of capture is evaluated; distance sampling, based on the idea
that detection is a function of distance; and sightability models, where observable
covariates for factors that influence detectability are used.
Field methods must be documented and made available to anyone who wishes to
replicate the sampling. Variations from a standard method or protocol must be
documented and rationalized. Any changes made to sampling method during the study
must be recorded with the data so data analysis will give a true picture of the sampled
population.
9
Resources Inventory Standards Committee (RISC) website: http://www.ilmb.gov.bc.ca/risc/about.htm
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 46 -
3.2.8 Planning the data analysis
Finally, one should consider a priori how the data will be analyzed and what will be
reported. If the study objectives have not been clearly defined, then this step will be
impossible. Use past research, pilot data, or expert opinion to describe the proposed
approach. Once sampling has been completed, describe the observed data for the
approach taken (it may not be exactly what was proposed). The following questions and
examples are intended to provide an idea of what data analysis should consider or might
address, but are by no means an exhaustive list of possible analytical approaches.
What does the data look like?
o Are the data a direct measure or an index?
o Describe the response or index of interest
 abundance
 density
 survival
 diversity index
 health index
How are the data distributed?
o continuous (e.g., height, weight)
o categorical (e.g., colour, sex)
o strictly positive (e.g., count data)
o bounded between 0 and 100 (e.g., percent cover)
o binomial (e.g., each observation can only take one of two values)
What exactly is the statistic of interest?
o mean, median, maximum, minimum, rolling mean, range, variance
What questions are of interest?
o Does the response differ by year, group, or location?
o Is there a relationship between environmental covariates and the response of
interest?
o Does the response exceed a threshold or target?
What statistical tests are appropriate?
o Parametric tests such as a t-test, require an assumption about the distribution of
the data (i.e., that it is “Normal”)
o Non-parametric tests such as the Mann-Whitney test may be appropriate if the
distribution is not obvious
o Regression analyses describe the relationship between environmental covariates
and the response of interest
o Multivariate tests, such as the multi-response permutation procedure (MRPP) may
be appropriate for community level questions where there are multiple responses
(i.e., the abundance of several species)
How to select significance levels and confidence intervals?
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 47 -
o Selecting the significance level of statistical tests should be done before the data
are collected (the level of significance will be used to determine sample size –
refer to section 3.2.5) and confirmed before the data are analysed.
o The significance level (or “alpha level”) is the probability of rejecting the null
hypothesis 10 when in fact it is true – that is, concluding that there is a difference
between two values when in fact there is no difference. This is known as a “Type
I” error.
o Significance levels are often 0.01 or 0.05 - the former represents a more rigorous
test than the latter and provides more certainty about your conclusion, and is
employed where errors in rejecting the null hypothesis have serious implications
to humans or the environment.
o Confidence intervals describe the amount of uncertainty associated with a sample
estimate of a population parameter, and are calculated from the significance level,
the sample statistic itself, and the margin of error (the range of values above and
below the sample statistic). For example, we might find our sample population did
not differ from the true population value by more than, say, 5 percent (the margin
of error) 90 percent of the time (the significance [alpha] level).
Check the assumptions
o Any analysis should include a discussion of the assumptions made about the data
or the outcome of the protocol: what the assumptions were, whether or not they
were met, and any weaknesses in the assumptions.
3.2 Sampling Design – in practice
Determine and describe the sampling design by:
 describing life history of the target species
 defining the target population
 choosing appropriate sampling unit(s)
 determining how sampling unit(s) should be chosen
 determining sampling effort (sample size), frequency and permanence
 describing the details of the field methods
 describing intended/proposed data analysis
10
The null hypothesis states that there is no significant difference between the results obtained and the
results expected: for example, there is no difference between these two values; or this value is greater than
(or less than) that value. A null hypothesis is either “rejected” by statistical testing, or “not rejected” – a
null hypothesis is never “accepted” by statistical testing because the testing is based on probabilities.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 48 -
4 IMPLEMENTING A PROJECT
If you have...
1. determined where your project is in the planning and design process (What
has already been done? Is an existing protocol? Is pilot testing needed?)
2. determined the need, and current knowledge, and developed the conceptual
model and effectiveness indicators to be monitored (Section 2), and
3. established a study design with a statistically rigorous sampling design
appropriate to where your project is in the development process (Section 3) ...
then you are ready to implement your project.
4.1 Confirm Agency Support
A project charter should be developed for approval by the appropriate supervisor or
manager. The charter should indicate how the project is aligned with program priorities.
The project charter should include a title, objectives, project lead and team, an estimated
budget or resources required and assessment of priority of the project.
4.2 Prepare a Monitoring Plan
A Monitoring Plan should be prepared for each WHA or UWR effectiveness evaluation
project before the project gets underway. Because effectiveness monitoring will continue
over some period of time, the plan must include proposed activities, timelines and
deliverables for the duration of that period along with full details of the work planned for
the first (or current) year. Monitoring Plans should be updated annually as the project
progresses from pilot to full evaluation project, documenting obstacles and delays, results
of pilot tests and decisions such as amending data collection or analytical methods.
Annual updates from a stable project (one in which no amendments are made to methods)
could be included in an annual Progress Report.
The Monitoring Plan should include:
 Identify management objectives/questions to be addressed,
 description of indicators and parameters to be measured,
 any thresholds or management triggers that will be used,
 study design,
 sampling design,
 description of project efficiency indicators to be measured,
 standard data collection methods or description of those being developed,
 equipment and training requirements,
 quality assurance framework and quality control measures
 proposed analytical methods and desired level of statistical significance (where
appropriate),
 project milestones, timelines, and deliverables,
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 49 -

plans for reporting and extension.
4.3 Data Management and Quality Assurance
FREP has adopted a quality assurance framework11 that establishes a process of checks
and balances to ensure all efforts are made to meet data security and quality standards,
from point of collection to final storage. Although data collected in wildlife resource
value projects are not currently integrated into the FREP information management
system, all wildlife resource value projects should adhere as much as possible to FREP’s
quality assurance and quality control commitments.
To meet the FREP commitment to quality management, wildlife resource value projects
should adopt a process than ensures data collected is copied, transferred, stored, etc., such
that (i) integrity of raw data is maintained, (ii) missing or erroneous data is addressed,
(iii) data are entered, checked, archived and analysed on a regular annual cycle relevant
to management, reporting and further study, and (iv) details of all results of all summaries
and statistical analyses are available for review and archived.
5 REPORTING
Knowledge gained from monitoring may range from procedural improvements to data
collection and designs thru to furthering our knowledge of the complexity of natural
systems. The purpose of reporting is to share this knowledge with those responsible for
making resource management decisions, those interested or affected by this knowledge
and the general public.
Reporting out provides material for discussion and recommendations for changes to
management practices and monitoring protocols. The wildlife resource value
effectiveness evaluation program is one step in an adaptive management cycle whereby
decision makers are informed of new science, recommendations for change can be made
and management can be improved.
5.1 Report Templates
The FREP Communications Strategy 12 and the FREP Reporting Guidelines 13 provide
overall direction for communications and standard layout and procedures for various
types of FREP reports and publications.
11
See http://www.for.gov.bc.ca/HFP/frep/qmgmt/index.htm
http://www.for.gov.bc.ca/ftp/hfp/external/!publish/frep/archived/PM-FREP_Communications_Plan.pdf
13
http://www.for.gov.bc.ca/ftp/hfp/external/!publish/frep/qmgmt/QCProtocol4-ElectronicForms-Dec92008-MASTER-BLANKFORM.pdf
12
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 50 -
There are a number of types of reports that could stem from effectiveness evaluation
projects. Background and planning reports will come first. Results of an effectiveness
monitoring project in its early stages of development will include an assessment of the
method(s) used to assess the effectiveness, in addition to the actual evaluation the
effectiveness of the WHA or UWR to protect habitat for a species. Final reports will
summarize the effectiveness evaluation and make recommendations for improvements.
5.2 Input to Decision-Making and Resource Management
It is very important that an effectiveness evaluation program and project results connect
to resource management planning or practices; if statutory decisions are to be made to
change a practice then clear documentation of the data used and the information being
brought forward must be documented for evaluation – because the results will guide
decisions about resource management planning and practice.
Improvements to resource management might be made informally through decisions to
change management practices (e.g., the size of a specific WHA) or formally through
government decisions to amend policy or regulation as necessary (e.g., provincial policy
that defines WHA size for a species) . The feedback of project results to land managers,
industry and government staff and/or executive will serve as the crucial “closing-theloop” step in a cycle of adaptive management and continuous improvement.
6 LITERATURE CITED
Buckland, S.T., Anderson, D.R., Burnham, K.P. and Laake, J.L. 1993. Distance Sampling: Estimating
Abundance of Biological Populations. Chapman and Hall, London. 446pp.
http://www.colostate.edu/Dept/coopunit/download.html
Bunnell, F. 2005. Biodiversity Monitoring Guideline: Terrestrial Biological and Physical Monitoring
and Aquatic Biological and Physical Monitoring. Forest Investment Account (FIA) Activity
Guidance Document. 21 pp. Available at:
http://www.env.gov.bc.ca/fia/documents/monitoring_guidelines.pdf
Cochran, William Gemmell. 1977. Sampling Techniques. John Wiley & Sons, Inc., New York, NY.
Doran, G. T. (1981). There's a S.M.A.R.T. way to write management's goals and objectives.
Management Review, Volume 70, Issue 11(AMA FORUM), pp. 35–36
Edwards, D.: 1998. Issues and themes for natural resources trend and change detection. Ecol. Applic.
8(2): 323–325.
Elzinga, Caryl L., Daniel W. Salzer, John W. Willoughby, and James P. Gibbs. 2001. Monitoring
plant and animal populations. Blackwell Science, Inc., Malden, Massachusetts.
Gordon, D.R., J.D. Parrish, D.W. Salzer, T.H. Tear, and B. Pace-Aldana. 2005. The Nature
Conservancy’s Approach to Measuring Biodiversity Status and the Effectiveness of Conservation
Strategies. Ch. 3 In: Principles of Conservation Biology, 3rd Edition. 2005. Groom, M.J., G.K.
Meffe, and R.C. Carroll, eds. Sinauer Press: Sunderland, MA.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 51 -
Houde, I., Bunnell, F.L., and S. Leech. Assessing success at achieving biodiversity objectives in
managed forests. BC J. Ecosys. and Manage. 6(2):17-28.
Hurlbert, S. H. 1984. Pseudoreplication and the design of ecological field experiments. Ecological
Monographs 54:187-211.
Jessen, R. J. 1942. Statistical investigation of a sample survey for obtaining farm facts. Iowa Agric.
Exper. Station Res. Bull. 304: 54–59.
Kinley, T. A. 2009. Effectiveness monitoring of badger wildlife habitat areas: summary of current
areas and recommendations for developing and applying protocols. B.C. Min. For. and Range, and
B.C. Min. Envir., Victoria, B.C. Forest and Range Evaluation Program report.
Krebs, Charles J.1989. Ecological Methodology. Harper Collins Publishers Inc., New York, NY.
Lint, J., B. Noon, R. Anthony, E. Forsman, M. Raphael, M. Collopy and E. Starkey. 1999. Northern
spotted owl effectiveness monitoring plan for the Northwest Forest Plan. Gen. Tech. Rep. PNWGTR-440. USDA Forest Service, Pacific Northwest Research Station. Portland, Oregon.
Lohr, Sharon L. 1999. Sampling: Design and Analysis. Brooks/Cole Publishing Company. Pacific
Grove, CA.
Marcot, B.G., R.S. Holthausen, M.G. Raphael, M.M. Rowland, and M.J. Wisdom. 2001. Using
Bayesian belief networks to evaluate fish and wildlife population viability under land management
alternatives from an environmental impact statement. For. Ecol. and Manage. 153:29–42.
McDonald, Lyman L. 2004. Sampling Rare Populations. pp.11-42 in William L. Thompson ed.
Sampling Rare or Elusive Species. Island Press. Washington, DC.
McDonald, T.L. (2003) "Review of Environmental Monitoring Methods: Survey Designs",
Environmental Monitoring and Assessment, v. 85, p.277-292.
Meyer, Paul J (2003). "What would you do if you knew you couldn’t fail? Creating S.M.A.R.T.
Goals". Attitude Is Everything: If You Want to Succeed Above and Beyond. Meyer Resource
Group, Inc.
National Park Service. 2012. Guidance for designing an integrated monitoring program. Natural
Resource Report NPS/NRSS/NRR—545. National Park Service, Fort Collins, Colorado. Available
at: http://science.nature.nps.gov/im/monitor/guidance.cfm
Newhouse, N. J., T. A. Kinley, C. Hoodicoff, and H. Page. 2007. Wildlife Habitat Area effectiveness
evaluations. Protocol for monitoring the effectiveness of badger wildlife habitat areas. Version 1.4.
Prepared for Ministry of Forests and Range, Victoria, BC.
Noon, B. R. 2003. Conceptual issues in monitoring ecological resources. Chapter 2 (pp. 27 – 71) in D.
E. Busch and J. C. Trexler (editors), Monitoring Ecosystems. Interdisciplinary Approaches for
Evaluating Ecoregional Initiatives. Island Press, Washington.
Nyberg, B. 2010. Conceptual models for wildlife effectiveness evaluations: a recommended approach.
B.C. Min. For. Ran., For. Prac. Br., Victoria, B.C. FREP Ser. 24. Available at:
http://www.for.gov.bc.ca/hfp/frep/publications/reports.htm#rep24
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 52 -
Pollack, Kenneth H., Helene Marsh, Larissa L. Bailey, George L. Farnsworth, Theodore R. Simons,
and Mathew W. Alldredge. 2004. Separating Components of Detection Probability in Abundance
Estimation: An Overview with Diverse Examples. pp.43-58 in William L.Thompson ed. Sampling
Rare or Elusive Species. Island Press. Washington, DC.
Schueller, S.K., S.L. Yaffee, S.J. Higgs, K. Mogelgaard and E.A. DeMattia. 2006. Evaluation
Sourcebook: measures of progress for ecosystem and community-based projects. Ecosystem
Management Initiative University of Michigan, 214 pp. Available at:
http://www.snre.umich.edu/ecomgt/evaluation/planning.htm
Shaffer, T. L. and D. H. Johnson. 2008. Ways of learning: observational studies versus experiments. J.
Wildl. Manage. 72(1): 4-13.
Skalski, John R., Kristen E. Ryding, and Joshua J. Millspaugh. 2005. Wildlife Demography: analysis
of sex, age, and count data. Elsevier Academic Press. SanDiego, CA.
Thompson, Steven K. and George A. F. Seber. 1996. Adaptive Sampling. John Wiley & Sons Inc.
New York, NY.
Thompson, William L. 2004. Sampling Rare or Elusive Species. Island Press. Washington, DC.
Urquhart, N. Scott and Tomas M. Kincaid. 1999. Designs for detecting Trend from Repeated Surveys
of Ecological Resources. Journal of Agricultural, Biological, and Environmental Statistics. 4:404414.
Vesely, D.; McComb, B.C.; Vojta, C.D.;Suring, L.H.; Halaj, J.; Holthausen, R.S.; Zuckerberg, B.;
Manley, P.M. 2006. Development of Protocols To Inventory or Monitor Wildlife, Fish, or Rare
Plants. Gen. Tech. Report WO-72. USDA, Forest Service, Washington, DC. 100p.
Wieckowski, K., D. Pickard, M. Porter, D. Robinson, D. Marmorek, and C. Schwarz. 2008. A
conceptual framework for monitoring Fisheries Sensitive Watersheds (FSW). Report prepared by
ESSA Technologies Ltd. for BC Min. Environ., Victoria, BC. 61 p.
Yoccoz, Nigel G., James D. Nichols and Thierry Boulinier. 2001. Monitoring of biological diversity
in space and time. Trends in Ecology and Evolution 16:446:453.
U.S. Environmental Protection Agency. 2006. Guidance on systematic planning using the data quality
objectives process. EPA report QA/G-4. 111 pp. Available at:
http://www.epa.gov/quality/qa_docs.html
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 53 -
Appendix 1. Example study designs
Based on definitions from Schwarz (2006), specific examples of each study design are
provided using WHA monitoring for examples.
Descriptive study
A WHA is selected and an indicator (s) (e.g., road density) is measured. The information
collected is only relevant to the WHA sampled.
Observational study
A non-designated area and a WHA are selected; selected indicators are measured in both.
Comparisons between the two areas can be made, but the results are only applicable to
the areas sampled. Using this approach, it is not possible to conclude whether any
observed differences are representative of the differences between the areas. Descriptive
and observational studies involve non-randomly selected sampling units; as a result the
information obtained is limited to the sites actually observed.
Analytical survey
A random sample of WHAs of two or more categories is selected and road density is
measured in each. An estimate of the mean road density with known precision can be
obtained for each category. The estimates from the categories can be compared; however,
it is possible that another unknown factor (besides WHA designation) is actually
responsible for the difference.
Impact and control-impact surveys
The goal of this approach is to assess the impact of some change, in this case the
designation of a WHA. A variety of impact designs with increasing levels of effort and
increasing degrees of inference exist. Mellina and Hinch (1995) provide a summary of
different impact designs and describe how each might be used to assess watershed
restoration. Schwarz (2006) and Underwood (1994) provide a good description along
with examples for a range of impact studies, as well as evaluating their respective
strengths and weaknesses. The simplest impact studies look at a single location before
and after some event. Obtaining multiple observations before and after an event improves
the ability to determine if an observed change is ‘real’ by taking into account the natural
year to year variability. Since obtaining ‘before’ samples is often difficult, it may be
possible to get variance estimates by randomly sampling from similar but undisturbed
habitats (Underwood 1994). This approach can be considerably improved by adding a
control site, where the control site is similar to the treatment site with respect to general
characteristics (e.g., region, annual precipitation, size, etc.). This approach is termed a
‘Before After Control Impact’ (BACI) design. BACI designs are intended to address the
question of whether a particular action has resulted in a change at the treatment/impact
site relative to the control site, while simultaneously adjusting for extraneous co-variables
that might be similarly affecting both impact and control areas. In most cases, the use of
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 54 -
controls greatly increases the power of detecting treatment/impact effects; however,
poorly chosen control sites can decrease the power of detecting an effect (Korman and
Higgins 1997; Roni et al. 2003). For example, a lack of randomization in assigning
impact sites prevents us from inferring whether the impact will occur elsewhere.
Alternatively, if there is only a single impact/control pair, how do we know that the
results are not just a consequence of the choice of sites?
The example in Figure5 illustrates the value of including a control site for assessing the
effects of an impact/treatment on populations which are highly variable naturally over
time. In this hypothetical example the ability to detect improved salmon parr to smolt
survival after a habitat restoration treatment would not be possible without a control
(Hayden Creek), due to high variability in survival (5A). Evaluation of the average
annual difference between fish survival in the treatment vs. the control stream (5B),
however, indicates that survival in the treatment stream is much greater relative to the
control in the years subsequent to the restoration action.
Mainstem
Hayden
parr to smolt survival
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
1990
1995
2000
2005
2010
2015
2020
2025
2030
Year
0.10
treatment - control survival
0.08
0.06
0.04
D(POST)
0.02
0.00
D(PRE)
-0.02
-0.04
1990
1995
2000
2005
2010
2015
2020
2025
2030
Year
Figure 5.
Value of BACI design for inferences - A) Hypothetical time series of mainstem river and
Hayden Creek (control area) parr-to-smolt survival rates, with habitat restoration actions
assumed to happen simultaneously in 2015 (dark arrow) in the mainstem river. Time series for
both impact and control streams track erratically over time. B) Hypothesized difference
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 55 -
between mainstem river and Hayden Creek parr-to-smolt survival rates over the time series.
D(PRE) and D(POST) represent average survival difference pre- and post-impact in 2015 in
the mainstem river. Difference between control and impact streams shows much greater parrsmolt survival in the treatment stream vs. the control, indicating a benefit of the restoration
action over time that would not have been apparent given annual variation without a control
system for comparison (source CSMEP 2006).
BACI designs can be grouped into three types: BACI, BACIP, and MBACI (Downes et
al. 2002). In simple BACI designs, measurement are made before an impact at control
and treatment locations and then after the impact. However paired measurements through
time pre- and post impact are better able to avoid spurious results (Green 1979, StewartOaten et al. 1986). Downes et al. (2002) refers to this as BACIP (P for paired
measurement through time). In BACIP paired measurements are taken in both the impact
and control sites at multiple random times pre- and post- impact. Keough and Mapstone
(1995) further extended the BACI design to contain multiple controls and if possible
multiple impact sites, referred to as MBACI (Downes et al. 2002). MBACI designs have
been developed to address questions about the impacts of an action across broader scales.
Multiple treatment and control locations are chosen randomly from a group of potential
locations, thereby providing the means to extrapolate to a larger area. If it is not possible
to randomly assign treatments and controls, but the same pattern is observed in multiple
pairs, it is reasonable to assign a causal relationship (Schwarz 2006). The MBACI design
compares a fixed period of time before the manipulation to (in ideal situations) a similar
period of time after the manipulation.
Designed experiments
In a designed experiment, the investigator has control over the treatment and can
randomly assign experimental units to treatments. The degree of control the investigator
has on a study affects the ability to show causation. The ability to make inference to other
sampling units depends on random selection of samples or assignment of treatments.
There are many good text books available on the subject of designing experiments
including: Schwarz 2006, these online course notes geared to the environmental scientist
and are probably the best place to start; Montgomery 1997, is a solid introductory book
which is probably more than enough for most environmental studies; Box et al.1978, is a
traditional reference; Wu and Hamada 2000, is a more recent and extensive reference.
References
Box, G., E.P., W.G. Hunter, and J.S. Hunter. 1978. Statistics for Experimenters An introduction to
Design, Data Analysis, and Model Building. John Wiley and Sons Inc, Toronto.
Downes, B.J., L.A. Barmuta, P.G. Fairweather, D.P. Faith, M.J. Keough, P.S. Lake, B.D. Mapstone and
G.P. Quinn. 2002. Monitoring ecological impacts: concepts and practice in flowing waters.
Cambridge University Press, Cambridge, UK.
Green, B.F. 1979. Sampling design and statistical methods for environmental biologist. Wiley
Interscience, New York, New York, USA.
Keough, M.J. and B.D. Mapstone. 1995. Protocols for designing marine ecological monitoring
programs associated with BEK mills. Technical Report No. 11. CSIRO, Canberra, ACT.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 56 -
Korman, J. and P.S. Higgins. 1997. Utility of escapement time series data for monitoring the
response of salmon populations to habitat alteration. Can.J. Fish. and Aquatic Sci. 54:20582097
Mellina, E., and S.G. Hinch. 1995. Overview of Large-scale Ecological Experimental Designs and
Recommendations for the British Columbia Watershed Restoration Program. Province of
B.C. , Min. Environ., Lands and Parks and Min. Forests. Watershed Restoration Management
Rep. No. 1.
Montgomery, D.C. 1997. Design and Analysis of Experiments. 4th edn. John Wiley and Sons, New
York.
Roni, P., T.J. Beechie, R.E. Bilby, F.E. Leonetti, M.M. Pollock and G.P. Pess. 2003. A review of
stream restoration techniques and a hierarchical strategy for prioritizing restoration in the
Pacific Northwest watersheds. North Amer. J. Fish. Manage. 22:1-20.
Schwarz, C. 2006. Course notes for beginning and intermediate statistics. Available at:
http://www.stat.sfu.ca/~cschwarz/CourseNotes.html. Accessed on: March 20, 2008.
Stewart-Oaten, A., W.W. Murdoch and K.R. Parker. 1986. Environmental impact assessment:
“pseudoreplication” in time? Ecology 67:929–940.
Underwood, A.J. 1994. On Beyond BACI: Sampling Designs that Might Reliably Detect
Environmental Disturbances. Ecol. Applic. 4:3-15.
Wu, J. and M. Hamada. 2000. Experiments Planning, Analysis, and Parameter Design
Optimization.John Wiley and Sons Inc, Toronto.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 57 -
Appendix 2. Examples of sampling designs
Cluster sampling
What: A random sample of clusters is selected and a census of all sampling units within
the cluster is taken (Figure 7). For example: if we are interested in the average number of
bicycles per household, we could select a sample of city blocks and then visit every
household within the block. It is often relatively cheap to collect more information once
at a site. This improves the estimate of the cluster (city block in this case) but may only
have limited improvement to the overall precision. This is a special case of the two-stage
sample where the second stage is a census rather than a sample.
When: This kind of strategy is useful when the cost of moving between sites is expensive
relative to the cost of obtaining many more samples within a given site. Given
information about the cost of moving between clusters and preliminary estimates of
variability within and between clusters we can determine if this is a suitable approach.
Figure 6. This figure illustrates a cluster sample (from Lohr 1999, p133).
Multistage sampling
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 58 -
What: Similar to cluster sampling except that rather than a census of all sampling units
within the primary sampling unit (psu), a secondary sample (e.g. SRS, SysRS,
stratified…) is taken. For example: if a random sample of transects is taken from the
population of possible transects (psu’s), then in a cluster sample we’d take a census of
each transect but in a multi-stage sample we might take a systematic sample every 10m
along the transect –secondary sampling units (ssu’s). In Figure 6, this would be
illustrated by taking a sample of the squares within the three selected clusters (psu’s)
rather than every small square (ssu) within each cluster.
When: A multi-stage sampling strategy is often employed when the cost of measuring the
units within the ssu is small compared to the cost of moving between psu’s. One must be
cautious to use the correct estimate of variance and therefore confidence intervals when
using mult-stage estimates. The observations can’t be treated as a random sample from
the target population, but rather a random sample from the psu (eg. Transect). Increasing
the number of observations within each psu will improve the precision of the estimate of
the psu, but will have limited effect on the precision of the overall estimate. Increasing
the number of psu’s is generally required to substantially improve the precision of the
overall estimate. The optimum sample size for both psu’s and ssu’s can be determined (as
explained in the sample size calculator section) using preliminary estimates of the
variability within and between psu’s combined with the cost/effort of moving between
psu’s vs. the cost/effort of sampling more units within each psu.
Another interpretation of a multi-stage sample is where the lowest stage refers to the
actual measurement protocol and so refers to the measurement error associated with the
protocol. For example if counting all individuals within a transect, there may be
measurement error involved if the animal is difficult to detect (Hankin 1984).
Adaptive sampling
What: The major difference with adaptive sampling is that the sample units are not fully
described before the study begins. The strategy begins like any other sampling design
with a random selection of sampling units, but in adaptive sampling additional sampling
units may be added based on the observed values in the initial sample. This is particularly
useful for clustered populations where the presence in a given sampling unit may increase
the likelihood of presence in neighbouring populations. Additional samples are added
based on a pre-determined rule (i.e. if at least one whale is present in the sampling unit
then look in the 4 adjacent sampling units). The same or a different rule is then applied to
the new sampling units. For rare clustered populations this method may be more efficient
than a traditional sampling approach as more individuals are likely to be encountered for
the same overall number of samples resulting in more precise estimates (Thompson and
Seber 1996). This strategy is more difficult to implement than a traditional sampling
design and a statistician should be consulted for the design and analysis. MacDonald (in
Thompson 2004) found mixed feedback on this approach in a survey of statisticians and
biologists and found that logistical difficulties with this approach may arise, particularly
in surveys with large spatial scale.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 59 -
When: rare and clustered populations
Examples from Thompson and Seber 1996:
•
Estimating whale populations in the ocean, where whales tend to group in pods
and cover a small fraction of the possible search area.
•
Contagious diseases, if a person tests positive for a rare disease, then test friends
and family of people who have had recent contact with the person
•
Rare plant and animal species
•
Cases where pollution levels are generally light, but where a few hotspots exist.
•
Shrimp trawl surveys, where the shrimp tend to be clustered, but the cluster
locations are not fixed due to the mobility of the shrimp.
•
Deep sea fish (Orange Roughy) form large spawning aggregates making it
difficult to use traditional sampling methods to estimate abundance.
Figure 7.
This figure shows an example of an initial random sample of units on the left and the resulting
adaptive cluster sample on the right (From Thompson 1990).
Ratio estimation
What: When two quantities are measured on each sample unit. For example if you
measure yi = bushels of grain harvested from field i and xi = acreage of field i, and you
are interested in B = average yield in bushels per acre or the ratio of y:x. (Lohr 1999)
When/why: (Lohr 1999)
a) if the ratio itself is of interest, this is a straightforward application
b) if N is unknown but can be related to some other metric such as weight (i.e. total
weight of fish in a net and average weight per fish)
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 60 -
c) to improve the precision of an estimate of the total or mean. If you can collect two
types of information (x and y) on the same sampling unit and x and y are
correlated, then you may be able to improve the precision of the estimate using
the relationship between x and y. These estimates may be biased but may still be
worth it if the reduction in variance is sufficient. Lohr (1999) gives a good
description of the conditions where this method is suitable, showing how the
precision and bias changes with sample size and the strength of the correlation
between x and y. This method is essentially a special case of a regression model
based estimate.
d) Adjust estimates from the sample based on ratios observed from different groups
(such as gender) using ‘post-stratification’. See the discussion on poststratification.
e) Can also be used to adjust for non response such as might occur in a survey of
businesses, Non-response in an environmental study may occur when access to a
sampling unit is denied by a land owner or is not accessible for safety reasons.
Double sampling framework
What: In ratio estimation we measure multiple variables on every observation. However,
if it is easier or cheaper to collect auxilliary information than the actual variable of
interest, it may be convenient to take more samples of the auxiliary information than the
variable of interest (Thompson 2002). The term ‘double sampling’ has not been
consistently used. Our definition of double sampling follows Thompson 2002 and
Pollock et al. 2002: where an initial sample is taken and only the auxiliary information is
observed, but then a smaller second sample (often a subset) is taken where both the
variable of interest and the auxiliary information is observed. The observed relationship
between the two variables is used to improve the overall estimate for a given cost.
When: This strategy may be suitable when the cost of measuring the variable of interest
for a sufficient number of sites is prohibitive. This approach is also used where there are
multiple approaches for estimating the variable of interest that differ in cost and accuracy.
Examples:
•
Aerial vs. ground counts
•
‘eyeball’ estimates of volume of trees in a stand vs. felling trees and measuring
•
Time constrained vs. area constrained searches for tailed frogs
•
Number of ponds vs. waterfowl counts
Sequential sampling
The general idea is that sampling continues until a prespecified number of events are
observed (Seber 1986 quotes- Rao, Portier and Ondrasik, 1981). It may also be
implemented by sampling until a particular precision level is reached (Elzinga et al.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 61 -
2001). A first sample may be used to determine how many more samples to take (Lohr
1999), the variance estimates need to account for this.
Network sampling and snowball sampling
These strategies all follow a link tracing design where information about the links
between sampling units are used (Thompson 2002). In network sampling, if one person is
selected then everyone related to them (i.e. all siblings) are also sampled. These strategies
can be useful in hard to access and elusive populations such as drug users or commercial
sex workers. The idea is that members from a rare population know one another, so if you
interview one homeless person you can then ask them to identify additional homeless
people. These ideas are also used in internet search engines.
Post stratification
Often at the outset of an experiment you don’t know what proportion of the sampling
units belong to a particular group or stratum (males vs. females, grassland vs. forest…)
and so it is impossible to stratify the sample prior to the implementation of the study.
Post-stratification is when the sample is grouped into strata after the study is complete
and stratified estimates are obtained. There is a major danger to post-stratification if ‘data
snooping’ occurs. If you choose your strata after you see the data you can obtain
arbitrarily small variances (Lohr 1999). Holt and Smith (1979) describe post-stratification
as “a device for protecting the statistician’s inference against those occasions when his
randomization gives an unbalanced or unrepresentative sample”. However, they like
Lohr, note that it may be possible to stratify to the point where there are strata with only a
single individual and Royall (1968) notes that “the statistician needs to consciously
ignore some of the information in the data so that meaningful inference is possible”. A
safe compromise to prevent datasnooping might be to choose the groups that you will
stratify on ahead of time, but assign the observations to each group after you see the data.
In order to use post-stratification the relative size of each stratum must be known, or they
must be estimated (Thompson 2002). The variance estimate is slightly different with
post-stratification and approximations vary among texts. Thompson (2002) provides a
brief summary and justification for the variance estimate he recommends.
Quota sampling is a variation on this where the surveyor continues sampling until they
obtain a pre-specified number from each stratum. If the samples are chosen at random
then this is equivalent to a stratified random sample (Cochran 1977) but in practice this
would require significant effort near the end of the study as many of the individuals
drawn would be from strata that already had their quota filled. As a result quota samples
are often allowed to be drawn in some convenient and not random manner. If the samples
are not obtained using probability sampling we can’t draw inference from these samples
except in a model-based approach (Lohr 1999).
Examples:
•
a useful strategy in estimating sea otter populations that occurred in both large
groups and very small groups (Thompson 2004, p30)
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 62 -
Rotating Panel Design
A rotating panel design provides a method to balance the conflicting goals of monitoring
population status vs. monitoring the trend in populations. These designs have been
referred to by many names: revisit, repeated, rotating, split-panel, replacement etc…
McDonald (2003) provides an excellent review of these designs and develops consistent
terminology. These designs provide a compromise between temporary and permanent
sites that allow the estimation of both status and trends. For example, in order to estimate
status, 100 different sites might be monitored each year for five years, providing a total
sample size of 500 for that period of time. For trend detection, it is best to revisit sites
each year. Consequently, for the above example, it would be better to revisit the 100 sites
visited the first year in each of the subsequent four years, yielding a total sample of only
100 sites over the 5-year period. A rotating panel design balances the requirements for
both status estimation and trend detection (while also recognizing budget limitations
related to sampling effort) by sampling a new set of sites each year over a particular time
cycle (for status estimates), but then revisiting these sites according to some schedule (for
improved trend assessments). Various versions of a rotating panel design allow for
visiting a subset of sites every year, revisiting some sites on longer cycles, or
incorporating new sites each year along with the revisit schedule.
References
Cochran, William Gemmell. 1977. Sampling Techniques. John Wiley & Sons, Inc., New York, NY.
Elzinga, Caryl L., Daniel W. Salzer, John W. Willoughby, and James P. Gibbs. 2001. Monitoring
plant and animal populations. Blackwell Science, Inc., Malden, Massachusetts.
Hankin, D.G. 1984. Multistage sampling designs in fisheries research: Applications in small
streams. Can. J. Fish. and Aquatic Sci. 41:1575-1591.
Holt, D. and T.M.F. Smith. 1979. Post Stratification. Journal of the Royal Statistical Society: Series
A (General). 142:33-46.
Lohr, S.L. 1999. Sampling: Design and Analysis. Brooks/Cole Publishing Company. Pacific Grove,
CA.
McDonald, Lyman L. 2004. Sampling Rare Populations. pp.11-42 in William L.Thompson ed.
Sampling Rare or Elusive Species. Island Press. Washington, DC.
McDonald, T.L. 2003. Review of Environmental Monitoring Methods: Survey Designs.
Environmental Monitoring and Assessment 85: 277-292
Pollack, K.H., J.D. Nichols, T.R. Simons, G.L. Farnsworth, L.L. Bailey and J.R. Sauer. 2002. Large
scale wildlife monitoring studies: statistical methods for design and analysis.
Environmetrics. 13:105-119.
Royall, R.M. 1968. An old approach to finite population sampling theory. J. Amer. Statist. Ass.,
63:1269-1279.
Seber, G.A.F. 1986. A Review of Estimating Animal Abundance. Biometrics. 42: 267-292.
Thompson, S.K. and G.A.F. Seber. 1996. Adaptive Sampling. John Wiley and Sons Inc. Toronto,
ON.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 63 -
Thompson, S.K. 1990. Adaptive cluster sampling, J. Amer. Statistical Assoc. 85:1050-1059.
Thompson, S.K. 2002. Sampling 2nd Edition. John Wiley & Sons Inc. New York, NY.
Thompson, W.L. 2004. Sampling rare or elusive species. Island Press. Washington DC.
A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft
- 64 -