CIR1528
Measuring the Effectiveness of Lagoon Quest: A Case
Study in Environmental Education Program Evaluation1
Judith Chen-Hsuan Cheng, Martha C. Monroe, and Alison Gordon2
1. This document is CIR1528, one of a series of the School of Forest Resources and Conservation Department, UF/IFAS Extension. Original
publication date August 2008. Reviewed January 2015. Visit the EDIS Web site at http://edis.ifas.ufl.edu.
2. Judith Chen-Hsuan Cheng, research assistant; Martha C. Monroe, professor, School of Forest Resources and Conservation, UF/IFAS Extension, Gainesville, FL
32611; and Alison Gordon, Brevard Zoo.
The Institute of Food and Agricultural Sciences (IFAS) is an Equal Opportunity Institution authorized to provide research, educational information and
other services only to individuals and institutions that function with non-discrimination with respect to race, creed, color, religion, age, disability, sex,
sexual orientation, marital status, national origin, political opinions or affiliations. For more information on obtaining other UF/IFAS Extension
publications, contact your county’s UF/IFAS Extension office.
U.S. Department of Agriculture, UF/IFAS Extension Service, University of Florida, IFAS, Florida A & M University Cooperative Extension Program, and
Boards of County Commissioners Cooperating. Nick T. Place, dean for UF/IFAS Extension.
CIR 1528
Measuring the
Effectiveness of
Lagoon Quest
A case study in environmental
education program evaluation
Judith Chen-Hsuan Cheng
Martha C. Monroe
School of Forest Resources
and Conservation
University of Florida
Alison Gordon
Brevard Zoo
Photo by G. Cohen
Acknowledgments
This is both a report of the evaluation of the second year of Lagoon Quest’s countywide implementation and a case study of program evaluation written to accompany
professional development programs for environmental educators. The format
parallels the process described in the North American Association for Environmental
Education’s workbook, Evaluating Your Environmental Education Programs. We hope
that NAAEE members as well as those from the Association of Zoos and Aquariums,
US Fish and Wildlife Service, US Forest Service, and National Park Service will find it
helpful as they work to measure and improve their program successes.
We gratefully acknowledge all of the zoo staff and the fourth-grade teachers and
students in Brevard County who participated in Lagoon Quest evaluation. Our activity
and this publication were made possible by the leadership, support, and special skills
of the following individuals:
Keith Winsten, Executive Director, Brevard Zoo
Andrea Aubert, Education Program Coordinator, Brevard Zoo
Ed Short, Elementary Science Coordinator, Brevard Public Schools
Patty Leonard, Sherwood Elementary School, Brevard Public Schools
Kathy Lueck, Sherwood Elementary School, Brevard Public Schools
Linda Garrett, Quest Elementary School, Brevard Public Schools
Neyda Riley, Assistant Director of the Office of Accountability, Brevard Public Schools
Anne Graham, Technology Integrator, Brevard Public Schools
Kathy Hill, IRL Education Coordinator, Indian River Lagoon National Estuary Program
Eleanor Sommer, Editor
Shannon Paulin, Designer, JS Design Studio
Funding for this work was provided by:
National Fish and Wildlife Foundation School
of Forest Resources and Conservation
UF/IFAS
Printed by Storter Childs on FCS-Certified paper
Credited Photos courtesy of the Indian River Lagoon National Estuary Program
© 2008 University of Florida
I feel the program helps our students
understand more of the world around them.
—Teacher
L
ocated in East Central Florida, Brevard
County is home to approximately 531,250
people,1 most of whom live near the
Atlantic Ocean in the largest cities,
Palm Bay and Melbourne. A large number of
tourists visit the ocean and NASA’s Kennedy
Space Center at Cape Canaveral. Paralleling the
coast is the Indian River Lagoon, a linear, 156mile estuary, which greatly enhances Brevard
County’s agricultural, fishing, economic, and
recreational opportunities.2 Like other subtropical
estuaries, the Indian River Lagoon has very
high biological productivity and is the home of
many endangered and threatened species. To
help protect this important treasure, Brevard Zoo
developed an environmental education program,
called Lagoon Quest, to enhance environmental
stewardship.
Lagoon Quest was originally developed to increase knowledge of local environmental issues
and estuarine ecology for students in fourth grade
through eighth grade classes. Because of the success
of this voluntary program, Brevard Public Schools
included the program in their fourth-grade science
curriculum beginning in 2005. Now all fourth-grade
students and teachers are required to participate in
Lagoon Quest. The purpose of the program’s expansion is to encourage all students to respect and be
responsible for the Indian River Lagoon.
In the summer prior to the required expansion of the
Lagoon Quest program, zoo staff and six fourthgrade teachers developed the Teacher Guide and
Student Guide (Lagoonie Logbook). The Teacher
Guide provides an introduction to the study trip and
instructions for the Lagoon Quest in-class activities, while the Student Guide provides worksheets
for students and their parents to complete. Twelve
in-class activities were designed to bracket the
one-day study trip to the Indian River Lagoon,
which is staffed by Brevard Zoo. During the study
trip students learn about the Indian River Lagoon,
collect aquatic organisms while wading in the water,
test water quality, discuss water pollution with an
Enviroscape model, and hunt for litter and evidence
of animals on a beach scavenger hunt using GPS
technology.
Brevard Zoo asked students and faculty
at the University of Florida’s School of Forest
Resources and Conservation to assist with the
program evaluation. When the National Fish
and Wildlife Foundation (NFWF) agreed
to provide funding for the evaluation, the
process was enlarged to accommodate their
interests and the graduate student’s research
agenda. This training case study discusses
the development of the program evaluation
and provides the results of the evaluation for
Lagoon Quest.
1. U.S. Census Bureau (Retrieved, 2006, 07, 04) http://www.census.gov/Press-Release/www/releases/archives/facts_for_features_special_editions/006838.html
2. South Florida Water Management District (Retrieved, 2006, 07, 04). http://www.sfwmd.gov/site/index.php?id=67
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 3
The purpose of the evaluation is to understand if
and how the program increases students’ knowledge about the Indian River Lagoon and attitudes
toward caring for the Lagoon and its watershed.
Focusing the Evaluation
Zoo staff and Brevard school district administrators
helped articulate the purpose of the evaluation: to
understand if the program is increasing students’
knowledge about the Indian River Lagoon and helping
to develop a conservation ethic for taking care of
the Lagoon and its watershed. Additional interests
were exploring the impacts of expanding a voluntary
environmental educational program to the entire fourth
grade, identifying changes to improve the program,
and identifying indicators that could be used over time
to track the development of a conservation ethic in
the county. The results obtained from this study will
be used for program improvement and new program
development. The program stakeholders include the
zoo staff, school district administrators, and fourthgrade teachers (including those who helped write the
program materials). The program audience includes
about 5500 fourth-grade students, about 260 fourthgrade teachers, and about 900 parent chaperones that
accompany the study trips.
To become familiar with the program, the evaluators
created a logic model for Lagoon Quest. The logic
model was presented to zoo staff to confirm that it
included all of the elements of the program and to
help identify those elements that should be included
in the evaluation (Table 1).
n Situation: The Indian River Lagoon is
a biologically diverse estuary that is threatened
by rapid population growth, development in the
watershed, and increased boat traffic.3 The longterm health of the Lagoon is dependent upon the
knowledge and actions of local citizens, their
willingness to support regulations and policies that
protect the Lagoon, and their investment of resources.
Brevard Zoo is a local environmental education
facility that developed Lagoon Quest. Brevard Public
Schools has adopted the program for all fourthgrade classes to support science education and help
strengthen a conservation ethic in the county.
n Inputs: The Lagoon Quest inputs are
funding (from NFWF and Brevard Schools),
written materials (Teacher Guide, Lagoonie
Logbook from Brevard Zoo, and Indian River
Lagoon Activity Book from St. Johns River
Water Management District), personnel (zoo
staff, teachers, chaperones), and program
equipment (from schools and zoo).
n Outputs: An in-service teacher workshop,
in-class activities, study trips, family activities,
and family festival are outputs. The workshop
introduced the Lagoon Quest teaching materials,
in-class activities, study trip, and evaluation tools.
The family activities are part of the Lagoonie
Logbook, and the family festival is a zoo-sponsored
exploration and celebration of the Indian River
Lagoon for families of fourth-grade students. The
program participants are fourth grade public school
students, teachers, and chaperones. The festival
participants are fourth grade public school students
and their families.
3. St. Johns River Water Management District (Retrieved, 2007, 08, 16) http://www.sjrwmd.com/programs/index.html
4
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
n Outcomes: Lagoon Quest has three types
of outcomes: short-term, medium-term, and
long-term. Short-term outcomes are students’
knowledge and teachers’ knowledge, experience,
and skill enhancement. Medium-term outcomes
are students’ environmental attitudes and pride
in the Indian River Lagoon. The long-term
outcomes are enhancement of conservation
behaviors, interest in conservation careers, and the
improvement of the biophysical environment.
n Assumptions: A number of assumptions about
education and learning are built into this program:
Hands-on, real-world learning experiences are
meaningful and memorable. Teacher-led classroom
activities help introduce and reinforce study trip
concepts. One school-based experience can help
motivate students to engage their parents in additional
lagoon-related activities. The activities of collecting
organisms, picking up litter, and being in the lagoon
help motivate students to care for the lagoon.
Table 1. Logic Model for Lagoon Quest Program
Situation
Inputs
(What we invest)
Like other subtropical
estuaries, the Indian
River Lagoon has very
high biological productivity and is home to
many endangered and
threatened species.
To protect this important treasure, Brevard
Zoo developed an
environmental education program that
enhances environmental stewardship:
Lagoon Quest.
Outputs
Activities
Participation
(What we do)
(Who we reach)
Funding
• school board
• NFWF
In-service
teacher
workshop
Materials
• SJRWMD
• student/
family guide
• teacher guide
Activities
• pre-trip
• study trip
• post-trip
activities
• family
activities
• family
festival
Personnel
• teachers
• zoo staff
• chaperones
• fourth grade
students
• fourth grade
teachers
• parents
• chaperones
• families
Equipment
• school
• field
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t Outcomes
Learning
(Short-term)
Student
Knowledge:
• Indian River Lagoon (IRL)
ecology
• geography
•w
atershed issues
•w
ater quality
Teachers’ and chaperone’s
impression of
program and environmental
attitudes becomes more positive
Action
(Medium-term)
Students’
attitudes:
• more positive
toward
environment
• increased
connection
to nature
• feelings of pride
regarding IRL
Impact
(Long-Term)
Behavioral
intentions and
conservation
behaviors
(personal and
family):
• Conservation careers
explored
• Improved
environmental health
of IRL
Teaching skills
• comfort and confidence to
teach about IRL
• a ddress state standards through IRL
• a ddress real world issues
• a ddress critical thinking skills
5
Logistics and Timeline
The Brevard Zoo asked students and a professor
from University of Florida to lead the program
evaluation process. Funding for the evaluation
came from NFWF in a two-year matching grant.
Evaluators became familiar with Lagoon Quest
through observations and interviews in fall 2005
(the first season of the expanded program) and
developed the logic model. In January 2006 a
stakeholder meeting was held to review and
improve the evaluation plan, logic model, and
timeline.
In spring 2006, student and teacher interviews
were conducted to develop evaluation tools in the
form of student, teacher, and chaperone surveys.
Following the pilot-test, the revised evaluation
tools were distributed to the stakeholders
for review. The data collection process was
implemented from fall 2006 to spring 2007 and
the data analysis was initiated in spring 2007.
The Brevard school administrators preferred
requiring the evaluation activities as part of the
program for all students and teachers. The school
system posted the surveys and test of knowledge
on its Web site and expected full participation
(Box 1). As a result, there is no reasonable control
group (Box 2) for this evaluation. A completed
evaluation focus form is provided in Table 2.
Box 1. Conduct a Sample or a Census?
Evaluations depend on obtaining good data from
participants. In general, evaluators can do a better job
of obtaining, managing, and working with data from
a smaller group of people. If the population can be
sampled, and if the sample will be representative of
the whole, this is often the preferred strategy. In the
Lagoon Quest program, however, school administrators wanted everyone included in the evaluation to
build ownership and treat everyone equally. Because
the administration made the evaluation a required
part of the curriculum, they were confident that every
teacher would participate.
Box 2. Who Could Be in a Control Group?
Evaluations typically compare the results of people who have participated in a program to people who have not to
measure the value of the program. Since the Lagoon Quest program involved all fourth grade teachers and students, a
control group would have to be from another county. That raised questions of whether the control group would really
be similar to the treatment group on the characteristics that matter—access to the Lagoon, socio-economic status,
awareness of the estuary, interest in aquatic ecology. Because the program is site specific, i.e., the Indian River Lagoon,
we decided a control group was not feasible.
6
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
Table 2. An Evaluation Focus for Lagoon Quest
A. Purpose for Evaluation
Who is the intended user of the evaluation results?
n Brevard Zoo and Brevard Public Schools
n Training courses in environmental education evaluation
n National Fish and Wildlife Foundation
What is the intended use for the evaluation results?
n To better understand if program outcomes are being achieved.
n To identify potential areas for further program improvement.
nTo provide an example for other agencies that want to expand their
programs.
Evaluation purpose statement:
nTo measure the program’s effects on students’ knowledge about the
Indian River Lagoon.
n
n
n
To explore the impacts and consequences of expanding a voluntary
environmental educational program to the entire fourth grade in
Brevard County.
To suggest changes to improve the program.
To identify indicators that could be used over time to track the
development of a conservation ethic in Brevard County.
Are you able to reach consensus among major stakeholders on the
purpose of the evaluation? Yes, we met with all the stakeholders and
reviewed the logic model and evaluation tools.
Is the intended use and user of the evaluation clear, specific, and
well-defined? Yes, please see the answers on the left.
Is what is at stake in this evaluation appropriate for an “in-house”
evaluation? No, Brevard Zoo hired evaluators from University of Florida to
evaluate the program.
Can the evaluation purpose be addressed in a way that respects the
rights and dignity of those involved? Yes, confidentiality was maintained
and the participants had right to withdraw their consent.
Will evaluation results be used? Yes, please see the answers on left.
Will decisions be made be based on the data that are collected?
Yes, Brevard Zoo and Brevard Public Schools will use the evaluation results
for further program improvement.
B. Description of program to be evaluated:
See the logic model in Table 1
Are the program objectives well-defined? Yes.
Is it possible for the program objectives to be achieved with the
intended audience? Yes.
Is the program as carried out similar to the way it was intended? Yes.
Is the program grounded in sound assumptions?
Yes: experiential education, science education, environmental education.
Is the program likely to achieve the stated goals/objectives based
on the program’s inputs, outputs, and/or assumptions? The program
objectives intend to achieve short-term and medium-term outcomes. The
long-term outcomes will be explored in the future evaluation opportunities.
Does the program have the potential for sufficient impact,
thus warranting the time and expense of evaluation? Yes, since the
program is required of all students. We would like to understand the long-term
impacts from a future longitudinal study.
C. Logistical Considerations:
Available staff for the evaluation:
Evaluators from University of Florida, Zoo staff
Is the desired evaluation purpose feasible given available staff, time,
and resources? Yes, two years were available and needed.
Information needed by:
Brevard Zoo, Brevard Public Schools,
National Fish and Wildlife Foundation
Given logistical constraints, can an evaluation be carried out that
would yield useful and relevant information? Yes, and the evaluation
report will be produced for AZA, NAAEE, and FWS training courses.
Resources available for the evaluation:
n Funding and staff from Brevard Zoo and NFWF
n Evaluators from University of Florida
Political context/external factors:
nBecause full participation is expected, there is no reasonable
control group for this evaluation.
n Extreme weather conditions could delay the program
or create negative attitudes.
n Mandatory participation might create negative attitudes.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 7
Developing an
Evaluation Plan
The evaluation plan began with brainstorming
evaluation questions that could be asked about
the program. Because of the various aspects such
as the in-service training, in-class activities,
study trip, and family festival, many potential
questions were suggested and discarded.
Although the program discussed conservation
behaviors, evaluators initially believed that
the fourth-grade program did not emphasize
conservation behaviors that youth could undertake
in a manner that would create a measurable
change. Stakeholders, however, suggested that
students’ intention regarding picking up litter and
returning to the Lagoon would be appropriate
indicators of program success, so these
conservation behaviors were included. Because
the family festival was not initiated until spring
2007, it was removed from the evaluation plan.
The final evaluation questions are explained later
and are summarized in Table 3.
Table 3. Evaluation Plan Matrix
Evaluation
questions
8
Indicators
Sources of
Information
Methods /
Tools
Sampling/
Design
A. How was Lagoon
Quest implemented
and perceived?
• Response to teacher surveys
• Response to student surveys
• Staff observations
• Fourth grade public
school teachers
• Fourth grade public
school students
• Staff
• Teacher surveys
• Student surveys
• Staff observations
• Post only
• All fourth grade public school teachers and
students in Brevard County, Florida, who
participated in Lagoon Quest program
B. Does the Lagoon
Quest program
influence students’
knowledge of Indian
River Lagoon?
• Responses to knowledge test
• F ourth grade public
school students
• Student test of
knowledge
• Pretest/posttest
• All fourth grade public school students in
Brevard County, Florida, who participated
in Lagoon Quest program
C. Do students enjoy
their experience
with the Lagoon?
• Response to student surveys
• F ourth grade public
school students
• Parents who chaperoned
the study trip
• Student surveys
• Chaperone survey
• Post only
• All fourth grade public school students in
Brevard County, Florida, who participated in
Lagoon Quest program
• Parents who chaperoned the study trip
D. Do teachers’ previous
experiences in environmental education affect
their use of the Lagoon
Quest unit?
• Teachers’ previous experience
in environmental education
• Students’ knowledge test
scores
• Fourth grade public
school teachers
• Fourth grade public
school students
• Teacher surveys
• Student test of
knowledge
• Post only
• All fourth grade public school teachers and
students in Brevard County, Florida, who
participated in Lagoon Quest program
E. How can Lagoon Quest
Program be improved?
• Suggestions from teacher and
chaperone surveys
• Comments to staff
• Observations by staff
• Chaperones
• Staff
• Fourth grade public
school teachers
• Chaperone survey
• Teacher survey
• Staff records
• Post only
• Parents who went on the Lagoon Quest
study trips
• The zoo staff observation forms filled out
during the study trips
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
Photo by A. Humphrey
Tool Development
and Modification
The initial student knowledge test was developed
by the Lagoon Quest team of teachers and zoo staff
in 2005. Evaluators in the University of Florida
analyzed the results after the first year of the program
and offered some suggestions for vocabulary and
additional questions, based on student interviews.
The other evaluation tools were initially designed
by graduate students from University of Florida in
fall 2005 after observing the Lagoon Quest study
trip and interviewing teachers. In spring 2006,
stakeholders reviewed these evaluation tools and
offered suggestions. School district staff confirmed
their interest in the answers to all five evaluation
questions and suggested that evaluators develop an
additional tool, the chaperone survey. The student
attitude survey was significantly revised after pilot
testing and interviews with students. The teacher
attitude survey was tested with teachers from the
planning committee. The following tools were used
in the Lagoon Quest evaluation:
• Teacher attitude survey
• Student knowledge test
• Student attitude survey
• Staff observation form
• Chaperone survey
Additional details about these tools are provided in
the methods and tools portions of each question.
? Evaluation Question A:
How was Lagoon Quest implemented
and perceived?
n Indicators: Responses to student attitude
surveys, teacher surveys, and staff observations.
n Sources of information: Fourth-
grade public school teachers, fourth-grade public
school students, and staff.
n Methods and Tools: The tools used to
collect this information were a student survey,
a teacher survey, and staff observation forms.
Before the student survey was developed, an
interview guide was created to explore students’
interest in the natural environment and their daily
outdoor activities. Fifteen group interviews were
conducted in four fourth-grade classes. After
the survey was developed, two pilot tests were
conducted in fourth-grade classes to check the
vocabulary and language. The survey included
questions that explored students’ in-class and
study trip experiences.
The teacher survey was developed to gauge
teachers’ previous experience in using Lagoon
Quest and other environmental education materials;
to determine teacher’s attitude toward Lagoon
Quest; and for teachers to report what portion of the
program they completed, their perception of support
from their principals, other teachers, and parents,
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 9
and their level of knowledge of and comfort with
teaching about the lagoon.
Zoo staff completed an observation form at the
end of each study trip. The form was created
after the evaluators observed study trips and
included questions about the weather conditions;
species diversity; and the preparation of teachers,
chaperones, and students.
n Sampling and Design: Post program
only, with all fourth-grade students and teachers
who participated in Lagoon Quest and zoo staff who
implemented the Lagoon Quest study trips. Fourthgrade teachers helped implement the student survey.
Teachers were asked to have students complete
the Web-based or paper-and-pencil surveys after
completing Lagoon Quest. The teacher survey was
made available to teachers through the school district’s
Web site, in the Teacher Guide, and through the school
mail. The staff observation form was distributed to the
zoo staff who conducted the study trips.
? Evaluation Question B:
River Lagoon, as measured by a test of knowledge.
n Sources of information: Fourth-grade
n Sampling and design: Evaluation
participants consisted of all fourth-grade public
school students. The design was pretest/posttest.
The test was distributed through the school system
Web site and for those who requested a paper
version, the mail.
? Evaluation Question C:
Do students enjoy their experience
with the Lagoon?
survey.
n Indicator: Students’ knowledge of the Indian
10
knowledge quiz was developed by teachers
and zoo staff and based on the Lagoonie
Logbook and Indian River Lagoon Activity Book. It
was tested in the first program season and revised to
increase variability. The second version of the test
consisted of eighteen questions ranging from simple
to difficult. The test was developed to explore
students’ ecological concepts of Indian River
Lagoon and the basic concepts of ecology.
Teachers reviewed and approved of the test.
n Indicator: Responses to a student attitude
Does Lagoon Quest influence students’
knowledge of Indian River Lagoon?
students who participated in Lagoon Quest.
n Methods and Tools: An initial
n Sources of information: Fourth-grade
students who participated in Lagoon Quest.
n Methods and Tools: The tool used to
collect the information was a student attitude survey
described in evaluation question A.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
n Sampling and Design: All fourth-grade
public school teachers were asked to use the
Lagoon Quest materials, including the student
survey. The survey design was posttest only,
because attitudes are not likely to change very
quickly and some questions required the program
experience to answer. Teachers were asked to
have students complete the Web-based or paperand-pencil surveys after completing the Lagoon
Quest unit.
? Evaluation Question D:
Do teachers’ previous experiences in
environmental education affect their use
of the Lagoon Quest unit?
n Indicators: Responses to teacher survey
and students’ knowledge scores from their Lagoon
Quest knowledge test.
n Sources of information:
Fourth-grade students who participated in Lagoon
Quest.
n Methods and Tools: The development
of the teacher survey was described in evaluation
question A, and the student test of knowledge was
described in evaluation question B.
n Sampling and Design: Post program
only, with all fourth-grade teachers who
participated in Lagoon Quest. The survey was
Photo by K. Burke
In the end of the year, we ask students
to pick their favorite activity and most
of them choose Lagoon Quest. —Teacher
distributed to teachers by the school district on
their Web site, in the Teacher Guide, and by
paper through the school mail.
? Evaluation Question E:
How can the Lagoon Quest be improved?
n Indicators: Teachers’ and chaperones’
suggestions and staff observation records.
n Sources of Information: Fourthgrade public school teachers, students’ parents who
attended the study trips as chaperones, and zoo staff
who implemented the study trip activities.
n Methods and Tools: Surveys and
observations were used to obtain the information.
The survey for chaperones consisted of several
open-ended questions to understand their attitudes
toward Lagoon Quest and their idea about how the
program affected their children.
n Sampling and Design: Teachers and
chaperones who accompanied the study trips, and
the staff, were the study population. The surveys
were distributed to the chaperones by the zoo
staff upon the completion of each study trip. The
chaperones sent the surveys back to the evaluators
by business reply envelopes enclosed with the
surveys. The staff completed the observation form
at the end of each day. Teachers completed the
teacher survey on the Web or on paper.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 11
Photo by S. Chalmers
Lagoon Quest affected my child in a very positive
way by teaching him about the animals in our
Lagoon as well as how important it is to preserve
their habitat. —Chaperone
Data Collection
Box 3. Using Computers to Collect Data
In keeping with the technology requirements
of state education reform efforts, the school
administration wished to encourage the use of
technology with the Lagoon Quest evaluation. In
addition, the program developers saw this as a good
opportunity to develop a long-term tracking system
for students’ environmental knowledge, attitudes,
and intentions. As a result, they requested that all
students and all teachers be involved in the program
evaluation and that the data collection be conducted
in the schools’ technology class, using the school
system’s Web site (Box 3).
Instructions for the evaluation implementation
were provided in the Teacher Guide to the Lagoon
Quest program, reinforced during an in-service
workshop, and mentioned in teacher updates sent
by the teacher coordinators of the program.
We discovered several challenges to using computers
to collect data. Unfortunately, all schools do not have
a computer lab and although all classrooms have a
computer, it is not always easy for teachers to enable
every student to enter his or her own data. Teachers may not have been aware that mistakes made
while entering information on the Web form can be
corrected before submitting it. After reviewing the
results, evaluators felt the paper version was more
accurate and accessible.
In August 2006, three forms were posted on
the school system’s Web site: student test of
knowledge (to be completed before and after
the program), student attitude and experience
survey (post program), and teacher attitude and
experience survey (post program). Students and
teachers were asked to use their school-assigned
identification number to maintain confidentiality,
allow evaluators to group students into the proper
classroom, and link the classes to the teachers.
Some teachers voiced concerns that they either
could not or did not wish to use the school’s Web
site or computer facility. Those teachers were sent
packets of student tests and surveys on paper.
The act of requesting a paper survey may have
increased a teachers’ commitment to using it.
Chaperones were handed their survey and a return
envelope by zoo staff during the study trip. Zoo
staff completed their observation forms daily and
submitted them at the end of October, November,
and April.
12
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
As the paper versions of the tests and surveys
were sent to the evaluators, they were entered
into a spreadsheet. The school system
compiled the computer-based Web responses
into parallel databases and gave the evaluators
access to their system. By September 29, 2235
pretests of knowledge had been submitted.
By November 27 this number had risen to
2772 but only 1470 posttests were completed.
Reminders to complete the surveys were
distributed to teachers in November 2006
and March 2007. By April 18 a total of 3463
pretests, 2369 posttests, 1084 student surveys,
and 99 teacher surveys were received.
Data Analysis
Student knowledge was measured by eighteen
questions regarding Indian River Lagoon, estuary
ecology, and water quality. Because the test was
administered before and after the program, one
can assume that something in the intervening time
produced any changes observed in the answers. It
could have been a TV program, increased parental
interest in the Lagoon, or teacher-led activities
and instruction. Although any one student could
have watched a TV program on the Lagoon, it was
not likely that thousands of students did. Given
the large sample size, an increase in knowledge
was deemed likely to have been a function of the
program. Knowledge of the Indian River Lagoon
Box 4. Determining Impacts on Students
Originally, we planned to correlate students’
Lagoon Quest knowledge test with their FCAT
scores, the state achievement test. However,
fourth grade students take the FCAT in reading
and writing but not science. Not having science
achievement data to work with, we used Title I
school classification. These are schools with at
least 40 percent of the students from families
whose income is below the poverty level. Because
of correlations between income, health care,
home environment and achievement,4,5 we could
determine how the Lagoon Quest program works
for disadvantaged students by comparing Title I and
non-Title I student scores.
was defined as understanding the concepts
presented in Lagoon Quest: watershed, local
biodiversity, estuarine ecology, and water quality.
Several other variables were used to interpret
test results, including Title I status (Box 4)
and teacher experience with environmental
education. Teacher experience and familiarity
with the lagoon could have led to a difference in
instructional quality, amount of time spent on the
material, and therefore student knowledge.
Data were entered, student numbers were matched,
and the student tests were compared against the
correct answers, providing a student score for
correct responses. After data cleaning, 3398 pretests
and 2093 posttests were useable, but only 1616
were matched pairs. This enabled evaluators to
determine the difference between using matched
4. Borman, G. and J. D’Agostino. (1996). Title I and student achievement: A meta analysis of federal evaluation results. Educational Evaluation and Policy Analysis. 18(4), 309-326.
5. Duncan, G and J. Brooks-Gunn. (1997). The effects of poverty on children. The Future of Children. 7(2), 55-71.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 13
and unmatched data. Both the matched pairs and
unmatched data were analyzed with t-tests. The
students whose entire class had perfect scores
in posttest were excluded in data analysis (Box
5). The random missing data were replaced
by the mean value of each variable (Box 6).
Descriptive statistics were conducted to describe
the materials that were used in classes, the
teachers’ perception of Lagoon Quest and study
trips. ANOVA was used to assess the difference
between student scores from Title I and nonTitle I schools. Correlation analysis was used
to assess the relationships between teacher
experience and student scores. Descriptive
statistics were used to describe student’s
enjoyment of Lagoon Quest. Independent t-test
was used to determine if teachers’ previous
experiences with environmental education
affected the number of Lagoon Quest activities
they completed. And finally, chaperones and
teachers’ suggestions were categorized into
different themes, and direct quotes were
provided.
Box 6. Dealing with Missing Data
Missing data is a problem when conducting data analysis
because some statistical tests are sensitive to the number
of cases in each variable. Evaluators need to have their
own standards to address missing data. We coded missing
values when the data were entered and then dealt them
in two ways. Surveys in which an entire section was
missed were considered incomplete, and the entire survey was deleted. For the surveys that contained random
single missed questions, the mean for those variables
were used to replace the missing values.
Box 5. Perfect Posttest Scores?
Six classrooms submitted perfect posttests for every student in the class. Because schools with strong records in
science achievement were not scoring as well, evaluators wondered what occurred with a few classrooms in a few
schools. The teachers might have effectively taught the content that the knowledge test covered. On the other hand,
the teachers might have reviewed the correct answers with students before the post-tests were submitted.
From talking to teachers, evaluators learned that some teachers asked students to complete the posttest on paper,
and then enter their own responses on the computer. Some teachers might use the opportunity of the quiz to continue teaching as they review the correct responses. It may have been the case that some teachers asked students
to input the corrected version of their posttest rather than the initial version. Not knowing exactly how these high
scores were obtained, we excluded these classes from the data analysis.
14
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
I learned more about the lagoon and its
inhabitants than I knew before. —Chaperone
the pre-trip activities and followed the directions in
the Teacher Guide.
Results
? Question A
How was Lagoon Quest implemented and
perceived?
One hundred and thirty-one teachers responded to the
teacher survey. Among these teachers, nearly all completed the first two activities and about 80% completed
activities 3, 4, and 6. About 75% of teachers completed
activities 5 and 9 while about 65% of teachers completed activities 7 and 12. However, less than 65% of the
teachers completed activities 8, 10 and 11 (Table 4).
Zoo staff reported in the observation forms that more
than 80% of the classes seemed prepared for their
study trip, suggesting that most teachers conducted
More than 50% of the teachers indicated that
Lagoon Quest was very helpful for increasing
students’ awareness of Indian River Lagoon, while
more than 45% of teachers said that Lagoon Quest
was very helpful for increasing environmental
responsibility. About 38% of teachers indicated
that Lagoon Quest was very helpful for promoting
conservation ethic. Regarding educational
standards, most of the respondents indicated that
Lagoon Quest was only somewhat helpful for
addressing state standards in language arts, social
studies, and math, while 40% of the respondents
stated that Lagoon Quest was very helpful for
addressing science standards (Table 5).
Table 4. Lagoon Quest Activities Used by Teachers
Activity
used before trip (%)
used after trip (%)
Total used (%)
1.Brainstorming the Lagoon!
87
3
90
2 .The IRL Activity Book
88
9
97
3.Can I get there from here?
76
2
78
4 .What’s for supper?
75
6
81
5.Taking a closer look (I)
63
11
74
6.How healthy is the lagoon (I)
59
21
80
7.Who am I?
47
20
67
8.Wetland in a pan
42
20
62
9.Indian River Lagoon Reflection
12
60
72
10.Changing waters
18
36
54
11.Taking a closer look (II)
17
40
57
12.How healthy is the lagoon (II)
21
44
65
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 15
About 70% of the teachers indicated that the zoo
instructors were very knowledgeable, and only
9% of the respondents strongly agreed that the
zoo instructors could improve their facilitation
skills. About 65% of the teachers stated that it was
easy to get parent chaperones, and 80% of them
indicated that students were comfortable while
in the water. Most teachers agreed that the study
trip experiences were connected with classroom
activities. Indeed, the teachers also specified that
the study trips encouraged them to learn more
about the environment. More than 90% of the
teachers agreed or strongly agreed that the study
trip was an essential element of the Lagoon Quest
program (Table 6).
About 50% of the teachers indicated that the
Lagoon Quest Teacher Guide, Indian River
Lagoon Activity Book, and Lagoonie Logbook had
very high educational value. They also indicated
that students were better able to use the Indian
River Lagoon Activity Book than Lagoonie
Logbook (Table 7).
There were six family activities in the Lagoonie
Logbook. About 42% of the students indicated
that they did some family activities. Among
1392 students who completed the student attitude
survey, about 10% reported that they completed
family activity 1. About 19% of them completed
family activity 6, and 14 % of them completed
activity 3. However, less than 10% of the students
16
completed activity 2, 4 and 5 (Table 8). The
Brevard Zoo received 61 certificates of completion
from families that enabled 112 adults and 117
children to take advantage of the opportunity for
free admission to the zoo.
These various data suggest that a large
majority, but not all, of the responding teachers
implemented Lagoon Quest as intended. The
post-trip activities on water quality were not used
as often as the other activities. The teachers are
generally quite pleased with the program and its
impact on student awareness and attitudes, even
admitting that they learned more about the Lagoon
from the program. As designed, the program
is recognized as supplementing the science
standards. For the most part, the family activities
were not implemented. It is not possible to know
if the teachers did not encourage students to take
them home, if parents were too busy, or both.
? Question B
Does Lagoon Quest influence students’
knowledge of Indian River Lagoon?
There was a significant increase in knowledge
about the Indian River Lagoon because posttest
scores were significantly higher than pretest scores
in the matched pairs (i.e., the p value is less than
.01, assuming a 99% confidence level) (Table
9). The results of unmatched data (Table 10) are
nearly identical.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
Results
Table 5. Teachers’ Perceptions of the Lagoon Quest Program (percent responding)
How helpful was this program for….
Very Little
Increasing your students’ awareness of Indian River Lagoon
Enhancing your students’ sense of environmental responsibility
A little
Somewhat
Much
Very Much
0
1.6
7.1
34.1
57.1
0.8
0.8
14.3
38.9
45.2
Promoting your students’ conservation ethic
0.8
1.6
24.8
35.2
37.6
Addressing Language Art Sunshine State Standards
8.0
7.2
43.2
24.8
16.8
Addressing Social Studies Sunshine State Standards
5.6
9.6
37.6
24.8
22.4
Addressing Science Sunshine State Standards
0.8
1.6
19.2
38.4
40.0
Addressing Math Sunshine State Standards
13.7
17.7
41.1
18.5
8.9
Table 6. Teachers’ Perception of the Lagoon Quest Study Trip (percent responding)
Statements
Strongly
Disagree
Zoo instructors had sufficient knowledge of program content
Zoo instructors’ could improve their facilitation skills
Disagree
Neutral
AGREE
STRONGLY
agree
0
0.8
4.0
24.0
71.2
16.1
33.9
25.8
15.3
8.9
It was easy to get parent chaperones for the Lagoon Quest field trip
8.0
11.2
16.8
38.4
25.6
Students were comfortable in the water
0.8
5.6
13.6
58.4
21.6
Field trip experiences were not connected with the classroom activities
42.7
41.9
9.7
3.2
2.4
0
3.2
29.8
47.6
19.4
0.8
0.8
6.4
20.8
71.2
ok
high
very high
Field trip experience encouraged me to learn more about environment
Overall, field trip was an essential element of the Lagoon Quest program
Table 7. Perceptions of the Lagoon Quest Materials (percent responding)
Statements
very low
low
Your perceived educational value of the Lagoon Quest Teacher Guide
1.7
2.6
14.7
29.3
51.7
Your perceived educational value of the Indian River Lagoon Activity Book
1.0
1.0
14.4
35.1
48.5
Your perceived educational value of Lagoonie Logbook
3.0
3.0
19.8
27.7
46.5
The students’ ability to use the Lagoonie Logbook
5.0
4.0
26.0
32.0
33.0
0
3.2
18.1
34.0
44.7
The students’ ability to use the Indian River Lagoon Activity Book
Table 8. Lagoonie Family Activities
Reported by Students
Family Activity
1. Build a plankton net
Table 9. Matched Cases of
Pre and Post Test Scores
completed
(%)
Pretest
Mean (N)
SD
posttest
Mean (N)
SD
T
p
10.1
15.73 (1616)
3.32
21.06 (1616)
4.23
47.89
0.00
2. What happened to all that rain?
8.7
3. Who am I
13.6
4. Monofilament activity
6.2
5. Salt matters!
9.4
PRETEST Mean (N)
SD
POSTPOST Mean (N)
SD
6. Clean it up!
18.7
15.70 (3398)
3.39
21.03 (2094)
4.28
Table 10. Unmatched Cases of Pre and Post Test Scores
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 17
Photo by P. Corapi
An analysis can be conducted on each of the
eighteen questions in the knowledge test to
indicate where initial misunderstandings were
changed and where learning had not yet occurred
(Table 11). In this case, the greatest increase in
learning is with questions 11, 13, 15, and 16,
which covered topics of lagoon ecology, habitats,
endangered species, and water quality. Over
75% of the students already knew the correct
answers to questions 2, 4, 5, and 9, which included
information about conservation ethics and the
roles of plants and animals. Less than 50% of
the students were able to answer questions 14,
15, 16, and 17 correctly after the program. These
questions covered more challenging aspects of the
topics of wetlands, endangered species, and water
quality.
Thirty-one schools in Brevard County qualify
for Title I status, represented by 677 matched
pretests and posttests from respondents. The
remaining 939 students with matched pretests and
posttests attend non-Title I schools. There was a
significant difference in the pretest scores between
Title I and non-Title I schools, with the Title I
schools scoring significantly lower than the
non-Title I schools. After the program, there was
still a significant difference between the posttest
scores at the 95% confidence level (Table 12).
However, the improvement in Title I schools
is significantly greater than the improvement
in non-Title I schools. Because Title I students
18
averaged lower pretest scores than non-Title I
students, they had more potential to improve
(Table 13).
? Question C
Do students enjoy their experience with
the Lagoon?
The students rated how much they liked the study
trip activities and classroom activities from one
(did not like at all) to five (liked very much).
Some data were missing, perhaps due to errors
in using the computer system. To avoid the bias
caused by missing data, the missing data were
replaced by the mean value for that item.
Regarding Lagoon Quest study trip, a majority
of students indicated that they very much liked
catching fish in the lagoon and spending time in
the water. Between 3% and 14% of students did
not like any of the activities. Seeing birds and
plants, finding things around Lagoon, and testing
water quality in the Lagoon were less popular
(Table 14).
With regard to Lagoon Quest classroom activities,
the majority of students liked learning about
animals and the Indian River Lagoon while
learning about salinity and turbidity were not
popular (Table 15). All teachers; however, did
not use the classroom activity that covered water
quality. Approximately 36% of teachers used all
twelve lessons.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
Results
Table 11. Frequencies of Correct Answers in Pretest and Posttest
Pretest
Posttest
Increase
Question #
Frequencies
PERCENTAGE
Question #
Frequencies
PERCENTAGE
PERCENTAGE
1
1106
68.4
1
1476
91.3
22.9
2
1225
75.8
2
1403
86.8
11
3
815
50.4
3
1163
72.0
22.4
4
1487
92
4
1564
96.8
4.8
5
1345
83.2
5
1531
94.7
11.5
6
658
40.7
6
1041
64.4
23.7
7
976
60.4
7
1351
83.6
23.2
8
1118
69.2
8
1494
92.5
23.3
9
1494
92.5
9
1553
96.1
3.6
10
958
59.3
10
1157
71.6
12.3
11
485
30.0
11
979
60.6
30.6
12
653
40.4
12
1021
63.2
22.8
13
417
25.8
13
888
55.0
29.2
14
157
9.7
14
550
34.0
24.3
15
63
3.9
15
472
29.2
25.3
16
79
4.9
16
691
42.8
37.9
17
385
23.8
17
759
47.0
23.2
18
860
53.2
18
1204
74.5
21.3
Table 12. Knowledge Scores for
Title I and Non-Title I Schools
Title I Schools
Test
Table 13. Improvement between
Pretest and Posttest Scores
Non-Title I Schools
Mean (N)
Title I Schools
SD
Mean (N)
SD
F
Sig.
Pretest
15.05 (677)
3.22
16.22 (939)
3.31
50.39
0.00
Posttest
20.80 (677)
4.35
21.24 (939)
4.14
4.31
0.04
Non-Title I Schools
Mean (N)
difference
SD
Mean (N)
difference
SD
F
Sig.
5.75 (677)
4.76
5.02 (939)
4.23
12.85
0.00
Table 14. Student Enjoyment of Study Trip Activities (percent responding)
not at all
a little
somewhat
much
very much
MEAN*
Spending time in the water
3.2
2.9
5.4
21.3
67.2
4.51
Catching fish in the lagoon
3.7
4.2
5.6
23.1
63.4
4.42
Finding things around the lagoon
10.9
9.6
27.1
19.5
32.8
3.62
Testing water quality
14.0
19.0
32.3
19.7
15.0
3.03
Seeing birds at the lagoon
12.9
25.4
28.7
13.5
19.4
3.01
Seeing plants in the lagoon
12.8
35.4
20
16.4
15.4
2.96
* Scale of one to five in which one is “not at all” and five is “very much”
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 19
To investigate students’ overall enjoyment
of Lagoon Quest experience, two composite
variables were created. The first variable explored
the overall enjoyment of Lagoon Quest study
trip while the second variable measured the
overall enjoyment of Lagoon Quest classroom
experience. Both variables indicated that students
liked their Lagoon Quest experience (Table 16).
? Question D
Do teachers’ previous experiences in
environmental education affect their use of
the Lagoon Quest materials?
Because Lagoon Quest is now required of all
fourth-grade teachers, we wanted to explore
teachers’ attitudes toward Lagoon Quest. We
wondered if their previous experience with Lagoon
Quest or their experience with other environmental
education programs could predict their willingness
to use these materials. We received 131 teacher
surveys by the end of May; however, only 96 were
completed because of a computer glitch. With
the term ending, school administrators asked that
we not bother these teachers with a request to
complete the survey. Among the 96 teachers, 62
did not have previous experience in environmental
education while 34 had prior environmental
education experience. Independent t-test was used
to determine if teachers’ previous experience with
environmental education affected the number of
Lagoon Quest activities they completed. The results
suggested that there was no significant difference
in the number of Lagoon Quest activities used
between teachers with and without environmental
education experience. Among the 131 teachers, we
received student data for only 35 of the classrooms.
Of these 35 teachers, only 20 of them provided
their environmental education experience (Box
7). Of these 20 teachers, only 6 teachers had
experience in environmental education. Among the
20 teachers, there was no significant difference in
students’ knowledge based on teacher experience
with environmental education (Table 17).
Similarly, there was no difference between
teachers who had participated in Lagoon Quest
Box 7. Increasing Response and Completion Rates
A successful evaluation requires not only high response rate, but also a high completion rate. Twenty-four percent
of the teacher respondents to the Lagoon Quest survey did not complete the entire survey due to the following:
1) a computer glitch failed to bring up additional pages, 2) the perception that some questions were not relevant,
3) mistaken skips. In addition to issues of confidentiality and anonymity, the Institutional Review Board that approves research with human subjects expects that participants will be able to skip a question if they choose. Most
respondents, however, will answer an entire survey if they believe their responses will be helpful. Providing clear and
frequent information about how to complete questions and why each section will be useful can reduce the number
of incomplete surveys. Paper-based instruments have been shown to have fewer skipped question than computerized
instruments.6 Paper-based surveys are also easier to read and can easily to be noticed if people keep the surveys
instead of throwing them away.
6. Paolo, A. and Gibson, C. and Partridge, T. and Kallail, K. (2000). Response rate comparisons of E-mail and mail-distributed student evaluations. Teaching and Learning in Medicine. 12(2), 81-84.
20
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
Results
Table 15. Student Enjoyment of Classroom Activities (percent responding)
Classroom Topics
not at all
a little
somewhat
much
very much
MEAN*
Indian River Lagoon
1.8
2.9
5.4
16.6
73.4
4.6
Animals
1.6
4.5
8.4
26.9
58.6
4.38
The watershed
14.8
15.3
26.3
18.0
25.7
3.27
pH
13.0
18.1
27.7
18.2
23.0
3.22
Plants
10.9
24.3
26.4
19.1
19.3
3.12
Turbidity
16.8
28.3
22.7
14.7
17.5
2.96
Salinity
15.3
30.7
22.3
16.4
15.4
2.93
*Scale of one to five in which one is “not at all” and five is “very much”
Table 16. Students’ Enjoyment of Lagoon Quest Program
Enjoyment
Scores*
* Scores
3 = Somewhat like
Study Trip Experience
3.59
1 = Did not like at all
4 = Like much
Classroom Experience
3.50
2 = Like a little
5 = Like very much
Table 17. Association between Environmental Education Experience and Pretests and Posttests
Test
Experienced
Teachers Mean (N)
SD
less experienced
teachers Mean (N)
SD
F
Sig.
Pretest
15.63(6)
1.27
15.90(14)
1.15
0.32
0.58
Posttest
21.31(6)
3.58
21.20(14)
2.90
0.45
0.51
before and those who had not. The
nonsignificant results might be due to the
small sample size, but it might also suggest
that the program was designed and the
curriculum was written well enough for
less experienced teachers to use the Lagoon
Quest program as well as those with more
experience. It could also mean that other
variables are more indicative of success than
previous experience (Box 8).
Because of the small number of teachers
included in this analysis, additional data were
collected through focus groups at several schools
where we did not receive many surveys from
teachers. We visited 5 schools and spoke to 13
Box 8. Contacting Non-Respondents
If the response rate is not high (more than 70%) an evaluator should wonder if the respondents are truly representative of
the population. The best way to find out is by contacting those who did not respond. Because of confidentiality concerns
with the school system, however, surveys were coded with identification numbers, but the evaluators were not allowed to
match those numbers to people to contact them. Other strategies to address the low response rate would be to compare
the respondents’ demographic data to census data and to compare early respondents to late respondents. Neither strategy was feasible with the Lagoon Quest evaluation because fourth grade teachers and students may not be representative
of the county population and response time was a function of the study trip schedule, not necessarily respondent interest.
To achieve higher response rate, sending multiple reminder letters may help increase the response rate as long as recipients are tolerant of them.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 21
teachers, 11 of whom had participated in Lagoon Quest.
Included were teachers with a great deal of experience
from the days of voluntary participation and teachers
who were brand new to the program. Interest in using
Lagoon Quest ranged from very strong to rather weak.
Teachers described the way they used the material
and some of the barriers associated with completing
the post-trip activities. With regard to Question D, we
believe that prior voluntary Lagoon Quest experience
is not an indicator of interest and commitment to using
Lagoon Quest because some teachers were told to
sign up for the study trip by their principals. However,
it appeared that teachers who had prior experience
in teaching science and had stronger commitment
to Lagoon Quest may be more likely to spend more
time using Lagoon Quest activities, to use additional
teaching materials that supplement Lagoon Quest, and
to reinforce conservation ethics.
? Question E
How can Lagoon Quest be improved?
n Chaperones’ Suggestions
for the Program
After the Lagoon Quest study trips were completed
in 2007, 171 surveys from parents who attended
the study trips as chaperones were reviewed. These
surveys provided useful suggestions to Brevard
Public Schools and Brevard Zoo for further program
improvement. Suggestions included modification
of program components and improvement of the
environment around the trip areas. The suggestions
were categorized into several perspectives, and some
direct quotes are provided in the following paragraphs.
A majority of the chaperones enjoyed the study trip,
appreciated the program, and credited the passionate
and knowledgeable staff.
The two young people who spoke and helped were
very knowledgeable, educated, very well spoken.
Full of energy and you could tell they loved what
they were doing! They also liked kids. Thank you
for the fun day. Sincerely!
It was great, the presenters were well informed and
in touch with the kids and my daughter and I both
learned a lot.
22
Regarding program components, the chaperones
recommended including more explanation on the
threats to the health of the Lagoon and the strategies
to conserve the Lagoon and its wildlife. Moreover, the
chaperones suggested that more reflection could be
included after each activity, especially after students
got out of water. More information on how and why
to help the lagoon might move the discussion toward
conservation actions.
Some kind of summary discussion would
have helped tie the day together. The
introduction was great, the water experience
was great, the pollution demonstration was
well worth it, but can we see evidence of
this pollution right here in the lagoon? Do
the tests prove it? What can your school
do, what can you do at home? Why is the
diversity so important? We were left with
these questions and I am sure the kids
thought they had a great day of fun, but now
what? (Six similar comments.)
Let kids see they can make a positive difference
in ecology with care and forethought. (Two
similar comments.)
Chaperones also suggested reducing the use of
handheld devices such as GPS because they were not
age appropriate and it was difficult for chaperones to
assist their children. Other chaperones suggested that
more resources such as nets could be provided, and
that more instructions could improve the experience.
Do not use handheld device on treasure hunt.
Children could not work them and neither
could the adults. (Two similar comments.)
Seining was very difficult for some of the
children. Maybe a little more instruction by
Lagoon Quest instructors. The children were
shown in school but it is not the same when you
enter the water.
Extending the length of the study trip was
suggested. Some chaperones indicated that
more time should be allowed for discussion of
findings and sharing of experiences.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
Follow up on scavenger hunt. I’m not sure,
but a lack of time apparently kept us from
discussing our findings. It might be better to
have each group focus on one part of scavenger
hunt then allow each group to share their finds
and comment about them.
Many chaperones also raised a concern about weather
conditions. They mentioned that picking a warmer day
will be helpful for children, teachers, and chaperones.
The field trip was excellent information was great.
The temperature of the water was terrible; it was
too cold to stand in. It was the worst time to go. I
could not stand the burning in my legs. The other
adults would not go in the water. This trip should
be planned in warmer weather. Go during warmer
weather not in the winter.
Schedule during warmer months as sea life
in the shallows was too limited. (Six similar
comments.)
Regarding the facilities in the trip area, the chaperones
raised some concerns about the nets, the need for
more trashcans, more bathrooms, and the possibility
of changing the trip site.
I think that the group should have had more
green nets for the kids. One net wasn’t enough.
I would suggest at least three per group. The
kids caught more interesting things with the
green net than the large net. Great field trip.
(Two similar comments.)
Have better bathroom facilities and more
trashcans around. (Two similar comments.)
The chaperones also proposed having more staff for
trips with larger numbers of students. The chaperones
indicated that they were not familiar with the
procedures, so more helpers would be an advantage
for children to learn better.
Needed more knowledgeable assistants to help
the kids in the river. Although the chaperones
tried to help the kids, we were unable to catch
anything. (Four other similar comments.)
n Teachers’ Suggestions
for the Program
One hundred and thirty-one teacher surveys were
received by the end of May and only 96 were
completed. Because we were unable to identify the
nonrespondents, we cannot generalize to the entire
teacher population. We can only summarize the
comments that were submitted. More than 10 teachers
stated their appreciation for the program while some
teachers indicated that their students were enjoying
the program. Following are some examples.
My students thoroughly enjoyed the program
and our visit to the IRL. The IRL staff was well
organized, knowledgeable and helpful.
I really love the curriculum and the trip and
appreciate that the background work is so well
prepared and thorough. There might even have
been too much between both books we used.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 23
so the focus can be on teaching Lagoon Quest after
students have concluded the other requirements.
I was very pleased with the educational
knowledge and experience the Lagoon Quest
program provided for me and my students.
A few comments indicated that some respondents
were not thrilled with the program. They felt the
requirement took too much time from their teaching.
This is a great optional program, but when it
becomes another requirement it’s not as much
fun for the teachers as it could be. We have so
many things we are required to do, and this
becomes just one more thing. I completed my
responsibilities and tried to make it fun for
the students, but I know many teachers in the
county who didn’t. My curriculum is already
mapped out for the year, and this had to be
worked into my classroom theme.
Due to the FCAT reality, Lagoon Quest was not a
priority in fourth grade.
I have enjoyed this program for many years. I do
not like being scheduled to go on the trip. There are
often other things happening which I should prefer
not to change for the trip. I am also concerned for
the ecology of the areas to which the thousand of
students tramp through for weeks on end.
Many teachers provided suggestions for program
improvement, specifically on the subject of the
schedule, teaching materials, facilities at the trip sites,
and evaluation tools.
Regarding the teaching schedule, several teachers
suggested that the study trips be scheduled after FCAT
24
The ideal time to take Florida’s fourth-grade
students is in the spring after FCAT testing.
(Five similar comments.)
Some teachers did not receive sufficient Lagoon Quest
materials or they received the materials too late to
prepare for the program. Some teachers wanted to
be able to link these materials to the school science
requirement so that they could correlate the program to
their teaching schedule. Teachers also stated that there
was a need to make the teacher guide and Lagoonie
book coordinated.
Please make sure that the water management
district sends enough student books to every
school. (Three similar comments.)
We needed to get the materials earlier in the
year. I felt rushed to get it done. I would like
to see this correlate a bit more with the county
required science test so that I don’t have to
go back and reteach from the Lagoonie to get
ready for the test.
The teacher manual needs further explanation
of some activities. It also needs to reflect what
is in the Lagoonie Logbook.
Teachers also commented on the facilities at the trip
sites. Some areas did not have sufficient restrooms and
trash cans to accommodate large groups of students.
They suggested looking for a more appropriate place
that could accommodate large groups.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
Find a location that has enough bathrooms to
accommodate sixty children and twenty plus
adults. Kelly Park patrons were angry and
verbally abusive because they had to wait outside
the bathroom while sixty kids changed one at a
time in two stalls. (Three similar comments.)
Some teachers suggested reducing the test and surveys
for Lagoon Quest program evaluation. Additionally,
they would like to have a paper version of the
evaluation instead of a computer version.
Reduce teachers’ paperwork regarding lagoon
quest activities (online surveys). Make sure
teachers and students receive appropriate materials
needed for this field trip in a timely manner.
The online test was awful. Please stick to the
hard copy. Computer access/dependability
can be a problem.
Besides program improvement, the teachers also
provided the subjects that they want to learn more about
in a teacher in-service training day. These topics are
listed in Table 18.
The focus group discussions also revealed
suggestions for program improvement. Because
these focus group participants were selected from
schools that did not submit many teacher surveys,
it is likely that their voices were not included in
the previous summary of themes from the survey.
Although we initially believed these teachers
might be less enthusiastic about Lagoon Quest,
some focus group comments were extremely
positive. Teachers said their students count the
Lagoon Quest study trip as one of the best things
of the entire semester. Teachers found the Indian
River Lagoon Activity Book easy for students to
use and informative and suggested the Lagoonie
Logbook be written in the same style. They also
suggested making the Lagoonie activities more
relevant to the study trip to enhance student
interest.
Not all teachers completed the program, however.
Some were stymied by the need to obtain a variety
of equipment or materials to conduct the science
experiments and would appreciate receiving kits
for the activities. Others were severely limited by
the lack of time to teach science and requested
more activities that integrate the Indian River
Lagoon into reading and writing exercises. Some
requested that the zoo staff demonstrate all the
experiments during the in-service training so the
teachers will be more confident to conduct these
experiments in class.
Table 18. Topics of Interest to Teachers
Topics
Activities
Teaching Method
4
How to fit in all of the pre-field trip activities before the actual field trip?
How to incorporate more inquiry methods into the experiments?
4 How to incorporate activities in the classroom to impact all students
4
Program
Iwould like to know the agenda for the day at the lagoon, so I can let my students know what to expect
How can this be related to FCAT
4 I would like to do some of the family activities such as making a plankton net and demonstrating storm water runoff
4 Water and program safety
4
4
Water
4
4
4
ater experiments
W
What does water quality indicate?
Water quality and its effect
Ecology
4
4
4
Identify species
Animals and plant life around the lagoon
Integrated Florida history and geography of IRL and animals identification into the field trip
Environmental Issues
4
4
4
ecycling
R
Development along the IRL
Temperature and weather in general, and as it relates to the health and activity in the lagoon
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 25
The program has a great affect on my
children.They learn about the health of
nature around them while having fun.
—Chaperone
Discussion
We believe that this program provides many
benefits for students, especially for those who
do not have previous experience with the
Lagoon. It is unrealistic to say that one single
program creates change in all students, but it may
bring some environmental awareness for many
students. For those students who want to participate
in additional activities, more information is needed.
Opportunities for additional activities and resources
would be useful for students, teachers, and families
who wish to learn about and experience the Indian
River Lagoon.
The results of the Lagoon Quest knowledge test
suggested that students significantly improved
their knowledge of estuarine ecology after
they participated in Lagoon Quest. This is
true for students in both Title I and non-Title
I schools. According to the student surveys,
students enjoyed Lagoon Quest. Some activities
successfully attracted students’ attention, for
example, the water activities and learning about
animals. Activities such as salinity and turbidity
were less popular, which might be because those
concepts were new to the students and might
have been difficult for them to understand.
Some chaperones mentioned that their children
expressed interest in returning to the Indian
River Lagoon.
According to teacher and chaperone surveys, there
are a few challenges for future implementation of
Lagoon Quest.
Photo by C. Fifer
n Challenges for Lagoon Quest
26
Teachers prefer to use Lagoon Quest during
warm weather and after FCAT. However, on the
current schedule, there are not enough weeks in
the school year to accommodate everyone after
FCAT. If it is possible to hire more Lagoon Quest
staff, the four trip sites could be filled at the same
time. All the trips could be finished earlier to
avoid cold weather, and the staff would have
time to make up the trips that are canceled due
to bad weather.
Teachers and chaperones suggested having sites
that can accommodate large groups of students
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
Photo by R. Dubin
and avoid encounters with other visitors. Also,
chaperones suggested having more staff to
instruct children and more equipment (dip nets)
for students to use during the experience. Are
other sites available? Could a site accommodate
two groups instead of one? If other sites are not
available, it would be helpful to recognize these
concerns and explain the limitations to teachers
and chaperones. Additionally, an analysis of
student knowledge by trip location suggested that
the locations with less biodiversity are just as
effective as other locations.
After the study trips, some chaperones asked
questions such as, “So what now?” “How can
people help the lagoon?” “What can kids do
at home?” The curriculum focused on estuary
ecology and science, but one of the program’s
long-term intentions was to support students’
conservation behavior. To increase conservation
behavior, science education could be connected
with conservation education. Some of this can
be done with discussion questions at the end
of the day. The zoo could also design a fact
sheet to provide tips for students, teachers,
and chaperones to help the lagoon and the
environment around them. The staff could
consider modifying the scavenger hunt to allow
more time for discussion of students’ findings
and reinforcement of conservation practices. An
additional follow-up activity in the Lagoonie
Guide might also be useful.
We recognize that more than half of the teachers
did not complete surveys, and that many
responding teachers did not complete the entire
unit. Encouraging greater participation will
take a concerted effort from the schools and the
zoo. Teacher in-service time, required all-grade
meetings, and follow-up support may be needed.
Additional research to better understand the needs
of nonrespondents would be helpful.
It appears that few families completed the
family activities in Lagoonie Logbook and took
advantage of their free admission to the Zoo. The
primary purpose of the family activities was to
encourage parents to learn about Indian River
Lagoon with their children. However, some
parents did not know about these activities because
the students did not bring the Lagoonie Logbook
home. Perhaps multiple reminders could be sent
to the fourth-grade teachers to encourage fourthgrade parents’ participation in Lagoonie family
activities.
This evaluation suggests that Lagoon Quest is an
effective and well-received program. It provides
a foundation for future study of an important
local ecosystem. By offering this program every
year, teachers will increase their knowledge
and improve their abilities to lead the in-class
activities. This report provides ideas for the inservice training and other strategies that zoo staff
might use to support fourth-grade teachers.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 27
Photo by C. Fifer
This is the study trip the kids always talk
about and remember. Kids still come back
to me and tell me that was the best trip
they have ever had. —Teacher
participants is most important, shorter forms
with fewer questions and more reminders to
complete them may prove successful.
Recommendations
for Evaluation
In reflection, we would do some things differently
and therefore recommend that future evaluation
efforts consider the following suggestions:
1. Data from the Web-based surveys were
much easier to analyze, but for young
students answering questions accurately
with a computer may be a barrier. Access
to computers may also be a barrier. Some
teachers requested paper versions of the
surveys and others duplicated forms from
the Teacher Guide. Until computers are more
accessible in all schools, and youngsters and
their teachers are more computer savvy, it may
be best to use paper versions of a survey.
2. There are several ways to imagine future
evaluation efforts, depending on what is
important. We did a census instead of sample
because the school system wanted to treat
students and teachers equally. However, we
were not able to obtain 100 percent response.
If evaluating the program is paramount, it
would be possible to randomly sample the
study population and work more diligently to
collect all of the forms from a fewer number
of participants. If program ownership by all
28
3. We requested students’ and teachers’
identification numbers for matching and
research purposes. Although we emphasized
confidentiality, some participants were more
comfortable providing their suggestions
without their identification numbers. If
program evaluation is the primary purpose,
emphasizing anonymity and the usefulness
of responses are recommended, but collecting
identification numbers will be essential to
track respondents over time.
4. The focus groups were added because of the
poor response rate, but they provided such
valuable insights that we recommend that
future evaluators consider ways to collect
in-depth data from a small sample of
teachers and youth.
5. Providing the answers to the knowledge
quiz enables teachers to score the quiz. It
also enables teachers to teach to the test
and undermines its validity. A different
posttest, if it reliably measures the same
concepts, may be considered. Correct
answers could be provided after tests
were submitted.
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
Photo by E. Moon
Lagoon Quest is an extremely beneficial
addition to the Brevard County
curriculum. Most teachers, chaperones,
and students who participated in the
evaluation were very pleased with the
program and committed to making it
even better in future years. Because the
program involves all fourth graders,
the long-term commitments from
Brevard Public Schools and Brevard
Zoo to this program promise to bring
significant opportunities for shaping the
development of conservation attitudes,
ethics, and career interests among the
entire population of students. There
are many possibilities for launching
supplementary units and activities that
reinforce the concepts that Lagoon
Quest introduces and for monitoring and
assessing changes over time to both the
residents and the lagoon. The potential
for long-term tracking of the entire
school-aged population suggests an
important opportunity for research and
evaluation.
The program requires a significant amount
of instruction, equipment, and time.
Responding teachers have demonstrated
that they can teach the concepts, though
there is variation in program success.
Providing the teachers who are less
comfortable with the program with
supportive material, relevant training,
and assistance will be challenging.
Fortunately, the thrill that fourth
graders feel when catching organisms
in the Indian River Lagoon should help
bolster and sustain teacher interest and
participation in Lagoon Quest.
Photo by D. Stewart
Conclusion
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t 29
Lagoon Quest gave me a greater understanding
of how special the Indian River Lagoon is and
how important it is to protect it. —Chaperone
Follow-Up
The development of this evaluation report has involved several meetings with zoo and school
staff and has resulted in their commitment to work toward the following types of changes:
1.Require that only new teachers attend the fall In-service Training.
2.Provide equipment and materials for in-class science experiments.
3.Provide a To Do List and Sunshine State Standard checklist in the Teacher Guide.
4.Include information about good environmental practices to help the Lagoon in the
Lagoonie Logbook.
5.Use the study trip to reinforce concepts introduced in the pre-trip lessons.
6.Simplify the Lagoonie Logbook.
7.Create a package for parents that includes information about Lagoon Quest family
activities, preparation for the study trip, Lagoon Quest family festival,
and environmentally friendly practices that can help the Lagoon.
8.Design the knowledge test to better reflect the information covered in the study trip
and pre- and post-trip activities.
30
M e a s u r i n g t h e e f f e c t i v e n e s s o f La g o o n Q u e s t
this document is cir 1528,
UF/IFAS school of forest resources and conservation .
original publication date: April 2008
visit the edis website:
http://edis.ifas.ufl.edu
The Institute of Food and Agricultural
Sciences (IFAS) is an Equal Opportunity
Institution authorized to provide research,
educational information and other services
only to individuals and institutions that
function with non-discrimination with
respect to race, creed, color, religion, age,
disability, sex, sexual orientation, marital
status, national origin, political opinions or
affiliations. For more information on
obtaining
other
UF/IFAS
Extension
publications, contact your county’s UF/IFAS
Extension
office.
U.S. Department of Agriculture, UF/IFAS
Extension Service, University of Florida,
IFAS, Florida A & M University Cooperative
Extension Program, and Boards of County
Commissioners Cooperating. Nick T. Place,
dean for UF/IFAS Extension.
Copyright Information
This document is copyrighted by the University
of Florida, Institute of Food and Agricultural
Sciences (UF/IFAS) for the people of the State
of Florida. UF/IFAS retains all rights under all
conventions, but permits free reproduction
by all agents and offices of the Cooperative
Extension Service and the people of the State
of Florida. Permission is granted to others
to use these materials in part or in full for
educational purposes, provided that full credit
is given to the UF/IFAS, citing the publication,
its source, and date of publication.