2012-13 Montana GEAR UP
Evaluation Resutls
January 15, 2013
Montana GEAR UP
1
Research Question 1: What services do GEAR
UP schools implement with program funds?
1. Montana GEAR UP offers a full range of services that address all
three GEAR UP goals. The average hours for students receiving
GEAR UP services ranged from two hours (financial aid counseling)
to over 80 hours (rigorous academic curriculum).
2. There is a core group of five key services on which schools spend
the majority of funds, spend the majority of time implementing, and
reach the largest numbers of students. These services are:
• College Visits
• Tutoring and homework assistance
• Academic enrichment activities or STEM activities
• Summer programs
• Counseling, advising and academic planning
And what is the level of
implementation of those services?
1. There is wide range in the quality of implementation of GEAR UP
Services, from fair to very good. Overall, based on student and
parent survey ratings and evaluator site visits, data suggest that most
services had an average level of implementation.
2. The service that appears to have the overall strongest
implementation is college visits, which is best described as having an
above average implementation.
Research Question 2. To what extent do GEAR
UP schools implement college & career
readiness best practices?
1. Based on the educator survey, GEAR UP schools appear to have a fairly
high level of college and career readiness best practices in all three practice
areas of academic preparation, college aspirations and expectations, and
(for high schools) college knowledge and taking action. On a seven point
scale, most items and subscales had average ratings in the 5.0 to 6.0
range…somewhat agree to agree.
2. But there are some areas, where there is room for growth in the
implementation. For example, many schools struggle with finding role
models and mentors to help GEAR UP students, particularly our American
Indian students, navigate the road to college access and success.
3. Together, the data suggest that the real value of the educator survey
results is for state program leaders and individual school leaders to look at
the relative results, targeting those practice areas and individual practices for
improvement that are rated lower than others.
Research Question 3. Do GEAR UP activities achieve
the following program goals and the Montana state
performance indicators under each goal?
• Montana GEAR UP established three goals and a total of 21
indicators to measure the success of the program. Currently
data are available 11 of these measures to assess progress
from one year to the next.
• The available results are mixed at this early stage in program
implementation. For every annual benchmark that Montana
GEAR UP met for an indicator, Montana GEAR UP did not meet
the annual benchmark for a similar indicator.
• On the positive side, Montana GEAR UP has already achieved
the final benchmarks for two indicators regarding the percent of
students who take the PLAN and EXPLORE.
School Indicator Flags: Examines selected
state performance indicators and their
relationship
• MTGU & PRA developed a few School Indicator Flags to more
accurately gauge student performance by identifying consistencies
and discrepancies of student performance on different indicators.
• Based on a series of School Indicator Flags, Montana GEAR UP
students seem to perform better on indicators that measure local
standards, such as course completion, than on indicators that
measure state and national standards, such as performance on state
CRT and national ACT testing.
• For example, Montana GEAR UP students increasingly are taking
courses that align with the Montana University System college
preparatory curriculum, but they are also outperformed on state and
national assessments, such as the ACT, that can point to college
readiness and success.
Research Question 4. Are GEAR UP services related to
achievement of program goals and outcomes for all
students and student subgroups, particularly Native
American students?
• The evaluation examined the relationships between the amount of time
that students and parents participated in GEAR UP services with
predicted student performance on short-term outcomes.
• The results found positive associations between the set of Goal 1
services—such as academic tutoring and homework assistance,
summer credit recovery, and academic enrichment activities—with
Goal 1 outcomes—student completion of pre-Algebra in grade 8 and of
Algebra I in grade 9.
• The evaluation also found other positive associations between
increased hours of participation in Goal 2 services—
counseling/advising/academic planning, college visits, and student
workshops—and Goal 2 outcomes, such as being on track to meet
MUS college preparation requirements.
Research Question 4 (Cont)
• The evaluation found fewer relationships between Goal 3
associated services—such as financial aid advising, parent
workshops on college preparation, and family events—with
Goal 3 associated outcomes, such as expectations about
student enrollment in college, knowledge about financial aid,
and FAFSA completion.
• The key finding for Goal 3 was the number of hours that parents
spent in Goal 3 services was positively related to FAFSA
completion.
• For the most part, all of the positive results across the three
goals held equally true regardless of student gender or ethnicity.
However, the positive results were not consistent across all
grade levels measured.