External
Evaluation
Report Year 1
“FAST FORWARD” GRANT
San Juan College
Godfrey Heritage, LLC
Laura M. Godfrey, Evaluator
John Tohtsoni Jr.
3/18/2013
External Evaluator’s Report
San Juan College, “Fast Forward” Grant
PR Award# P382C110007
Overview
During the first year of its Title III NANTSI grant (Accelerating AA/AS Degree and Certificate
Completion for Native American Students aka “Fast Forward”) San Juan College has successfully
implemented the program as outlined in the grant, hired key staff, and made progress towards
meeting the objectives delineated in the grant. All of the key personnel plus the External
Evaluator have met frequently to refine the implementation and evaluation of the program.
Many aspects of the grant are in compliance with the timelines established in the proposal and
adjustments have been made to meet timelines during Year 2. The grant is “audit-ready” with
full documentation available and easily accessible. The team has established comprehensive
evaluation and tracking systems for all aspects of the project. Responsibilities are clearly
delineated for collecting and reporting both quantitative and qualitative data.
Adequacy of Data
The External Evaluator has reviewed all data sources and data collected. Data is provided by
the Institutional Research Department, the Grants Compliance Office, the Project Director/Staff
and the External Evaluator. The data reported is reliable and accurate. Data sources include:
• Datatel SIS collected by the San Juan College Institutional Research Department
• Audit Review Records collected by project director and reviewed by Grants Compliance
Office and External Evaluator
• Grant Staff collects and reports quantitative data such as number of participants,
number of reports, number of web pages, etc.
In addition to collecting data that documents progress towards achieving program objectives,
the College funds a full time Grants Compliance Officer who is responsible for “regulatory
compliance of all college grants”. The College requires each grant program to compile a record
keeping system for 16 items on the “Audit Review Checklist” (attached). Items include:
approval documents, budget files, procedures manuals, inventories of equipment, travel
reports, budget ledgers, project participants and all financial, performance and annual reports.
The checklist for the Fast Forward program has been reviewed by the Project Director and
Grants Compliance Officer and a system established for record keeping. The Evaluator has
confirmed that Fast Forward program is in compliance with SJC policies, procedures and
financial documentation requirements and is “audit-ready”. Each year there is an annual review
of audit documents by the External Evaluator in conjunction with the Grants Compliance Office.
Timely Reporting
The College has submitted thorough, accurate reports to the granting agency in a timely
manner. The Annual Performance Report (APR) submitted by the grantee in January 2013
contains complete results for the goals and objectives delineated in the grant. Results show
that the grantee has systematically collected quantitative and qualitative data demonstrating
substantial progress towards meeting all of the performance measures. In the few instances
where benchmarks could not be met a thorough analysis has been completed and adjustments
to targets or programs made accordingly.
Relationship with ED Program Grants Officer
The project staff has established a positive working relationship with the Department of
Education Program Officer. During the first year of grant implementation, the College realized
the need to change several timelines for achieving benchmarks and to hire an additional staff
person to work directly with students. The Fast Forward staff worked closely with the Program
Officer, thoroughly justified the proposed changes and obtained permission for the changes.
External Evaluator’s Report
The report that follows is a synthesis of interim and annual reports submitted by the grantee,
data collected by the Evaluator and analysis of documentation that demonstrates compliance
with College and Federal guidelines for grant activities.
Critical Questions for Evaluation
The following is a synopsis of the extent to which the College has addressed the following:
1. To what extent has the grantee achieved the Goals and Objectives set forth in the
proposal within the timelines specified?
2. To what extent has the grantee addressed challenges to achieving objectives and taken
steps to overcome those challenges?
3. To what extent has the grantee delivered high quality products and programs?
4. To what extent has the grantee used data to improve programs?
1. Critical Question One: Achieving Goals and Objectives within timelines specified and
2. Critical Question Two: Addressing challenges when Goals and Objectives were not
met
Goal 1. Improve degree completion for Native American AA/AS Students
Performance Objectives:
Performance Measure Objectives
1.a. Increase the number of Native American students who
graduate with AA/AS degrees or transfer certificates to
1.b. Increase the percentage of first-time full-time AA/AS degree
seeking Native American students who complete an AA or AS within
three years to
1.c Increase the percentage of first-time full-time AA/AS degree
seeking Native American students who complete an any SJC degree
or certificate within three years to
1.d Improve the ration of AA/AS enrolled students per each AA/AS
graduate to
Target year 1
18
Actual year 1
13*
4.6%
4.5%*
6.5%
9.9%
15
22.5
Challenges and Changes:
*The grantee recognized the crucial role direct student contact plays in student success.
Therefore in consultation with the Program Officer the grantee added a “Transfer Specialist”
position. In year 2 and beyond the College will be able to work directly with students to
identify AA/AS Native American students and guide them towards graduation and completion.
This important change should result in the grantee achieving the grant objectives in Year 2.
Goal 2: Improve transfer rates for Native American students
Performance Measure Objective
2.a. Increase the percentage of first-time full-time degree-seeking
Native American students who successfully transfer to a four year
college by
Target year 1
13.5%
Actual year 1
7.2*
Challenges and Changes:
*The same challenges and changes discussed for Objectives 1a and 1b above apply to this goal
of the grant. Again, the personal contact with students that the Transfer Specialist will have
ensures the grantee will achieve this goal during year 2 and beyond.
Goal 3: Improve course success rates in key AA/AS disciplines
Performance Measure Objective
3.a Improve course success rates for all students in English,
Business Administration, Mathematics and Chemistry
Target year 1
65.5%
Actual year 1
63*
Challenges and Changes:
*Although the grantee was statistically very close to meeting this objective, the College has
taken steps to ensure improvement in this area. The grantee is developing an Annual Data
Summary Report which will be shared with the Academic schools of the College. This report
can be used by instructors to inform their instructional delivery methods and thereby increase
student success in their courses.
Goal 4. Expand capacity for San Juan College to better serve Native American AA/AS students
Performance Measure Objectives
4.a Increase the percentage of General Education classes offered
outside the M-Th 8 a.m. – 4 p.m. traditional format to
4.b Increase the number of full-time and adjunct faculty who
participate in grant-sponsored professional development each year
4.c Increase the number of students who participate in grantsponsored assessment of technology skills each year to
4.d Increase the number of students who participate in grantsponsored remediation of technology skills each year to
4.e Increase the number of distinct web pages enhanced each year
by this grant to
Target year 1
42%
Actual year 1
34.7*
8
500
19 (38 including
staff)
0**
25
0**
20
20
Challenges and Changes:
*To increase the number of classes offered outside the traditional format the grantee is
developing a Community Survey to be conducted in Year 2. Results of this survey will be used
to launch a pilot program of new course selection for alternative times which should ensure
more courses offered outside the traditional format.
** During Year one the grantee planned to purchase a technology readiness assessment and
remediation program. However, it has been very difficult to find a program or system that will
interface with the College’s Datatel Colleague Student Information System and include a
remediation module that is fully automated and integrated with the assessment component.
The Online Grassroots Committee is charged with approving technology for the College. During
Year one the Committee was focused on distance delivery for accreditation purposes. During
Year two, the focus of the Committee will shift to identifying and purchasing a technology
assessment and remediation program.
Goal 5. Improve the ability of SJC personnel to collect, analyze and utilize student data to
better serve Native American students
Performance Measure Objectives
5.a Administer retention assessments to the following number of
students each year
5.b Through the grant, survey the following number of Native
American students regarding satisfaction and needs each year
5.c Based on surveys of SJC personnel the following percentage of
survey respondents will indicate that they are using grantsponsored data in strategic planning each year
5.d Build the following number of customizable reports
Target year 1
350
Actual year 1
0*
500
101**
50%
0***
20
109
Challenges and Changes
*A retention assessment was identified during Year one. Unfortunately the College
experienced a significant number of leadership changes in top Administration which slowed the
purchasing process for a retention assessment. The College has since revisited its institutional
priorities and is on track to purchase and implement a retention assessment in Year two of the
grant.
**The grant staff has determined that the survey needs to be more formalized and
standardized in order to have a higher number of students participate. The formal,
standardized survey will be administered in Year two and beyond.
***The intention of the grant staff was to administer this personnel survey once the institution
had received the “Native American Student Profile” which would guide the questions.
Unfortunately the College did not receive the Student Profile until September, at the end of
grant year one and not in time for this report. Staff is currently working on the personnel
survey which will be administered during Year two.
3. Critical Question: Quality of Products and Programs
During its first year the grantee has consistently offered high quality products and programs
that cut across all of the objectives listed in the grant. A few examples are:
Building partnerships with Native American Serving Four-Year Institutions
•
Collaboration meeting were held between San Juan College and the University of New
Mexico, Fort Lewis College, and Northern Arizona University. In these meetings student
preparation, transfer articulations, transfer processes and support services, and creating
a transfer network were discussed with greater understanding between partners.
•
•
Through grant activities, Native American students from SJC visited four-year
institutions including the university of New Mexico in March and July and Fort Lewis
College in May, July and September. A total of 29 students participated in these visits.
The grant staff organized and convened the Fast Forward Native American Advisory
Council to assist with building partnerships.
Improving the Transfer Process
•
•
•
•
The grantee has created a formal advisement support system to aid students in their
transfer process through a central Transfer Center on campus
Transfer information and opportunities have been created for students via workshops, a
hallway bulletin board, flyers, electronic messaging and posters
Grant staff participated in, and shared information at, Fall Student Rush, Student
Resource Day and at Fall Convocation. The grant also sponsored “Wear Your College
Colors Day.
The Fast Forward project participated and supported Native American Center events
and activities such as the NAC Graduation Reception and the Scholarship and Transfer
Fair
Strengthening Professional Development for Faculty
•
•
The Bisti Writing Project, provided through grant funds, provided two professional
learning community (PLC) summer workshops for 16 SJC faculty, including a five day
workshop in June. These writing workshops provided professional development
learning communities for both faculty and adjunct instructors who teach a significant
percentage of first-year courses.
In conjunction with the Advising and Counseling Department the Fast Forward program
conducted an Advisor Summer Workshop in July which focused on catalog changes, a
review of Native American student data and transfer data. This group provided valuable
input on the preliminary Transfer Website.
Enhancing Interactive Electronic Student Resources
•
•
•
The staff completed design and development of the new Transfer Center Website
The staff created a SJC YouTube channel explaining the student transfer process
The staff enhanced web pages for two Associates of Science degree programs and
increased the distinct number of web pages to 20 during Year one.
Collecting Extensive Quantitative and Qualitative Data to Improve Native American Student
Success Through Data-Based Decision Making
•
•
•
•
•
Trend reports were built for the School of Sciences and the School of Humanities.
Comprehensive trend data was collected on Native American students at SJC
A quote and contract was solicited from Ellucian Services for the Fast Forward Program
to complete the customized reports using Proclarity software
The Native American Student Profile, which shows trends in headcounts and
enrollments over the past five years, was completed in September and presented to
College faculty and professional staff. The Profile has now been made public on the
College’s website.
The staff prepared a total of 109 customizable reports, far exceeding the Year one target
of 20
Providing Cultural Enrichment
•
•
•
Staff assisted with the SJC Pow-Wow
Participated in the Title IIIA “Launch Program” Grant’s Cultural Immersion Program
Provided culturally relevant student workshops on the transfer process and college
success.
4. Critical Question: Use of Data to Improve Programs
The grantee has systematically used data to improve its programs. Critical Questions 1 dealt
extensively with how well the College has achieved the Performance Objectives as stated in the
Grant. Critical Question 2 dealt with the Challenges the grantee faced in meeting some of the
Objectives during year one and more importantly the Changes the grantee is making in order to
achieve those Objectives in future years. In all cases the grant staff has systematically analyzed
what has worked and what has not. Changes have been made based on this analysis of data,
and experience. The grantee is on track to improve its performance in all area and achieve the
stated Objectives during Year two.
For example, the grantee realized that they could not improve the percentage of Native
American students who either transfer to a four-year college or graduate with a degree or
certificate to the level stated in the grant. They analyzed the information and decided that the
program was lacking a direct student service component, in other words a person to identify
the eligible students and guide them towards graduation, completion or transfer. As a result a
request was made to add a Transfer Specialist using grant funds. The Transfer Specialist is
having great success advising students and should greatly impact the success of the College in
helping more Native American students graduate and/or transfer.
Evaluator’s Conclusions
The Fast Forward Grant faced numerous challenges during its first year including delays in
hiring key project staff, changes in key administrative leadership positions and delays in
purchase of programs to conduct technology assessments and remediation. Nevertheless, the
staff has done a commendable job of implementing the grant and has achieved many of the
Objectives stated in the proposal. The staff has learned a great deal during the first year and
has taken data-based steps to overcome the challenges they have faced; all of the Objectives
should be met during Year two.
The grantee has faithfully carried out the Fast Forward Grant in a professional manner and has
adhered to activities and programs proposed in the grant narrative.
The External Evaluator finds that:
.
• The grantee has met or exceeded benchmarks for many of the performance objectives.
In the instances when the targets for year 1 were too ambitious the grantee negotiated
a change in schedule with the program officer and is on track to meet or exceed all
targets in year 2.
• The grantee determined that direct student support was needed in order to achieve
degree completion and transfer objectives. The grantee then requested, and was
granted, a program change to allow for the creation of a Transfer Specialist position.
• The grantee has followed federal guidelines and college compliance requirements for
expenditures, record keeping and is “audit-ready” in all areas of grant compliance.
• The grantee has delivered high quality services and reporting that will accelerate degree
completion and transfer.
• The grantee has improved transfer support and relationships between San Juan College
and regional four-year Colleges and Universities
• The grantee has increased data collection and analysis of Native American students
which should aid in more data-based decisions which will aid in student success
• The grantee has developed culturally based professional development opportunities for
faculty and staff
• The grantee has improved and expanded electronic resources for students.
• The staff is professional, enthusiastic and dedicated to achieving greater success for
Native American students. Under the leadership of the Senior Director of Student
Engagement, the grant is well run, well organized and well positioned to fulfill its goals.
• The staff coordinates and cooperates with other stakeholders on campus. In particular
the relationship with the Native American Center, the Launch Grant and the Academic
Schools are noteworthy.
Next Steps in Evaluation
The Evaluator recommends the following and is ready to assist in their design and
implementation:
•
•
Systematic qualitative evaluations of programs and workshops including College Visits,
and Faculty Development Workshops.
Follow up surveys to determine the perceived effectiveness and impact of web pages,
customizable reports and other deliverables to faculty and staff.