Planning for Successful Student
Learning Outcomes Assessment
for the
Voluntary System of Accountability
with
Presentation for Fitchburg State
January 13, 2009
© 2009 ACT, Inc.
All Rights Reserved.
1
Webinar Agenda
•
•
•
•
•
CAAP Overview
CAAP and the VSA
CAAP Usage Strategies
Plan for Success
CAAP Resources
2
Collegiate
Assessment of
Academic
Proficiency
3
The standardized assessment
program from ACT that enables
postsecondary institutions to
assess, evaluate, and enhance
the outcomes of their general
education programs.
4
Selected as one of the options
for measuring student learning
outcomes for the Voluntary
System of Accountability.
5
CAAP Benefits
Systematically measure your
students’ achievement levels
and growth in your general
education program
6
CAAP Benefits
Institution and individual student level
data to help you identify opportunities
to enhance student learning
7
CAAP Benefits
Help satisfy the need for greater
accountability in higher education
8
CAAP Benefits
Help meet accreditation
self-study requirements
9
CAAP Benefits
Help evaluate individual
student readiness for upper
division coursework
10
CAAP Benefits
Not to evaluate faculty.
Not as a sole evaluation in
high stakes environments.
11
CAAP Features
•
•
•
•
•
Modular format (6 content areas)
Flexible administration
Comprehensive reports
Extensive national user norms
Value-added measurement
through linkage to COMPASSTM
and the ACT® test
• Identification of action items with
Content Analysis Reports
12
CAAP Test Modules
• Writing Essay
• Writing Skills
• Mathematics
• Reading
• Critical Thinking
• Science
13
CAAP Administration
• Convenient in-class administration
• 40 minutes per test
• Pencil and paper
• Proctored
• Can be embedded in required
freshman or capstone course
• 2-week test window
• Nine local questions can be added (except
for VSA testing)
14
CAAP Standard
Reporting Package
• Institutional Summary Report
• Mean scores for each demographic and
subscores in math, reading, & writing skills
• Student Score Reports – 2 copies
• One for institution and one for student
• Certificate of Achievement
• For students scoring at or above the national
mean
• Student Roster Report
• Listing of each student with their CAAP
results
15
CAAP User Norms
to
16
CAAP User Norms
• Norms for each test module
(and subscore)
• Broken out by:
• Institution type
• Year of students tested
• Institution ownership
• Three-year rolling norms
• Custom groupings available
17
CAAP and the VSA
• Voluntary System of Accountability
• A joint initiative developed in partnership
between NASULGC and AASCU
• Designed to help institutions
• Demonstrate accountability and
stewardship to the public
• Measure educational outcomes to
identify effective educational practices
• Assemble information that is accessible,
understandable, and comparable
• Currently over 60 NASULGC/AASCU
institutions use CAAP for their outcomes
assessment
18
CAAP and the VSA
• VSA College Portrait
• Consumer Information
• Student Experiences and Perceptions
• Student Learning Outcomes
• VSA participants measure critical
thinking, analytical reasoning, and
written communication
• CAAP
• Measures critical thinking and written
communication using two test modules
• CRITICAL THINKING
• WRITING ESSAY
19
VSA College Portrait Pilot
http://www.voluntarysystem.org
http://www.collegeportraits.org
• VSA Web Site
•
•
•
•
College Portrait data elements
College Portrait prototype
VSA Background/development
VSA Enrollment
20
CAAP and the VSA
Student Learning Outcomes
Freshman to Senior Learning Gains
• Critical Thinking
• Writing Essay
• The increase in learning on the performance
task is “_____” what would be expected at an
institution with students of similar academic
abilities.
• “well above”, “above”, “below” or “well below”
what would be expected
21
College Portrait
Prototype
College Portrait
Learning
Outcomes Reporting
22
CAAP Writing Essay
• Two independent prompts
• 20 minutes each
• Essays scored at ACT by two trained
raters
• ACT scoring rubric 1-6
• http://www.act.org/caap/tests/holistic.html
23
CAAP Writing Essay
• Formulating an assertion about a given
issue
• Supporting that assertion with evidence
appropriate to the issue, position taken,
and a given audience
• Organizing and connecting major ideas
• Expressing those ideas in clear,
effective language
24
Writing Essay Sample
Your college administration is considering
whether or not there should be a physical
education requirement for undergraduates.
The administration has asked students for
their views on the issue and has announced
that its final decision will be based on how
such a requirement would affect the overall
educational mission of the college. Write a
letter to the administration arguing whether or
not there should be a physical education
requirement for undergraduates at your
college.
25
CAAP Critical Thinking
• 32 items/4 passages
• Sample questions on-line
• Clarifying, analyzing, evaluating,
and extending and argument
• Subject matter—Social Sciences,
Natural Sciences, Humanities
• Formats include:
• Case Studies
• Dialogues
• Debates
• Editorials
• Statistical Arguments
• Experimental Results
• Overlapping Position
26
CAAP and the VSA
• Sample Size
• ACT requires a minimum 200 freshman and
200 seniors tested on each module
• Test Dates
• Freshmen must be tested early in the fall
semester
• Seniors must be tested within 6 months of
graduation
• Both groups tested during the same
academic year (September-May)
• ACT (or SAT) Score Required
• Institutions must provide data file with CAAP
tested students’ ACT or SAT scores – see
CAAP VSA Guidelines for more details
27
CAAP and the VSA
• Random selection
• Institution determines what constitutes a
random representative sample
• Value-added VSA Model
• Cross-sectional
• Longitudinal Model
– Consider the minimum number of records
needed at the time of post-testing and then
calculate back to establish the number for
pre-test, and test 3-4 times that number
28
CAAP Scoring
• Return materials to ACT for scoring and
reporting
• Critical Thinking
• ACT proprietary scanning
• Writing Essay
• Two ACT-trained scorers
• Three-week turn-around time for
standard reporting package
29
CAAP VSA Report
• Provided upon completion of all freshmen and
senior testing – late summer
• Provided upon receipt of your institution data
file (with student matched ACT or SAT and
CAAP scores)
• Report will include:
• Average scores for both freshmen and seniors
on both test modules
• Student learning gains on each test module
• Average ACT (or SAT) for students tested
30
31
CAAP Administration
for VSA
• Administer at least once every 3
years
• Report within 4 years of becoming
VSA participant
• Updated at least every 3 years
32
Additional CAAP Test
Modules
• CAAP VSA Materials Order Form
provides option for additional modules
• Administer to a different group of
students than those tested on the VSA
modules
• Option for local questions for only
those who are not testing for VSA
33
Any questions so far?
34
Usage Strategies
35
CAAP Usage Strategies
• Outcomes Measurement
• Value-Added Measurement
• Student Readiness Evaluation
36
Outcomes Measurement
37
Outcomes Measurement
• Establish benchmarks
– (Scale scores, percentiles)
• Identify relative strengths and
weaknesses
– (Subscores, Content Analysis Report)
•
•
•
•
Evaluate issues
Propose solutions - set goals
Implement changes
Track results
38
Content Analysis Report
TEST MODULE
C.A.R. DOMAINS
Writing Skills
- Usage/Mechanics
- Rhetorical Skills






Punctuation
Grammar
Sentence Structure
Strategy
Organization
Style
Mathematics
- Basic Algebra
- College Algebra






Prealgebra
Elementary Algebra
Intermediate Algebra
Coordinate Geometry
College Algebra
Trigonometry
Reading
- Arts/Literature
- Social/Nat. Sciences


Reasoning Skills
Referring Skills
Science



Analysis
Generalization
Understanding
Critical Thinking



Analysis of Elements of Arguments
Evaluation of Arguments
Extension of Arguments
39
40
Content Analysis
Report (Math Sample)
41
42
Value-Added Measurement
43
Value-Added Measurement
63
62
62.7
62.4
62.4
61.9
61
60.7
Inst. X
60
59
Inst. Y
Nat. Mean
59.3
58
57
Fresh
Senior
44
Value-Added Measurement
• Longitudinal (CAAP to CAAP)
• Longitudinal (Linkage)
• ACT to CAAP
• COMPASS to CAAP
• VSA Model – Cross-Sectional –
Freshman to Senior Learning Gains
45
Value-Added Measurement
Longitudinal (CAAP to CAAP)
• Test cohort with CAAP upon program entry
• Test same cohort with CAAP upon
completion of program
• Change in scores is a direct measure of
“value-added” performance gain
• Evaluate results in terms of expectations
• Investigate opportunities for improvement
46
Value-Added Measurement
Longitudinal (ACT or COMPASS to CAAP)
• Use cohorts’ ACT or COMPASS scores as a
benchmark
• Test cohort with CAAP upon completion of general
education coursework
• Linkage report evaluates cohorts’ results in terms of
“expected gain”
• Evaluate results in terms of expectations
• Investigate opportunities for improvement
47
Value-Added Measurement
Cross-sectional - VSA Model
• Test a cohort of incoming freshmen at
the beginning of the fall term
• Test a similar group of seniors within 6
months of graduation
• Differences between the two group
mean scores may indicate performance
gain
• Challenge of this strategy is matching
student characteristics across groups
48
Student Evaluation
49
Student Evaluation
• Determine student skill expectations
• Establish appropriate cut scores
• Propose appropriate remediation
50
PLAN FOR SUCCESS
51
Plan for Success
#1 Confirm the fundamentals
• Review your institutional mission statement.
• Evaluate the objectives of your institution’s
general education program.
• Evaluate your institutional general education
core curriculum.
52
Plan for Success
#2 Specify usage of results
• Who will review and analyze the data?
• How will the results be used to influence
program evaluation and planning?
• Will results be tracked on a longitudinal
basis?
53
Plan for Success
#3 Define population
• Specify characteristics of the student body
that is the focus of the study.
• Specify characteristics of subgroups.
54
Plan for Success
#4 Sampling
• Test entire population or sample?
• How will sample be selected?
• How will subgroup sample(s) be selected?
• Is sample random?
• Is sample representative of student
population?
55
Plan for Success
# 5 Faculty motivation
• Specify plans to communicate with faculty.
• Specify plans to enlist faculty support.
• Specify plans to share results with faculty.
56
Plan for Success
#6 Student motivation
• Specify plans to communicate with students.
• Specify plans to motivate students.
• Specify plans to share results with students.
57
Plan for Success
#7 CAAP Test Administration
•
•
•
•
•
•
•
•
•
•
Submit CAAP-VSA Participation Agreement
Confirm test modules, if additional*
Confirm optional local questions*
Confirm reporting requirements
Confirm test period (two-week window)
Confirm test venues
Confirm students to be tested
Confirm proctors
Schedule training for proctors
Order test materials at least one month prior to
test dates
• *For non-VSA test modules only
58
CAAP Testing Costs
• Critical Thinking - $13.50/student
• Writing Essay - $13.50/student
– Includes Standard Reporting Package and CAAP
VSA Report
• Additional CAAP Modules
– Writing Essay - $13.50/student
– Single Objective Test - $13.50/student
• 501+ - $12.70/student
– Multiple Objective Tests - $ 13.50/student
• 501+ - $19.20/student
59
CAAP RESOURCES
60
CAAP Planning Workbook available online at
http://www.act.org/caap/pdf/planguide.pdf
Portions of Guide are available at
http://www.act.org/caap/materials.html
61
Portions of this can be found at
http://www.act.org/caap/techhandbook
62
Norms available on-line at
http://www.act.org/caap/norms/
63
Available on-line at
http://www.act.org/caap/pdf/userguide.pdf
64
http://www.act.org/caap
http://www.act.org/caap/vsa
65
Recap
•
•
•
•
•
CAAP Overview
CAAP and the VSA
CAAP Usage Strategies
Plan for Success
CAAP Resources
66
ACT Contact Information
ACT Postsecondary Assessment Services
500 ACT Drive
P.O. Box 168
Iowa City, Iowa 52243-0168
Phone: 800-294-7027
Sandra Stewart
[email protected]
Direct # 319-337-1051
FAX: 319/337-1790
E-mail: [email protected]
67