Assessment of Teacher Preparation:
Issues and Illustrations
2005-2008 Louisiana Value Added Assessment
George H. Noell, Ph.D.
Overview of today’s talk

Early work that lead to VAM initiative


Evolution of the early work


Data, funding, flaws, and relationships
Ramping up to a sustained effort


How connections & opportunities can emerge
Coordination, funding, flaws, and relationships
Some interesting findings
Some consideration of the
complexity of the journey

The scope of the effort in Louisiana

The complexity of the work


IT, models, IRB, software
Ramping up to a sustained effort

Coordination, funding, flaws, and relationships
Acknowledgements

Commissioner of Higher Education E. Joseph Savoie, Past
State Superintendent Cecil Picard, Current State
Superintendent Paul Pastorek, and Assoc. Commissioner
Jeanne Burns.

Louisiana Department of Education Division of Planning,
Analysis, and Information Resources (David Elder – Division
Director, Allen Schulenberg, Robert Kaufman, Kelvin
LaCroix, Steve Gunning, Sam Pernici, Bobby Franklin, and
Roth Aymond)

LSU Research Team (Veronica Gulley, Bethany Porter, Anna
Beth Ball, Maria Patt, Amanda Dahir)
Redesign in Louisiana
& the Blue Ribbon Commission
quality
Louisiana’s Teacher Preparation
Programs: Four Levels of Effectiveness
Level 4: Effectiveness of Growth in
Student Learning
(Value-Added Teacher Preparation
Program Assessment)
Level 3: Effectiveness of Impact
(Teacher Preparation Accountability System)
Level 2: Effectiveness of Implementation
(NCATE – Comprehensive Assessment System)
Level 1: Effectiveness of Planning
(Redesign of Teacher Preparation Programs)
How redesign lead to
Louisiana’s Assessment

The principle of meaningful assessment

Putting all of the key players in the same room

Creating a climate of striving and mission

Creativity and shared resources ($)
Challenges in Assessing
Teacher Preparation
 The challenge of measures
 Achievement versus opinions
 Geography
 Heterogeneous schools & classes
 Data management
 Technical issues
 The plausible counter factual (Rubin)
General Assessment
Approach
1. Establish empirical expectations
2. Measure Performance
3. Compare Expectations & Performance
4. Act on the Results
Detailed follow-up & policy issues
Moving to sustained work

Funding then and now

Establishing a small work team


Establishing the big statewide team as well
Working in public view and listening
Selected methodological
issues with VAM in Ed.

Meaning of the data you have
 Teacher experience variable
 Scaling of your tests
 Who “taught” the student versus tested
 Reliability of identifiers: students vs. teachers

Modeling issues
 “Just fit fixed models” because that is the right thing to do
 Just drop the Special Ed. kids, what a mess
 Of course you need cross year school fixed effects
Selected issues 2

How big is a BIG effect?


How important is 0.1 SD or 5 points
Poverty SLD and 3 to 12 points

What is a reasonable apportioning of the variance
 What would that mean

What do demographic effects “mean” is a massively
specified VAM?
 Learning disability and free lunch effects & change in
time
Selected Issues 3
Value added models, growth models, and
their union and disunion
When effects don’t make sense
Is it too complicated yet or already too
complicated
2004-2008 Study
Breakdown of Data

Years 2004-2005, 2005-2006, and 2006-07

Student Grade Levels: Grades 4-9

Content Areas: Mathematics; Science; Social Studies;
Language Arts; and Reading

Tests: Iowa Test of Basic Skills, i-LEAP, and LEAP-21

Pathways for New Teachers: Undergraduate and Alternate
Certification Programs (Master of Arts in Teaching, Practitioner
Teacher Program, and Non-Master’s/Certification-Only)
New & Experienced Teachers
 New Teachers:
 1st and 2nd year teachers
 Regular certificates;
 Completed TPP within 5 years
 Experienced Teachers
 3rd or subsequent year teacher with a regular certificate
 Teaching within area of certification.
Important Decisions
Defining New Teachers
(2007 study data)
ELA Teacher Years Experience Effect
2
1.5
1
Teacher Effect
0.5
0
-0.5
-1
-1.5
-2
-2.5
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Teacher Years Experience
Criteria for Inclusion of
New Teachers in Study

Inclusion for each content area





25 or more new teachers in grades 4-9
At least 10 new teachers per year
Teaching within certification
Remained with student full academic year
Not all universities had sufficient new teachers to
be included in the analysis. Another year of data
will be added and will increase the number that
exceed 25 graduates.
Size of the Data Base
for the Study Per Year

Districts: All school districts in Louisiana

Students:
approximately 250,000

Teachers:
7,000+

Schools:
1,300+
Hierarchical Linear Models:
Nesting and Interacting by Year
Teacher Preparation Effect
Estimates
355
Prior (2005)
Achievement
Current (2006)
Achievement
350
Gifted
African American
Free Lunch
10 days absent
Male
Post-Redesign Programs &
Pre-Redesign Programs

Post-Redesign Programs: Programs that
were redesigned for grades PK-3, 1-5, 4-8, and
6-12 and began admitting pre-service teachers
on July 1, 2003.

Pre-Redesign Programs: Universities
stopped admitting candidates to the programs
on July 1, 2003. A phase out period is
occurring for pre-redesign programs.
Effect Estimates for Post-Redesign
Alternate Certification Program
Mathematics
Performance Bands
Effect Estimates
Level 1: Programs more effective
than experienced teachers.
The New Teacher Project (+3.1)
Level 2: Programs comparable in
effectiveness to experienced
teachers.
University of Louisiana at Monroe (+1.1)
Northwestern State University (+0.8)
Level 3: Programs comparable in
effectiveness to new teachers.
Louisiana College (-2.7)
University of Louisiana at Lafayette (-2.9)
Louisiana Resource Center (-3.2)
(Note: Mean for new teachers = -2.7)
Level 4: Programs less effective
than new teachers.
Level 5: Programs significantly less
effective.
Effect Estimates for Post-Redesign
Alternate Certification Program
Language Arts
Performance Bands
Effect Estimates
Level 1: Programs more effective
than experienced teachers.
University of Louisiana at Monroe (+2.7)
The New Teacher Project (+1.6)
Level 2: Programs comparable in
effectiveness to experienced
teachers.
Louisiana College (+1.5)
Northwestern State University (+0.5)
Nicholls State University (-0.3)
Level 3: Programs comparable in
effectiveness to new teachers.
Louisiana Resource Center (-1.8)
(Note: Mean for new teachers = -1.8)
Level 4: Programs less effective
than new teachers.
Level 5: Programs significantly less
effective.
University of Louisiana at Lafayette (-4.6)
Effect Estimates for Post-Redesign
Alternate Certification Program
Reading
Performance Bands
Effect Estimates
Level 1: Programs more effective
than experienced teachers.
The New Teacher Project (+2.2)
Louisiana College (+2.1)
Level 2: Programs comparable in
effectiveness to experienced
teachers.
Northwestern State University (+0.6)
Level 3: Programs comparable in
effectiveness to new teachers.
University of Louisiana at Lafayette (-2.4)
(Note: Mean for new teachers = -1.8)
Level 4: Programs less effective
than new teachers.
Level 5: Programs significantly less
effective.
Louisiana Resource Center (-6.2)
ACT Scores of New Teachers

ACT scores of new teachers within
programs did not account for variance in
teacher preparation program
effectiveness.

ACT mathematical scores of individual
new teachers across programs was a
modest predictor of teacher effectiveness
in mathematics.
Impact of Teachers Who Are Not
Content Certified
Teachers who are
certified in the
content area they are
teaching are more
effective than those
who are not certified
to teach that content.
Content
Coefficient (CI)
Mathematics
-3.50
(-4.70, -2.28)
Reading
-1.27
(-1.72, -0.82)
Language Arts
-4.09
(-4.70, -2.28)
Science
-1.58
(-2.34, -0.82)
Social Studies
-3.32
(-4.61, -2.03)
Placing Results in Context:
Mathematics from 2007 study
4
2
0
-2
-4
-6
-8
-10
-12
-14
ar
Le
g
nin
Dis
.
r
s
C
ch
AC
he
roj
ce
n
c
e
n
nA
P
u
a
r
g
e
r
L
e
e
e
s
e
t
T
o ll
ch
Ab
es
w
ree
a
C
e
w
F
0
e
h
1
N
T
La
g.
ort
w
e
N
Av
N
ilit
ab
y
Ma
le
Summary Observations from
the Data
•
•
•
•
•
Some TPP prepare new teachers contribution to achievement
is similar to experienced teachers.
Effectiveness varies across and within TPP.
ACT scores of new Louisiana teachers do not account for
variance in teacher preparation program effectiveness.
ACT mathematics scores of individual teachers is a modest
predictor of teacher effectiveness in mathematics.
Certified teachers are more effective than teachers who are
not certified to teach the content.
Summary Observations from
the Process
 Sustaining system change and/or the long term
research agenda
 Data & Policy
 Relationships & funding
 Working in the public view
 Visibly responding to criticism
 Sustaining institutional support
 The many agendas in play
 Louisiana’s statewide research team