Session 2:
Understanding Program Theory
and Implementation Theory
Objectives for Today
Formative v. Summative Evaluations
Program Theory v. Implementation Theory
Example: STOP-DWI
Stages of Formative Evaluation
Assign Groups for Focus Group/Survey
Project
Formative Evaluation
What is being implemented?
– Program description
– Stages? Steps? Services? Procedures?
Are the necessary steps in service delivery
being adequately coordinated and
executed?
– Consistent with expectations or guidelines?
– How do clients experience the program?
Summative Evaluation
How well did the program achieve its
intended impact?
Are there any unintended or unforeseen
consequences?
Identify changes in conditions, behavior, or
status that are directly related to the
program
Formative or Summative?
Number of visits to
food pantry
Satisfaction with job
referral system
Lower rates of heart
disease
Response time of
operators to hotline
callers
Cleaner drinking
water
Citizen involvement in
decision-making
Sustained
employment for 6
months
Lower infant mortality
rates
Program v. Implementation Theory
Program Theory = mechanisms of change
– “specifies a chain of causal assumptions linking
program resources, activities, intermediate outcomes,
and ultimate goals”
– “the mechanisms that mediate between service
delivery (and receipt) of the program and the
emergence of the outcomes of interest”
Implementation Theory = program activities
– Translation of program objectives into services and an
operating program
Enable Assistive Technology Flow Chart (PROPOSED)
New/Intake
Needs Evaluation
Information and
Referral form
Helped By Assistive Technology
Need equipment/services from Assistive Technology
Did you get
service?
Referral
Yes
 Pre-Assessment
 Training
Did you get
equipment?
No
No
Why?
Yes
 Equipment loan
 Equipment
purchase
Post-Assessment
Outcomes:
Carecall
STOP-DWI in NY State
In 1981, NY State passed STOP-DWI law to
reduce drunken-driving and alcohol-related
accidents.
What is the implementation theory?
– What activities or services comprise the STOP-DWI
program?
What is the program theory?
– How will the program affect behavioral change?
How would we know if it was being implemented
well?
What might affect successful implementation of
STOP-DWI programs?
Outcomes of STOP-DWI in NY
Revenues from DWI fines:
– 1982 fines = $156,313; 2000 fines = $925,666
36 alcohol-related fatalities in 1981, 2
alcohol-related fatalities in 2000 (–94%)
-67 fatalities total in 1981, 46 in 2000 (-31%)
DWI arrests in Onondaga County:
– 1980 = 1,069; 1990 = 2,191; 2000 = 1,461
Is STOP-DWI a success?
Outcomes of STOP-DWI in NY
How do we know if the program is having
an impact?
Program v. non-program effects
What factors affect the number of
accident-related fatalities?
What factors other than the program might
affect drunk-driving rates?
Stages in Developing an Evaluation
Stage One –Defining the Evaluation
– Program goals, implementation theory, type of
evaluation, stakeholders
Stage Two - Develop Analysis Strategy
– Tracking implementation and outcomes
– Proper data for the question(s) of interest
Stage Three - Implementation of Data Collection
Stage Four - Analysis, Report Writing, and
Presentation
Enable Assistive Technology Flow Chart (PROPOSED)
New/Intake
Needs Evaluation
Information and
Referral form
Helped By Assistive Technology
Need equipment/services from Assistive Technology
Did you get
service?
Referral
Yes
 Pre-Assessment
 Training
Did you get
equipment?
No
No
Why?
Yes
 Equipment loan
 Equipment
purchase
Post-Assessment
Outcomes:
Carecall
Readings for Next Time:
Trochim, Chapters 1 – 3
Also, in-Class Data Activity
next Monday