T4T: BAA UNIVERSITY, DAY 4
EVALUATION &
TRANSFER OF LEARNING
L E S L I E W. Z E I T L E R , L C S W
B AY A R E A A C A D E M Y / C S U F R E S N O
O A K L A N D | M AY 1 8 , 2 0 1 7
INSPIRED BY THE PUBLIC CHILD WELFARE TRAINING
A C A D E M Y ’ S M A R C H 2 0 1 6 T 4 T O N E VA L U AT I O N & T O L
INTRO ACTIVITY:
STARTING WITH NUMBERS…
Let’s do this together…
2
AGENDA
1. Welcome & Intros
2. Evaluation: Purpose, Definitions, Framework
3. Evaluation-Informed Curriculum Design
4. Current Training Evaluations (BAA & Statewide)
5. Teaching to Skill
6. Pulling it all Together
7. Closing
3
GROUP AGREEMENTS
Revisit from yesterday…do they still work?
Or do we need new ones?
If so, what are they (briefly)?
4
HOPES (& PARKING LOT?)
What do you hope to get out of this session?
5
EXPERIENCE WITH TRAINING
EVALUATION? RAISE YOUR HANDS IF…
1. Who has written curricula for a class with a
knowledge or skill evaluation? What were some
lessons learned?
2. Who has taught a class with a knowledge or skill
evaluation? What were some lessons learned?
6
WHAT’S THE WIIFM?
What’s in it for me?
7
LEARNING OBJECTIVES
Review today’s learning objectives…
Which learning objective stands out to you?
Why?
8
• Purpose
• Framework
• Definitions
9
PURPOSE(S) OF EVALUATION
What are some purposes for evaluation?
10
FRAMEWORK FOR EVALUATION
There are multiple frameworks for evaluation. For
today’s purposes, we will use American Humane
Association’s (1999) Levels of Training Evaluation.
Let’s take a look at that framework now -
11
EVALUATION DEFINITIONS –
GAME!
1. Clear your tables (no trainee content, no phones, etc.)!
2. Wait until all tables have a set of cards. Leave them face
down until further instructions.
3. When trainer says “go”, turn your set of cards face up
and match evaluation terms (in bold) with their
respective definitions (in italics).
4. The first table to correctly match all of the terms with
their definitions wins a prize!
12
13
• Knowledge
• Skill
• TOL
14
TEACHING TO KNOWLEDGE
We want…
1. Clear and measurable LOs
2. Lesson Plan and Trainer Guide that identifies
where the LOs will be covered
3. Range of activities that reinforce the knowledge,
like…
15
RANGE OF ACTIVITIES TO
REINFORCE KNOWLEDGE:
 Demonstrations*
 Case Studies
 Guided teaching
 Group inquiry
 Information search
 Study group
(From Silberman, 2006)
16
TEACHING TO SKILL
We want…
1. Clear and measurable LOs (ideal: 1 skill LO/day)
2. Lesson Plan and Trainer Guide both identify where the
LOs will be covered
3. Include explicit instructions in curriculum for trainers
re: 4 steps of training to skill
4. Allow time in training day for the 4 steps of training to
skill for each skill LO
5. Range of activities that reinforce the skill, like…
17
RANGE OF ACTIVITIES TO
REINFORCE SKILL:
 Demonstrations*
 Role plays
 Simulations
 Writing tasks
 Action learning
(From Silberman, 2006)
18
TEACHING TO SKILL: 4 STEPS
INCLUDE these steps in Trainer’s
Guide/curriculum AND training day (e.g., THIS is
the ‘Holy Grail’…):
1. Trainer explains the skill
2. Trainer demonstrates the skill
3. Trainees get adequate time to practice the skill
4. Trainer provides feedback to trainees on their
skill practice.
19
TRANSFER OF LEARNING (TOL)
What can we as trainers do to support TOL?
Let’s take a look at some highlights from the TOL
Guide (PCWTA)…
20
PCWTA TOL GUIDE: HIGHLIGHTS
1. Clear course goals and objectives.
2. Connect new knowledge and skills to future use.
3. Use authentic learning tasks similar to job
requirements…
4. Use activities and questions that tap into trainees’
existing knowledge and skills…
5. Help trainees develop job aids/checklists…
and more in TOL guide…
21
PRACTICE SCENARIO, INSTRUCTIONS
PART 1:
Each table group:
1. Select a recorder to write down answers (and
would then report out during debrief portion).
2. Get one piece of chart paper + marker to write their
answers
3. Gets 1 LO per table to work on for this activity. At
table groups, ask trainees to work together to
answer the following questions… (see next slide)
22
PRACTICE SCENARIO, INSTRUCTIONS
PART 2
At your table groups, work together to answer the following
questions:
K1 and K2 groups: What type of activity would you use to
train to knowledge for this content area? (Incl. basic
outline)
S1 and S2 groups: What type of activity would you use to
train to skill for this content area? (Incl. basic outline)
S3 group: Describe at least 2 TOL strategies you would
include with this curriculum. (Incl. basic outline)
23
HOW TRAINING EVALUATORS
CAN HELP YOU…
1. Participate in LO development
2. Participate in curricula review
3. Design framework for evaluation
4. Design knowledge and skill evals (incl rubrics)
5. Assess pilots, assess fidelity of training
6. Obtain feedback from trainees, trainers, & others
7. Use data to improve training
8. And more!
24
•
BAA
•
Statewide
25
26
•
Headlines
•
5 “Test” Questions
27
TEACHING TO SKILL
28
TRY A NEW SKILL:
STEP 1. TRAINER EXPLANATION
Origami (art of folding paper)
At table groups, there is a folder with 4 roles in it (Trainer,
Curric Writer, Evaluator, Trainees) and a set of instructions
 Trainer & CW draft LO for learning the new skill
 Trainer & CW obtain feedback from training evaluators on
whether the LO is SMART
 Trainer will demonstrate the origami
 Trainees will practice
 Trainer will provide feedback to trainees
 Training Evaluator will evaluate!
29
STEP 2(A). TRAINER DEMO (LOS)
Writing LOs (By the end of this session…):
1. Trainees will be able to demonstrate at least 3
basic folds central to making origami.
2. Trainees will be able to make at least 1 piece of
origami that is a container or looks like an animal.
* Feedback from training evaluators (and others): Are
these Specific? Measurable? Achievable? Realistic? Timelimited?
30
STEP 2(B). TRAINER DEMO - USE
PAPER AT TABLES
A. Demonstration: I’ll show you and you try it!
•Mountain fold
•Valley fold
•Kite fold
B. Make Swan Origami:
•Let’s experiment, Part 1 (written instructions)
Debrief: What worked? What could be
different?
•Let’s experiment, Part 2 (I’ll show you & you try it!)
31
STEP 3(A). TRAINEE PRACTICE
(THAT’S YOU!)
Making more origami…at each of your table groups! Find…
1. One set of ‘role’ cards: Choose your role for skill practice
– Trainer, Curriculum Writer, Evaluator, or
Observer/Trainee)
2. One set of instructions: Read and follow instructions .
Note: Observers/Trainee Roles: As Observers, sit in a minifishbowl/half circle around Curriculum Writer, Trainer, etc, as you
watch the process unfold. When you transition to being ‘trainees’,
take your place at the table for ease of crafting origami.
32
STEP 3(B). TRAINEE PRACTICE (INSTRUC.)
1.
2.
3.
4.
5.
Each group develops an LO around learning to make an
origami container. (5 min)
Curriculum Writer: then writes brief instructions to teach
making an origami container (5-7 min)
Trainer: using instructions, 1) practices making an origami
container and 2) then explains, demonstrates making origami
container to trainees in your group. Trainees practice, and
trainer provides feedback to trainees. (10 min total)
Evaluator: Reviews LO(s), trainer’s teaching, implements
evaluation, and then asks trainees what worked well/what
could be different (5 min)
Observer/Trainees: Observe the curriculum
writer/trainer/evaluator process. Learn to make an origami
container.
33
STEP 4. TRAINER FEEDBACK TO
TRAINEES
Did that happen?
34
DEBRIEF, PART 1
1. By
group/pod (group 1, 2, etc.):
 What worked well in your process of designing your
own training with the roles involved?
 What could have been different/worked better?
2. By role (Curric writers, Trainers, Evaluators &
Observers/Trainees)
 How was your given role useful?
 How was your given role frustrating?
 What was critical about each of the roles in designing
training – esp. to skill?
35
DEBRIEF, PART 2 (REFLECTION)
1.
How does teaching making origami relate to
teaching child welfare workers new skills?
2.
Reflect on your current understanding of this
topic, and respond to each of these sentence
stems:
a. I used to think…
b. Now I think…
3.
Based on this session, what will you DO now
that you didn’t do before?
36
37
FOR BAA UNIVERSITY
DAYS 1-4
38
INDIVIDUAL REFLECTION & THEN SHARE
1. Which TOL techniques have we used today?
2. (From BAA Univ Days 1-4 LOs): Which TOL
techniques could they use in their training
practice?
3. Work on your own TOL Action Plan (in trainee
content; ID 2 strategies)
39
SMALL GROUP ACTIVITY
Count out from 1-4 around the room. Then:
1. Group 1 (Adult Learning Concepts)
2. Group 2 (Curriculum Development)
3. Group 3 (Presentation Skills)
4. Group 4 (Evaluation & TOL)
Each group goes to a different corner to work.
40
SMALL GROUP ACTIVITY (CONT’D)
Each table:
1. Select recorder who will write down answers on chart
paper. Recorder: Draw line down middle of chart paper
page to prepare for writing.
2. Table groups:
A. Review the Learning Objectives for the respective
table group (BAA Day 1, 2, 3 or 4)
B. Discuss their answers with table mates re which TOL
techniques covered in training could they apply in
their own training practice?
41
SMALL GROUP ACTIVITY (CONT’D)
Each table: Recorder writes down the answers on
chart paper to the following questions:
1. (Left side of chart paper): What did they hear from
other table mates that they would like to add to
their own TOL plan?
2. (Right side of chart paper): What are they planning
to transfer from BAA Days 1-4 to their own
trainings?
42
REPORT OUT
 What are the top 3 answers you heard from your
table mates that you’d like to add to your own
TOL Action Plans?
 What is one take-away that you’d like to transfer
from across BAA University (Days 1 through 4)
and implement in your own trainings?
43
WRAP UP!
Last Questions?
A Visual Reminder of the importance of TOL (if
there’s time)…
44
Thank you!!!
(and please fill out the satisfaction surveys!)
45