The Half-Time Strategy: Saving Your
Organization Time and Money with
Diagnostic Testing
John LoBianco
Steven Torkel
PwC
2013 Users Conference
Baltimore | March 3rd – 6th
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Your Presenters

John LoBianco


Steven Torkel


Slide 2
Senior Associate, Evaluation & Assessment Team
Director, Evaluation & Assessment Team
Special thanks to Sean Farrell, Senior Manager on the
E&A Team, who has been instrumental in this work.
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

A global network of professional services firms


People development is a top priority


Slide 3
180,000 staff members and partners across the network
PwC was awarded #1 in Training Magazine’s Top 125 for 3
years in a row, and was placed into the “Hall of Fame”
Highly mobile and virtual workforce
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Session Objectives
Slide 4

How our approach to diagnostic assessment has
evolved over the past three years

Best practices for building defensible assessments for
the purpose of testing-out of training

Half-time Strategy

Modeling tool
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Diagnostic Assessment


Slide 5
Measures current level of
knowledge and skill in
specific topic areas.
Results help drive decisions
about future development
activities.
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Why do diagnostic
assessments?
Slide 6
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How our approach has evolved
2010
• Tax 1 self-studies
• New employees
2011
• Tax 1 self-studies
• New employees
• Enhancements
2012
• Mandatory Training
• Experienced Partners
Future
Slide 7
• Classroom Training
• Integrating assessments with on-the-job learning
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
The Tax 1 Program

New tax hires must attend Tax 1 after joining the firm



College graduates with varying levels of experience
First professional job
First exposure to PwC technical training
Four years ago:

Slide 8
Program shortened
from two weeks to
one week
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Tax 1 Re-design

Several self-studies were created as advance preparation


Slide 9
New hires were required to complete these self-studies before
attending Tax 1
Learners could not
always find the time to
complete the self-studies
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How could we ease the burden on our
learners while still maintaining quality?

Test-outs for three of the self-studies were
created:
New hires could save time by not having to
complete the self-studies
 Optional
 The three test-outs were comprised of 25, 28, and
21 questions
 “One and done”

Slide 10
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How much time did we save our learners?


On average, there was a slight increase
in training time across the learner
population after the test-outs were
implemented
Why didn’t we achieve the benefit we were
hoping for?


Slide 11
Not enough learners passed the test-out
Test-outs were too long
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How did we improve the test-out process?

Combined the three test-outs into one
“consolidated” test-out

Identified and removed poorly performing items

Adjusted cut score

Required learners to register for test-out

Implemented “half-time” strategy
Slide 12
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Half-Time Strategy
Slide 13
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Tax 1 Test-out Half-time Strategy

Each section of the test-out had 20 questions


Each section of the test-out was split in half
Learners needed to pass the first half of each
section to proceed to the second half of
that section
Learners answered anywhere from
30 to 60 questions
 Saved learners time by bypassing
questions they would probably
answer incorrectly

Slide 14
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Test-out Half-time Strategy
Course A
(10 questions in
1st
half)
Learner
scores 80%
or above
2nd
half)
Learner is then taken to
Course B
Slide 15
Course B
(10 questions in
1st
half)
Learner
scores 80%
or above
Course A
(10 questions in
Learner
scores less
than 80%
Course B
(10 questions in
2nd
Learner
scores less
than 80%
Course C
(10 questions in 1st half)
Learner
scores 80%
or above
half)
Learner is then taken to
Course C
Course C
(10 questions in 2nd half)
Learner reaches end of
test-out
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
End of Test-out Feedback

Learners were told their results for each section
Course A
Course B
Course C
Slide 16
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Time savings were modest

58 hours of training time
(about 5%) were saved
871 total hours taking the
self-studies
 164 total hours taking the
test-outs
 1093 potential total training
hours

Slide 17
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Mandatory Training test-out option

Mandatory Training for Partners

Yearly training event

Option to take a short online test in
lieu of completing a 90 minute selfstudy learning activity

Testout designed to be more
challenging than the end of course
assessment

Available only to partners with 2+
years experience
Slide 18
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Benefit: Expected time savings for the
practice


Difficult to quantify as it depends on the number of partners
who take the optional test-out, as well as the percentage who
pass
Sample estimates based on:
 A learner population of 1500 partners
 90 minute self study learning activity
 18 minute test-out activity
Good
Better
Best
% Taking Test out
50%
70%
90%
% Passing
40%
60%
75%
225 (10%)
630 (28%)
1114 (50%)
Training Hours Saved
Slide 19
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Rigorous test development procedures
improved the quality of the test-out

Parallel test development: Developed over 50 questions
mapped to course content for both the test-out and self-study
end of course assessment

Pilot-tested all questions with 68 partners and received 400+
comments

Used statistical item analysis to isolate best performing
questions with appropriate level of difficulty

Limited testing time while still maintaining quality


Slide 20
17 question test-out
Half-time strategy
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Test-out Half-time Strategy

17 question test split in two parts:





Passing score of 80% (14 of 17)
Need to “pass” first half to proceed to second half
First half passing score:


Slide 21
First half 9 questions and second half 8 questions
Split the questions to even out content and difficulty
where possible
6 of 9 which meant 8 of 8 was needed on the second half
First half passing score set to allow total pass if perfect on second half
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Partners were satisfied with the test-out
process
“I thought this was a great option. Saved a lot of time
and it is still not easy to take the test so it also
ensures appropriate focus and knowledge.”
“I thought it was a great improvement,
and permitted those of us who watch
these areas closely to get through the
requirement (which is a good
requirement to have) easily and
efficiently.”
Slide 22
86%
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Success Factors
Participation Rate
Pass Rate
Time Saved
Partner Feedback
Slide 23
• 231 Partners took the test-out; 20% of target audience
• Reasons for not participating:
- Want to take the self-study
- Could not find time to complete
• 67% Pass Rate
• Of those who did not pass, 31% were stopped at the
half-way point, saving them time
• 14 minutes average test time; 9 minutes half-time
• Saved 236 hours in total; 11% of training time
• 86% satisfied with test-out
• Increased flexibility for partners
• Favorable feedback
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Modeling Tool Interactive Demo
Assumptions
Data
Learner Population
1000
% Taking Test-Out
50%
% Passing Test-Out
50%
Minutes for Training
90
# Questions in Test-Out
15
Minutes Per Question
1
Additional minutes for test out (e.g. instructions)
3
Change assumptions to estimate hours saved by
using a test out strategy.
Total Minutes for Test-Out
18
Total Possible Training Hours
Path
# of Learners
Test-Out Hours
Learner passed the test out
250
75
Learner did not pass the test out
250
75
Learner did not attempt the test-out
500
Total
1000
150
225
15%
Hours Saved
Slide 24
1500
Training Hours
Total Hours
75
375
450
750
750
1125
1275
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Future Deliverables


Classroom Training
Integrating assessments with on-the-job learning

Traditional training model
 Curriculum-based
 Mandatory courses
Slide 25

On-the-job learning {70 / 20 / 10}

Use diagnostic assessment to better target on-the-job learning as
well as formal training
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.