Can you prove it?
Seven Ways Assessments Fortify Compliance
John Kleeman
Executive Director and Founder
Questionmark
Brian McNamara
Marketing Director
Questionmark
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing
Limited. All other trademarks are acknowledged.
To ask questions,
use the “Q&A” feature
Slide 2
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Why are we holding this webinar?
Auto Manufacturer to Spend $14.7 billion
to Settle Emissions Case
Roofer's first-day injury costs
Florida contractor $152K OSHA fine
Universities fined millions for how labs how
their labs managed chemical waste
Pollution Settlement to Cost Motorcycle
Manufacturer US$12 Million
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Agenda
Context for “assessment”
​ Seven Ways Assessments Fortify Compliance
Getting assessment results you can trust
Q&A
Slide 4
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
About Questionmark
Background
• Founded in 1988
• Offices in US, UK, Canada; Partners worldwide
• Assessment technologies and portal solutions that enable organizations to
measure knowledge, skills, and attitudes securely for certification, regulatory
compliance and improved learning outcomes.
Customers
• Customers worldwide in Corporate and Government (Learning/HR function);
Higher Education; Awarding Bodies and Test Publishers
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Assessment Management System
Reporting to analyze results and
inform stakeholders
Collaborative, secure authoring of
surveys, quizzes, tests, and exams
Multilingual:
36 Languages
Global
24/7 Support
Data Centers in US
and Europe
Blended delivery to support a broad
range of devices and security needs
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Types of “assessments”
Slide 7
Quizzes, Tests, Exams
Surveys
Observational Assessments
• Questions and scoring algorithms
• Scores and feedback at question,
topic and assessment level
• Questions relate to a person’s
opinions on a topic
• Conclusions drawn based on
analyzing frequency of responses
• Course evaluations, employee
attitude surveys
• Someone assesses or rates a
person’s performance, knowledge,
skills, and abilities
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How our customers use assessments
Measure and
Enhance Learning
Qualification and
Certification
Regulatory
Compliance
Sales Channel
Verification
Technical Channel
Verification
•Measure knowledge,
skills and attitudes –
before, during and
after learning
•Correct
misconceptions and
increase knowledge
retention
•Deliver reliable,
defensible tests to
certify employees
•Ensure sound
personnel decisions
•Ensure knowledge of
essential laws,
regulations and
health/safety
guidelines
•Demonstrate
compliance and
reduce risk
•Ensure competence
and product
knowledge of sales
professionals
•Increase revenues
and customer
satisfaction
•Ensure competence
and expertise of
technical and service
professionals
•Reduce costs and
improve customer
satisfaction
Credentialing
•Efficiently create and
maintain valid,
reliable tests
•Generate revenue
from exams and
practice tests
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
​ Seven Ways Assessments Fortify Compliance
John Kleeman
Executive Director and Founder
Questionmark
Slide 9
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
7 reasons why assessments fortify compliance
Regulators
encourage them
Demonstrate
commitment
Increase knowledge
retention
Slide 10
Find problems early
Reduce costs and
training time
Document
understanding
Reduce human error
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #1 : Regulators encourage them
Slide 11
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Some examples

US FDIC:


WHO good manufacturing principles


“Contractors should consider using tests or other mechanisms to determine the trainees’
comprehension of the training concepts presented.”
UK FSA (now FCA)

Slide 12
“Continuing training should also be given, and its practical effectiveness periodically
assessed.”
US Department of Health & Human Services


“Once personnel have been trained on a particular subject, a compliance officer should
periodically assess employees on their knowledge and comprehension of the subject
matter”
“where staff understanding has not been tested, it is hard for firms to judge how well the
relevant training has been absorbed”
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #2 : Demonstrate commitment
Slide 13
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Why is it important to demonstrate top level
commitment?

Because employees follow their leaders

Charles Jennings (August 2012)
“… common thread that runs through almost all high-profile
compliance catastrophes. It is that the top-tier executives and
middle managers in the organisations simply don’t model the
behaviours that will lead to a culture of compliance”
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How do assessments demonstrate commitment?

If you require all employees and managers to




Slide 15
Take assessments every 6 or 12 months
Remedial training and retake assessment if fail
Assessments seen as fair
Major stride in demonstrating commitment
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
US Department of Justice
“Prosecutors also should determine whether
the corporation's employees are adequately
informed about the compliance program and
are convinced of the corporation's
commitment to it. This will enable the
prosecutor to make an informed decision as
to whether the corporation has adopted and
implemented a truly effective compliance
program”
Slide 16
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
2014 SCCE survey suggests widespread use of
assessments for codes of conduct
Which methods do compliance function use to measure effectiveness of Code of Conduct?
60%
50%
40%
30%
20%
10%
0%
Employee compliance
knowledge
assessments
Slide 17
Employee culture of
integrity and
compliance surveys
Other
Tracking code
Employee focus groups Tracking employee Tracking code release
communication
utilization of code dates against employee
initiative dates against
resources on intranet
reports
employee reports
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #3 : Find problems early
Slide 18
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How tests/quizzes/exams can find problems early

Look at topic and assessment
scores






By department
By geography
Over time
Look at wrong answers
Look at test failures
Assessments touch every
employee: a unique tool
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How surveys can find problems early


Surveys identify issues in culture and ethics
Most employees want to help. From employee of bank involved in 2012
LIBOR scandal:
“Always happy to help, leave it with me, Sir”.

UK Ministry of Justice Bribery Act Guidelines
“Staff surveys, questionnaires and feedback from training can also provide an
important source of information on effectiveness and a means by which employees and
other associated persons can inform continuing improvement of anti-bribery policies”.
Slide 20
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #4 : Document understanding after training
Slide 21
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Why document training for compliance?

Because regulators require it

common regulatory requirement, e.g. OECD recommends on ethics/compliance
programme:
“measures designed to ensure periodic communication, and documented training for all
levels of the company”

Because makes business sense


Trained employees more effective
“If you can not measure it, you can not improve it.“
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Common Methods of Documenting Training
Test to check
understanding
Get employee
signatures
Take
attendance
Slide 23
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Take attendance

Pros
Easy to do
 Measures who turns up


Cons
Did they stay the whole time?
 Did they pay attention?
 Did they get it?

Slide 24
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Get employee signatures

Pros
Gets employee confirmation
 Employee committed to follow rules


Cons


Slide 25
People may sign but not take
seriously
People may sign without attending
or understanding
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Does getting an employee to sign give legal protection?

Not where safety is involved, according to 2005 United States Appeal
Court:
“merely having an individual sign a form acknowledging his responsibility to read the
safety manual is insufficient to insure that the detailed instructions contained therein
have actually been communicated.”

Slide 26
Complete General Const. Co. v. OSHRC, 2005 U.S. App. LEXIS 5197 (6th Cir.), March 29, 2005
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Test to check understanding

Pros
Genuinely check understanding
 Allows improvement of weaknesses
 Feedback to improve training


Cons


Slide 27
More effort
You have to follow through & take
action on failures
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Beyond the Headlines…

September 2016: Large bank fined …but also ordered to bring in an
independent consultant to review various things including:
“whether Respondent’s employees are required to undergo training
reasonably designed to prevent Improper Sales Practices and other salesintegrity violations; whether such training is adequate, complete, and
timely updated, provided when employees join Respondent, and repeated
at sufficient recurring intervals during their employment to reinforce such
training; whether training records are complete, accurate and adequate”
ADMINISTRATIVE PROCEEDING 2016-CFPB-0015
Slide 28
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #5 : Increase knowledge / reduce forgetting
Slide 29
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Compliance Training
Receive
information
Slide 30
Working
memory
Long term
memory
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Training useful if can retrieve
• Question
• Other stimulus

Research shows retrieval practice
helps future retrieval
• Retrieve from long term
Search memory

Successful retrieval modifies
memory making it more
retrievable in future
Cue
Act
Slide 31
• Act or respond
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Roediger & Karpicke experiment 2006



120 students divided into groups randomly
Given reading comprehension passages used in TOEFL
Some students


Some students



Slide 32
Study for 14 mins
Study for 7 mins
Recall material for 7 mins
Compared results after 5 mins and 1 week
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How much learners retained
90%
80%
70%
60%
50%
Restudy
40%
Quiz
30%
20%
10%
0%
After 5 mins
Slide 33
After 1 week
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Learning in the Actual World
They don’t get it All!
Slide 34
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Learning Curve
Purpose:
• Content repetition
• Memory retrieval practice
• Strengthens memory recall
Learning
Experience
Learning
Experience
Questions
Learning
Experience
Questions
Knowledge/Skills
Ouch! The
Forgetting
Curve
Time
Slide 35
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
In the Actual World
They Forget It!
Slide 36
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Slow the Forgetting Curve
Purpose:
Questions
Questions
Learning
Experience
Learning
Experience
Questions
Learning
Experience
Questions
Knowledge/Skills
• Memory retrieval practice
• Strengthens memory recall
• Reduce Forgetting Curve
Time
Slide 37
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #6 : Reduce cost & time for compliance
training
Slide 38
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Testing out of compliance training
Take test
Slide 39
If pass, skip
training
Otherwise
do training
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Saving money by “testing out”
Management Report Example
A) Number of people
1,000
B) Time for training if no testing (2 hours each)
2,000 hours
C) Time to take test (20 mins / test)
333 hours
D) 25% pass test and skip training
Slide 40
E) 75% do compliance training
1,500 hours
F) Hours of training avoided (B – C - E)
177 hours
G) Average cost / hour
$100
H) Money saved (F * G)
$17,700
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Will the regulator allow?

Many regulators permit “testing out” to reduce compliance training

ABA Bank Compliance magazine article 2009 interviewed some US regulators
 OTS: Testing out acceptable
 OCC: Agree, providing testing program well structured. May be some exceptions
 FDIC : Agree

Check with your regulator
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #7 : Reduce human error
Slide 42
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Root causes of human error
(data from UK MHRA 2011, graph by Questionmark)
6%
5%
22%
Process/procedure incorrect
15%
Procedural steps omitted
Concentration error
Training misunderstood
Training missing
23%
Poor communication/rushing
29%
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Assessments

Assessments can fix
Training misunderstood (15%)
 Training missing or out of date (6%)
 Procedural steps omitted (23%)
 Process or procedure incorrect
(22%)

6%
Process/procedure
incorrect
5%
22%
15%
Concentration error
Training misunderstood
23%
29%

Other causes of error


Slide 44
Procedural steps
omitted
Training missing
Poor
communication/rushing
Poor communication / rushing (5%)
Concentration error (29%)
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
7 reasons why assessments fortify compliance
Regulators
encourage them
Demonstrate
commitment
Increase knowledge
retention
Slide 45
Find problems early
Reduce costs and
training time
Document
understanding
Reduce human error
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Getting assessment results you can trust
Brian McNamara
Marketing Director
Questionmark
Slide 46
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
What can we learn from assessments?
Did they
like it?
Did they
learn it?
Can they
do it?
Can we
prove it?
Trustworthy results are essential to answering these questions
Slide 47
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Trustable results require assessments be…


Reliable: dependable, repeatable, consistent
Valid: measures appropriate knowledge and skills
Figure 1:
Reliable but not Valid
Slide 48
Figure 2:
Not reliable, not Valid
Figure 3:
Reliable and Valid
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Validate assessment content using
Job Task Analysis
Ask Subject Matter Experts:
• Do you do the task or supervise it?
• How Difficult is it to do this task?
• How Important is it to do this task?
• How Frequently do this task?
Slide 49
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Job Task Analysis (JTA) Reports
Slide 50
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
JTAs valuable for “blue printing” exam content
Electrical Skills Test - Blueprint
Section 1: AC Motors
% of test
# items
45%
34
Skill level
Criticality
1.a - Lorem ipsum dolor
21
3
3
1.b - Duis aute irure
8
3
3
1.c - Excepteur sint occaecat
5
2
3
Section 2: High-voltage switchgear
25%
19
2.a - Neque porro quisquam
12
2
2
2.b - Modi tempora incidunt
7
3
2
Section 3: Synchronous Motors
30%
23
3.a - Totam rem aperiam
11
2
2
3.b - Eaque ipsa quae
12
2
3
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reliable results: Basics of Item Analysis
Difficulty index (aka P “value”)

What it measures:



% answered correctly
Number from 0.0 to 1.0

Too high = too easy
 Too low could mean…
 Too difficult
 Confusing/ambiguous
 Miscoded

Correlation between
item score and test score
How it is expressed:

How to interpret:

What it measures:

How it is expressed:


Item correlation discrimination:
Number between
-1.0 and +1.0.
How to interpret:

Generally…
 < 0.2 low
 > 0.3 good
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Correlation Discrimination
Difficulty/
P-Value
Slide 53
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Slide 54
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Pop quiz!
Which of the following is the most appropriate passing score for a
compliance related test or exam?
A.
B.
C.
D.
E.
70%
80%
90%
100%
It depends on how difficult the questions are
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Are my pass/fail scores defensible?
Not so good practice



Slide 56
Guess
Roll dice
Pick a number out of a hat
Good compliance practice

Set pass/cut score to reflect
minimally acceptable
competence

Passing test demonstrates
competence
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Angoff Method

Based on this question:


What is the % chance that a marginal
candidate will get question right?
How it works
Poll SMEs
 Consider marginal candidates and the
probability of getting specific questions
right (0-100%)
 Make a mathematical calculation to
determine cut score

Slide 57
Why use this method?
One of our customers in the US military
summed up the benefits this way:
• Defensible
• Easy to use and implement
• Widely accepted
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Angoff Method Example
What is the % chance that a borderline candidate will get question right?
Q1
SME A
75%
SME B
75%
SME C
75%
Total
75.00%
Q2
70%
80%
80%
76.67%
Q3
65%
75%
70%
70.00%
Q4
60%
85%
90%
78.33%
Q5
80%
80%
85%
81.67%
Q6
80%
80%
80%
80.00%
Q7
75%
80%
75%
76.67%
Q8
65%
90%
65%
73.33%
Q…..
75%
80%
75%
76.67%
Q50
65%
85%
65%
71.67%
71%
81%
76%
76%
Totals
Slide 58
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Scenario-based questions:
Test “above knowledge”

What does a yellow traffic light
mean?



Slide 59
Stop
Go
Caution

If you are driving toward an
intersection and the light turns
from yellow to red, what should
you do?
Speed up and cross the intersection
 Stop suddenly
 Stop gradually

Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Observational Assessment
Observer “rates”
the participant
Example Applications
Sales Training
Certification
Medical and Dental
(OSCE)
Smartphones/tablets ideal for
observational assessment
Equipment operation
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Case Study: Observational assessments for learning &
compliance

$10 billion medical equipment
manufacturer


Sales reps trained on new product
Field trainer then…
Observes rep demonstrate the
product’s use
 Using an iPad, completes an
observational assessment
 Rates performance on a 3-point scale:

 “Performed without Prompting”
 “Performed with Prompting”
 “Unable to Perform.”
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Summary on good practice





Slide 62
JTA for valid content
Item analysis for reliable questions
Defensible cut scores
Scenario questions – test above knowledge
Observational assessments
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Recommended White Papers
www.questionmark.com/go/whitepapers
Slide 63
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
More good practice
Criterion-Referenced
Test Development:
Technical and Legal Guidelines
for Corporate Training
Sharon Shrock and
William Coscarelli
Slide 64
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Looking for help?



Contact
Glen Budgell PhD
[email protected]
1.866.574.7041 x272
Slide 65
Questionmark Partner: HRSG
Leader in the field of competency-based talent
management
Can assist in developing and implementing
compliance assessment programs on the
Questionmark platform
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How to Evaluate
Slide 66
•
Request a one-on-one demo
The Questionmark team will contact you to arrange a
demonstration tailored to your needs and questions
www.questionmark.com/go/request-demo-enus
•
Request a 30-day trial of Questionmark OnDemand
Complimentary technical support during trial
www.questionmark.com/go/od30us
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Thank you for attending!
We hope to see you at a future webinar!
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing
Limited. All other trademarks are acknowledged.