Slide 1
© 2001 By Default!
Heriot-Watt University
6th February, 2002
Interoperability: futureproofing CAA
Niall Sclater
CETIS
Manager, Educational Systems
University of Strathclyde
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 2
© 2001 By Default!
Why do I need to know about
QTI?

I’m authoring a bank of questions in my
subject area

I’m involved in the selection of a CAA system
for my institution

I have a general interest in online learning
and assessment
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 3
© 2001 By Default!
The basics
IMS QTI is a means for exchanging
assessment information
 It uses XML
 2 key parts of the spec:

– Assessment Section Item
– Results Report
Also QTI lite
 Can be implemented separately

A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 4
A Free sample background from www.pptbackgrounds.fsnet.co.uk
© 2001 By Default!
Slide 5
Response types
Multiple choice
 True false
 Multiple response
 Image hot spot
 Fill in the blank
 Select text
 Slide
 Drag object
 Drag target
 Order objects
 Match item
 Connect the points

A Free sample background from www.pptbackgrounds.fsnet.co.uk
© 2001 By Default!
Slide 6
© 2001 By Default!
The concept of rendering

The same item can be “rendered” in different
ways to allow for different
– delivery platforms
– technical capabilities
– participant challenges (eg visual, audio, physical
etc)
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 7
© 2001 By Default!
Rendering example
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 8
© 2001 By Default!
Basic response & render types

Response types
–
–
–
–
–
logical identifier
xy coordinate
string
number
logical group
A Free sample background from www.pptbackgrounds.fsnet.co.uk

Render types
–
–
–
–
–
choice
hot spot
fill in blank
slider
‘object’
Slide 9
© 2001 By Default!
True false example
Paris is the Capital of France ?
True
False
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 10
© 2001 By Default!
True false example XML
<questestinterop>
<qticomment>
This is a simple True/False multiple-choice example.
The rendering is a standard radio button style.
No response processing is incorporated.
</qticomment>
<item ident="IMS_V01_I_BasicExample001a">
<presentation label="BasicExample001a">
<material>
<mattext> Paris is the Capital of France ? </mattext>
</material>
<response_lid ident="TF01" rcardinality="Single" rtiming="No">
<render_choice>
<response_label ident="T">
<material><mattext> True </mattext></material>
</response_label>
<response_label ident="F">
<material><mattext> False </mattext></material>
</response_label>
</render_choice>
</response_lid>
</presentation>
</item>
</questestinterop>
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 11
© 2001 By Default!
Response processing
<item>
…
<resprocessing>
<outcomes><decvar/></outcomes>
<respcondition title="Correct">
<conditionvar>
<varequal respident="TF01">T</varequal>
</conditionvar>
<setvar action="Set" >1</setvar>
<displayfeedback feedbacktype="Response" linkrefid="Correct"/>
</respcondition>
</resprocessing>
<itemfeedback ident="Correct" view="Candidate">
<material><mattext>Yes, you are right</mattext></material>
</itemfeedback>
…
<item>
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 12
© 2001 By Default!
Multiple choice example
Which one of the listed standards committees is responsible
for developing the token ring specification ?
IEEE 802.3
IEEE 802.5
IEEE 802.6
IEEE 802.11
None of the above.
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 13
© 2001 By Default!
Multiple choice example XML
<questestinterop>
<item title="Standard MC Item" Ident="IMS_V01_I_BasicExample002">
<presentation label="BasicExample002">
<material><mattext>Which one of the listed standards committees is responsible
for developing the token ring specification ? <mattext>
</material>
<response_lid ident="MC01" rcardinality="Single" rtiming="No">
<render_choice shuffle="Yes">
<response_label ident="A">
<material><mattext> IEEE 802.3 </mattext></material>
</response_label>
<response_label ident="B">
<material><mattext> IEEE 802.5 </mattext></material>
</response_label>
<response_label ident="C">
<material><mattext> IEEE 802.6 </mattext></material>
</response_label>
<response_label ident="D">
<material><mattext> IEEE 802.11 </mattext></material>
</response_label>
<response_label ident="E" rshuffle="No">
<material><mattext> None of the above. </mattext></material>
</response_label>
</render_choice>
</response_lid>
</presentation>
</item>
</questestinterop>
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 14
© 2001 By Default!
Response processing
<item>
…
<resprocessing>
<outcomes>
<decvar vartype="Integer" defaultval="0"/>
</outcomes>
<respcondition title="Correct">
<conditionvar>
<varequal respident="MCb_01">A</varequal>
</conditionvar>
<setvar action="Set" >1</setvar>
<displayfeedback feedbacktype="Response" linkrefid="Correct"/>
</respcondition>
</resprocessing>
<itemfeedback ident="Correct" view="Candidate">
<material><mattext>Yes, you are right.</mattext></material>
</itemfeedback>
…
<item>
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 15
© 2001 By Default!
Multiple response example
Which of the following elements are used to form water ?
Hydrogen
Helium
Carbon
Oxygen
Nitrogen
Chlorine
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 16
© 2001 By Default!
Image hot spot example
Identify the VDU.
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 17
© 2001 By Default!
Fill in blank (string) example
Complete the sequence:
Winter, Spring, Summer, _ _ _ _ _ _ .
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 18
© 2001 By Default!
Numerical example
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 19
© 2001 By Default!
Composite example
Which city is the capital of England
and name another city in England ?
Sheffield
London
Manchester
Edinburgh
Another city:
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 20
© 2001 By Default!
Results reporting
Specific to a single participant
 4 constructs

– summary (eg max score, num attempts)
– assessment
– section
– item

Each construct contains a context which
specifies details such as name of participant
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 21
© 2001 By Default!
Where does QTI fit in with other
specs?
IMS Meta-data Specification
 IMS Content Packaging Specification
 IMS Learner Information Packaging
Specification
 IMS Digital Repositories Interoperability
Specification

A Free sample background from www.pptbackgrounds.fsnet.co.uk
IMS Spec Development Process
IMS
Members
Final
Approval
Public
Elect
Technical
Board
Feedback
Drafts
Public Draft
Working
Group
Base Document
Scope
Scope
Slide 23
© 2001 By Default!
What’s next with QTI?

V1.2 currently being approved
– selection and ordering of sections and items
– scoring roll-up in sections and assessments
(from question scores to test scores)
– results reporting
Question and test metadata
 Harmonisation with other specs
 Integration with ADL SCORM
 Implementations now needed


V1.3 early in 2002
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 24
© 2001 By Default!
UK Interoperability
Developments
Strong representation in QTI working group
 CETIS QTI SIG

– meetings
– online discussions
– repository of open source tools
– hosting workshops
CAA Centre & SCAAN
 JISC MLE CAA Project
 e3an

A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 25
© 2001 By Default!
Products which say they are
implementing QTI
E-Test 3, RIVA Technologies Inc
 Question Mark Perception
 Can studios
 Oracle i-learning
 Blackboard
 WebCT
 Ultimate assessment
 E3AN

A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 26
© 2001 By Default!
IMS and other initiatives
IMS Europe
 SCORM
 Open Knowledge Initiative
 Prometeus
 Questionmark -> Blackboard & WebCT

A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 27
© 2001 By Default!
www.imsproject.org
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 28
© 2001 By Default!
QTI page
A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 29
© 2001 By Default!
QTI: some problems

For vendors
– implementing QTI is costly
– stops them from protecting their user base
Vendors want simple specifications; users
want broad, well-defined ones
 It is difficult to define specifications which do
not have an effect on functionality

A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 30
© 2001 By Default!
Finally: why bother with QTI?

Avoid lock in
– allow data to be moved in standard formats
– allow one system to replace another
Integrate systems from different vendors;
allow choice of best assessment tool
 Facilitate exchange of questions and tests
between institutions, allow market to develop
 Enable separation of content from
presentation
 Standard assessment records will facilitate
student mobility

A Free sample background from www.pptbackgrounds.fsnet.co.uk
Slide 31
© 2001 By Default!
Further information
www.sclater.com
www.imsproject.org
www.cetis.ac.uk/assessment
A Free sample background from www.pptbackgrounds.fsnet.co.uk