Semantic models
in healthcare education
What is it and how it can improve formative assessments
MedBiquitous Annual Conference 2012
May 2-4 2012 - Baltimore, MD
Muriel Foulonneau
Younes Djaghloul
Raynald Jadoul
Nabil Zary
7/28/2017
2
7/28/2017
3
Challenges
Two main challenges:
• Item variability in an assessment
-
generate items from a model in order to avoid repeating items
save time and resources, as assessment resource creation is a time and
resource consuming activity
• Learning adaptivity
-
adapting question forms or assessment path in formative assessment according
to candidate answers or profile
We strive toward: Efficient Approach to Automate/Assist the
generation of assessment resources.
28/07/2017
OVACS-AIGLE-TAO
4
How the challenges are addressed
•Knowledge sources




Expert
Social / Crowd sourcing
Repository
Textual
• Question generation
• Keep the initial semantic
• Semantic Inference
• Adaptivity
How ?
Assessment resources
28/07/2017
OVACS-AIGLE-TAO
5
In summary the goal was to
• Enable the automatic generation assessment questions based on
formal models of knowledge
• Knowledge oriented approach based on semantic technologies:
•
•
•
The creation of a streamline exploring the use of semantic technologies for eassessment
Semantic for model checking
Semantic for inference ( to discover knowledge)
• Needs to have models with formal representation (such as RDF)
• Four questions
28/07/2017
How to build a domain model?
How to validate the proposed model by non IT expert ?
How to generate assessment questions from the refined model?
How to build a flexible delivery environment for these questions?
OVACS-AIGLE-TAO
6
7/28/2017
7
The vision
28/07/2017
OVACS-AIGLE-TAO
8
Overview on approach
Knowledge: Informal models
Experts,
repositories,
social media
4.Delivery strategy
•TAO Delivery Module
•TAO QTI viewer
1.Model building: data mining, human
methodology
Knowledge: Formal models
The Final test
Formal but
not validated
Validate questions
Experts for question validation
List of assessment
questions
2.Model validation
•Experts for model validation
•OVACS : to assist experts and to
hide the complexity of he formalism
(OWL, description logic )
28/07/2017
Final
Ontology
OVACS-AIGLE-TAO
3.Question generation
•AIGLE tool, Automatic QTI based
questions generation
•Semantic similarity techniques
9
The process
Capturing information from an expert (in our
case, a teacher)
• Creation of a domain ontology
Validating the ontology
• Evaluation by the experts using OVACS
Generating assessment items
• Evaluation of the assessment generation approach with a
teacher
Delivering the assessment items
• Delivery of the assessment test using the TAO platform
28/07/2017
OVACS-AIGLE-TAO
10
Origins of OAT
28/07/2017
OVACS-AIGLE-TAO
11
OVACS
Ontology VAlidation for Common uSers
How to validate formal knowledge
model by questions
28/07/2017
OVACS-AIGLE-TAO
12
OVACS: what ?
• Question based strategy for validation
•
•
•
Question to for the validation of the domain not for the assessment
Generate question based on existed knowledge element ( automatic)
More simple for the expert than modifying formal model ( OWL )
• Four ontological components (OC) to validate (RDF schema)
•
•
•
•
Instance of
All property value
Sub class
Property of a class
• 12 types of feedback
• For each OC Accept, remove, don’t Know
• Templates for textual question
• Generic (Subject, Predicate, Object)
• Dedicated
28/07/2017
OVACS-AIGLE-TAO
13
OVACS architecture
Source Ontology
OWL
Validated
ontology
OVACS Engine
Evaluated ontology
(Semantic web
technologies)
Ontology of
management
feedback
Generated Question
(Web based)
•Manage history
•Get past questions
Expert feedbacks
28/07/2017
OVACS-AIGLE-TAO
14
OVACS interface
http://crpovacscaries.elasticbeanstalk.com/
28/07/2017
OVACS-AIGLE-TAO
15
AIGLE
Assessment Item Generator
in Learning Environment
28/07/2017
OVACS-AIGLE-TAO
16
AIGLE – Assessment item generator
-
Security issue (variability)
Adding variability to an item
no expected variation of the construct
-
Model-based learning (adaptivity)
Generating items from knowledge represented as a model
the construct is modified for each item
Stem
variables
Auxiliary
information
Options
Key
28/07/2017
OVACS-AIGLE-TAO
17
IMS-QTI item generation process
Generating items from Web data sources
28/07/2017
OVACS-AIGLE-TAO
18
Calculating the semantic similarity between
distractors and the correct answer
Gabon -- Libreville
Ulan Bator
Libreville
Manila
Maputo
Port Louis
Libreville
No SemSim
With SemSim
Adapted 3 semantic similarity strategies to large
scale semantic graphs
28/07/2017
OVACS-AIGLE-TAO
19
Results of user test
Clear decrease of performance in the population when using SemSim
(optimizing the similarity between the correct answer and the distractors)
28/07/2017
OVACS-AIGLE-TAO
20
User testing with countries and their
capital
28/07/2017
OVACS-AIGLE-TAO
21
TAO
Testing Assisté par Ordinateur
(Computer-Aided Testing)
28/07/2017
OVACS-AIGLE-TAO
22
TAO – assessment and feedback loop
The TAO platform is based on semantic web paradigm, i.e. it manages
question items decorated with any needed ad-hoc properties
The TAO platform delivers questionnaires that can also be featured
with any desiderated extra semantic properties
The TAO collects all answers and behaviors of the test-takers
 If extra properties like the “provenance” (i.e. the source model built
with OVACS and used by AIGLE) are attached to the question items
or to the questionnaire, these properties are stored in tests results
 The analysis of the tests results will enforce be used by as feedback
loop for a validation process impacting the AIGLE & OVACS phases.
OVACS
28/07/2017
AIGLE
OVACS-AIGLE-TAO
TAO
23
7/28/2017
24
Experiment with a dentistry teacher
28/07/2017
OVACS-AIGLE-TAO
25
Original hypothesis
The creation of the domain ontology can use semi-automatic strategies,
or third party encoders, or a collaborative work: can we ask an
expert to validate the assertions in the ontology?
- What is lost in the expert’s speech when creating the ontology?
- Does the expert understand automatically generation questions?
- Does the expert flag the errors?
28/07/2017
OVACS-AIGLE-TAO
26
Creating the ontology
An ontology of the caries
- A one hour interview where the teacher explained
the caries, their description, their causes,
how to handle them,
how to prevent them,
how to set a diagnostic
- Definition of a list of concepts / keywords
- Creation of classes, instances, and properties
- Creation of the OWL ontology
28/07/2017
OVACS-AIGLE-TAO
27
Test set up
Labels on stand alone
Selected a subset of the ontology
to keep the test short:
 instanceOf (13 items) and
 subClassOf (11 items)
Only Boolean questions
+ “I do not know” option
24 questions
2 intentional mistakes:
on the content (causes of caries) and
spelling (emanel instead of enamel)
Objective:
- verify whether the teacher would find the validation mechanism usable
- Verify whether errors would be detected and corrected
Video recording of the teacher
28/07/2017
OVACS-AIGLE-TAO
28
OVACS interface
http://crpovacscaries.elasticbeanstalk.com/
28/07/2017
OVACS-AIGLE-TAO
29
Test conclusions
Confusion between
the role of domain expert validating knowledge
and
the role of teacher who prepares questions for students
 Objective was not well understood  rework experiment conditions
According the comments of our expert:
“Difficulty level of the generated questions is generally low”
“But with very different variations in the difficulty level”
The OVACS validation questionnaire led to:
6 removals (2 subClassOf, 4 instanceOf)
16 accept (9 for subclassOf, 7 for instanceOf)
2 answers “I do not know” for subclassOf  meant not relevant
28/07/2017
OVACS-AIGLE-TAO
30
Next steps
OVACS
Enrich collaborative features
AIGLE
Ensure a validation / feedback on the generated items
AIGLE generates distractors from an open model (large dataset from
the Web) using semantic similarity, but needs to identify relevant
distractors in the case of a bounded model (in this case a model for
caries)
Predicting item difficulty? Initial test for general culture questions using
a Web mining approach. Would need to be tested for medical
knowledge.
28/07/2017
OVACS-AIGLE-TAO
31
http://tao.lu
OVACS-AIGLE-TAO