Experiment „Natural Language Processing“
Christian Gütl, IICM, TU-GRAZ
Instructions for Experiment
„Natural Language Processing“
Experiment „Natural Language Processing“
The Experiment Procedure
 Topic: “Natural Language Processing (NLP)”
 Objective of the Experiment
 Gain insight about extracting and evaluating concepts and
generating questions
 Informed consent
 Please fill in the informed consent
28.07.2017
Christian Gütl, IICM, TU-GRAZ
2
Experiment „Natural Language Processing“
The Experiment Procedure






Step 1: Learning the text
Step 2: Questionnaire 1
Step 3: Learning Activity I (Questionnaire 2)
Step 4: Test (Questionnaire 3)
Step 5: Learning Activity II (Questionnaire 4)
Step 6: Questionnaire 5a
 Please allow yourself breaks between tasks!
28.07.2017
Christian Gütl, IICM, TU-GRAZ
3
Experiment „Natural Language Processing“
Step 1
Please learn the text (max. 30 minutes)
 See email (Subject: „NLP – text“)
28.07.2017
Christian Gütl, IICM, TU-GRAZ
4
Experiment „Natural Language Processing“
Step 2
Please fill in Questionnaire 1 (max. 10 min)
See email (Subject „phd Questionnaire 1“)
Please give all answers in English!
Please do not use the text while answering the
questions!
Click on the „Submit“-Button once you have finished
the task!
28.07.2017
Christian Gütl, IICM, TU-GRAZ
5
Experiment „Natural Language Processing“
Step 3a: Extracting concepts
 Please extract the main concepts of the NLP-text,
using Questionnaire 2 (max. 10 min)
 (Example: Speed of light; theory of relativity, Einstein).
 Please separate your concepts with semicolons (;) or
commas (,)
  See email (Subject „ phd Questionnaire 2“)
 After you have finished this step, please continue working
on Questionnaire 2 (Step 3b) as described on the next
slide
28.07.2017
Christian Gütl, IICM, TU-GRAZ
6
Experiment „Natural Language Processing“
Step 3b: Generating questions
 Please generate eight questions from the NLP-text, using
Questionnaire 2.
 Construct two Open Ended questions using the text
(Example: What do you know about theory of light?)

Construct two Fill In The Blank questions using the text
(Example: The founder of this university was ________.)


Construct two Single Choice questions using the text. Their answer
should be “no”! (Example: Albert Einstein was born in Norway)
Construct two Multiple Choice questions with three distractors for each
chapter. Only one of the distractors should be correct!
(Example: The sun is _____.
blue, green, yellow, red)
Note that you have to generate one question for each category first and then another one afterwards.
28.07.2017
Christian Gütl, IICM, TU-GRAZ
7
Experiment „Natural Language Processing“
Step 4
Please fill in Questionnaire 3 (max 15 Minutes) and
allow yourself a break afterwards.
See email (Subject „ phd Questionnaire 3“).
Please do not use the text while answering the
questions!
28.07.2017
Christian Gütl, IICM, TU-GRAZ
8
Experiment „Natural Language Processing“
Step 5a: Evaluation of concepts
Evaluation Task I (max. 10 min.)
 Please evaluate the concepts you will find in
Questionnaire 4. Do you think the concept is relevant for
the text?
 1 = not relevant; 5 = very relevant
See email (Subject „ phd Questionnaire 4“).
After you have finished this task, please continue working on Questionnaire 4
(Step 5b) as described on the next slide
28.07.2017
Christian Gütl, IICM, TU-GRAZ
9
Experiment „Natural Language Processing“
Step 5b: Evaluation of questions
Evaluation Task II (max. 35 min)
Please evaluate the questions provided in Questionnaire 4, using the
criteria as described below. For each criterion described below, please
assign a score between 1 ( = very bad) and 5 (very good) to each
question.
Pertinence: Relevancy of the formulated questions. Has the question a precise
and logical connection to the topic?
Terminology: Appropriateness of the words chosen to formulate the question.
Level: Extent in approaching the topic. Is the question trivial or expresses it a
significant meaning? (1 = trivial, easy; 5 = difficult, significant meaning)
Answer (only if provided!): Is the provided answer relevant?
Distractors (only for multiple choice!): Quality of the computed distractors of the
multiple choice questions
28.07.2017
Christian Gütl, IICM, TU-GRAZ
10
Experiment „Natural Language Processing“
Step 5c General Survey
Please fill in the last section of Questionnaire 4 (max.
15 min)
28.07.2017
Christian Gütl, IICM, TU-GRAZ
11
Experiment „Natural Language Processing“
Step 6 Evaluating questions
Evaluation Task III
Please evaluate the questions provided in Questionnaire 5a (see email),
using again the criteria as described below. For each criterion, please
assign a score between 1 ( = very bad) and 5 (very good) to each
question.
Pertinence: Relevancy of the formulated questions. Has the question a precise
and logical connection to the topic?
Terminology: Appropriateness of the words chosen to formulate the question.
Level: Extent in approaching the topic. Is the question trivial or expresses it a
significant meaning? (1 = trivial, easy; 5 = difficult, significant meaning)
Answer (only if provided!): Is the provided answer relevant?
Distractors (only for multiple choice!): Quality of the computed distractors of the
multiple choice questions
28.07.2017
Christian Gütl, IICM, TU-GRAZ
12
Experiment „Natural Language Processing“
Thank you for participating in this Experiment!
28.07.2017
Christian Gütl, IICM, TU-GRAZ
13