Classifying questions for sampling in a task-given-type examination
Toshihiko Takeuchi* and Akiyuki Sakuma*
Department of Industrial and Systems Engineering, College of Science and Engineering,
Aoyama Gakuin University
Toshihiko Takeuchi
Postal address: Daizawa 4-45-15, Setagaya-ku, Tokyo, Japan
E-mail: [email protected]
Phone & fax number: tel 81-3-3419-1248, fax 81-3-3419-1248
Akiyuki Sakuma
Postal address: Sasazuka 1-62-3-1202, Shibuya-ku, Tokyo, Japan
E-mail: [email protected]
Phone & fax number: tel 81-3-5371-7085
Abstract
To promote efficient science and technology education, we propose a task-given-type examination
(TGTE). In this system, questions are given to the learners before the examination and questions in the examination
are chosen from them.
One issue related to the TGTE is the method of choosing questions. As criteria of evaluation for sampling
questions, we propose simplicity, predictability, and proportionality between effort and achievement. Classifying
questions satisfies simplicity and predictability. In cases when the amount of effort needed for each question differs
and in cases when there is relation between knowledge elements needed for each question, we propose a method of
classifying questions satisfying the proportionality.
As a result of a case study in fluid mechanics, 182 questions could be divided into 16 classes satisfying the
criterion to the effect that each of them require(s) nearly equal total(or additive) amount of effort for answering all
the questions in each class, thus proving the system to be effective.
Keywords: task-given-type examination, TGTE, science and technology education, classifying questions, fluid
mechanics
1.
1.1
Introduction
Background
learners a sufficiently long time before the examination.
Then, questions in the examination are chosen from the
informed group. For example, in a term examination in
We propose a task-given-type examination (TGTE). This
college, 100 questions for the examination, hereinafter
is an effective method to help improve achievement since it
called candidates (for questions of the examination), are
encourages learners to study by themselves.
given to the learners on paper or on a Web site. Then, 20 of
A TGTE is carried out as follows: A group of questions
in an examination related to the learning is given to the
the 100 questions are randomly chosen and included in the
term examination.
Giving the questions in advance enables many people to
cannot be used in a TGTE either.
check the questions in a TGTE. Therefore, we can expect
an improvement in the quality and objectivity of the
1.3
Purpose of this study
questions in an examination and the exclusion of tricky or
The purpose of this study is as follows. On sampling
difficult questions. In addition, we need not worry about
questions for a TGTE, we define three criteria of evaluation
leaking of questions, so we can improve morals.
called
Furthermore, dependence on chance in achievement in a
between effort and achievement. As a method satisfying
TGTE is less than in achievement in a usual examination.
them, we propose the idea of "classifying questions" and
Thus, earnest efforts are rewarded in a TGTE and this
apply it to real questions in order to verify the effectiveness
encourages learners to study by themselves.
of the method.
simplicity,
predictability,
and
proportionality
Examples of carrying out a TGTE have not reported, but
Takeuchi et al.(1)(2) presented theoretical studies into a
system for a TGTE.
2.
Sampling criteria and an outline of classifying
questions
In this section, we will state criteria that should hold
1.2
Previous studies related to sampling questions
One issue in preparing a TGTE is to sample questions
when questions for a TGTE are sampled from candidates
and outline a method to divide questions into classes.
for the examination from the candidates adequately. This
paper discusses methods to sample questions for the
examination from the candidates. There are previous
2.1
Conditions that should hold in sampling
questions
studies regarding sampling adequate questions from a large
When we sample questions for a TGTE, we need to
number of questions. For example, selecting questions
sample them adequately so as not to discourage learners to
using item response theory,
(3)
a trial-and-error individual
study aid system using fuzzy theory,
fuzzy theory,
(5)
(4)
sampling using
study by themselves. Thus, we will propose the following
three criteria in this study:
and a study using Bayesian estimation to
(Evaluation criteria 1) Simplicity: If the system for the
select questions(6) have been reported. In (3) and (4), for
examination is too complex, it will be difficult for learners
learners who have not studied enough, the system estimates
to make a strategy to study and they will be discouraged to
an adequate question that they should review. Then, it gives
study by themselves. In addition, confusion will occur
them the question. In (5), the system sets the most adequate
when we carry out and mark the examination, so it will be
question according to the study record of the learners. In
difficult to manage smoothly.
(6), the system selects a question that maximizes mutual
(Evaluating criteria 2) Predictability: Predictability is the
information by Bayesian estimation engine. The above
accuracy with which a learner predicts his/her achievement.
studies consider an examination after a lesson in (3) and (4),
Concretely, when a learner is assumed to take the
at each time of learning in daily individual lessons in (5),
examinations many times, this is the standard deviation of
and at questions in (6). All of these studies strongly depend
the distribution of his/her achievement. If this value is
on data of the correctness or incorrectness of answers to
small, learners can predict their achievement with high
questions by learners. However, since the learners can
accuracy, so we consider that they are encouraged to study.
know the answers before a TGTE, understanding and the
(Evaluating criteria 3) Proportionality between effort
rate of correct answers do not have a strong relation.
and achievement: We consider that, if achievement and
Therefore, it is inadequate to use the data of correctness or
effort of study are not proportional, then achievement will
incorrectness.
increase in the degree of study, so learners are encouraged
A study for sampling questions for review(7) also exists.
to study by themselves.
This study, however, assumes that data of acquisition on
There are other criteria needed for sampling adequate
each question of the learners can be used. These data
questions other than the above. For example, from structure
among pieces of knowledge to study, Akahori calculated
.
subordination, basics, and relation. He then proposed a
method to calculate difficulty and usefulness of each
element to study.(8)
That is, the sum of questions selected from every class is
n. An example of classifying questions is as follows. If
In this study, however, we consider only the three criteria
there are 80 candidates and 10 of them are set, the
above for the following reason. In the simplest TGTE that
candidates will be divided into 10 classes of eight questions,
we can consider, the most adequate method to set questions
and one question from every class will be set.
that we consider satisfies the conditions. Therefore, also in
Learners can also easily understand classifying questions
complex cases, we think that they are more essential than
since the method to set questions is simple, so this satisfies
other conditions and we think that it is necessary to
simplicity. In addition, it is certain that one or several
consider them.
questions in each class will be set, so the predictability
We will explain the above consideration in a concrete
strengthens.
example of a test of 100 irregular verbs. Before this test, a
For example, among 100 candidates Q1 - Q100, suppose
teacher presents 100 questions of irregular verbs and on the
that a learner studied 50 questions Q1 - Q50 and that he/she
date of the examination, all of the 100 questions are set. By
can answer the learned questions without fail and cannot
regarding an irregular verb as a knowledge element, there
answer the other questions. Let the perfect score be 100.
are 100 knowledge elements in the test. We can consider
Let marks be allotted equally. If 20 questions are randomly
that the knowledge needed to correctly answer each
set without classifying questions among the 100 candidates,
question is independent of that required to answer other
then the mean (expectation) of his/her achievement is 50
questions and that the effort needed to study each question
and the standard deviation is 100/√99 ≒ 10.05. We can
is approximately the same. Therefore, this is the simplest
derive these from the formulae of sampling without
TGTE.
replacement. Classifying questions, however, lower the
Now, in this test, if we need not consider the time of the
standard deviation without lowering the mean. For example,
test or the load to answer or mark the test, it is the best to
assume that the 100 candidates are divided into 20 classes
set all questions. The reasons are as follows. [1] The
of five continuing questions and that the learner is
method to select questions is simple and there is no place
informed that one question in every class is set. Then
for randomness. [2] Learners can perfectly predict their
his/her achievement is always 50, and the standard
achievement before the test. [3] It is clear to learners that
deviation is zero. Generally, classifying questions tends to
achievement and time of effort are proportional. These
lower the standard deviation to those who study in order. In
features are simplicity, predictability and proportionality
other words, this strengthens predictability.
between effort and achievement.
Therefore, in this study, we consider only the above
three criteria.
Note that for those who can answer questions at an equal
rate for every class, we see that this weakens predictability
by the formula of sampling without replacement. However,
this is not important for our argument.
2.2
Definition of classifying questions and its
advantage
2.3
Classifying questions is a method that among N
Proportionality between effort and achievement
Classifying
questions
does
not
always
satisfy
informed candidates for questions of the examination, if n
proportionality between effort and achievement. For
(≦ N) of them are set, they are divided into r (≦n)
example, consider cases satisfying the conditions in 2.2 and
disjoint classes and learners are informed that ni of the ith
the following two conditions:
class will be set, where,
(1)
Q1 - Q100 are mutually independent with respect to each
r
Σ
ni = n
i=1
Elements needed to correctly answer questions
question.
(2)
The effort needed to study to correctly answer
questions Q1 - Q100 can be measured as, for example, the
unit time of study, and the effort for each is equal to the
question number.
Condition 4: The set of knowledge elements needed to
solve each question is known.
Fields that may satisfy the conditions above are the
fields in which knowledge is explicitly structured, such as
Then, to study all questions of Class 1 (Q1 - Q5)
mathematics, physics, and computer language. For example,
requires only 15 units of effort, while Class 20 (Q96 -
S-PLUS, which is a computer language for statistical
Q100) requires 490 units of effort. Therefore, to those who
analysis, has many functions, such as the function seq to
study Classes 1 - 20 in order, effort and achievement are
make an arithmetic sequence and the function rev to make
not proportional. So classifying questions does not always
a sequence of the reversed order. So acquiring them is the
satisfy proportionality between effort and achievement.
main content of study and each function can be considered
If relation among elements of study is simple, it is easy
a knowledge element. In addition, the order of studying
to classify questions so that proportionality between effort
functions is apparent: To study the function apply, which
and achievement holds. For example, in 2.3, study needed
applies a function to rows or columns of a matrix, we need
for each candidate is assumed independent. Thus, to satisfy
to study the function matrix to make a matrix in advance.
the condition appropriately by neglecting difference in each
Furthermore, the knowledge elements for a given question
class, it is sufficient to distribute questions in approximate
become clear by checking functions in a program that is the
proportion to (the amount of effort to study each class)/(the
answer to the question.
total amount of effort to study all questions), or to allot
marks similarly. In systematic subjects such as mathematics
and physics, however, it is difficult to make classifying
questions in which the amount of study of each class is
equal.
3.2
Conditions to satisfy proportionality between
effort and achievement by classifying questions
In this paper, we consider the following cases: One
question is set from every class, and equal points F/n,
where F is the perfect score and n is the number of
3.
Proposition of a method of classifying questions
In this section, we will propose a method of
classifying questions that satisfies proportionality between
questions that are set, is allotted to every question.
To
satisfy
proportionality
between
effort
and
achievement, we need to classify questions considering the
following two conditions:
effort and achievement, even if the simple method of
Condition 1: The additional amount of effort to study all
classifying questions is inadequate owing to systematic
questions in the class should be nearly equal with respect
knowledge.
to every class.
As the questions in a class become difficult, the effort to
3.1
Conditions assumed in order to apply this
method of classifying questions
study each class will become larger. However, when we
study a difficult question, we have already acquired
We will state the conditions assumed for knowledge in
knowledge elements for easy questions, so we need only
fields of study in which this method of classifying
effort to get new knowledge (hereinafter called the
questions can be applied.
additional amount of effort). Therefore, if the questions
Condition 1: The items to study can be divided into finite
knowledge elements.
Condition 2: Order can be specified to the knowledge
elements to study as follows: Before study of knowledge i, it
is necessary to study knowledge j and knowledge k.
Condition 3: There is no loop of order in knowledge
elements to study.
satisfy the requirement that we can study all questions in
each class with nearly equal additional effort with respect
to every class, then the achievement and time to study
become proportional.
Condition 2: In the same class, every question can be
answered with nearly equal amount of effort (similar
difficulty).
If the difficulty within the same class is extremely
different, then by studying only easy questions, learners
can increase expectation of achievement. So this condition
should hold.
For a method to relate knowledge elements, see a study
by Sato.(11)
Procedure 3: Assigning an amount of effort to each
knowledge element
The amount of effort is the quantified value of load of
3.3 Procedure for classifying questions satisfying
study needed to acquire knowledge Ki when we have all
the needed knowledge to acquire it. One who has already
proportionality between effort and achievement
We will state the procedure for classifying questions
studied the subject expediently gives an amount of effort
satisfying the two conditions in 3.2 if the knowledge in a
by units of time in which average learners acquire the
field of study satisfies the conditions in 3.1.
knowledge as an indication and considering a mental
Procedure 1: Dividing items into knowledge elements
burden.
Divide the items to study of the subject into L
Investigations have been conducted by Akahori(8) and
knowledge elements K1, . . . , Kn. With respect to methods
Nakamura et al.(12) to determine the difficulty of questions
of division, we can refer to methods to make a hierarchic
even if test data of learners do not exist. However, the
structure, such as an investigation for a method to make
former is an investigation to determine difficulty of each
(9)
question in relation to the entire structure among
and a schema for a structure of teaching material by
knowledge elements. The latter is an investigation to
teaching material for study in procedure by Kuwabara
Sato.
(10)
determine difficulty of deduction when the necessary
Outlines of the methods above are as follows: For the
knowledge has already been acquired. They are different
method to make teaching materials(9), we extract technical
from investigations needed here to determine an "additive
terms from explanation in each question, collect the related
amount of effort" to answer question i when a learner can
terms, and form groups of ideas. To make a structure of
answer question (i  1). Development of objective and
(10)
teaching materials
, a teacher extracts elements of study
from a textbook, makes cards of the elements, and arranges
them in different order.
Procedure 2: Expressing the data of the structure of
knowledge
We express the relation between knowledge elements by
an L × L adjacency matrix D, and a reachability matrix is
denoted by M.
concrete methodology to determine an amount of effort is
earnestly wanted.
Procedure 4: Making a quasi-adjacency matrix between
knowledge elements and questions
A quasi-adjacency matrix between knowledge elements
and questions C defines which knowledge each question
directly needs as follows:
C = (cij) for i = 1, . . . , L; j = 1, . . . , N;
D = (dij) for i, j = 1, . . . , L;
where
M = (mij) for i, j = 1, . . . , L;
cij = 1 if knowledge i is directly needed for question j;
where
dij = 1 if knowledge i is needed for knowledge j;
cij = 0 otherwise.
dij = 0 otherwise;
between questions
Procedure 5: Making a quasi-reachability matrix
mij = 1 if it is reachable from knowledge i to knowledge j
From the quasi-adjacency matrix between knowledge
(i.e., mik[1] = mk[1]k[2] = … = mk[h]j = 1 for some h, k[1], . . . ,
elements and questions C and the reachability matrix
k[h]);
between knowledge elements M, we will calculate the
mij = 0 otherwise.
quasi-reachability matrix between questions U. For
Note that when we say that knowledge A and knowledge
candidates for questions of the examination Qi and Qj, we
B are needed for knowledge C, it means that before
say that the order from Qi to Qj holds if the set of all
studying knowledge C we need to study both knowledge A
knowledge elements to which Qi is traced back is included
and knowledge B.
in the set of all knowledge elements to which Qj is traced
back. We define a quasi-reachability matrix between
"any of knowledge 2, 4, or 5" in advance, but "all of
question U such that the (i, j) element is 1 if the order from
knowledge 2, 4, and 5" in advance.
Qi to Qj holds and is 0 otherwise as follows:
Procedure 6-a: A state of study begins from the state
U = (uij) i, j = 1, . . . , N;
without studying any knowledge or any question.
where
Procedure 6-b: From the order between knowledge
Wi = {j | mjk = cki = 1 for some k};
elements, we make a quasi-reachability matrix between
uij = 1 if Wi ⊆ Wj;
knowledge elements (Figure 2). For example, according to
uij = 0 otherwise.
Figure 1, Q5 is traced back to knowledge 5 and 10, while
Q1 is traced back to only knowledge 5. So Q1 should be
Procedure 6: Arranging the questions in order
Every question in the same class should be answered
studied before Q5 and Figure 2 says Q1 → Q5.
with a nearly equal amount of effort (cumulative time to
6
Q9
Q10
12 9
study from the state before study). For this purpose, we
will arrange the questions from the structure scheme of
7
11
3
10
Q2
questions and the data of additive amounts of effort, each
of which is needed to acquire each question. The elemental
20
2
2
10
3
1
1
3
9
7
8
10
10
Q5 10
12
4
6
1
3
Relation between
knowledge elements
Question number
Knowledge-element
number
Amount of effort
Q1 5
10
State before study
idea of the arrangement is the following: "When learners
who can already answer i questions study to answer the (i +
8
Q4
Q8
Q6
10
Legend
Q3
Q7
Figure 2. Scheme for order between questions
1)th question, they select the question with the least
Procedure 6-c: From the present state of study, we list
amount of effort." We will explain the procedure of the
the questions that can be studied next. We list the
arrangement using an example. Suppose there are ten
questions Q1 - Q10 and 12 knowledge elements. In Figure
1, question numbers are surrounded with a rectangle with
unanswered
questions
without
an
arrow
from
an
unanswered question (Figure 3). Here, they are Q2, Q6, Q8,
and Q1.
Q9
Q10
round corners and a knowledge-element number is
Q3
surrounded with a circle. A thin arrow signifies the order
Q7
Q4
between knowledge elements, and the amount of effort is
Q5
Q2 1 3
printed in white on a black rectangle.
Q6 1 3
Q8
Q10
11
20
Q7
7
Q6
6
7
Relation between
knowledge elements
Q4
3
10
Legend
9
11
10
Relation between knowledge
elements and questions
8
10
Q5
10
Question number
12
2
4
Q2
1
Amount of effort
State before study
Q3
12 9
Q9
Relation between
knowledge elements
Question number
9
Q1 1 0
2
Legend
6
Q8
5
Knowledge-element
number
10
Amount of effort
Figure 3. Questions to select
Procedure 6-d: For each question Qj listed in Procedure
6-c, we calculate a temporal additional amount of effort to
answer Qi, say a[i] (i = 1, . . . , L). This is the total amount
Q1
3
State before study
Figure 1. Knowledge-structure scheme of questions
A thick arrow in Figure 1 from a knowledge-element
number to a question number signifies the order of
knowledge to study. Multiple arrows to a question signifies
not "or" but "and": For example, Q9 needs not "either
knowledge 6 or 7" but "both knowledge 6 and 7." In
addition, a thin arrow from a knowledge-element number
to another signifies the order between knowledge elements
to study. Multiple arrows to a question signifies not "or"
but "and": For example, studying knowledge 3 needs not
of effort for all knowledge elements to which Qi is traced
back and that has not been acquired yet. For example, a[8],
which is the additional amount of effort for Q8, is 9. This is
the combination of the amount of effort 3 for knowledge 1
and the amount of effort 6 for knowledge 4 (in Figure 3,
a[2] = 13, a[6] =13, a[8] = 9, a[1] = 0). We select the
question with this temporal smallest additional amount of
effort. In Figure 3, a[8] is the smallest, so we select Q8. If
there are plural questions with the temporal smallest
additional amount of effort, we choose one of them
randomly. When a question is selected, we determine a[j],
the additional amount of effort for the selected question.
Procedure 6-e: Knowledge elements to which the
Sec[Sj], is as follows: (Here [x]* is the integer to which x is
rounded up.)
question selected in Procedure 6-d is traced back (here,
X[Sj] = Σ
knowledge 1 and 4) are made to be the state already
Y =Σ
studied.
N
k=1
j
k=1
b[k]
b[k]/m
Sec[Sj]=max([X[Sj]/ Y]*,1)
Procedure 6-f: We repeat from Procedure 6-b to
Figure 5 shows the result of classifying questions.
Procedure 6-e, and we end when all questions have been
Class 5
Q9 20
studied. It is not necessary that all knowledge elements
have already been studied. Figure 4 shows the order of
Class 4
Q10 9
Class 3
Relation between
Questionss
Q7 21
questions when it is determined.
Question
Knowledge
Q8
1 4
5
2
Sj
s(i)
S1
s(8)
S2
s(1)
S3
s(2)
S4
s(6)
S5
s(5)
S6
s(4)
S7
s(3)
Cumulative additive
amount of effort
X[Sj]
X[Sj] / Y
Sec[Sj]
9
19
29
29
41
51
58
79
88
108
0.42
0.88
1.34
1.34
1.90
2.36
2.69
3.66
4.07
5.00
1
1
2
2
2
3
3
4
5
5
Class 2
Q6 0
Q8 9
Class
State before study
easy
Q1
Q2
Q6
Q5
Q4
Q3
Q7
Q10
difficult
Q9
10
8
9
3 7
12
6
3
6
10
10
0
12
10
7
10 11
9
20
9
10
10
0
12
10
7
21
9
20
Question number
Q5 12
Q2 10
Amount of Additive amount
effort
of effort
e[i]
a[i] = b[j]
Legend
Q4 10
Q3 7
S8
s(7)
S9
s(10)
S10
s(9)
Class 1
Figure 5. Result of classifying questions
Figure 6 shows the additive amount of effort needed to
answer each of the classes correctly.
Class 1
Q1
Q8
Class 2 Q2,Q6
Figure 4. Order of questions
In Figure 4, the order is Q8 → Q1 → Q2 → Q6 →
Class 3
Q5 → Q4 → Q3 → Q7 → Q10 → Q9, but there is a
Class 4
possibility of Q8 → Q1 → Q6 → Q2 → Q5 → Q4
Class 5
→ Q3 → Q7 → Q10 → Q9. The result of classifying
questions in Procedure 7 does not change. Whether Q2 or
Amount of effort
Q1 10
Q3
19
Q5
Q4
22
17
21
Q7
Q10
Q9
29
Figure 6. Additive amount of effort for each class
4.
Simulation
Q6 is studied earlier, the additive amount of effort to
answer the other question is zero. Thus X[Sp], which is the
To confirm the effectiveness of the proposed method, we
cumulative value of b[j] in order from the easiest question
carried out a simulation experiment and considered the
to the pth question, does not differ between these cases.
result.
Therefore, by classifying questions in Procedure 7, Q2 and
Q6, will be assigned to the same class.
Procedure 7: Classifying questions
4.1
Field to which we applied the simulation
We applied the simulation to the field of physics, which
We divide questions into n classes so that the total of the
is a representative field in the education of science and
additive amount of effort in each class is equal, where n is
technology. We carried out simulation considering the use
the number of questions sampled from the candidates for
of questions in Computations in Fluid Mechanics,(13) which
the examination.
in an introduction to hydrodynamics, as candidates for the
We define s(i): For i, j = 1, . . . , N, if the original
questions of the examination. This introduction consists of
question number Qi is the jth question in the order of this
eight chapters beginning at managing units followed by
procedure (Sj in Figure 4), then Sj = s(i). For example, the
hydrostatics, hydrodynamics, flow in a pipe, flow
first question determined here is Q8, so S1 = s(8).
measurement, force on an object, pumps, and calculation of
The additive amount of effort for the jth question
determined in Procedure 6-d is denoted by b[j] for j =
1, . . . , N, that is, if Sj = s(i), then b[j] = a[i]. For example,
waterwheels. Each chapter has questions.
We will consider whether fluid mechanics satisfies the
conditions in 3.1.
in Figure 4, b[1] = 9, b[2] = 10, . . . , and b[10] = 20. The
In fluid mechanics, there are several ideas defined
class number to which question Qi belongs, denoted as
clearly, such as Torricelli's law of efflux, gauge pressure,
and orifice coefficient. We can determine their order. For
answering a question. In addition, we compared with
example, we cannot understand Torricelli's law of efflux
simulation of the half number (= 8) of classes and the
without knowing the Bernoulli theorem. We cannot
double number (= 16) of classes. Furthermore, we
understand a Pitot tube without knowing Torricelli's law. In
compared with the simulation of selecting two questions
addition, we can consider that there is no loop of order in
from each of eight chapters, which are original questions
knowledge elements to study.
from the question book.
Furthermore, explanations of questions are objective
sentences with keywords of knowledge elements, so we see
4.3
Procedure of classifying questions
that the set of knowledge elements directly needed to
We will explain procedure of classifying questions in the
answer each question is known.
experiment. In preparation for the experiment, we selected
Therefore, fluid mechanics satisfies the four conditions
a subject in the fourth year of university who was willing to
in 3.1, so we judged that the method in this study is
study fluid mechanics. He was asked to study until he
effective.
could answer all questions in Computations in Fluid
Mechanics by intensive study in three weeks (from the
4.2
beginning to the middle of October 2000). He was asked to
Conditions of the simulation
In the examination, we selected 16 questions among 182
record knowledge, formulae, and keywords of ideas needed
candidates about basic knowledge of fluid mechanics and
to answer each question in a notebook for reference. In
many small questions: We divided the 182 candidates into
addition, we asked him to record the time in minutes
16 classes and selected one question from every class. The
needed to study each question. We then carried out the
reason for choosing 16 is as follows. For an examination in
simulation according to Procedures 1–7, as follows:
90 minutes, we consider 5 minutes to be adequate for
Legend
Relation between
knowledge elements
Knowledge-element
92 double acting 3
57orifice coefficient α
2
60 Pitot tube
coefficient
8
90 pumping rate of a
reciprocating pump
8
48 λ=64/Re
93 pumping rate of a gear pump
6
45 Weisbach
9
5
86 specific speed Ns
4
5
41 Bernoulli theorem
3
6
85 impeller lift
106 number of
revolutions
of a waterwheel
9
39 law of continuity
84 impeller power 4
40 heavy fluid
3
9
7
5
22 power
2
75 drag coefficient
2
31 hydraulic principle
3
18 viscosity u
2
69 form drag 4
4
3 Pa
1
2 at
1
1 kgf,N
1
1
4 Pa・ｓ
13 densityρ
2
68 thrust
97 B number 4
1
9 mmHg→SI
73 capitation
8
78 total head H
1
7
77 actual head h
8
94 Pelton wheel
1
15 specific gravity
37 restoring force T
3
36 buoyancy F 8
5
47 Moody diagram
46 roughness
88 hydraulic
pump power
8
State before study
4
4
74 Mach
5
16 compressibilityδ
2
64F=ρQ(v-u)
7
1
10 J
2
20 capillarity
2
7 N/m3
79 how to obtain H 4
65 F=ρQvsinθ
12 specific weight
89 piston speed
5
5 N・m
1
70 boundary layer
4
34 moment
91 volumetric
efficiency
67 F=√(FX2-FY2)
3
14 r=ρg
2
4
6
72 lift
17 resistivity Ｆ
71 drag 5
50 Alievi’s equations
10
11 PS
1
2
76 lift coefficient
7
95 Pelton wheel power
3
83 angular 6
velocity ω
51 hydraulic
mean depth
30 Pascal’s law
4
98 Francis turbine
53 Hazan-Williams
equation
52 Chezy
formula
27 atmospherec
pressure
3
8 Aq 1
80 theoretical
power
4
96 maximum power
5
4
24 head
4
99 Francis turbine power
5
3
4
82 mechanical efficiency
3
81 pump efficiency
2
23 total
pressure
3
19 kinematic
viscosityμ
100 maximum of
a Francis turbine
54 Venturi meter
102 effiency of a
waterwheel
25 absolute
pressure
7
38 Reynolds number
3
66 F=ρA(v-u)2
5
3
26 equilibrium
of pressure
32 mean pressure
4
42 Torricelli’s law of efflux
44 loss factor
87 Ns for drawing 3
3
4
liquid at
both sides Ns
55 flow coefficient C
4
35 pressure on arc
3
33 gauge pressure
61 weir 4
103 efficiency of a
power plant
28 horizontal
water tube
4
4
4
3
62 90°V-notch weir
3
4
29 gas pressure
63 flow formulae
in JIS
2
43 friction coefficient of a pipe
101 net head
4
58 opening ratio m
3
3
4
59 Pitot tube
9
49 ζ=(1-A1/A2)2
5
104 Ns of a waterwheel
5
56 orifice
6
105 table of Nｓ
Amount of effort
2
8
6 mmHg
3
21 PV=RT
3
2
Figure 7. Knowledge-structure scheme
162
166 175
179
145 156
69
75
76
117
91
92
61
105
67
70
22
23
80
167 168 177 178
114
176
102 103
104
51
63
68
144
72
65
169
182
180 170
143
108 109 110
73
88
154 155
71
142 153
74
64
172 181
Legend
97
115 116
89
49
171
161 149
84
85
86
87
90
48
50
57
58
59
99
98
Relation between
Questions
112
140 129
Question number
124
130 141
132
27
41
44
45
66
21
37
52
20
25
30
46
36
39
24
15
56
133
173 163
43
26
14
40
62
119
78
53
29
77
18
10
6
3
35
31
120
82
107
94
17
126
125 136
For simplicity
159
150 160
151 152
131
95
Note
83
158
147 157
93
47
148
16
34
9
122 123
96
118
121 134 135
79
81
100 101 111 113
165 174
33
4
2
55
164
42
106
13
54
38
28
32
19
60
is abbreviated a
146
128 127 137 138 139
8
11
7
5
12
1
State before study
Figure 8. Scheme for order between questions
Question number
162
Class16
179
145 156
166 175
Class8-1
76
115 116
91
105
92
89
67 70
49
Class3-1
27
51
64
45
14
10
73
88
180 170
143
142 153
99
71
140 129
85
86
87
130 141
90
98
Legend
Class8-2
124
132
122 123
50
57
58
59
60
62
81
96
120
47
37
54
55
100 101 111 113
165 174
164
53
133
42
26
106
29
78
77
18
5
121 134 135
40
35
31
79
93
17
94
95
125 136
Class10-1
12
Class13
159 150 160
173 163
43
107
Class4
82 83
119
28
8
7
56
Class3-2
9
Class1
84
48
34
6
11
Class14
Class6
Class2
13
3
65
20
36 38
19
154 155
74
52
4
2
72
46
15
32 33
169
112
108 109 110
66
21
39
24 25
30
63
23
22
41
44
68
104
61
97
Class9
114
172 181
144
Class10-2
176
102 103
167 168 177 178
182
Class11-1
69
80
117
Class7
75
171
Class15
161 149
16
Class5
118
131
151 152
148
158
147 157
126
146
Class12
128 127 137 138 139
Class11-2
1
Figure 9. Result of simulation for classifying questions
Chapter 1
Q1-Q11
Chapter 2
Q12-Q43
Chapter 3
Q44-Q66
Chapter 4
Q67-Q97
Chapter 5
Q98-Q105
Chapter 6
Q106-Q141
Chapter 7
Q142-Q162
Chapter 8
Q163-Q182
Procedure 1: Extracting knowledge elements from fluid
includes the set of all knowledge elements to which A is traced
mechanics
back.
The subject extracted 106 knowledge elements from items to
study fluid mechanics, such as "kinematic viscosity μ ,"
"Hazen-Williams equation," and "Torricelli's law of efflux,"
from his notebook.
Procedure 2: Expressing the data of the knowledge elements
of fluid mechanics
Procedure 6: Arranging the candidates in order
We arranged the candidates for questions of the examination
in order.
Procedure 7: Classifying questions
We divided questions into 16 classes so that the total of the
additive amount of effort in each class is equal.
We expressed the relation between knowledge elements in a
106 × 106 adjacency matrix D and a reachability matrix
4.4 Result of simulation for classifying questions
denoted by M. The subject defined the relation between the 106
Figure 9 shows the result of simulation for classifying
knowledge elements by explanation of questions and his
questions. Each class is surrounded with a thick curve. In each
experience of study and then made a matrix expressing the
class, a number in an ellipse signifies a question number. The
structure of knowledge (Figure 7). In Figure 7, knowledge
same design of ellipses means that they are of the same chapter
elements are surrounded by rectangles with round corners, and
in the question book. The class numbers 1 - 16 are in order of
A → B means that we should study A before studying B.
the arrangement in Procedure 6-f. Questions in a class of a
Procedure 3: Assigning an amount of effort to each
knowledge element
We had the subject assign an amount of effort to each
knowledge element when the knowledge needed to acquire it is
small number are generally easy. In Figure 9 and schemes for
classifying questions (Figures 12, 14, and 16), descriptions of
Classes k-1, k-2, . . . , k-h mean that we draw Class k divided
into h parts to make figures simple.
already obtained e1, . . . , e106. We had him refer to his record of
the study time and consider the mental burden. Figure 7 shows
the amounts of elements used for the experiment of simulation.
Procedure 4: Making a quasi-adjacency matrix between
knowledge elements and questions
4.5 Consideration on the result of the experiment in
classifying questions
We consider whether the result of the simulation satisfies the
two conditions in 3.2.
Figure 10 is a graph of the additive amount of effort for each
between knowledge elements and questions C, referring to his
class. The abscissa represents the class number and the ordinate
notebook, his experiment to study, and explanation and answers
represents the additive amount of effort to acquire all questions
of questions.
in the class.
Procedure 5: Making a quasi-reachability matrix between
questions
From the quasi-adjacency matrix between knowledge
elements and questions C and the reachability matrix between
knowledge elements M, we calculated the quasi-reachability
matrix between questions U. Figure 8 shows the relation
between questions. A number surrounded with an ellipse
represents a question number, and the notation A → B means
that the set of all knowledge elements to which B is traced back
additive Amount of effort
The subject expressed data in a quasi-adjacency matrix
40
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16
Class No.
Figure 10. The additive amounts of effort for each
class
Figure 10 shows that the additive amounts of effort to study
all of each class are nearly equal, so it satisfies Condition 1.
when the valve is fully open?
Generally, if the class number is large, much knowledge has
to be acquired so the questions become difficult.
Table 1 shows that questions assigned to each class are often
of the same chapter or the next or previous chapter. For every
class, we counted the number of questions from the most
4.6
Comparison with classifying question by simple
division of chapters
common chapter and the next and previous chapters. Then, we
We compare the result of the proposed method of classifying
found that the total is 156 out of 182 questions. This shows that
questions (Figure 10) with a simple method. We simply divided
questions in the same class are closely related and are easy to
each chapter using the original order of questions so that the
study together.
numbers of questions of the former and latter parts is equal.
order of questions so that the numbers of questions of the
former and latter parts are equal. When the chapter number is
odd, then the middle question is included in the former part.
90
Chapter 8, former part
8
章
前
半
Chapter 8, later part
7
章
前
半
Chapter 7, later part
Chapter 6, later part
6
章
前
半
Chapter 7, former part
Chapter 5, later part
5
章
前
半
Chapter 6, former part
Chapter 4, later part
4
章
前
半
Chapter 5, former part
For example, out of 23 questions of Class 1, we see that 11
3
章
前
半
Chapter 3, later part
in each class and chapter
2
章
前
半
Chapter 4, former part
1
章
前
半
Chapter 3, former part
0
Chapter 2, former part
追
加
努
力
量
Chapter 2, later part
Table 1. Classifying questions and the number of questions
this simple method. We divided each chapter using the original
Chapter 1, former part
For each class, a gray background signifies
the most common chapter or the next or previous one.
Figure 11 shows the additive amount of effort for each class for
Chapter 1, later part
C hapterC1hapterC2hapterC3hapterC4hapterC5hapterC6hapterC7hapterTotal
8
11
12
23
9
1
10
9
4
13
2
2
6
10
2
3
2
7
15
2
6
3
26
4
15
4
3
26
1
1
6
1
9
2
1
3
6
8
2
10
1
5
6
1
5
6
5
5
3
3
6
8
9
17
2
2
11
32
23
31
8
36
21
20
182
additive amount of effort
C lass1
C lass2
C lass3
C lass4
C lass5
C lass6
C lass7
C lass8
C lass9
C lass10
C lass11
C lass12
C lass13
C lass14
C lass15
C lass16
T otal
questions are from Chapter 1 (about units in fluid mechanics)
and 12 questions are from Chapter 2 (questions mainly about
Figure 11. Additive amounts of effort in simple
density). Out of 10 questions of Class 2, we see that nine
division of chapters
questions are from Chapter 2. Therefore, the result of
We see that the additive amount of effort for each class varies
simulation also satisfies Condition 2, so it satisfies the two
greatly, so this is not an adequate method of classifying
conditions in 3.2.
questions.
In addition, we obtained the following result. If the class
number is large, questions become difficult. For example,
Question 1 in Class 1 is as follows:
Question 1: There is fluid that weighs 1,000 kgf. What is the
weight in SI units, N (Newtons)?
We can also solve other questions in Class 1 if we study once.
On the other hand, Question 61 in Class 6 is as follows:
Question 61: In a horizontal pipe having an inside diameter
4.7 Relation between the number of questions and the
number of classes
We will consider the relation between the number of classes
and that of classes.
Figures 12 - 15 show the results of classifying questions into
eight classes, which are half the number, and 16 classes, which
are double the number.
of 20 cm, the pressure is 392 kPa when the downstream valve is
If only the number of classes is changed while other
fully closed and decreases to 294 kPa when the valve is fully
conditions are the same, then the method is the same until
open. What are the mean flow velocity and the stream flow
Procedure 6 in 3.3, that is, arranging the questions in order.
Therefore, for example, the result of division for half the
a question (Class 31) appears. This phenomenon might occur
number of classes (Figure 12) is the same as merging two of
when the additive amount of effort a[i] needed for a single
original classes (Figure 9) in order. The result of division for
question Qi is greater than Y, which is the additive amount of
double the number of classes (Figure 14) is a subdivision of the
effort that we want to assign to a class.
original division (Figure 9).
In the example of Figure 15, if we add 1 to the number of
When the number of classes is sufficiently smaller than the
classes to make it 33, classify questions again, and neglect
number of questions, we can more evenly distribute questions
classes without a question, then the number is resolved.
to each class. For example, the additive amount of effort for
However, the additive amount of effort of the class that is just
each class for half the number of classes (Figure 13) is more
after the class without a question is large. The number of
evenly distributed than for the original division (Figure 10).
classes might generally increase in this method.
The additive amount of effort for each class for double the
Generally, if the number of classes is large compared to the
number of classes (Figure 15) is less evenly distributed than for
number of questions, it is difficult to distribute the additive
the original division (Figure 13). What is worse, a class without
amount of effort to each class.
Legend
Chapter 1
Q1-Q11
Chapter 3
Q44-Q66
Chapter 5
Q98-Q105
Chapter 7
Q142-Q162
Chapter 2
Q12-Q43 Chapter 4
Q67-Q97
Chapter 6
Q106-Q141 Chapter 8
Q163-Q182
Question number
179
161 149
145 156
171
167 168 177 178
172 181
169
182
6-1
Class
4
80
115 116
Class
91
105
92
89
67 70
49
68
114
102 103
104
51
64
61
97
5-1
63
72
73
88
84
85
86
87
71
140 129
99
74
65
7
112
143
142 153
108 109 110
Class
180 170
154 155
Class
176
75
76
144
69
117
98
90
124
130 141
132
2-1
23
22
Class
27
41
44
45
46
55
50
56
57
58
59
60
62
81
96
120
15
32 33
13
14
107
2-2
Class
164
28
42
82
53
133
26
83
106
29
78
77
18
8
7
35
31
118
131
121 134 135
40
9
6
11
159 150 160
173 163
43
34
10
47
100 101 111 113
165 174
119
19
3
54
20
36 38
39
25
4
2
48
Class
37
52
24
30
3
66
21
122 123
94
95
17
126
125 136
0
1
146
2
3
4
5
6
7
8
Class No.
6-2
5-2
12
70
128 127 137 138 139
Class
Class
5
1
158
147 157
79
93
151 152
148
16
additive amount of effort
162
8
Class
166 175
1
Class
Figure 12. Result of division for half the number of classes
Figure 13. Additive amount of effort
for each class for half the number of
classes
Legend
Chapter 1
Q1-Q11
Chapter 3
Q44-Q66
Chapter 5
Q98-Q105
Chapter 7
Q142-Q162
Chapter 2
Q12-Q43 Chapter 4
Q67-Q97
Chapter 6
Q106-Q141 Chapter 8
Q163-Q182
Question number
179
171
30
Class
161 149
145 156
167 168 177 178
172 181
169
69
117
13
14
49
105
92
Class
102 103
104
51
64
63
29
72
18
73
112
Class28
143
142 153
99
74
88
180 170
154 155
17
108 109 110
65
Class
Class
144
Class
68
114
Class
89
67 70
61
20-1
97
176
Class
91
80
115 116
75
76
27
182
21
Class
15-1
Class
Class
84
85
86
87
90
48
50
57
58
59
15-2
Class
71
140 129
16
Class
130 141
98
124
132
6-1
Class
23
22
27
41
44
45
3
Class
30
2
Class
52
28
19
13
14
26
106
54
55
11
Class
42
43
4
1
Class
62
81
122 123
96
120
47
18
107
7
78
35
31
17
Class
119
121 134 135
118
16
131
29
19
Class
94
95
Class
20-2
159
147 157
9
Class
146
22
158
25
Class
24
Class
25
0
1
4
7
10
13
16
19
22
Class No.
23
Class
128 127 137 138 139
Class
150 160
151 152
148
126
125 136
1
26
83
6-2
Class
12
82
53
Class
79
93
8
133
10
77
5
100 101 111 113
165 174
164
Class
34
8
7
56
173 163
Class
40
9
6
11
60
Class
20
36 38
15
32 33
10
46
5
39
24 25
3
12
Class
37
Class
4
2
66
21
additive amount of effort
162
32
Class
166 175
Figure 15. Additive amount of effort
25
28
31
Figure 14. Result of division for double the number of classes
for each class for double the number of classes
4.8 When amounts of effort are varied
by different amounts of effort. This is the result of the following
The person who assigns the amounts of effort has already
simulation. We varied each amount of effort given in Procedure
studied the subject and can estimate the time in which the
3 in 4.3 within five percent using a uniform random number.
average learner can acquire the knowledge and can consider the
The other conditions are the same as those in 4.4.
mental burden. However, these amounts may vary according to
the teacher. Figure 16 shows the result of classifying questions
Question number
15
Class
162
16
Class
179
161 149
171
145 156
166 175
167 168 177 178
172 181
169
11-2
182
Class
117
8-3
Class
6-4
Class
115 116
76
105
92
89
51
8-3
143
142 153
Class
104
63
72
74 86
84
80
68
64
22
1-2
10-2
23
41
45
21
Class
73
65
66
20
88
48
6-1
Class
87
85
57
52
46
36
39
24 25
54
55
59
56
60
62
96
120
15
32 33
14
3-1
42
119
6
11
1-1
173 163
43
121 134 135
3-2
Class
26
34
106
53
82
6-2
79
Class
16
77
8
7
18
5
35
31
93
94
17
95
Q44-Q66
7-3
Chapter 4
Q67-Q97
Chapter 5
Q98-Q105
Chapter 6
Q106-Q141
Chapter 7
Q142-Q162
Chapter 8
Q163-Q182
125 136
10-1
9-2
Class
2-2
Class
151 152
148
146
13
Class
158
147 157
5
126
107
83
29
78
Chapter 3
159 150 160
118
131
Class
9
4
10
28
40
4-2
133
Class
2-1
100 101 111 113
165 174
164
Q12-Q43
47
122 123
Class
Class
38
Class
3
132
Q1-Q11
Chapter 2
Class
7-2
4-1
2
98
Chapter 1
6-3
124
Class
37
Class
13
90
Legend
81
50
8-2
30
99
58
Class
19
8-1
Class
71
140 129
130 141
70
Class
44
112
114
102 103
67
61
27
180 170
176
91
49
14
Class
154 155
108 109 110
97
75
Class
144
9-1
69
7-1
Class
12
Class
128 127 137 138 139
11-1
Class
Class
12
1
Class
Figure 16. Result of simulation for classifying questions when amounts of effort are varied within five percent
Comparing Figures 9 and 16, we see that the results of
their achievement with a small load, and they can easily predict
classifying questions differ. The method of classifying
their achievement. So we believe that they will be encouraged
questions proposed in this paper gives different classifications
to study.
when the amount of effort differs. Therefore, it is a future
important challenge to develop a more robust method.
5. Conclusion
We proposed a task-given-type examination (TGTE) as an
4.9 Summary of simulation experiment
effective method to encourage learners to study by themselves.
In this simulation, we divided 182 questions of fluid mechanics
As criteria of evaluation for sampling questions, we proposed
into 16 classes. Seeing the result, we find that the additional
simplicity, predictability, and proportionality between effort and
amount of effort needed to acquire all questions in each class is
achievement. We showed that classifying questions satisfies
nearly equal. In addition, every question in the same class can
simplicity and predictability. In cases when the amount of effort
be answered with a nearly equal amount of effort (nearly equal
needed for each question differs and in cases when there is a
difficulty). Therefore, if we set questions according to the
relation between the knowledge elements needed for each
method of classifying questions proposed in this paper, by
question, we proposed a method of classifying questions
studying in order of class numbers, learners can surely improve
satisfying proportionality between effort and achievement. By
simulation in the field of fluid mechanics, we confirmed that
this is effective.
(8) Akahori, K.: "Grouping Questions in an Example of
Mathematics in Junior High School”, Journal of Japan
Educational Technology, Vol. 15, No. 2, pp. 57-71 (1991).
Acknowledgments
We sincerely thank Mr. Taga Aoki of Aoyama Gakuin
(9) Kuwabara, T.: "A method to make Procedural Study
Materials and its Evaluation”, Journal of Japan Educational
Technology, Vol. 20, No. 2, pp.133-140 (1996).
University for helping us to write this paper and to design the
(10) Sato, T.: "ISM Structural Studying Method”, Meiji Tosho,
experiment and Mr. Yasunori Kitajima of Aoyama Gakuin
Tokyo (1987).
University for helping us to write this paper.
(11) Sato, T.: "Determining a Hierarchic Structure of Studying
Elements by ISM method", Journal of Japan Educational
References
Technology, Vol. 4, No. 1, pp. 9-16 (1979).
(12) Nakamura, Y., Kuwabara, T. and Takeda, K.: "Analysis of
(1) Takeuchi, T. and Sakuma, A.: "Development of a
Factors in Determining Difficulty of Questions on Teaching
System to Support Making Questions in a Task-given-type
Material about Computer Language”, IEICE Trans., Vol.
Examination”, Preprints of Fall Congress of Japan Industrial
J-82-D-1, No. 5, pp. 644 654 (1999).
Management Association, pp. 242-243 (2000).
(2) Takeuchi, T. and Sakuma, A.: "Development of a
System to Support Introducing a System of an Examination in
(13) Morita, Y.: "Computations in Fluid Mechanics, in Series
of Computations of Machinery”, Tokyo Denki University Press,
Tokyo (1999).
a Task-given-type Examination”, Preprints of 6th Nation-wide
Congress of Association of Sciences Related to Educational
Technology, Vol. 2, pp. 363-364 (2000).
Toshihiko Takeuchi was born in
Meguro ward, Tokyo, in 1970. He
(3) Kyo, K. and Shigemasu K.: "Optimization of Order of
graduated from Aoyama Gakuin
Presenting Tasks by Item Response Theory and Expression of
University, College of Science and
Hierarchic Structure”, Japan Journal of Educational Technology,
Engineering,
Vol. 14, pp. 73-80 (1990).
Industrial and System Engineering in
Department
of
(4) Isomoto, Y., Yamasaki, H. and Yoshine, K.: "Scenario of
1993, and then Aoyama Gakuin
Courseware and Trial-and-error Study in Frame-type CAI”,
College of Science and Engineering, Graduate School of
Japan Journal of Educational Technology, Vol. 17, pp. 29-38
Management Engineering in 1996 and accomplished credits
(1993).
and withdrew from Aoyama Gakuin University, College of
(5) Saito, N. and Nakaura, S.: "Development of a System to
Science and Engineering, Department of Management System
Support Individual Study Adopting a Method of Selecting the
Engineering in 2001. He joined Aoyama Gakuin University,
Optimal Question”, Japan Journal of Educational Technology,
College of Science and Engineering, Department of
Vol. 22, No. 4, pp. 215-225 (1999).
Management System Engineering as an assistant professor in
(6) Kato, H. and Akahori, K.: "A Method of Selecting
2001. He joined the Tokyo Institute of Engineering as a
Questions for a System of Adaptive Questions by Bayesian
researcher at the Center for Research and Development of
Estimation”, IEICE Trans., J82-D-II (1), pp. 147-157 (1999).
Educational Technology in April 2005 specializing in
(7) Koizumi, N., Matsui T. and Takeya M.: "A Method to
Entertaining Education. Presently, he is a guest researcher of
Sample Questions for Review Reflecting Structure of Teaching
the Development Center of Human Resources for e-Learning
Materials”, Japan Journal of Educational Technology, Vol. 19,
(eLPCO) at Aoyama Gakuin University.
No. 1, pp. 1-13 (1999).
Akiyuki Sakuma graduated
from Keio University, Graduate
School of Engineering, with a
major
in
Engineering
Mechanical
in
1950,
and
joined Keio University, College
of Science and Engineering,
Management Engineering, as
an assistant professor in the
same year. He joined Aoyama Gakuin University, College of
Science and Engineering, Department of Industrial and System
Engineering as full-time lecturer in 1965, and was then
promoted to associate professor. He was assigned as Professor
of the Department of Management System Engineering at the
same University in 1976 and awarded Doctor of Engineering.
During that time, he held prominent positions including
member of the Science Council of the Ministry of Education,
chairman of the Japan Industrial Management Association and
provisional committee member on the Industrial Structure
Council. He studied global warming and environmental
management. He retired from the University, and is now
Professor Emeritus of the Department of Management System
Engineering of Aoyama Gakuin University.