QUALITY ITEMS – Part II
Writer
Consultants
Editor
: Bakri b. Baharom (CISEC PTSS)
: Abdulbrani b. Yunus (TPA PMU)
Engku Muhammad b. Engku Ali (JKM PKB)
: Azlida bt Ahmad (JPA PTSS)
It is said that in the old days, like during the time of 1Confucius (551 – 479 BC), the scholars’ tests
were not subjected to any curriculum. Could it mean that the scope of assessment items then was
anything between heaven and earth? How some scholars could excel in multiple unrelated disciplines is
miraculous. 2Avicenna (980 - 1037) was reputed to be an authority in medicine, philosophy, astronomy,
mathematics, physics, alchemy (chemistry), geology, psychology and poetry. 3Beethoven (1770 - 1827)
perfected his concerto although he had hearing disability. However, the greatest of all was 4Leonardo da
Vinci (1452 - 1519), the figure behind the mystifying smile of 5Mona Lisa, and hidden message in the
6
Last Supper. He was a painter, sculptor, architect, musician, scientist, mathematician and an engineer
who built a 7flying machine resembling a helicopter.
In modern times, it is said that the sum total of human’s knowledge double-up every ten years.
A friend put it in a very interesting perspective. At SPM level, you study SOMETHING about MANY
THINGS. That is why you hear the local examination syndicate has to limit the number of subjects
taken to 20. This is to avoid the nightmare of trying to accommodate the aspiring candidates. At
Diploma level, you continue to study SOMETHING about A FEW THINGS. At Degree level, you begin
to specialise, you study MANY THINGS about SOME THINGS. At Master level, you go deeper, trying
to understand EVERYTHING about ONE THING. The ultimate is when you pursue PhD, to know
EVERYTHING about NOTHING. Now, I understand why the acronym is also known as Permanent Head
Damage.
Who is Benjamin Samuel Bloom (1913–1999), or just Bloom? He was an American educational
psychologist who anchored the classification of learning objectives which came to be known as
Bloom’s Taxonomy (1956). The taxonomy is divided into three domains; namely COGNITIVE,
AFFECTIVE and PSYCHOMOTOR. The Cognitive deals with Knowing (head), Affective deals with
Feeling (heart) and Psychomotor deals with Doing (hands). A holistic education means the
curriculum covers the three domains. The Cognitive Domain, which concerns thinking and
knowledge, is divided into six levels; namely Knowledge, Comprehension, Application, Analysis,
Synthesis and Evaluation. In our polytechnic curriculum system these are written as C1, C2, C3, C4,
C5 and C6 respectively. The C1 and C2 are considered to be LOWER ORDER levels. Thus, HIGHER
ORDER levels begin from C3 to C6. Final Examination would only involve Cognitive Domain type of
items. Now, there is a variation of the six classification terms used; as in the Table 1 below. The
change is proposed by Lorin Anderson (2001). Note also that C5 and C6 are interchangeable. Using the
taxonomy will help us think clearer, employing a standardised framework and common vocabulary to
discuss items.
1
BLOOM
ANDERSON
C1
Knowledge
Remembering
C2
Comprehension
Understanding
C3
Application
Applying
C4
Analysis
Analyzing
C5
Synthesis
Evaluating
C6
Evaluation
Creating
Table 1
The next step is to understand the taxonomy levels from C1 (the easiest) to C6 (the most difficult).
A search on the net should produce some explanation as in the following Table 2:
(C1)
KNOWLEDGE
- defined as the
remembering of
previously
learned material,
the recall of the
wide range of
material, from
specific facts to
complete
theories.
(C2)
COMPREHENSION
- defined as the
ability to grasp the
meaning of material:
translating material
from one form to
another by
interpreting material
and by estimating
future trends.
(C3)
APPLICATION
- refers to the
ability to use
learned material
in new and
concrete
situations,
application of
rules, methods,
concepts,
principles, law
and theories.
(C4)
ANALYSIS
- refers to the
ability to break
down material
into its
component
parts so that its
organizational
structure can
be understood.
(C5)
SYNTHESIS
- refers to the
ability to put
parts together
to form a new
whole, pattern
proposal or
structure.
(C6)
EVALUATION
- concerns
with the ability
to judge the
value of
material based
on definite
criteria for a
given purpose.
Table 2
How does the level of studies correspond to the Cognitive levels? The general consensus is,
DIPLOMA level should be at C3, DEGREE at C4 and MASTER at C5. However, it does not strictly mean
that a Diploma level cannot exceed C3. It is just that the most dominant item will be at level
C3, with some share of lower and higher level questions. For example, to make C3 dominant,
the percentage of item belonging to this level must be about 60%. However, our Curriculum
Department has decided that; firstly, if the curriculum states that the topic is for C3, then we
MUST NOT EXCEED C3. Secondly, if it is set at C4, it means we may include item at C4 but the
dominant number of questions (and marks too) STILL COME FROM C3 with the rest from C1
and C2.
C3
A real understanding of cognitive levels is of utmost important in setting quality items. Let’s
take a look at our curriculum Course Learning Outcome (CLO) and Final Examination Item Specification
Table (FEIST) with regards to the topics in the syllabus (Table 3).
Course: FP101- Programming Principles
Final Examination Format: Objective (40); Subjective (Structured) (2)
COURSE LEARNING OUTCOME (CLO)
Upon completion of this course, students should be able to:
CLO1 Explain the basic computer and programming fundamentals with appropriate examples of language
and technology. (C2)
CLO2 Apply the different types of algorithm to solve problem efficiently. (C3,P4)
CLO3
Solve problem effectively by applying related theories of the basic programming language to a given
particular scenario using programming life cycle. (C4,A2,P4)
CONTEXT
COURSE LEARNING
OUTCOME
2
TAXONOMY
DOMAIN
TYPE OF ITEM
Topic
CLO1
CLO2
CLO3
√
√
√
√
√
√
Introduction To Programming Language
√
Problem Solving Method
√
Fundamentals of Programming Language
√
Basic Programming Codes
√
TOTAL ITEM
Cognitive
Level
Objective
(40)
C2
C3
C4
C4
3
13
13
11
40
Subjective
(Structured)
(2)
Q1
Q2
2
Table 3
For FP101 course, the committee has decided to have 40 Objective and 2 Structured type of
item. The three Course Learning Outcomes (CLO) consist of Cognitive Levels ranging from C2 to C4. The
table also shows the topics and its related CLOs, cognitive level, the number of Objective questions for
each topic and the combination of topics for the two Structured items. Those given the task to set item
for FP101 will have to follow this specification. Consider the following convention (Table 4) in writing
Cognitive levels for Diploma level courses:
C1 – C3
C1 – C4
C2
C1, C3
Items CAN BE from C1 to C3, but C3 MUST be the most dominant.
Items CAN BE from C1 to C4, but C3 MUST be the most dominant.
Items MUST BE C2 only.
Items MUST BE C1 and C3 only.
Table 4
One good practice to produce quality item is to ensure that the stem consists of 3 parts; namely
VERB, CONTENT and STANDARD (VCS). The following are two sample questions based on VCS (Table 5):
1.
2.
VERB
State
Calculate
CONTENT
THREE (3) computer components that use the
the moment so that the bar is in equilibrium
STANDARD
South and North Bridge.
to within one decimal point.
Table 5
The words STATE and CALCULATE are verbs associated with Cognitive levels. Look at the
following Table 6 for a short list of 10 action verbs for each Cognitive level in alphabetical order. Refer
to a longer list for more action verbs.
(C1)
KNOWLEDGE
Count
Define
Draw
Label
List
Match
Name
Recall
State
Write
(C2)
COMPREHENSION
(C3)
APPLICATION
(C4)
ANALYSIS
Classify
Compare
Compute
Describe
Differentiate
Distinguish
Interpret
Rewrite
Summarize
Translate
Apply
Calculate
Choose
Conduct
Demonstrate
Illustrate
Implement
Modify
Produce
Solve
Analyze
Compare
Deduce
Differentiate
Examine
Explain
Identify
Illustrate
Investigate
Relate
Table 6
3
(C5)
SYNTHESIS
Arrange
Build
Combine
Compile
Construct
Create
Develop
Generalize
Prepare
Produce
(C6)
EVALUATION
Appraise
Argue
Assess
Contrast
Evaluate
Grade
Interpret
Judge
Justify
Recommend
The verb STATE in the first example tells us that it is a Knowledge item. The second example
uses the verb CALCULATE that belongs to Application level. Select the verbs meticulously as some verbs
may belong to more than one cognitive level. As we move higher in the cognitive level, from C1 to C6
we notice the increased degree of difficulty in responding to the task. IMPORTANT: Avoid the following
verbs (Table 7) as they are not suitable for specific outcome as they sound very general.
Know
Appreciate
Learn
Enjoy
Understand
Do
Realize
Table 7
Another tricky problem is when we have to specify the item to be one of the three categories in
term of difficulty; LOW, MEDIUM and HIGH. On what basis do we say that the question is easy or
difficult? Does it depend on the cognitive level? Question like,
What is the name of our capital city?
is definitely C1 and LOW. What about question like;
What is so special with our capital city?
is also C1 but the difficulty is HIGH. Do we just base our judgement on guess work or feel alone? Is the
ratio 5:3:2 (Low:Medium:High) in line with the spirit of having C3 as Diploma level ideal weightage?
These questions beg answers from our 7,570 (as of April 2012) lecturers in the polytechnic system.
Let’s revisit the Objective type item. It is said that it is more suitable to cover
items of lower level like C1 and C2. We are not obliged to use verbs as in Table 6. More
often than not, word-questions like WHAT, WHICH, WHEN, WHERE and WHY are used.
However, do not overuse any of the word-questions because it will make our item set
monotonous. The 9 word-questions are as follow:
WORD
What
PURPOSE
Things/Person
EXAMPLES
What are the five (5) main components of a computer?
Which
Things/Person
Which of the following is a form of DeMorgan’s Theorem?
Who
Person
Who is considered to be the first inventor of microchips?
Whose
Person
Whose ideas are sequencing, looping and repetition?
Whom
Person
Whom should you speak to when the computer system is down?
When
Time
When doing problem analysis, what are the three (3) items needed?
Where
Place
Where does a computer add and compare data?
Why
Reason
Why is JK flip-flop better than clocked SR flip-flop?
How
Method
How does clock play a role in computer processing?
Table 8
4
The objective items below are taken from the Final Examination, June 2011. Are they quality
questions? Do they follow the principles of objective items? What further Continuous Quality
Improvement (CQI) could you think of?
Q1.
Which of the following is NOT a merit of free market economy? (Q3, PB102-Microeconomics)
A. Social welfare is prioritized
B. Economic efficiency is achieved
C. Economic decisions are made quickly
D. Development in technology and innovation are encouraged
Q2.
When a party has performed all of their obligations in accordance to the terms of the contract
that party has discharged the contract by
(Q3, PB302-Commercial Law)
A. mutual discharge C. operation of law
B. agreement
D. performance
Q3.
What is the function of an interpreter?
(Q2, F1029-Programming Principles)
A. To detect syntax error for high level language
B. To translate the source code to machine language
C. To convert assembly language to machine language
D. To translate a high level language to machine language
Q4.
Where is a flip-flop used?
i) Register
ii) Memory Device
iii) Switch
(Q30, F1030-Digital Logic)
iv) Counter
A. i, ii, iii
C. ii, iii, iv
D. i, iii, iv
Q5.
B. i, ii, iv
Why does an ISP require a DSLAM device on their network? (Q6, F2034-Network Fundamentals)
A. To connect end users using cable technology
B. To connect end users using analog technology
C. To connect end users using wireless technology
D. To connect end users using digital subscriber line technology
…..to be continued…..
** Note: Quality Items: Part I is available at www.ptss.edu.my (Penulisan Artikel)
5