Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
WORKFORCE EDUCATION DEVELOPMENT OFFICE
Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
The Workforce Education Development Office (WEDO) was established in January 2007 to provide research and leadership, deliver professional development, and establish a
state skills assessment system that meets the needs of Career and Technical Education (CTE) in Arizona. The outcome ensures industry-validated standards for all CTE
programs and an online skills assessment delivery system that serves secondary, comprehensive Career and Technical Education, Joint Technological Education Districts, and
Community Colleges.
Working principally with the Arizona Department of Education (ADE) Career and Technical Education unit and the Arizona Skill Standards Commission, WEDO collaborates
with multiple in-state and out-of-state entities to strengthen human capital development within the Arizona education system. The Arizona CTE delivery system includes Joint
Technical Education District (JTED) comprehensive high schools within unified and union high school districts in the state, and other state-funded CTE providers. Articulation
between secondary and post-secondary institutions (community colleges, trade and technical schools, and four-year colleges) is required by both state and federal mandates
and within the WEDO scope of work.
WEDO works closely with Arizona business and industry to validate skill standards and measurement criteria, the basis of assessments developed to measure student skills in
50+ CTE programs. The Arizona Skill Standards Commission (ASSC) was established to validate and recognize state-established skill standards that guide curriculum
development, skill assessment specific to CTE programs, and documentation of skill attainment for students preparing to enter the workforce and/or transitioning to postsecondary education. The ASSC awards certificates and transcripts to students who complete a CTE program and pass technical skill assessments.
The Workplace Employability Skills (WES) model and rubrics have been developed under ASSC leadership and are currently positioned for implementation and further
development. WEDO is also engaged in academic crosswalks of common core standards as applied in the context of CTE. WEO is currently involved in a proof-of-concept
study regarding adaptation of assessments for other uses (e.g., diagnostic, guidance, benchmarking learning) and users (i.e., Arizona Department of Economic Security
American Job Centers – formerly WIA One-Stop Centers).
STAFF:
Maggie Mangini, Executive Director, [email protected], 602.827.2332
Robert Keim, Associate Director, [email protected], 602.827.2330
Judy Balogh, Project Director, [email protected], 602.827.2328
Melanie Ohm, Senior Program Coordinator, [email protected], 602.827.2331
Lynne Bodman Storms, Senior Program Coordinator, [email protected], 602.827.2329
MAILING ADDRESS:
Workforce Education Development Office
University of Arizona
714 E Van Buren Street
Phoenix, AZ 85006-3364
FAX: 602.827.2304
GUIDELINES FOR WRITING MULTIPLE CHOICE ITEMS
The item—
Options:
1. — is based on a valid skill (measurement criterion.)
1. Each item has four options.
2. — uses correct grammar, punctuation, and spelling.
2. The options are homogeneous in content.
3. — is not a trick item that would mislead or deceive
students into answering incorrectly.
3. The length of the options is fairly consistent.
4. — uses either the best answer or the correct answer
format.
4. Options are phrased positively, not negatively.
5. — does not have textbook, verbatim phrasing.
5. There are no clues through the use of grammatical
construction. (For example, grammatical inconsistencies
involving "a," or "an" give clues to the correct answer.)
6. — is based on important aspects of the content area not
on trivial material.
6. The phrase “none of the above” is never used because the
options are randomized.
7. — does not contain insensitive content or language.
The stem of the item—
Distractors—
1. — is stated as a question.
1. — are plausible and logical.
2. — is clearly worded and lets the student know exactly
what is being asked.
2. — incorporate common errors of students.
3. — includes enough information to make the question clear.
Additional considerations:
3. — use familiar yet incorrect phrases.
4. — may be true statements but do not correctly answer the
item.
The correct option—
1. If an item is stated negatively, the negative word is
capitalized.
1. — is randomized so that it appears about the same number
of times in each possible position for a set of items.
2. The phrase “all of the above” is never used.
2. — is the one and only correct, or clearly best, answer on
which experts would agree.
3. Specific determiners such as “never” and “always” are not
used
1 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
WORKSHEET FOR MULTIPLE CHOICE ITEMS
(Program Title)
(Developed by)
Standard/Measurement Criterion: (Use standard number and measurement criterion number, e.g., 1.1; 2.3; 10.2)
Bloom’s Classification: (Use Knowledge Dimension and Cognitive Process Dimension, e.g., A1, B3, C4)
Question:
Correct Answer:
a)
Plausible Incorrect Options:
b)
c)
d)
If above item requires a graphic/image—
•
•
•
•
Attach a copy or provide it in electronic format, e.g., GIF, PING, or JPEG—5” to 6”.
If from a copyrighted source, cite the source and provide written permission to use the image.
Images may need to be found and scanned.
Make a rough drawing that can be turned into good line art with dimensions, arrows, etc.
2 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
—
—
The Cognitive Process Dimension
The Knowledge
The
Knowledge
Dimension
Dimension
A. Factual Knowledge
The basic elements students must know to be able
to solve problems, i.e., terminology, specific
details, and elements.
1. Remembering
2. Understanding
List
Summarize
Respond
Select
Check
Recognize
Classify
Provide
Differentiate
Determine
Recall
Clarify
Carry out
Integrate
Judge
Retrieve knowledge from
long-term memory
(recognizing, recalling)
Construct meaning from
instructional messages
using oral, written, and
graphic communication
(interpreting, exemplifying,
classifying, summarizing,
inferring, comparing)
3. Applying
Carry out or use a
procedure in a given
situation (executing,
implementing)
4. Analyzing
Breakdown material into
its constituent parts and
determine how the parts
relate to one another and
to the overall structure or
purpose (differentiating,
organizing, attributing)
5. Evaluating
6. Creating
Make judgments based
Put elements together to
on criteria and standards form a coherent or
(checking, critiquing)
functional whole;
reorganize elements into
a new pattern or
structure (generating,
planning, producing)
B. Conceptual Knowledge
The interrelationships among basic elements
within a larger structure that enable them to
function together, i.e., classifications and
categories; principles and generalizations;
theories, models, and structures.
C. Procedural Knowledge
How to do something, methods of inquiry, and
criteria for using skills, algorithms, techniques, and
methods, i.e., subject-specific skills and
algorithms; subject-specific techniques and
methods; and criteria for determining when to use
appropriate procedures.
D. Metacognitive Knowledge
Knowledge of cognition in general as well as
awareness and knowledge of one’s own cognition,
i.e., strategic knowledge; contextual and
conditional knowledge; and self-knowledge.
KEY:
YELLOW – CAUTION
Necessary foundational knowledge and understandings
GOLD – GO FOR THE GOLD
Target area for item development/summative assessments
GRAY – OUT OF BOUNDS
Exceeds limitation of multiple-choice assessment items
Note: The verb in each block is provided as an aid to help individualize and differentiate each classification, i.e., A1 vs. B1 vs. C1, and so forth.
Adapted from: A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives, edited by Lorin W. Anderson, David R. Krathwohl, Peter W. Airasian, Kathleen A. Cruikshank, Richard,
E. Mayer, Paul R. Pintrich, Jams Raths, and Merlin C. Wittrock. Addison Wesley Longman, Inc., New York, 2007. (ISBN 0-8013-1903-X)
3 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
Examples of Verbs for the Cognitive Process Dimensions
1. Remembering
2. Understanding
3. Applying
4. Analyzing
5. Evaluating
6. Creating
Retrieve knowledge from longterm memory (recognizing,
recalling)
Construct meaning from
instructional messages using oral,
written, and graphic
communication (interpreting,
exemplifying, classifying,
summarizing, inferring, comparing,
explaining)
Carry out or use a procedure in
a given situation (executing,
implementing)
Breakdown material into its
constituent parts and determine
how the parts relate to one
another and to the overall
structure or purpose
(differentiating, organizing,
attributing)
Make judgments based on
criteria and standards
(checking, critiquing)
Put elements together to
form a coherent or functional
whole; reorganize elements into
a new pattern or structure
(generating, planning,
producing)
Achieve
Advertise
Analyze
Appraise
Categorize
Classify
Compare
Contrast
Critique
Debate
Deduct
Determine
Diagnose
Distinguish
Differentiate
Distinguish
Examine
Explain
Identify
Infer
Investigate
Research
Separate
Specify
Appraise
Assess
Compare
Conclude
Critique
Debate
Decide
Defend
Determine
Envision
Estimate
Evaluate
Examine
Judge
Justify
Prioritize
Rank
Rate
Recommend
Review
Select
Support
Verify
Weigh
Actualize
Change
Combine
Compose
Construct
Create
Design
Devise
Forecast
Formulate
Generate
Hypothesize
Imagine
Invent
Originate
Plan
Predict
Pretend
Produce
Propose
Arrange
Define
Describe
Draw
Duplicate
Identify
Label
List
Locate
Match
Memorize
Name
Order
Recall
Recognize
Relate
Reproduce
Restate
Select
Spell
State
Tell
Underline
Characterize
Classify
Compare
Complete
Convert
Demonstrate
Describe
Discuss
Distinguish
Establish
Execute
Explain
Express
Generate
Interpret
Order
Outline
Paraphrase
Predict
Report
Restate
Summarize
Translate
Visualize
Administer
Apply
Calculate
Change
Choose
Classify
Complete
Conclude
Construct
Demonstrate
Determine
Employ
Examine
Experiment
Illustrate
Interpret
Manipulate
Modify
Operate
Perform
Practice
Produce
Solve
Use
4 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
CULINARY ARTS
KNOWLEDGE DIMENSION
A. Factual Knowledge
B. Conceptual Knowledge
C. Procedural Knowledge
Total:
1. Remembering
181 / 61.6%
10 / 3.4%
17 / 5.8%
208 / 70.7%
2. Understanding
21 / 7.1%
8 / 2.7%
2 / 0.7%
31 / 10.5%
3. Applying
1 / 0.3%
1 / 0.3%
40 / 13.6%
42 / 14.3%
4. Analyzing
1 / 0.3%
4 / 1.4%
5 / 1.7%
10 / 3.4%
5. Evaluating
0 / 0%
1 / 0.3%
2 / 0.7%
3 / 1.0%
Total
204 / 69.4%
24 / 8.2%
66 / 22.4%
294 / 100%
2. Understanding
21 / 10.9%
35 / 18.2%
12 / 6.3%
68 / 35.4%
3. Applying
4 / 2.1%
5 / 2.6%
19 / 9.9%
28 / 14.6%
4. Analyzing
6 / 3.1%
19 / 9.9%
3 / 1.6%
28 / 14.6%
5. Evaluating
1 / 0.5%
14 / 7.3%
5 / 2.6%
20 / 10.4%
Total
64 / 33.3%
85 / 44.3%
43 / 22.4%
192 / 100%
ENGINEERING SCIENCES
KNOWLEDGE DIMENSION
A. Factual Knowledge
B. Conceptual Knowledge
C. Procedural Knowledge
Total:
1. Remembering
32 / 16.7%
12 / 6.3%
4 / 2.1%
48 / 25.0%
5 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
UNDERSTANDING ASSESSMENT ITEM ANALYSIS
What is item analysis? Item analysis is the process of examining the performance of individual test items…or more simply put, identifying test items that
are not working well. Item analysis consists of three types of statistics or information that we use to improvement an item. This activity leads to item reliability
(how consistently a measurement of skill or knowledge yields similar results under varying conditions) and validity (the degree to which the instrument
measures what it’s supposed to measure over time):
1.
Difficulty Index. The difficulty index is determined by calculating the number of students who got an item correct (p value). A large number indicates
that many students have learned the content as measured by the item. When none of the students chooses the correct answer, the p value is 0.
When all of the students choose the correct answer, the p value is 1.0. (Note. The name of this index is counter-intuitive, as one actually gets a
measure of how easy the item is, not a measure of how difficult the item is.)
2.
Discrimination Index. The discrimination index measures the validity of an item. It shows the discrimination between those students who know the
content and those who do not know the content. This index is an indication of the extent to which overall knowledge of the content area or mastery
of the skills is related to the response on an item. Perhaps the most crucial validity standard for a test item is whether a student got an item correct
due to his level of knowledge or ability and not due to chance or test bias. A discrimination value of .3 and above indicates that the item accurately
discriminates between those who know and those who do not know the content. High values indicate good items; low indicate poor items.
An item is flagged when it falls below these parameters:
Difficulty Index of .45 with Discrimination Index of .3
or
Difficulty Index of .5 with Discrimination Index of .25
EXAMPLE #1: Difficulty Index: 0.76 / Discrimination Index: 0.26
(This item is not flagged.)
Which teaching method accommodates the auditory learner?
*156 (77.61%) – Singing the alphabet
29 (14.43%) – Writing the alphabet
9 (4.48%) – Coloring the letters of the alphabet
8 (3.98%) – Arranging magnetic letters of the alphabet
EXAMPLE #2: Difficulty Index: 0.3 / Discrimination Index: 0.16
(This item is flagged.)
What is the BEST exercise for preventing bone density loss in an elderly person?
*69 (30.53%) – Weight and resistance training
80 (35.4%) – Aerobic exercise
49 (21.68%) – Range of motion exercises
30 (13.27%) – PHF exercises
6 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
3.
Distractor Analysis. By calculating the proportion of students who chose each option, we can identify which distractors are "working" and which
distractors are simply taking up space and not being chosen by many students. To eliminate guessing that results in a correct answer purely by chance
(which hurts the validity of a test item), we want plausible distractors. Analyses of response options allow us to fine tune and improve items.
Some Things to Remember:
If an item is very easy so that nearly all students answered correctly, the discrimination will be near zero. Extremely easy
questions cannot distinguish among students in terms of their performance.
If an item is extremely difficult so that nearly all students answered incorrectly, the discrimination will be near zero.
The most effective items will have moderate difficulty and high discrimination values. The higher the value of discrimination,
the more likely it will be to distinguish between those students who perform well on the test and those who don’t.
Items with low or negative values of discrimination need to be reviewed for confusing language or an incorrect indicator.
A high level of student guessing will result in a discrimination value near zero.
7 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
REVIEWING/EDITING ASSESSMENT ITEMS
The question
Ensure that the right question is being asked in its simplest form and that it is matched to the correct standard and measurement criterion.
Improve the clarity and meaning of a question by using present or action conditions (what is; what are) rather than passive or progressive construction
(what would be; what should be; what will be).
Change “which of the following” to “which” or “what” plus a noun (e.g., term, issue, technique, statement, factor, action, purpose, step, etc.) and
eliminate “of the following” OR
rewrite the question entirely.
Capitalize words such as NOT, TRUE, USUALLY, COMMONLY, and PRIMARY.
*Note: Quotation marks and capped letters can be used; however, italics and symbols cannot not be used.
Distractors
Match structure (parallel structure), content, plausibility, and length of distractors.
Eliminate articles (a, an, and the) introducing distractors.
Remove “and” between items listed in a series as a distractor, when plausible.
Use the logical flow in answering the question to determine whether sentences, phrases, or gerunds are needed.
Eliminate same word used in distractors if the word is also used in the stem (e.g., What type of cells are used during tissue culture? a. Meristem cells; b.
Leaf cells; c. Stem cells; d. Root cells
Check for and change key words that appear in the question and are reused in distractors.
8 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
Overall
Remove unnecessary information/description in both the question and the distractors.
Use “that” for a restrictive clause (information that in essential to the understanding) and “which” for a nonrestrictive clause (information that is
nonessential to the understanding).
•
•
The college that Sonja has decided to attend is located in Michigan. (Restrictive: no commas needed)
Washtenaw Community College, which Sonja has decided to attend, is located in Michigan. (Nonrestrictive: commas needed)
Remove “you” in both stem and distractors.
Always use numbers with time, weight, and measurements. Spell out single-digit whole numbers and use numerals for numbers greater than nine. If a
numeral is used because one of the numbers is greater than nine, use numerals for all numbers in the question and/or distractor.
EXAMPLES OF HOW TO WRITE AND FORMAT ITEMS
(Don’t repeat “oven” in options.)
What kind of oven uses forced circulating air?
a. Convection
b. Conventional
c. Revolving
d. Deck
(Capitalize “not.”)
What information is NOT found on a balance sheet?
a. Revenue
b. Liability
c. Net worth
d. Asset
9 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
(Capitalize words such as “first” and “best.”)
What is the FIRST step when preparing to key a hand-written document?
a. Review the document to determine format.
b. Read the document thoroughly for content.
c. Key a rough draft for the document.
d. Proofread the document for errors.
(Each option is a complete sentence following by a period.)
What is the BEST method for determining seasonal sales promotion strategies?
a. Collect data on buying trends of the target market.
b. Place certain items on sale and calculate total revenue.
c. Review newspaper ads for local competition.
d. Circulate product coupons and track their redemption.
(Each option is parallel in structure and size.)
What is the BEST use of this tool?
a.
b.
c.
d.
Pushing and pulling
Prying and cutting
Cutting and pulling
Striking and prying
(Use a gerund* for all four options.) * A gerund is a verb ending in – ing and functioning as a noun.)
Which is a developmentally appropriate visual art activity for a 2-year-old?
a. Finger-painting with primary colors
b. Making a Mother’s Day card
c. Cutting out paper shapes
d. Coloring worksheets
10 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
EXAMPLES OF HOW TO IMPROVE ITEMS
#1. Problem: One question provides the answer to another question.
Question 1. A notarized document is unique because of what feature?
a. Seal
b. Commission
c. Signature
d. Licensure
Question 2. Which statement is an accurate reflection of the notary process?
a. The notary watches as the document is signed, affixes a notary seal to the document, and fills in the required information.
b. The notary checks drivers' licenses to verify the identity of the individuals after they sign the document.
c. The notary affixes a seal the document 24 hours after the individuals sign it.
d. The notary affixes a seal before the individuals sign the document.
Modified: The answer (“a”) in Question 2 was modified so as not to giveaway the answer in Question 1.
Which statement is an accurate reflection of the notary process?
a. The notary watches as the document is signed, marks it, and fills in the required information.
b. The notary checks drivers' licenses to verify the identity of the individuals after they sign the document.
c. The notary marks the document 24 hours after the individuals sign it.
d. The notary proofs and marks the document before the individuals sign it.
#2. Problems: The correct answer is more specific than the distractors, and the pronoun “you” is used.
While experiencing difficulties connecting to the internet, you notice that the IP number starts with "169." What has happened?
a. The computer was unable to get an IP number from the DHCP server and has defaulted to its Automatic Private IP Addressing (APIPA) number.
b. Nothing is wrong at the client end. The fact that you have an IP number means that you have connected to DHCP and the problem is with the server side
(ASP).
c. You probably have a virus that is blocking your IP connection to the server.
d. There is a problem with the Domain Name System (DNS) server resolving your IP number to a domain name.
Modified: The answer and distractors are equally specific, and the pronoun “you” is eliminated.
A web developer is experiencing difficulty connecting to the internet and notices that the IP number starts with "169." What has happened?
a. When the computer was unable to acquire an IP number, it defaulted to its Automatic Private IP Addressing (APIPA) number.
b. When the computer acquired the IP number, the server side (ASP) crashed and sent a faulty address.
c. The code indicates that a virus is blocking IP connection to the server.
d. The code indicates an issue with the Domain Name System (DNS) server resolving the IP number to a domain name.
11 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
#3. Problem: The stem leads the reader to the correct answer.
What type of search engine crawls the internet looking for recently created pages and/or updates?
a. Spider-based
b. RSS
c. Directory
d. Pay per click
Modified: The stem is adjusted to less “animated” language.
What type of search engine sifts through the internet identifying recently created pages?
a. Spider-based
b. RSS
c. Directory
d. Pay per click
#4. Problem: The conjunction “and” is used in lists.
What is an example of an email client configuration?
a. SMTP and POP3
b. FTP and POP3
c. FTP and SMTP
d. STMP and HTTP
Modified: The conjunction is eliminated.
Which pair refers to an e-mail client configuration?
a. SMTP, POP3
b. FTP, POP3
c. FTP, SMTP
d. STMP, HTTP
12 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013
#5. Problems: The assessment item is used to teach: the phrase “which of the following” and non-essential articles are used.
Server-side scripting, e.g., ASP, CGI PHP, etc., is used to generate dynamic web pages on which of the following?
a. The web server
b. The web browser
c. The client's computer
d. The network
Modified: Extraneous information and extra words are eliminated.
Server-side scripting is used to generate dynamic web pages in which location?
a. Web server
b. Web browser
c. Client's computer
d. Network system
#6. Problem: Passive “would be” is used when an active “is” is more direct. “Would be” is used also when “should be” is intended.
What would be the PRIMARY purpose of DHCP?
a. It assigns IP addresses.
b. It assigns protocol instruction to a server.
c. It assigns domain names automatically.
d. It assigns a new user name.
Modified: Question is in active voice. Distractors are shortened in length by using gerunds.
What is the PRIMARY purpose of DHCP?
a. Assigning IP addresses
b. Assigning protocol instructions to a server
c. Assigning domain names automatically
d. Assigning user name
13 | Arizona CTE Skill Standards and Assessment System / Arizona Department of Education in Partnership with the University of Arizona / May 2013