Saylor Academy’s Assessment Creation Guidelines Updating Objectives for Assessment Purposes Saylor’s existing learning outcomes were created to serve as guideposts during content pairing and assessment development. Ongoing learning outcome updates serve to improve the outcomes for assessment purposes. Saylor’s internal instructional design staff complete initial suggested revisions to existing learning outcomes and pass these new assessment objectives off to an SME. In addition to any specific questions noted in the learning objective document, SMEs should consider evaluating the objectives based on the following questions: ● Are the revised objectives appropriate for the course content and level of the course? ● Are the revised objectives of the same relative importance? (indicate any which are more or less important) ● Is the collection of objectives inclusive of the most important things a student should get from a course? ○ If there are missing objectives, are those objectives ​
best​
assessed via multiple­choice final exam? Once the updated objectives are finalized, assessment creation can begin in full. Assessment Categories Checks for Understanding:​
More informal quizzes, guided activities, or practice that primarily serve to facilitate content learning or other formative processes. These assessments are the most narrowly focused assessments and may be aligned to a single outcome or to practice with low­levels of Bloom’s taxonomy. The goal of these assessments is to engage students and support mastery. End of Unit Assessments (EOUAs):​
More formal, 10­20 item auto­graded quizzes that come at the conclusion of each unit. These assessments should be unit summative, and preparatory for the final exam. EOUAs include explanatory feedback that accompanies the answer choices (​
example​
). This feedback is provided once students have submitted their answers and serves to help students understand how or why their answers are correct or incorrect. Note that the explanation a student sees is contingent upon their answer selection, and each explanation should function as its own whole. Final Exam:​
This is the only component of a Saylor course that carries an official grade. Saylor final exams are course­summative and made up of multiple­choice items. Students must earn a 70% or higher on the exam to pass, and they may take the exam as many times as they’d like, with an enforced waiting period of 14 days between attempts. The goal of the final exam is to assess a student’s mastery of stated learning objectives. Final exam items should not include explanatory feedback. Typical Item Types Some items types will be more appropriate to assess particular learning objectives. Try to use a variety of types, but ensure that the item type is an appropriate match to the aligned objective. ● Multiple choice items should have 4 responses (one correct answer). ​
example ● Multiple response items should have 5 responses (2 correct answers). ​
example ● Open ended (fill­in the blank) questions for numerical input, dates, short vocabulary terms. Stems of this type should indicate the appropriate answer format (e.g. Provide your answer rounded to the nearest tenth.). This item­type should only be used when there is a very limited set of acceptable answers. If there are many possible answers that could be correct, a multiple choice/response is more appropriate. ​
example ● Matching questions for parallel concept. ​
example ● Questions with exhibits (scenarios, images, graphs) are used to expand assessment options, but exhibits must be relevant and openly licensed. ​
example Minimum Item Standards ● Each item should measure a single, clearly defined objective ● Items should be the best measure of whether a learning objective has been achieved. In other words, items should be relatively easy/straight forward for students who have learned the material. They should be difficult for students who have not learned the material. ● Items should be written at a language that is at or below the average reading level of the students targeted by the assessment (about 40% of Saylor students identify a language other than English as their first) ● Items should be stated positively (e.g. avoid “which is NOT” and similar) ● Items should be clearly written, unambiguous, and free of cultural bias ● Avoid trick questions ● Response should be of equivalent length and arranged in a logical order (by length, alphabetically, etc.) ● Avoid true/false; yes/no; all/none of the above, A and C, etc. as answer choices ● Incorrect choices should all be plausible and of equivalent plausibility ● Multiple response items should indicate the number of answers in the stem ● Exhibits should relate directly to the test question and not provide distracting or extraneous information ● Avoid items that may disclose answers to other items in the same test Example 1: Explanatory feedback The interest rate charged on financial aid is _______ data. A. qualitative B. qualitative discrete C. quantitative discrete D. quantitative continuous ANSWER: D EXPLANATIONS: A. Qualitative data is indicated by labels, not numbers, but interest rate is numerical. B. There is no such thing as qualitative discrete data. C. Quantitative discrete data is counted, but an interest rate is measured. D. An interest date is numerical data that can be measured, so it is quantitative continuous. OUTCOMES: 1a Example 2: Multiple choice (four responses, one correct answer) DO According to Locke in “The Second Treatise on Civil Government,” where does government’s legitimacy come from? A. Blood B. Divine right C. The general will D. The consent of the governed Which of the following principles best illustrates Machiavelli’s approach to politics? A. Public and private morality are two different things B. Traditional authority needs to be maintained at any cost C. The best way to unify a polity is to focus it on internal or external enemies D. A new prince must ruthlessly root out corruption in order to stabilize society DO NOT According to Locke in “The Second Treatise on Civil Government,” where does government’s legitimacy come from? A. Divine right B. Blood C. The consent of the governed D. None of the above According to Machiavelli, politics focuses on what? A. Power B. People C. Conflict D. Economics Example 3: Multiple response (five responses, two correct answers) DO Which of the following are types of political authority defined by Max Weber? (Choose two.) A. Moral B. Divine C. Traditional D. Managerial E. Charismatic According to Aristotle, which of the following represent “perversions” of virtuous forms of government? (Choose two.) A. Royalty B. Oligarchy C. Democracy D. Aristocracy E. Constitutional government DO NOT According to Aristotle, which of the following represent “perversions” of virtuous forms of government? A. Royalty B. Oligarchy C. Democracy D. B and C E. All of the above Example 4: Open­ended, short answer DO What subfield of politics considers the political relationships and interactions among countries, including the causes of war, the formation of foreign policy, international political economy, and the structures that increase or decrease the policy options available to governments? (Your answer should be two words. Answer not case­sensitive.) Possible answers: International relations, international affairs, foreign affairs, etc. Calculate the area of a circle with radius 3.5 cm. Please use 3.14 to represent the variable ​
pi and express your answer as a decimal to the nearest hundredth. You do not need to include the unit marker in your answer. ANSWER: 38.47 DO NOT Find the second derivative of f(x) = ­3x^4+7x^2­(⅙
)x­1. Example 5: Matching question DO Match each branch of the US federal government with the role it plays with regard to laws. A. Executive
1. Creates laws B. Legislative
2. Interprets laws C. Judicial
3. Enforces laws DO NOT Match the field or subfield to its description. A. The most common field of study; its subfields include public opinion, elections, national government, and state, local, or regional government. B. Studies constitutions, legal systems, civil rights, and criminal justice. C. Studies the role of the bureaucracy. It is the field most oriented toward practical applications within political science and is often organized as a separate department that prepares students for careers in the civil service D. Considers the political relationships and interactions among countries, including the causes of war, the formation of foreign policy, international political economy, and the structures that increase or decrease the policy options available to governments. E. Includes classical political philosophy and contemporary theoretical perspectives (e.g., constructivism, critical theory, and postmodernism). F. Focuses on politics within countries (often grouped into world regions) and analyzes similarities and differences among countries. G. Examines the passage and implementation of all types of government policies, particularly those related to civil rights, defense, health, education, economic growth, urban renewal, regional development, and environmental protection 1. International relations 2. Political theory 3. Domestic politics 4. Public administration 5. Comparative politics 6. Public policy 7. Public law Example 6: Exhibits DO Fruit preserves are made by heating fruit and sugar to temperatures hot enough to kill microorganisms in the fruit (at about 110 degrees C). A typical recipe for strawberry preserves calls for approximately 1 kg of crushed strawberries to be mixed with 1 k added sugar. The strawberries also contain some sugar of their own, which you can quantify using ​
Wolfram Alpha​
. What is the overall fraction by weight in the preserves before sterilization? (Please express your answer as a decimal with four decimal places. You do not need to include the unit marker in your answer.) Possible answers: 0.5245, .5245 The sculptures below are examples of which of the following genre? Terms of Use: (1) The image is released under a ​
Creative Commons Attribution­Share­Alike License 2.0​
. It is attributed to Rebecca Arnett. (2) This image is released under a ​
Creative Commons Attribution­Share­Alike License 2.5​
. It is attributed to Robert A. Estremo. A. Still life B. Portrait C. Allegory D. Religious DO NOT In 1971, President Augusto Pinochet implemented a program in Chile to dramatically reduce unemployment. It was fairly successful in meeting its goals in raising employment, but in the months following the implementation of the program, fewer entrepreneurs went into business for the first time. Why might a very low rate of unemployment warn entrepreneurs against going into business in Chile? A. More wage earners in the market makes it difficult for industries to keep up with excess demand. B. More wage earners in the market indicates that government will soon step in to regulate industry. C. More wage earners in the market curtails consumer spending because inflation drives industry prices up. D. More wage earners in the market forces industries to drive down prices to be able to compete for consumers’ spending.