USMA Course End Feedback System
Guides/Reminders:
 The USMA Course-End-Feedback system was
established to be a Formative System – not Evaluative
 Instructor level feedback is confidential for Instructors
within the system – up to instructors to share w/
mentors
 Only one (or two) Trusted Agents (TAs) per Dept. – no
one else should be made a TA when assigning roles.
 TA’s can see and do everything at and below their level
 Report Accessors (RA’s) can see reports and add
questions at their level only – not below.
 Only Instructors should be RA’s at the Instructor and
Section/Hour levels.
 Consider how you assign roles to courses in singleinstructor courses – anonymity of the data could be
compromised
 Watch out for free-text w/ rollup questions
Contact: Dr Mark Evans, Director, Center for Faculty Excellence
The below document was taken from:
http://www.dean.usma.edu/feedback/browser_help.htm
The End of Course Feedback System is Formative, not evaluative -In the early 1990s, the USMA Faculty Council established the paper-based
predecessor of this feedback system with the principle that its primary
function should be formative. That is, the information it provides is to
support the personal professional development of teachers. It is not to be
used as a mechanism for leaders to evaluate the performance of their
subordinates. While feedback reports may well be part of mentoring and
performance goal-setting, the Faculty Council deemed it important that
feedback reports be provided by the instructor to his/her mentors, not by
the system. As such this system, just as the paper system it replaces,
1
allows for the enforcement of confidentiality of all cadet responses and
faculty reports. Cadet responses are anonymous. Moreover, when the
system is properly operated, faculty members see only response
information regarding their own cadets. Reports for supervisors are
aggregated so as to preclude drawing conclusions about individual
instructors or cadets. It is the responsibility of trusted agents to properly
operate the system to ensure confidentiality is maintained.
Example Hierarchy
USMA
Skill-Activity
Swimming
Cognitive
Wrestling
Aero&NA
Civil & Mech Engr
Naval Arch
Aeronautics
NA301
Dr. Nimitz
NA301A01
Chemistry
NA342
Dr. von Tirpitz
NA301B01
The hierarchy of feedback nodes
For our purposes, a feedback node is a person or organization that:
• Writes or provides feedback survey questions and
• Receives reports summarizing the answers.
Many nodes will be people or organizations who conduct the acadmic
program -- departments, programs or divisions, courses, instructors, and
individual sections. In other cases, nodes correspond to sets of courses
where curriculum leaders and planners merely want summary reports for
assessment purposes. For example, Core and Elective nodes would result
in separate reports summarizing these kinds of courses within a
department or program. Nodes for academic disciplines may also be
useful: Discrete mathematics and Continuous, Cultural geography and
Physical, Digital electronics and Analog.
The system requires that nodes be arranged in a hierarchy. If we were
employing the usual convention of organizational wiring diagrams we
2
would draw the USMA node at the "top" and individual section-hours, e.g.
EN101A01, in the "fringe" at the bottom. Ultimately, every section-hour
where cadets are expected to provide feedback must appear exactly once
in the fringe.
It is in this sense that we use the words superior and subordinate in
referring to nodes above and below in the hierarchy, respectively. We say
that any set of nodes with the same immediate superior comprise a
level. Levels have names like Academy, Department, Program, and
Course.
Whenever the Feedback system shows the hierarchy or parts of it,
however, superior levels are shown to the left and subordinate levels to the
right. For instance, the C&ME Department might be shown on the left
margin of the screen with Civil and Mechanical divisions indented one level
to the right. This convention is similar to MS Windows "Explorer" and "File
Manager," so many will find it familiar.
This Help is about the Browser, a web page that Trusted Agents of the
Feedback system use to create and view the hierarchy. Report accessors
use the same Browser to edit questions and read reports.
Paths in the hierarchy
Every node in the hierarchy has a unique "path" that tells where the node
lies. For instance, CS105, Section 1, D Hour might have the path
USMA/EE&CS/CS105/LTC Smith/CS105D01 This signifies that the node
for Section 1D, CS105 is subordinate to another node for LTC Smith,
presumably the instructor, which in turn is subordinate to the course
CS105, which is subordinate to EE&CS, which is subordinate to USMA.
Access and report accessors
Any person with an e-mail account and a web browser can be designated
a Report Accessor (RA) for any node in the hierarchy. A RA for a node
can
• Edit questions for the node and
• Read the report ultimately produced for the node.
Report access is granted by Trusted Agents, discussed below.
3
Trust and trusted agents
Any person with an e-mail account and a web browser can be designated
a Trusted Agent (TA) for any node in the hierarchy. A TA for a node can
• Do anything that a RA can do at that node (edit questions and read
reports)
• Designate RA's for the node
• Create, change, and delete subordinate levels and nodes, and
• Delegate trust for any subordinate node to any person with an e-mail
account.
The last of these privileges is very broad. It means that a trusted agent has
complete control over any node where she is once given trust and all
nodes beneath in the hierarchy. For even if she does not initially have trust
at some subordinate level, she can delegate it to herself. This is why
trusted agents are called "trusted."
Who can see what
TA and RA privileges are exclusive. Only TAs can change the hierarchy,
and only TAs and RAs can edit survey questions and read reports. One
might be able to use "hacker" techniques to guess PINs or exploit obscure
security loopholes to "break in" to the system. As for many automated
systems used exclusively within the USMA, the security provisions of
Feedback 99 are designed to "keep honest people honest," not to
withstand attack by a determined adversary.
Reports support the confidentiality rules of the earlier paper-based
feedback system wherever possible. Where the new system provides new
capabilities, the implementers have attempted to maintain the spirit of the
earlier one.
The general principle is that reports summarize questions at the level
they are asked and also at each subordinate level down to and
including the level of the report itself. Another way of saying this is that
reports provide all summaries that are possible except where a summary
would provide explicit information about a subordinate node in the
hierarchy. This enforces longstanding confidentiality rules. Bosses do not
see survey results aggregated at the levels of their subordinates (unless
the subordinate provides them outside the system, which is highly
encouraged for professional development purposes).
4
Confidentiality leaks - Multiple choice questions
Most types of questions that the Feedback system supports (scale, choice,
anyall, rank) allow only multiple choice answers. (See Help for the
Question Editor.) Confidentiality for each individual instructor is protected
in higher level reports by aggregating data from sections of several
different instructors. Summaries take the form of histograms, means,
standard deviations, and min-max brackets.
Due care is necessary (the same as under the old paper system) to
anticipate and deal with confidentiality leaks through aggregates of small
size. For instance, if a course has only two sections, one taught by the
course director and the other by another instructor, then both could use the
Course and Section-Hour aggregates in their own Section-Hour reports to
infer responses of the other section and vice versa. Some departments
deem this unimportant. Others would omit the Course level node for this
course in order to avoid the undesired aggregation. Alternately, they could
allow access to only the Course level report for both instructors and let the
Section-Hour reports unread.
Confidentiality leaks - Free Text questions
Questions that ask for Free Text responses may also require some special
handling. In reports, summaries for Free Text questions are merely lists of
the responses of all cadets covered by the report. If a cadet mentions her
own name, the instructor's name, or other specifics, a high-level summary
containing her response may compromise confidentiality. For example, if a
Department Head, COL Glotz, asked a Free Text question, "How was your
course?" and a cadet replied "CPT Schmotz spent too much time washing
the blackboard, Sincerely CDT M.A. Cynic" this response could appear in
the COL Glotz's report, violating the principle, stated above, that "a
summary does not provide explicit information about a subordinate node in
the hierarchy."
To address this problem, the Feedback system offers two types of Free
Text questions, selectable in the Question Editor: Free Text (No Roll-up)
and Free Text (With Roll-up). For the former, response summaries appear
only in fringe (Section-Hour) level reports. Higher level summaries contain
only a note stating how many cadets responded. For the latter, summaries
of all Free Text responses are listed according to the tables shown below.
Therefore, No Roll-up questions guarantee confidentiality, while With Rollup questions maximize the amount of data in higher level reports, at some
risk to confidentiality. The decision on which to use falls to each Dept.
5
Examples
The discussion below is based on the fictitious Department of Aeronautics
and Naval Architecture (Aero&NA). It details what several people can see
in the reports they receive. For purposes of this example, we assume that
Aero&NA has a hierarchy of levels as follows: Academy/Department/
Program/Course/Instructor/Section-Hour In particular, the example
section is: USMA/Aero&NA/Naval Architecture/NA301/CPT Schmotz/
NA301A01 The trusted agent of Aero&NA has set Report Access as
follows:
Node
Aero&NA
Naval
Architecture
NA301
Level
Departme
nt
Program
Course
CPT Schmotz
Instructor
NA301A01
SectionHour
Report Accessor
COL Glotz, the Department Head
LTC Quonk, Naval Architecture Program
Director
Professor Baz, Course Director
CPT Schmotz, Instructor of section NA301,
A Hour, Section 1
CPT Schmotz, Instructor of section NA301,
A Hour, Section 1
We explain these points further in the form of answers to some likely
questions. Question: What can COL Glotz see and do? Answer: COL
Glotz can enter her own questions at the department level. She can also
read the Department level report. This report summarizes results at
various levels of aggregation as shown in this table:
Level where question is asked
Level of
summary
aggregation
USMA
Aero&NA
Naval Arch.
NA301
CPT Schmotz
NA301A01
USMA
Yes
Yes
No
No
No
No
Aero&
NA
Naval
Arch.
-Yes
No
No
No
No
--No
No
No
No
6
NA301
CPT
Schmotz
NA301
A01
---No
No
No
----No
No
-----No
The sole exception to this table is questions of type Free Text (Without
Roll-up). Answers to such questions appear only in fringe (Section-Hour)
level reports, without regard to the level they are asked. Question: What
can CPT Schmotz, an instructor, see and do? Answer:CPT Schmotz gets
at least two reports--one as instructor and another for section-hour
NA301A01. Presumably he also gets reports for each additional sectionhour he teaches. He can also enter his own questions at either (or both of)
the Instructor level or the Section-Hour level for his section-hour. This
section level report is the most detailed that the system generates. It
summarizes results at various levels of aggregation as shown in this
table:
Level where question is asked
Level of
summary
aggregation
USMA
Aero&NA
Naval Arch.
NA301
CPT Schmotz
NA301A01
USMA
Yes
Yes
Yes
Yes
Yes
Yes
Aero&
NA
Naval
Arch.
-Yes
Yes
Yes
Yes
Yes
--Yes
Yes
Yes
Yes
NA301
CPT
Schmotz
NA301
A01
---Yes
Yes
Yes
----Yes
Yes
-----Yes
Answers to Free Text (No Roll-up) questions appear only in fringe
(Section-Hour) level reports like this one, without regard for the level they
are asked. CPT Schmotz's Instructor level report is similar, except there
are no summaries aggregated at the section-hour level, and there are no
answers to questions of type Free Text (Without Roll-up). CPT Schmotz
probably won't bother reading the Instructor report because each section
level report contains the same information and more. The Instructor node
itself remains important, however, because its presence in the hierarchy is
necessary to receive summaries aggregated at the Instructor level, even in
the Section-Hour level reports. Question: What can Professor Baz, the
course director, see and do? Answer: Professor Baz can enter his own
questions at the course level. He can also read the Course level report.
This report summarizes results at various levels of aggregation as shown
in this table:
7
Level of summary
aggregation
USMA
Aero&NA
Naval Arch.
NA301
CPT Schmotz
NA301A01
USMA
Yes
Yes
Yes
Yes
No
No
Level where question is asked
Aero&
Naval
CPT
NA301
NA
Arch.
Schmotz
----Yes
---Yes
Yes
--Yes
Yes
Yes
-No
No
No
No
No
No
No
No
NA301
A01
-----No
The sole exception to this table is questions of type Free Text (Without
Roll-up). Answers to such questions appear only in fringe (Section-Hour)
level reports, without regard to the level they are asked.
The feedback cycle
There are three phases in the feedback cycle:
Phase
Setup
Surveying
Reporting
Approximate time
60 days before end of
semester through start of
Surveying phase
Last week of classes
through Term End Exams
Description
The browser allows changes to the
hierarchy and entry and edits of
questions.
Changes to the hierarchy and
questions are disallowed. Students
request, fill out, and submit survey
forms. Trusted agents and report
accessors use the browser to check
compliance. That is, they list students
in their sections with a Yes or No for
each showing whether s/he has filled
out a survey.
Starting one week after
Student responses are disallowed.
Term End Exams through Trusted agents and report accessors
lesson 10 of the following use the browser to read reports.
semester.
In addition, the system can be Closed while administrators do
maintenance.
8
The Browser Form
The browser form has spaces and buttons for viewing and navigating the
hierarchy, creating new levels and nodes, deleting them, granting access
and delegating trust, and editing questions. The trust and/or access
granted to you affects the way the browser appears. In general, buttons
and text boxes will appear and disappear as you navigate to different
portions of the hierarchy where you have and don't have access or trust.
The sections below discuss this in more detail.
The Login Page
The login page is where access to the Browser starts. Log in by typing
your e-mail id and your Feedback PIN in the respective boxes. Press the
button labeled Log in
If you've never logged in before or if you've lost your PIN, type your e-mail
id (e.g. xy1234) in the box marked Login and then press the button labeled
Email my PIN. A few minutes later, you'll receive an e-mail message
containing a PIN. The PIN will remain valid at least for the current
semester.
The Browser Path
The blue bar at the top of the browser contains the Browser Path, an
important bit of information. The path specifies a node in the hierarchy.
This is the node where the various functions performed by the Function
Buttons will act.
The Subordinate Level Listing
The white box within the browser page is the Subordinate Level Listing.
The top line of the listing gives the level name, a collective rubric that the
trusted agent enters. Beneath it is a list of individual subordinate nodes.
Each is preceeded by a "radio button" that can be "pressed" with the
mouse. Pressing the button selects it. The selected subordinate node
affects the following functions, performed by the Function Buttons.
• Go down - Browses "down" to the selected subordinate.
• Delegate trust down - Edits the trusted agent list for the selected
subordinate node.
Delete - Deletes the selected subordinate node and all of its subordinates.
9
The Message Area
The Message Area lies immediately beneath The Subordinate Level
Listing at the bottom of the Browser. It is often blank, but may contain a
message regarding the last action taken by the Browser. Errors and
advisories appear here.
The Function Buttons
The Function Buttons are the blue boxes on the right of the browser. Click
close to the white text to press a button. Following is a brief description of
the buttons and what they do:
• Go up - Browses up one level to the superior node.
• Go down - Browses down to the selected subordinate node.
• Set access here - Sets Report Access at the node given by the browse
path.
• Delegate trust down - Edits the trusted agent list for the selected
subordinate node.
• Edit questions - Opens the question editor for the node given by the
browse path.
• Delete - Deletes the selected subordinate node and all of its
subordinates (BEWARE! POWERFUL STUFF!)
• Check progress - Creates a new browser window with a list of all
course-section-hours that the Feedback system knows about.
The new subordinate level name and new subordinate node boxes
Use these boxes to type in the names of:
• A completely new subordinate level (then press the Create button).
• A new name for an existing subordinate level (then press the Change
button).
• A new subordinate node (then press the Add button).
Navigating the hierarchy
Upon a successful login, the Browser Path will be USMA and the
Subordinate Level Listing will display the Department level, consisting of a
node for each Department. Browse to any subordinate node by selecting
its "radio button" and then pressing Go Down. The Browser Path shows
the change.
10
Return to the superior node by pressing Go Up. Again note the change in
the Browser Path.
Creating a subordinate level
When you have browsed to the current bottom of the hierarchy of feedback
nodes, the Subordinate Level Listing will contain the note Subordinate
Level: None!. Create a new subordinate level by typing its name in the
box provided. Then press the Create button.
Singular nouns make the most sense in forms and reports created by the
system later using level names. Examples of good level names are:
• Academy
• Department
• Program
• Division
• Discipline
• Core/Elective
• Course
• Instructor
• Section-Hour
•
A known bug that nouns with irregular plural spelling will lead to
misspellings in forms (e.g. Categories). Avoid such level names if possible.
If the Create button is missing from the Browser, then the node you have
reached is on the fringe of the hierarchy. This means it is the name of an
individual course-section-hour. It is not possible to create a level
subordinate to such a node. The buck stops here!
Renaming a subordinate level
To rename the subordinate level currently shown in the Subordinate Level
Listing, first note that this name also appears in a text box with the label
new subordinate level name on the browser's right side. Change the
subordinate level name by typing the new name in this box. Then press the
Change button.
Adding a new node
To add a new node:
• Navigate to what will be the superior of the desired new node
11
• Create the node's level if it does not already exist. Until a level is created,
the Add subordinate node box and button will not appear on the
browser.
• Type the name of the new node in the box labeled "New [level name]",
where [level name] is the actual level name shown in the
Subordinate Level Listing.
• Check or uncheck the button labeled "Delegate trust to me!" When
checked, your Email id is automatically added to the trusted agent
access list of the new subordinate node. If you are creating the
entire hierarchy, this is a logical thing to do. If you want to have a
different person be the trusted agent for the hierarchy at and below
the new node you are creating, let the box unchecked. You will have
to delegate trust to that person later.
• Press the Add button.
After adding a node, check the Message Area. It will contain a note saying
either that the new node IS or IS NOT a course. This is important. The IS
a course message means that the node you have just added is on the
fringe of the hierarchy. No subordinate level can be created for a fringe
node. You can see which fringe nodes the system considers as remaining
to be entered using the Check progress button.
Delegating and removing trust to a subordinate node
To delegate or remove trust for a node:
• Navigate to its superior
• Click the "radio button" next to the name of the node where trust is to be
delegated
• Press Delegate trust down. The Access List containing all trusted
agents for this node will appear.
• To add a new trusted agent, type his/her Email id into the box labeled
"New login to add" and press Add.
• To remove one or more trusted agents, check the box(es) in front of the
respective Email id(s) and press Delete.
• When done adding and deleting trusted agents, press Return to
browser to return to the browser.
Granting and removing report access
To grant or remove report access for a node:
• Navigate to the node itself.
12
• Press Set access here. The Access List containing all report accessors
for this node will appear.
• To add a new report accessor, type the Email id into box labeled "New
login to add:" and press Add.
• To remove one or more report accessors, check the box(es) in front of
the respective Email ids and press Delete.
• When done adding and deleting report accessors, press Return to
browser to return to the browser.
Editing questions
To edit questions for a node:
• Navigate to the node itself.
• Press Edit questions.
See Help for the Question Editor for more information.
Deleting a subordinate node
To delete a node and all of its subordinates:
•
•
•
•
Navigate to its superior
Click the "radio button" next to the name of the node to be deleted
Think twice! Deletions cannot be undone.
Press Delete
Checking progress
Press the Check progress button to see which nodes in the fringe of the
hierarchy remain to be added. This is a listing of course-hour-sections from
the Academy database which have not yet been named as nodes by any
trusted agent.
The goal for the community of trusted agents is to create a USMA
hierarchy that includes a node for each course-hour-section. When this is
done, the Check progress list will be empty.
Summarizing trust, access, and questions
Trusted agents can press the Summarize button to see a summary of the
following information at the node given by the current browser path and all
13
subordinate nodes:
• Trusted agent list for each node
• Report accessor list for each node
• All questions and the multiple choice answers for scale, choice, rank,
and "any or all" questions.
This provides trusted agents with
• A check on trust and report access set on all nodes for which the agent
is responsible.
• A way to monitor the progress of report accessors in entering questions
and
• A way to enforce standards for style, correctness, and numbers of
questions.
Checking compliance
To check "compliance," that is, whether a cadet has or has not filled out a
form for a given course where s/he is enrolled:
• Navigate to the the fringe node for the section-hour of the course.
• Press Check compliance.
Reading reports
To read a report:
• Navigate to the the node for the desired report.
• Press Report and follow the instructions.
Note: The Report button will appear only if you have been granted trust or
report access for the node. If button is missing when it should be there,
see your trusted agent.
M. Evans, Director, Center for Faculty Excellence
14