SAA
Assessment of Student Learning
Constitution Day Project
Fall, 2007
April 4, 2008
Abstract: Students were asked to view a short video in association with Lake
Superior College’s observance of Constitution Day on September 17, 2007. They
then answered three multiple choice questions designed to assess their ability to
think critically by applying general principles from the video. Responses from 500
students were collected and analyzed. The scope of the project was subsequently
expanded to include both an analysis of student attendance on September 17, 2007
and an assessment of faculty use of technology. The results showed that (1)
students were generally adept at reasoning by analogy, (2) approximately two
thirds of the students registered for the classes that participated in the
Constitution Day observance were in attendance that day, and (3) the faculty as a
whole experienced a surprising and unacceptable degree of difficulty making
simple use of classroom technology.
1. Introduction
This project, undertaken in connection with the College’s federally- mandated
observance of Constitution Day, was designed to assess student
achievement of Lake Superior College’s College-wide Critical Thinking
Outcome. That Outcome is based on the FIRE Model of Critical Thinking,
which is used extensively at LSC to teaching critical thinking skills. Subpart
4.2 of the outcome states:
The student will be able to explore possible assumptions,
interpretations, or perspectives related to solving a problem.
Under the FIRE Model, this part of the outcome is interpreted to include
reasoning by analogy.
2. Research Questions
(1) Given general principles governing the free speech rights of students, can
students reason by analogy using those principles?
(2) What percentage of the students registered in classes that participated in
the Constitution Day observance were in attendance on that date, and how
does that percentage compare to a similar date much later in the semester?
(3) Are faculty members able to tune into the College’s closed circuit
television network or use the DVD players provided in their classrooms to
show their class a video?
3. Research Method
On Monday, September 17, 2007, all instructors in courses meeting on
Mondays with starting times at 9:00 a.m. and 12:00 p.m. were asked to show
their students a short (under 5 minutes) video.
The video gave students background information on two important U.S.
Supreme Court decisions that dealt with the First Amendment rights of public
school students (Tinker v. Des Moines and Frederick v. Morse). After
watching the video, students were asked to respond to three multiple choice
questions designed to assess their ability to reason by analogy about the First
Amendment rights of students. To determine the correct answers students
had to be able to make a specific application of the general principles
explained in the video. After students completed the response forms, they
were provided with an additional handout showing the correct answers and
the rationale for each.
A total of 52 instructors teaching a total of 65 course sections with
approximately 1300 registered students were included in the project. For
various reasons (e.g. unavoidable scheduling conflicts, instructor refusal to
participate, technical difficulties, or instructor inability to use classroom
technology), responses from exactly 500 students in 26 course sections
taught by 26 different instructors were actually collected. The student
responses were entered into eLumen, LSC’s software tool for conducting and
documenting assessment of student learning.
Due to the asynchronous nature of online courses and other impediments, the
Virtual Campus was not included in the project.
The materials used for the project were written by Kent Richards and Steve
Schneider, two faculty members with law degrees. The video was produced
by Brandon Leno, Media Studies and Broadcasting instructor.
Instructors were offered their choice of two methods for showing the video.
First, the video was broadcast over LSC’s closed circuit television network
commencing five minutes after class start times. Second, instructors were
offered the video on a DVD to show using the technology cart in their
classrooms. Either option required instructors to know how to use the
technology in their classrooms.
There were technical problems with the broadcast on the closed circuit
television network. In addition, it became clear that many instructors were
2
either unable to use the technology in their classrooms to show the video or
the technology in their classrooms was inoperable. Though most of these
problems were anticipated, they proved far more severe than expected.
Consequently, one unintended result of the project was an indirect
measurement of the extent to which faculty were able to use the technology
(i.e. show a DVD in their classrooms) that is included in all classrooms. To
capture that data, a supplemental survey about classroom technology was
distributed to all faculty members included in the project. A total of 27
instructors returned the supplemental survey. Of those, eight were instructors
who did not otherwise complete the project by returning student response
forms from their classes.
Finally, the use of eLumen provided a third opportunity. eLumen is intended
primarily to facilitate assessment of student learning outcomes at the course
level. To accomplish that, student registration data is imported to generate
individualized “scorecards” for each student within each course. It was
relatively simple to set up the Constitution Day project within eLumen.
Once the decision was made to use eLumen, it became apparent that the
mere fact that there was a student response indicated that the student was in
class on the day of the project. No student response indicated the student
was not in class that day. Consequently, only a limited amount of work was
necessary to generate attendance data for Monday, September 17th.
Instructors were then asked to take attendance a second time on Monday,
December 3rd.
4. Results
A. Critical Thinking
The video shown to the students described the specific relevant facts in two
U.S. Supreme Court decisions. In the first case the Court applied First
Amendment principles to expand the free speech rights of school students. In
the second case, the Court applied First Amendment principles in a way that
restricted the free speech rights of school students.
Question #1 asked LSC’s students “what difference between the two cases
provides the best reason to treat them differently?” From among the four
answers, 413 (82.6%) of the students selected the correct answer.
Question #2 asked LSC’s students “what similarity between the two cases
provides the best reason to treat them the same?” From among the four
answers, 383 (76%) selected the correct answer.
3
Question #3 asked LSC’s students to select the best reason supporting the
Court’s decision in the most recent case. From among the four answers, 430
(85.3%) selected the correct answer.
B. Classroom Technology
Aggregating the data on the supplemental surveys showed:
12/27 (44%) responding instructors reported they were able to show the
video without any difficulty.
4/27 (15%) reported they were able to show the video only after Bill Berg
[Computer/ITV Technician] or someone else helped them with the setup.
1/27 (4%) reported their classrooms did not have the necessary
equipment to show the video.
6/27 (22%) reported the video equipment in their classrooms was out of
order.
3/27 (11%) reported they often have difficulty using the video equipment in
their classrooms.
(Percentages do not equal 100 as the survey included additional
options and instructors were free to check more than one.)
Thus, seven (26%) of the instructors responding reported they were unable to
complete the project either because their classroom did not have the
necessary equipment or it was out of order. An additional four instructors
responding (15%) needed assistance to show the video.
The instructor comments from the survey forms further detail the difficulties
some experienced:
“As far as I could tell, I knew how to use the equipment, but the fact that the video was
late caused us not to watch it. At the time, I assumed that the system was not working
correctly and turned it off.”
“The timing was off but we finally got it. I did check it all out before class time because I
often find problems in various rooms.”
“My problem was with the sound. I was in S2984 and had [the video] on the screen, but
silent. Eventually I found out how to un-mute the sound, but by that time [the video was
over].”
“I was able to show the video because I had my laptop with me and I hooked that up to
the projector and speakers in the room. Otherwise, it would not play on the equipment
that was available in the room.”
4
“I use the equipment very often, but there seems to be new glitches all the time. Bill
[Berg] has been VERY HELPFUL, but the equipment and/or instructions (equipment
instructions) are not clear. I had SOME problem knowing what to “click” to get it working,
and the instructions did not match up with the equipment wording in the room.”
“Even Bill Berg couldn’t get the equipment to work! At least I know it wasn’t me.”
“The video equipment did not work.”
“I had to go to another classroom to do the video, but [then] had no trouble watching it.”
Other instructors had no difficulty. Their comments were generally quite
enthusiastic:
“Once I got a CD with good sound quality – no problem. I thought you did an excellent
job with this Constitution Day activity!”
“No problem – everything worked great in the 2 different rooms that I showed it (but I did
practice before the time arrived).”
“We had an interesting discussion in class about freedom of speech and expression
afterwards. I am amazed at how many of my students believe that they can’t express
their opinions freely. I have been to Russia, know people from China, from Albania, and
Ireland. We have letters to the editor, editorial pages, online forums, etc that simply do
not exist everywhere else. I am glad that you took the time to make the video. I also
think it would be a good idea to have more of these type of campus wide learning
experiences.”
“I was able to show the video to my class without any difficulties, in fact we watched it
twice, answered the questions, collected them and turned them in. Didn’t really have
time to have a discussion regarding video or questions, but I did ask for questions.”
“It worked fine for me.”
“It was a wonderful experience for everyone in the classroom and I look forward to
another opportunity to do it again.”
“I had no problems whatsoever.”
“It worked well!”
C. Attendance*
The 26 course sections in which students completed the Critical Thinking
assessment contained a total of 791 registered students. A total of 562 out of
791 registered students (71%) were present on Monday, September 17, 2007
(Constitution Day).
A total of six instructors took attendance in six separate course sections on
Monday, December 3, 2007 as part of the follow-up to the original project. A
total of 89 out of 122 registered students (73%) were present.
5
The raw data showed that attendance on December 3rd was actually higher in
five of the six course sections than the attendance on September 17th:
Course
ACCT 2420
ALTH 1410
ENGL 0460
ENGL 1109
MATH 0460
LGST 1400
9/17/2007
62.8%
77.2%
83%
84.1%
63.2%
80%
12/3/2007
70.6%
87.5%
86.4%
84.2%
38.1%
68.4%
This is thought to be an anomaly caused by a data issue resulting from the
importation of registration data into eLumen. The number and identity of
registered students in each course section is imported from ISRS.
Prior experience has revealed data integrity issues apparently caused by a
tendency of the ISRS database to preserve registrations for students who
both registered for and dropped a course prior to the start of the semester.
Many of these excess registrations are removed when the data upload is
repeated about the middle of the semester.
Comparison of the attendance data produced by instructors for December 3rd
with the data inferred for September 17th revealed the magnitude of the error
to be in the range of 5-10% with wide variation between courses. Thus, the
data may understate actual attendance on September 17th by as much as
10%.
5. Discussion
Critical Thinking
The results suggest a higher than expected level of critical thinking skills
among LSC’s students. Reasoning by analogy is a relatively difficult skill to
teach and learn. The results were far better than the architects of the project
expected.
It may be that the multiple choice format exaggerated student’s reasoning
abilities. It may be that the questions were too “easy” because the distractors
were too easy to spot and reject. Moreover, no attempt has been made to
validate the questions or statistically analyze the results.
Still, chance or lack of effort would dictate a 25% success rate on each
question. The actual success rates of 82%, 76% and 85% are substantial
and therefore difficult to dismiss.
6
Classroom Technology
The results regarding classroom technology were troubling.
First, it is clear that an unacceptable number of instructors are unable to use
the technology in their classrooms or have difficulty using it. LSC has made a
substantial investment in classroom technology and one might reasonably
question how much it is being used.
Second, at the time of the project there clearly was an unacceptable rate of
“out of order” classroom technology, even allowing for start-up difficulties in
the new classrooms during fall semester that have since been remedied.
Equipment that doesn’t work has been a perennial complaint of faculty
despite the best efforts and support of Bill Berg and other staff.
Third, it appears that LSC’s capability to broadcast important messages to all
classrooms by closed circuit television was doubtful. Between equipment that
doesn’t work and instructors who did not know how to use it, this important
capability was compromised. Recently, a new system for broadcasting to all
classrooms has been put in place that gives the system’s administrators more
control and relies less on instructors.
Attendance
Most instructors note a decline in attendance over the course of each
semester. It is often noted, as well, that the parking lots, are overflowing at
the beginning of each semester but become progressively less crowded as
each semester wears on.
However, few instructors take regular attendance in their classes and
attendance data is not aggregated in any way. Therefore, LSC does not have
data to support or contradict the notion that attendance drops precipitously
toward the end of each semester.
The attendance data produced by this project suggests attendance rates
between 70 and 80% early in the semester with little if any tapering off toward
the end. Possible explanations for this unexpected result include (1)
unknown data integrity issues, (2) confirmation bias on the part of faculty, and
(3) a decrease over the course of the semester in the amount of time outside
of class that students remain on campus.
This would be a relatively easy issue to explore further through a dedicated
attendance project. Although attendance data is an indirect measure of
student engagement and success, it could be cross-referenced with retention
data, petition rates, grades, and other existing data to yield additional insights.
7
Moreover, such a project would supplement existing efforts to increase
student retention.
General Conclusions
This type of project offers the opportunity to collect a large number of samples
of student work in an efficient manner. LSC should continue conducting a
college wide assessment project in conjunction with Constitution Day each
fall. Subsequent similar projects offer the opportunity to assess whether our
faculty’s ability to use the technology in their classrooms has improved.
7. Appendices
App 1 – Video Script
App 2 – Student Question and Response Form
App 3 – Answer Handout
App 4 – Supplemental Technology Survey
8
Appendix 1
Constitution Day
Congress shall make no law…abridging the freedom of speech…”
U.S. Constitution, Amendment I (1791)
Today, September 17th, is Constitution Day. On this day 220 years
ago, the 39 men who helped write the U.S. Constitution signed it,
creating the form of government we proudly claim to this day.
Educational institutions all over the United States are observing
Constitution Day. Here at LSC, we are setting aside fifteen minutes of
class time to give some thought to freedom of speech, which is just
one of the important rights guaranteed to each of us under the Bill of
Rights.
The Bill of Rights consists of the first ten amendments to the U.S.
Constitution. The very first one, the First Amendment, is the one that
guarantees us freedom of speech.
Have you ever wondered whether students in public schools enjoy the
same right to freedom of speech as everyone else in our society?
Does it surprise you to learn that the answer, even today, is not
entirely clear?
Whether students have a right to free speech under the Constitution
first came up in 1965, over forty years ago, when three students, Mary
Beth Tinker, her brother, and a friend, wore black armbands to school
in Des Moines, Iowa to protest the United States’ involvement in the
Vietnam War. They were suspended, but they fought their
suspensions, arguing in the courts that wearing armbands was a form
of political speech and that the First Amendment to the U.S.
Constitution, which promises all Americans freedom of speech,
prohibited school authorities from punishing them.
In the case of Tinker v. Des Moines, the U.S. Supreme Court agreed,
making it clear that public school students have a right to express
9
their political beliefs in school as long as they do so in a non-disruptive
manner.
Fast forward 37 years to Juneau, Alaska where, in 2002, the Olympic
Torch Relay was passing by a high school. A student, Joseph
Frederick, displayed a large banner with the statement “Bong Hits 4
Jesus” on it. When he refused his principal’s order to take down the
banner, she suspended him.
In Frederick v. Frederick, the U.S. Supreme Court approved the
suspension, reasoning that the banner could be interpreted as
promoting illegal drug use and therefore disruptive to the school’s
mission.
Did the Supreme Court treat Mary Beth Tinker and Joseph Frederick
consistently?
The U.S. Supreme Court tries very hard to be consistent. The Justices
strive to decide like cases the same way. Of course, every situation is
different, so one important question the Supreme Court must always
wrestle with is whether today’s case is similar or different from
yesterday’s case in important ways. Decisions that appear to be
inconsistent are usually the result of important differences between
the cases that cause the Justices to reach a different result.
We invite you now to think like a Supreme Court Justice by considering
three questions about the Supreme Court’s reasoning on a form your
instructor will hand out. When you are finished, we’ll give you the
correct answers and the reasoning behind them.
10
Appendix 2
Constitution Day
Name: ______________________________
Instructor: ___________________________
In the Tinker case, the Supreme Court decided a student could not be
suspended for wearing a black armband because doing so would
violate her right to free speech. (Under the First Amendment, “speech”
is considered to include traditional speech as well as written words and
symbolic actions, like wearing an armband, that have a message.)
In the Frederick case, the Supreme Court decided that a student could
be suspended for displaying a banner because doing so would not
violate his right to free speech.
That means the Supreme Court decided the two cases were different in
some important way. Is that true, or was the Supreme Court
inconsistent by treating Joseph Frederick differently than it had treated
Mary Beth Tinker?
Imagine yourself as a Justice on the U.S. Supreme Court for a few
moments and answer the following questions:
1. What difference between the two cases provides the best
reason to treat them differently? (check one)
_____A. Mary Beth Tinker was protesting against the Vietnam War.
Joseph Frederick was apparently promoting use of illegal drugs.
_____B. Mary Beth Tinker used actions, not words, to make her point.
Joseph Frederick made his point with words written on a banner.
_____C. Mary Beth Tinker lived in Iowa. Joseph Frederick lived in
Alaska.
_____D. Mary Beth Tinker’s case was decided 40 years ago. Joseph
Fredrick’s case just happened.
(over)
11
2. What similarity between the two cases provides the best
reason to treat them the same? In other words, what would be
the best reason the Supreme Court could have used to decide in
favor of Joseph Frederick? (check one)
_____A. Mary Beth Tinker and Joseph Frederick were both students in
public schools.
_____B. Mary Beth Tinker and Joseph Frederick were both children
and children shouldn’t have the same Constitutional rights as adults.
_____C. Mary Beth Tinker and Joseph Frederick both expressed points
of view not approved by most of society.
_____D. Mary Beth Tinker and Joseph Frederick both acted in ways
that were silly and pointless.
3. Based on what you know about the Supreme Court’s
reasoning in the Tinker and Frederick cases, what is the best
reason for concluding, as the Supreme Court actually did, that
Joseph Frederick’s right to free speech did not give him the
right to display a banner stating “Bong Hits for Jesus?”
(check one)
_____A. Joseph Frederick needs to learn to respect authority.
_____B. Allowing Joseph Frederick to display the banner would be
disruptive by suggesting the school allows use of illegal drugs.
_____C. Allowing Joseph Frederick to display the banner would
undermine the authority of the school administrators.
_____D. Separation of church and state prohibits mentioning Jesus in
public schools.
12
Appendix 3
Constitution Day Observance
Answers
By Kent Richards, attorney and LSC instructor.
1. What difference between the two cases provides the best
reason to treat them differently?
_X__A. Mary Beth Tinker was protesting against the Vietnam War.
Joseph Frederick was apparently promoting use of illegal drugs.
Ensuring that citizens can object, as Mary Beth Tinker was doing, to
the government’s policies when they consider those policies immoral
or unwise is exactly the kind of speech the First Amendment was
intended to protect. On the other hand, it’s not clear what Frederick
was doing. It appears he was either attempting to promote violation
of the law (rather than advocating change) or his statement was
nonsense. As a result, his speech is arguably not the kind of speech
the First Amendment was intended to protect.
_____B. Mary Beth Tinker used actions, not words, to make her point.
Joseph Frederick made his point with words written on a banner.
The question here is whether one can engage in speech without using
words. The Supreme Court has consistently held that actions (such as
burning a flag) can be a powerful form of speech that falls within the
First Amendment.
_____C. Mary Beth Tinker lived in Iowa. Joseph Frederick lived in
Alaska.
Both were U.S. citizens. The state of their residence makes no
difference when it comes to their rights under the U.S. Constitution.
_____D. Mary Beth Tinker’s case was decided 40 years ago. Joseph
Fredrick’s case just happened.
Yes, times have changed, but the First Amendment hasn’t changed.
Occasionally, the Constitution or its interpretation change to reflect
changing societal values (for instance, giving women the right to vote
and eliminating “separate but equal” racial discrimination), but one of
the strengths of our Constitution has been that its general principles
stay the same and don’t change with the passage of time.
13
2. What similarity between the two cases provides the best
reason to treat them the same? In other words, what would be
the best reason the Supreme Court could have used to decide in
favor of Joseph Frederick?
_____A. Mary Beth Tinker and Joseph Frederick were both students in
public schools.
The fact that both were students in public schools could be important,
because the First Amendment protects us only against suppression of
free speech by the government. These cases involved the First
Amendment because public schools are run by the government.
However, this similarity doesn’t help Joseph Frederick because even a
public school can restrict speech that is disruptive.
_____B. Mary Beth Tinker and Joseph Frederick were both children
and children shouldn’t have the same Constitutional rights as adults.
Other U.S. Supreme Court decisions have made it clear that while
students have Constitutional rights, those rights may be more limited
than the rights of adults. Thus, for instance, students have less right
to object to searches of their lockers and belongings than would adults
in a similar situation. However, the fact that students have more
limited rights doesn’t provide any reason why the Supreme Court
should have decided in favor of both Tinker and Frederick.
_X___C. Mary Beth Tinker and Joseph Frederick both expressed points
of view not approved by most of society.
People expressing points of view approved by society, or the
government, or a majority of their fellow citizens, don’t need to worry
about being allowed to express their views. Rather, it is those who are
expressing unpopular or minority views who need freedom of speech
the most and the U.S. Supreme Court has long recognized that.
_____D. Mary Beth Tinker and Joseph Frederick both acted in ways
that were silly and pointless.
While virtually any point of view could be considered silly and pointless
by some, the Supreme Court had already made it clear that Mary Beth
Tinker’s action was worthy of protection under the First Amendment.
Moreover, if only speech that is judged to be serious and have a point
is protected by the First Amendment then those in power will be free
to suppress unpopular views by simply deeming them silly or pointless.
14
3. Based on what you know about the Supreme Court’s
reasoning in the Tinker and Frederick cases, what is the best
reason for concluding, as the Supreme Court actually did, that
Joseph Frederick’s right to free speech did not give him the
right to display a banner stating “Bong Hits for Jesus?”
_____A. Joseph Frederick needs to learn to respect authority.
One purpose of schools is to socialize children, a process that includes
learning to respect authority. However, teaching Frederick to respect
authority is an entirely separate issue (not addressed in the
Constitution or the Bill of Rights) from his right to freedom of speech.
__X__B. Allowing Joseph Frederick to display the banner would be
disruptive by suggesting the school allows use of illegal drugs.
In Tinker the Supreme Court said students have a right to freedom of
speech, but only as long as they express themselves in a nondisruptive manner. Wearing an armband as a symbolic protest against
the Vietnam War did not disrupt the school (the administrators’
reaction did). But schools have very real problems with illegal drugs
and are expected to educate young people about the dangers of illegal
drugs. By displaying a banner that arguably promoted illegal drug
use, Frederick could reasonably be said to be disrupting the school’s
mission. Therefore, the rights given to Mary Beth Tinker did not apply
to him.
_____C. Allowing Joseph Frederick to display the banner would
undermine the authority of the school administrators.
School children are subject to the authority of school administrators.
However, the authority of school administrators is subject to (and
therefore limited by) the Constitution in general and the First
Amendment in particular. Therefore, any authority that would be
undermined if Frederick was allowed to display the banner is authority
the administrators did not have in the first place.
_____D. Separation of church and state prohibits mentioning Jesus in
public schools.
This is a misstatement. Separation of church and state (implied in the
right to freedom of religion, another right under the First Amendment)
does not prohibit mentioning Jesus in public schools. Rather, it
prohibits government from favoring one religion (such as Christianity)
over others. That is another topic entirely. Frederick didn’t claim to
be making any kind of religious statement with his banner.
15
Appendix 4
Constitution Day Evaluation
(Please Return to Kent Richards)
Name: __________________________
Please check all of the following that apply:
____ I was able to show the video to my class without any difficulty.
____ I was able to show the video to my class, but I had to overcome
some difficulty setting it up.
____ I was able to show the video to my class, but the sound quality was poor.
____ I was able to show the video to my class after Bill Berg or someone else
helped me with the setup.
____ I was unable to show the video to my class.
____ I was unable to show the video because it did not start at the announced
time.
____ The classroom I was in did not have the equipment necessary to show the
video.
____ I often have difficulty using the video equipment in the classroom I was in.
____ I don’t know how to use the video equipment in the classroom I was in.
____ The video equipment in the classroom I was in was out of order.
____ I didn’t understand the instructions for showing the video.
____ Other (explain):
Comments:
16