WRITTEN COMMUNICATION / January 2005
Creating the
Subject of Portfolios
Reflective Writing and the
Conveyance of Institutional Prerogatives
TONY SCOTT
University of North Carolina at Charlotte
This article presents research from a qualitative study of the way that reflective writing is
solicited, taught, composed, and assessed within a state-mandated portfolio curriculum.
The research situates reflective texts generated by participating students within the
larger goals and bureaucratic processes of the school system. The study finds that reflective letters are a genre within the state curriculum that regulates the substance and tone
of students’ reflections. At the classroom level, the genre provides a mode that students
adopt with the assurance that their reflections will meet state evaluators’ expectations.
At the bureaucratic level, the genre helps to continually validate the state’s portfolio curriculum through its strong encouragement of stylized narratives of progress. The study
demonstrates the importance of understanding how large-scale assessments shape
pedagogy and students’ writing.
Keywords: reflective writing; reflection; large-scale assessment; writing assessment;
portfolios; genre; accountability; Kentucky portfolio
Dear Reviewer: I have worked arduously over the past 2 years on these
pieces included in my portfolio. The particular products I have chosen
each show a bit about me as an imaginative person and an aspiring
writer. While some are very proficient, others are still at the point in
which they could be improved.
Even in the first sentence . . . “I have worked arduously over the past
few years,” I think it is BS. And that is what I try to avoid. But with these
Author’s Note: Special thanks to Lil Brannon, Boyd Davis, Christina Haas, Debra
Journet, Dottie Willis, and the many talented participating students and teachers in the
Kentucky school system who cannot be named. This work was supported, in part, by
funds provided by the University of North Carolina at Charlotte.
WRITTEN COMMUNICATION, Vol. 22 No. 1, January 2005 3-35
DOI: 10.1177/0741088304271831
© 2005 Sage Publications
3
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
4
WRITTEN COMMUNICATION / January 2005
things, I don’t know, I guess I didn’t care enough to give them an honest
voice or spend enough time to do so. It was just something that I
wanted to get out of the way.
The statements above were both made by “Barbara,”1 a high school
senior in a public school in Kentucky. The first quote is the opening
paragraph from her reflective letter to the reviewer, a required text in
a writing portfolio she submitted as a part of her graduation requirement. Consistent with the entire letter, this paragraph exhibits a generally positive tone and even conveys appreciation for various
aspects of portfolio pedagogy. In this case the student notes that the
portfolio afforded her the opportunity to review and revise texts. The
second quote is from one of two interviews I conducted with Barbara.
Consistent with the general substance and tone of those interviews,
Barbara sometimes expressed displeasure with the state’s required
portfolio and the ways that it affected her writing. In the reflective letter, the image is of an engaged student asserting authority of her own
work, in isolation from institutional processes, curricular requirements, and high-stakes assessments. In contrast, in interviews
Barbara talked at length about her frustrations with trying to write for
the state’s portfolio requirements. She described the specific, material
conditions of the portfolio’s production and revealed a sophisticated
understanding of how the portfolio fits into Kentucky’s large-scale
system of assessment and teacher and school accountability. The stark
difference in dispositions toward her writing portfolio that Barbara
exhibits in her reflective letter and in interviews was typical among
her classmates. How does one account for the difference?
During the 2000-2001 school year, I conducted a qualitative study
that examined how reflective writing is solicited, taught, composed,
and assessed within the state writing portfolio curriculum in Kentucky. The Kentucky Public School System is in many ways unique.2 It
was created in response to a 1989 decision by the Kentucky Supreme
Court, which found that the state’s school system did not meet basic
constitutional guarantees of equal education for its children. With this
ruling, the court struck down more than 700 laws that governed elementary and secondary education in the state. Prompted by the
court’s action, the Kentucky legislature took the opportunity to initiate a comprehensive overhaul of public education and passed the
Kentucky Education Reform Act (KERA) in 1990. The KERA innovations were progressive and broad, involving sweeping changes in
funding, administration, educational philosophy, and curriculums.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
5
Among the innovations was a large-scale portfolio assessment and
system of teacher and school accountability that has become
nationally recognized as a model (Hillocks, 2002).
Reflective writing in Kentucky is enacted within a portfolio curriculum that was developed, and has evolved for more than a decade, in
light of particular conversations in Composition and Education. This
study therefore approached the reflective writing of student participants as work that is situated within the bureaucratic system and culture created by these reforms. When students enter a writing classroom in Kentucky, they enter a space in which their roles and work
have already been somewhat preconstituted by the state’s system of
assessment and accountability and the pedagogical philosophy that
has informed it. In addition to studying curriculum and composition
in two senior English classrooms, the study therefore describes the
way that reflective writing—both as a concept and a practice
informed by particular ideas concerning literacy and pedagogy, and
as a specific generic form—fits within the bureaucracy of a particular
school system.
A growing body of research discusses the relationship between
large-scale “authentic” or “performance” systems of writing assessment, such as portfolio assessments and everyday classroom practices (see Camp, 1985; Freedman, 1993; Mitchell, 1992; Murphy,
Bergami, & Rooney, 1997; Simmons & Resnick, 1993; Underwood,
1999; Wiggins, 1989). This work points to a need to understand the
writing that is produced by students working within these large-scale
assessment systems in terms of their ecologies: focusing not just on
students, teachers, and classrooms but also on the larger practices and
goals that subsume classrooms. Below, I will present a reflective letter
that is deeply integrated within, and a product of, the practices,
assumptions, and goals of the state portfolio curriculum in Kentucky.
I will make the case that through the genre of reflective writing, the
system encourages the construction of a generic reflective subject that
reproduces the system’s ideal of a portfolio student. In the classes I
observed, the composition of the reflective letter is best described as
bureaucratic practice—a socializing process that reproduces the values of the sponsoring institution. Because the goals of reflection in this
instance appear more systemic than individual and dialogic, the
study highlights some of the problems with using reflection as an
aspect of writing assessment. Moreover, the study adds to existing
research that examines how large-scale assessments influence
everyday pedagogy.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
6
WRITTEN COMMUNICATION / January 2005
Reflective Writing
Reflective writing has become a widely used technique in composition classes. The popularity of the practice is perhaps largely due to
its enthusiastic promotion in the portfolio-oriented scholarship of the
early-to-mid 90’s, which lauds reflection for its potential to accomplish a variety of pedagogical goals, from fostering greater independence among students to making assessment more dialogic. However,
surprising, little research systemically examines how it actually functions in real school settings. Most scholarly discussion of reflection in
writing pedagogy is related to portfolios, and so a number of portfolio
studies mention reflective writing peripherally (for instance,
Belanoff, 1994; Camp, 1993; Camp & Levine, 1991; Hansen, 1992;
Lewiecki-Wilson, 1994; Murphy & Camp, 1996; Valencia & Calfee,
1991; Weinbaum, 1991). Some more recent scholarship has focused
more directly on reflective writing (for instance, Murphy, 1998;
Sunstein, 1998; Seale Swain, 1998; Yancey, 1996, 1998a). Although
substantial and certainly important, this work is primarily theoretical
and doesn’t systematically study reflective writing in a natural
setting.
A usual component of writing portfolios, reflective writing gives
students the opportunity to assess their own writing. Distinct from
certain types of religious reflection or journal writing, reflection in
writing pedagogy is not a practice that relies on an “inner voice.”
Rather, with reflection, students are encouraged to focus on certain
aspects of their writing, often in prescribed ways—for instance
through directive questions or writing prompts. Reflection as a learning tool in Composition Studies is thus used as a mode through which
students’ perceptions of their writing and composing practices are
formed through curriculum. Founded on Vygotsky’s articulation of
the relationship between learning and social development, reflection
is a type of scaffolding, a way of compelling students to see their work
through more sophisticated critical lenses. Teachers and large-scale
assessments that solicit writing portfolios usually require that they be
accompanied by a reflective letter that affords students the opportunity to reflect on the contents of their portfolios for evaluators. In the
letters, students might note tendencies across time and texts; they
might note that certain texts or passages in texts are particularly weak
or strong; they might even describe a general growth in writing ability
over time.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
7
However, assessment can complicate reflective practice considerably and raise a number of sticky questions concerning the practice.
Of particular concern are issues of authority and agency. Yancey has
recognized that reflection may be seen by students as an invitation to
read the teacher, and teachers may be compelled to evaluate and
respond based on the degree of success with which a student is able to
convincingly construct herself as the writer-in-development teachers
want to see. The problem of “schmoozing” as it has been termed by
Yancey and Weiser (1997) points to some ongoing discomfort with the
practice. Assessing reflective writing can put the evaluator in the
position of assessing honesty, determining which observations are
real and valuable, and which may be contrived or without merit.
Likewise, soliciting reflective writing could put students in the position of creating a persuasive persona—a subject who has convincingly adopted the particular critical stance that the curriculum is
designed to produce.
Another potential problem with reflective writing is that it is a
somewhat unwieldy hybrid of personal and public writing. Students
are often asked to reflect both for their own enrichment and to aid in
their own evaluation. The practice is intended to lend students independence while providing a wealth of information for assessment.
Some have been explicit about the institutional role of reflection. For
instance, Sunstein (1998) encourages teachers to use reflection as a
way to compel students to view their writing in terms of a set of
curricular standards:
As teachers, our job is not to assess the piles of artifacts students collect
and display in their portfolios; our job is to assist students to do it themselves, according to the institutional expectations we teach them, using
reflection and reflexivity as tools. (p. 42)
As a pedagogical practice, reflection can be used to blur the distinction between the personal and the institutional. Students are not only
typically expected to internalize curricular standards when they
reflect, they are also encouraged to perform that internalization for
what Sunstein calls the “institutional other.” In this view, reflective
writing is successful when students demonstrate that they can apply
the desired assessment standards to their own writing. At least on
paper, the personal becomes the institutional.
Yancey views this melding of the personal and the institutional as a
desired form of socialization—a natural outcome of the social view of
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
8
WRITTEN COMMUNICATION / January 2005
learning and writing that drives portfolio pedagogy (Yancey, 1998a,
p. 94). This socialization becomes complicated, however, when we
interrogate the position of the evaluator and/or her institution relative to that of the reflecting student. It is difficult to determine the line
between socialization and coercion. This line is especially indistinct
when reflective texts are a required part of an institutional system of
assessment and accountability, as is the case in Kentucky. The practice
highlights the tensions between the unique development of individual students and the broad aims and bureaucratic practices of an education system. The goals that teachers, students, state assessors, and
curriculum developers have for reflective texts are not easily congruent. A seemingly successful, insightful process of self-assessment for
a student doesn’t necessarily generate a reflective text that scores well
in the holistically scored state assessment.
Genre
Because it examines the ways that texts function within their contexts, the genre scholarship that has emerged during the past two
decades provides a useful analytical framework for the data collected
for this study. Carolyn Miller’s “Genre as Social Action” (1984) and
the English translation of Bakhtin’s Speech Genres and Other Late Essays
(1986) helped mark the beginning of the contemporary discussion of
genre in rhetoric and composition. Previous work with genre had
focused almost exclusively on textual conventions—for instance, the
standard sections of a research essay. This new work expanded the
focus of genre analysis from general textual features to the operations
of texts within their contexts. Miller described genres as part of an
ongoing relationship between rhetoric and context, a way of taking
action in specific situations. Her approach showed that genre study is
important because it emphasizes the social and historical aspects of
rhetoric that other perspectives had tended to omit.
This conception of genre spawned a still-expanding body of
research that systematically examines the complex relationships
between texts and the contexts within which they function. This
research accounts for a wide array of sometimes unwieldy elements
—such as social relationships, institutional hierarchies, material environments, heteroglossic languages, and individual and collective
agency—that have an impact on the composition and functions of
texts in natural settings (see, for instance, Bawarshi, 2000; Bazerman,
1997; Beebee, 1994; Berkenkotter & Huckin, 1993, 1995; Dias,
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
9
Freedman, Medway, & Paré, 1999; Freedman & Medway, 1994; Russell, 1997; Sauer, 1998; Schryer, 1993). Rather than viewing genre only
as a set of guidelines for texts, or as a tool that individuals learn to use
to achieve goals within specific contexts, these approaches see genre
as more constitutive of institutional power dynamics, values, and
subjectivity. Situated genres create rhetorical spaces that frame assertions and information, to a certain extent prescribing what can be said
and what is valuable. They also can create subject positions that people inhabit based on their function and rank within organizations
(Munger, 2000; Winsor, 2000).
Genres are therefore often not only regulative of texts, they are constitutive of activities and social orders. Drawing on the language of
Foucault, Bawarshi describes this constitutive nature of certain texts
as “the genre function.” Texts structure activities in ways that reinforce the ideological and social conventions that define the status quo
within a particular milieu. Given people’s positions within particular
organizations, genres help to determine what is appropriate and
inappropriate—what can or should be uttered in specific situations
by specific agents. They are thus an important factor in the shaping of
subjectivities.
BACKGROUND ON THE
PORTFOLIO CULTURE IN KENTUCKY
To understand the genre of the reflective letter in the Kentucky system, it is important to understand the state’s large-scale assessment.3
Among the important features of Kentucky’s system was a new
approach to assessment and accountability. The state assessment was
designed to promote the use of portfolios in classes and provide information concerning students’ learning for the state’s system of teacher
and school accountability. Each year, every public school in Kentucky
receives a numerical score based on various measurements of student
performance, including writing portfolios. This score is measured
according to the accountability index, and every 2 years, the average
of the scores are used to evaluate schools’ abilities to progress toward
the curriculum’s long-term numerical goals. All 12th-grade writing
portfolios are scored holistically according to a grading rubric developed by the state, with each receiving a rank of Novice, Apprentice,
Proficient, and Distinguished. These scores become a part of the
state’s system of accountability.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
10
WRITTEN COMMUNICATION / January 2005
The Kentucky curriculum is a culture of progression. The original
goal of the curriculum was to have every school in Kentucky averaging a proficient portfolio at the end of 20 years, and there is considerable pressure on teachers and schools to continually improve their
portfolio scores. This pressure was frequently manifested in the
classes I observed. For instance, in one of the classes, students were
not allowed to consider any piece—including the reflective letter—
finished until the teacher believed it would score proficient according
to the state’s holistic scoring guide. In both participating classes, students all had their own copies of the state’s scoring guide and were
encouraged to continually evaluate their work using its criteria.
As constituted during the year of the study, the Kentucky writing
portfolio completed by each graduating senior has five required
pieces from four categories: one reflective piece in the form of a letter
to the reviewer; one or two personal or expressive pieces in the form
of a personal narrative, a memoir, or a personal essay; one or two literary pieces in the form of a short story, a poem, or a play; and one or
two pieces of transactive writing in the form of editorials, letters, brochures, or feature articles (Writing Portfolio Development Teacher’s
Handbook, 1999, p. 20). After being completed by students and
turned in to teachers, the portfolios are evaluated locally by groups of
teachers at the schools at which they were compiled. Typically, these
groups are comprised of teachers from across subject areas and are led
by English teachers who have received some training in holistic scoring from the state. To ensure consistency and minimize the inflation of
scores, schools are regularly audited by the state. During these audits,
a sampling of portfolios is scored by a group from outside the school,
and the scores are compared to the scores given by the schools.
It should be noted that this system is designed primarily to assess
teachers, schools, and the system as a whole, not students—and the
rewards and penalties for teachers and schools under this system
have been significant. If the accountability score for a school exceeds
the target set by the state, the school receives public recognition and
its teachers can receive salary bonuses. If a school’s score does not
meet the target, it might get “assistance” from the state, which can
mean that the school gets direct intervention from state-appointed
educational consultants. On the school level, teachers are evaluated,
in part, according to how well their students are scoring on their portfolios. A school’s status within its community is also at issue, because
each school’s accountability scores are made public.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
11
Through assessment and accountability measures, the system not
only monitors its own success, but it enforces compliance: All students must turn in writing portfolios to graduate. If students are to
meet the requirements and score well on the state assessment, their
teachers must understand at least the basic tenets of the curriculum,
such as the characteristics of the required modes and how they will be
scored. Also, teachers in portfolio assessment grades are compelled to
center their curriculums on helping their students to produce portfolios that meet the requirements and score well. The portfolio in this
curriculum is therefore not just a learning tool for students or teachers: It is the embodiment of a state curriculum and assessment and an
important component in a number of bureaucratic practices. When
teachers and students who participated in this study referred to the
portfolio, they were not referring to a singular or isolated entity. The
term was synonymous with the state-mandated portfolio, with its
required elements, its established evaluation criteria, and its concrete
ramifications for teachers and schools. Each portfolio is interwoven
within a complex web of bureaucratic practices, and evidence of the
state’s curricular goals and accountability-based bureaucratic
processes abounded in the classes I observed.
There is a clear contrast between the bureaucratic and intended
pedagogical functions of the portfolio in Kentucky. From a wideangle, systemic view, the portfolio is characterized by requirements,
annual measurements, curricular consistency, and accountability. As
a pedagogical tool in particular classrooms, however, the same portfolio is intended to serve as a means through which students can gain
agency and a sense of ownership of their work. In its Writing Portfolio
Development Teachers Handbook, the Kentucky Department of
Education defines a portfolio as
a selection of a student’s work that represents his/her best efforts
including evidence that the student has evaluated the quality of his/
her own work and growth as a writer. The student, in conferences with
teachers, chooses the entries for this portfolio from the writing folder
which should contain several drafts of the required pieces. Ideally, the
writings will grow naturally out of instruction rather than being created solely for the portfolio. (Kentucky Department of Education, 1999,
p. 1)
It is interesting to note that this definition describes a process and a
culture as much as a material document. The definition also creates a
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
12
WRITTEN COMMUNICATION / January 2005
rather distinct portrait of the ideal portfolio student: A student who
makes her own choices concerning what to include in her portfolio, is
able to assess her own writing and growth, works in partnership with
her teacher, and writes the pieces that she wants to write—those that
“grow naturally out of instruction”—seemingly without regard to the
requirements of the portfolio. This is a description of the student subject that is found in most of the reflective letters to the reviewer collected for this study. However, this study will show that the bureaucratic functions of the portfolio exert considerable influence on how
students compile their portfolios and write their reflective letters.
THE STUDY
Although this study focused on two 12th-grade English classes, it
also accounted for many of the systemic elements that influenced—
and in many ways determined—the way that reflective writing was
conceptualized, composed, and assessed in these two classes. The
study therefore accounts for elements of the broad theoretical
assumptions that drive the state curriculum, the printed and verbal
articulations of the requirements for the state’s writing portfolio, the
state’s portfolio assessment process, and the system of teacher and
school accountability that is attached to that assessment. Students’
conceptions of the form and goals of the reflective letters are informed
by descriptions, models, guidelines, and checklists generated by the
Kentucky Department of Education and distributed by the school district to classroom teachers. These materials are, of course, supplemented by teachers’ instructions and feedback. In turn, students’
reflective texts provide information for teachers and administrators
about the efficacy of the teacher and/or the curriculum.
Method
The study resulted in more than 500 pages of transcriptions from
observations and interviews. It also incorporated hundreds of pages
of texts generated at the state, district, and classroom levels, such as
handbooks, curriculum support materials, assessment guidelines,
and student texts. The data were collected through interviews, observations, and textual analysis. Two high-ranking administrators were
interviewed and were also consulted in the design of the study. Both
had participated in the initial development of the portfolio system in
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
13
Kentucky and both had been administrators throughout the decade
since the system’s implementation. I sought teachers whom the state
considers among its most competent, teaching the writing portfolio in
the way that the state considers most ideal. Chosen in consultation
with a district official who has a high level of familiarity with the English teachers in the region’s schools, the two teachers whose classes
participated in this study were recognized for their skills and competence, both by their peers and among administrators outside of their
schools. Although the teachers had different backgrounds, both generally embraced the curricular principles and assessment processes of
the Kentucky portfolio system.
I conducted interviews and observations in two classes at two different schools. In consultation with the teachers, I chose 11 students to
be interviewed for the study. The students were chosen to represent a
broad range of attitudes toward writing (some were high-achieving
and confident writers, others struggled more with their writing).
Generally, reflecting the make-up of the student bodies at their
schools, the group was socioeconomically diverse: 7 were female and
4 were male; 5 were African American and 6 were Caucasian; 2 were
non-native speakers of English.
Interviews
I conducted 29 substantial interviews for this study (see Table 1).
The interviews were semi-structured. Astandard list of questions was
used for each round; however, in each interview, I also adjusted to the
information I was given (Patton, 1990; Rubin & Rubin, 1995). The
interview questions were designed to elicit information concerning
specific themes, but I also enabled the discovery of new themes
through open-ended and follow-up questions (see Appendix A).
Moreover, sometimes it was necessary to deviate from a particular set
of questions to make participants more comfortable and the interview
more conversational. I also often talked informally with the teachers
throughout the period of the study. When I felt that these conversations were relevant, I recorded them in observational notes.
During the initial phase of the research, I had interviews with two
state education officials and three teachers. These interviews were
designed to accomplish two primary goals: to enable me to better
understand the general systemic conception of portfolios and reflection in Kentucky public schools and to enable me to refine my initial
research questions and methodology based on their understanding.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
14
WRITTEN COMMUNICATION / January 2005
Table 1
Interview Source, Time, and Total
Participant Type
and Number
Number of
Interviews
Administrators, 2
1
Once before school year
2
Teachers, 3
2a
Once early in school year and again
at midyear
5
Students, 11
2
Times of Interviews
Once during portfolio compilation
(January to March) and again after
portfolios were completed (April to May)
Total
22
a. Only the two teachers whose classes participated in the study were interviewed
twice. The third participating teacher was interviewed once, early in the school year.
Their formulation was informed by my ongoing interaction with participants and my deepening understanding of the system and sites. I
interviewed the two teachers whose classes were the primary sites of
the study again at midterm. In the second round of interviews, each
teacher was given a different set of questions based on what I had
observed in their classes. During these interviews, I revealed some of
the general themes that I saw emerging from the data and asked the
teachers to address them.
I interviewed the students twice, once in the period between January and March as they compiled their portfolios and composed their
reflections, and again during the months of April and May after they
had turned in their portfolios. During the first round of interviews, I
asked the students to respond to a standard set of questions that were
designed to illuminate their individual understandings and conceptions of reflection and portfolios (see Appendix B). The second round
of interviews was primarily text-based: I read the students’ reflective
cover letters beforehand and formulated specific questions for each
participant. Mainly, I asked them to discuss specific rhetorical decisions they made in their reflective letters (for example, see Appendix
C). All of the interviews conducted for this study were audiorecorded, transcribed, and coded according to the procedures
described in Coding and Analysis below.
Observations
I conducted 12 class observations. Observation times were chosen
in conjunction with teachers, based on what would likely be most
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
15
Table 2
Varied Documents
State
District
Handbooks
Assessment guides
Calibration session materials
Curriculum support materials
Issues of Kentucky English Bulletin
Demographic information
Curriculum support
materials (various
handouts, assessment guides, and
instructions for the
relfective letter)
Model texts
Classroom
Handouts
Model texts
Student writing in a
variety of genres (rough
drafts and process
reflective writing)
Students’ reflective
letters to the reviewer
relevant to instruction concerning reflection and the state assessment.
In the spring, the observations were more frequent because this was
the period during which students compiled their portfolios and composed their letters to the reviewer. The observations were recorded in
two ways: I made audio-recordings and I took observational notes.
The observational notes served primarily to record conversations that
weren’t audio-taped, supply information that would be necessary to
contextualize the recordings, describe the physical characteristics of
sites, and reference handouts or other documents I collected that were
relevant to the recordings. Although I did make some analytical comments in the observational notes, I resisted analysis as much as possible in these notes with an eye toward enabling the coding process to
be as inductive as possible. The notes and audio tapes recording the
observations were transcribed according to the procedures described
in Coding and Analysis below.
Documents
I collected a wide variety of types of documents for this project that
were used in the analysis (see Table 2). These materials helped me to
gain an understanding of the culture created in the state’s school systems—its bureaucratic structure, its curriculum goals, its assessment
practices, what it encourages and values and what it discourages and
penalizes. Often, observations involved the discussion of texts, such
as a particular assignment sheet or a state-generated description of a
particular genre of writing, and during later analysis, these texts were
necessary for a deeper understanding of the event.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
16
WRITTEN COMMUNICATION / January 2005
Coding and Analysis
Though not always possible, I typically transcribed all interviews
and observations within 48 hours of their recording. This enabled me
to more accurately supplement the transcriptions with support notes
from my research log and observational notes that provide important
details, such as a description of a particular speaker’s gesture during
an important moment or the location of a specific text mentioned by
the speaker. All recordings were transcribed verbatim. Consistent
with an approach to data collection and analysis that has been termed
inquiry-guided (Mishler, 1990), reflexive (Atkinson, 1990; Emerson,
Fretz, & Shaw, 1995; Van Maanen, 1988), and dialectical (Emerson
et al., 1995), data analysis was a recursive process. I recognized that
the biases and preconceptions that I brought into the project affected
its design and my analysis. This reflexive approach was designed to
foster more awareness of these preconceptions and enable my conception of the direction of the project, research questions and methods
to evolve.
While collecting data throughout the school year, I wanted to recognize and value the participants’ voices and knowledge, and to let
the data suggest their own analytical possibilities, as much as possible. The process was designed to enable me to pursue alternative
research questions and new interpretive strategies (Miles &
Huberman, 1994). Midway through the school year, I began to analyze the data with an eye toward noticing significant categories of
data. Although I had entered the study expecting to gather data concerning portfolios and reflection, I was also getting significant
amounts of information about the formation and processes of the
Kentucky system of assessment and accountability (called KERA by
those who work in the system). Moreover, within these four categories—Portfolios, Reflection, KERA, and Assessment—certain subcategories—such as authority, agency, and accountability—were also
becoming salient. When a working list of categories and subcategories was developed from this analysis, I consulted with teacherparticipants concerning their importance and relevance. This clarified some issues and enabled alternative understanding of some of
the data. It also led me to explore certain issues with more depth as I
conducted the remainder of the study. For instance, in my ongoing
analysis of the various data I was collecting (from interviews, observations, support materials, etc.) I began to notice that the issue of students’ ownership of the writing they produced surfaced in a variety
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
17
of ways and in a variety of contexts. The system and participating
teachers clearly intended to foster feelings of ownership among students, but in observations I noticed some students expressing feelings
of disengagement from the writing they produced for their portfolios.
I therefore added ownership to the list of subcategories in the category
Portfolios and asked teachers and students directly about ownership
in interviews. Ownership proved to be among the most interesting
subcategories in the data.
When all data were collected, I refined the working list of categories and developed a list of corresponding codes. When satisfied that
the coding scheme was working well, I examined and coded all transcripts. I then further refined and verified the accuracy of the coding
scheme with a second reader through selective coding. A final list of
codes with definitions was developed as a result of this process. The
list included 14 subcategories under Portfolio, 17 subcategories under
Reflection, 8 subcategories under KERA, and 3 subcategories under
Assessment (see Appendix D). When data were organized according
to the codes, they were linked and analyzed with other documents
collected for the study (curriculum support materials, assessment
rubrics, handbook passages, etc.).
Analysis of Reflective Letters to the Reviewer
Fifty-six reflective letters to the reviewer were collected for the
study from the two participating classes. These letters were the final
drafts, turned in with the portfolios and submitted for assessment.
Initially, I read them with an eye toward identifying recurring rhetoric, content, and textual features. Based on a number of initial readings, I developed four main analytical categories that helped me to
define the typical letter to the reviewer. Letters were analyzed according to Form, Tone, Mention of Growth, and Appeals to the Reviewer
(see descriptions of categories below). Working independently, three
experienced researchers were given descriptions of the categories
along with illustrative examples from sample reflective letters. Only
agreement among all three readers was recorded as positive.
Form. Form refers to the organizational scheme of the reflective letters. The generic classification has three primary elements:
1. An introductory paragraph or paragraphs in which the writer makes
general statements about themselves as a writer. These introductory
paragraphs can vary somewhat in content and nevertheless be within
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
18
WRITTEN COMMUNICATION / January 2005
the bounds of the genre. I found that each English class observed for
the study formed its own subgenre based on what teachers emphasized. For instance, one of the participating classes emphasized literacy narratives, whereas the other focused more tightly on growth
through the portfolio. The reflective letters generated in these classes
were usually framed accordingly.
2. Middle paragraphs that provide a piece-by-piece description of contents. Each paragraph is devoted to one of the four pieces required for
the portfolio. Particularly sophisticated letters sometimes have transitional paragraphs and submerge this listing function of the genre
within analysis. More typical letters tend to begin each paragraph with
a formulaic sentence that identifies each piece by title and clearly
delineates which state-required mode the piece fits.
3. A conclusion that summarizes the letter in some general way—often
emphasizing positive elements of the portfolio.
Letters were categorized by readers as generic, general, or other.
Generic letters fit the above description closely. General letters also fit
the above description, with one exception: Not every piece included
in the portfolio is mentioned in the body of the letter: In some cases,
writers just emphasize a few pieces. Other refers to reflective letters
that did not follow this organizational scheme.
Growth. Another salient feature of the genre is that the reflective letters tend to be growth narratives. Writers mention specific ways that
they have grown throughout the year. Sometimes the mention is explicit (for example, “I feel that I have really grown as a writer”), and
these letters were classified explicit. At other times, the term growth or
an equivalent is not used, but the writer is clearly describing a progressive learning process (a trajectory from worse to better); these letters were classified implicit. Letters that did not clearly describe a
growth process were categorized no growth.
Tone. This category describes the general disposition of the writer
toward her work. If a writer is enthusiastic about the writing process
and the portfolio—topic development, revision, the opportunity to
explore various genres—in her reflective letter, then the tone was
coded positive. If the dominant disposition is negative or plaintive toward writing generally or the portfolio requirements particularly,
then the tone was categorized negative. Indeterminate letters have no
clearly dominant tone.
Appeals to the Reviewer. Some letters appeal directly to the reviewer
for a positive evaluation—typically in the final paragraph. These letters were classified explicit appeal. Other letters contain passages that
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
19
are obviously appeals to the reviewer, but the appeals are less explicit.
For instance, a letter might clearly be making the case for a positive
evaluation, but without explicitly mentioning the evaluation or
appealing directly to the reviewer. These letters were classified
implicit appeal. Letters that do not appeal to evaluators in any overt
way were marked none.
RESULTS
Analysis of Reflective Letters
The following reflective letter was written by “Katrina,” a successful 12th-grade student who planned to enter a premed program on
graduation. Katrina’s reflective letter fits the generic form identified in
the study. Of the 56 reflective letters collected for the study, 36 (64%)
were categorized as generic by all three raters. An additional 13 were
categorized by raters as either generic or general adoptions of the form.
Only four of the letters (7%) were considered other than the generic
form.
Katrina’s introductory paragraph stresses that she has grown as a
writer and mentions reflection as an important part of that growth:
Dear Reviewer,
The growth as a writer that I have experienced throughout my high
school years was ultimately expressed in the pieces that I have selected
for this portfolio. The hardest part of perfecting my portfolio was revision, for it was hard for me to find mistakes in what I had considered
some of my best work. However, the process of reviewing helped me
see a pattern of mistakes in my writing, and the weaknesses that I have
as a writer. Consequently, this has taught me to be more cautious about
making the same mistakes again.
Eighty percent of the reflective letters describe a growth trajectory.
Significantly, Katrina’s opening makes it seem as though she is engaging in this process for her own benefit—there is no indication that she
is required to write this reflection, nor does she mention that all of the
pieces included in her portfolio were written to fit the state’s required
modes. Moreover, she makes it clear from the beginning that she has
benefited from the curriculum, particularly the fact that it enables students to make revisions. Raters agreed that 32 (57%) of the reflective
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
20
WRITTEN COMMUNICATION / January 2005
letters collected for the study exhibit a clearly positive tone. No full
agreement was achieved for any particular letter in the negative
category.
After the introductory paragraph, generic reflective letters to the
reviewer move through a body comprised of a series of paragraphs
that describe each of the included pieces. These paragraphs are often
sequenced according to the way that the required pieces are
sequenced in the state’s curriculum support materials. The first piece
discussed is the personal narrative or memoir(s), followed by the literary piece(s), and then the transactive piece(s). In both of the classes
observed for this study, the students worked on their portfolio pieces
in this sequence—starting with the personal expressive and ending
with the transactive. The reflective letter to the reviewer was composed when the other pieces were completed.
Each paragraph in the body of Katrina’s letter is devoted to a different piece and, without being explicit about the state requirement,
makes it implicitly clear which of the state’s required modes each
included piece fulfills (personal narrative, literary piece, two
transactive pieces). Also, the paragraphs are sequenced consistently
with the sequencing that is found throughout the state’s support
materials for reflective letters. Below is Katrina’s description of her
personal narrative:
The first piece that I chose is titled. It is an extremely emotional piece,
for it contains the most intricate details about my life. This piece was
originally written in two thousand words and later shortened to five
hundred. The reason for which I chose to shorten this piece is that I felt
it contained certain personal information that was not necessary in the
context of the story. When revising it, it was very difficult to maintain
the main focus and include the most relevant details in merely one forth
of its former size. On the other hand, learning how to shorten long
works resulted [in my] being very productive for I often tend to write
extensively.
Katrina closes with a paragraph in which she again stresses her
growth and the importance of revision. There is also an unmistakable
note of humility that is common in the genre.
Even though my improvement as a writer was apparent when comparing my writing to that written four years ago, revising my best work
made me realize that I still have much to learn, and many skills to
develop in the art of writing.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
21
Others address the reviewer more explicitly in the closing paragraph.
Consider, for instance, the following three examples:
• “In the end, I would like to thank you for taking the time to read my
portfolio.”
• “I’m grateful to have another person invest in my work as well. Thank
you.”
• “Thank you for taking the time to read this portfolio entry.”
Seventy-three percent of the letters contained explicit or implicit
appeals to the reviewers.
Katrina’s reflective letter convincingly portrays her as a reflective,
developing writer. She makes it clear that the portfolio contains all of
the required pieces. She discusses each required piece with some
depth, indicating that she has grown as a writer and appreciates the
opportunities to revise and reflect that are afforded by the portfolio
curriculum. To use the phrasing of the definition of portfolios quoted
above from the teacher’s handbook, the work she presents in her portfolio seems to “grow naturally out of instruction.” She seems to have
composed these pieces because she wanted to, not because she was
required to produce work in four explicitly described modes. The
pieces appear to be chosen from a larger body of work, and the writer
appears to be working for her own benefit, without regard to the
state’s assessment, or its ramifications for her teacher and school.
However, in both classes observed for this study, much of the school
year was structured around enabling students to compose and revise
the five pieces that fit the state’s required modes. Classes moved
methodically through the modes one by one: Each required piece was
a class assignment and a distinct and structured unit. Students at both
schools thoroughly internalized the state portfolio requirements and
were familiar with its vocabulary and its assessment criteria. I
observed several class sessions that were devoted entirely to enabling
students to learn the scoring guide and apply its evaluative criteria
consistently, both to their own work and the work of peers. Nevertheless, they rarely explicitly mention particular pieces as assignments
for classes or as requirements in their reflective letters.
Interviews
The reflective letter presents a curricular success story. Katrina
seems to have adopted the critical lens of the “institutional other” so
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
22
WRITTEN COMMUNICATION / January 2005
naturally that she has been thoroughly socialized, and because the
standards and requirements themselves are rarely explicitly mentioned, they are universalized. After all, when someone is fully socialized, the arbitrary habits of a particular context have become natural
and intuitive.
I found, however, that this letter is at least as much a function of the
bureaucracy within which it was generated as a demonstration of sincere investment or deep socialization. The portrait of the student that
is presented in Katrina’s letter is significantly different from the student that I encountered during interviews. Indeed, student participants often expressed dramatically different views of the state writing curriculum and the contents of their portfolios in interviews than
they expressed in the reflective letters. For instance, in contrast with
the general tone of her letter, in interviews Katrina indicated that she
felt little sense of ownership of her portfolio and was not very
engaged in compiling it or writing the reflective letter. When
discussing her investment in the portfolio, Katrina said,
personally it is not something that I want to put too much time into,
because it is not something that is helping me in any way right now. I
see it more as busy work than anything else because I am kind-of being
pushed into doing it. I am not enjoying it as perhaps I should be. It is just
something that I have to have as a criteria to graduate. . . . I love writing,
but not when it is for a reason like this. (1.1 18-23)4
Katrina was even more explicit about who owns the portfolio and its
primary function:
Um, basically [the portfolio] is for the state. . . . It is to judge the school
and how the school is doing as far as writing goes and that is why I
don’t think it is a valid thing to do. . . . It is only to judge the school. (1.3
22-23; 4 1-4)
When I asked Katrina whether the portfolio helped her to become a
better writer, she brought her mouth close to the microphone and
answered,
Nooooooo. Because, like in my case for example—not just mine, but in
my English class—everybody is like kind of cramming up and just
doing it last minute, like focusing on it last minute to get it together so
the school will look better. You know what I am saying? It is not
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
23
something that we are taking the time to, you know, consider and write
about and do because we like it; it is something that we are obligated to
do. (1.6 7-14)
This is hardly the portrait of the student that shows up in Katrina’s
reflective letter. Indeed, although Katrina talked at length in interviews about the Kentucky curriculum and displayed considerable
knowledge of—and bitterness toward—its requirements and assessment processes, her letter mentions no requirements and indicates no
bitterness toward the curriculum whatsoever. Like most of her classmates, Katrina was conscious of a specific generic form that students
are expected to adopt for their letters. She mentioned that she had not
only seen models in this class (consistent with what I observed in class
sessions), but she had used models in her junior English class as well. I
found that the genre is conveyed and reproduced through a variety of
means: from state-generated models, checklists and prompts, to
explicit teaching in the classroom.
Katrina also indicated that there is a generally recognized form for
the reflective letter, and she simply adopted it for the sake of expediency: “I just tried to make it typical, like a typical letter to the reviewer.
I didn’t try to be creative or anything like that” (2.3 17-18). When I
asked Katrina whether she felt that she had benefited from writing
within the established form that she recognized for the reflective
letter, she answered,
no, I don’t think so. What I wrote was what I had already thought about,
and you know it is just trying to stick to the structure of what it is supposed to look like, but I don’t think I learned anything. (2.3 20-22)
The overall impression Katrina gave of her composition process was
that she recognized a standard form; she adopted it with little enthusiasm, and she didn’t think the process was very worthwhile for her—
it was a required exercise that served the state’s own assessment purposes. Katrina believed that she is merely fulfilling a requirement,
adopting a recognized, institutionally proven voice and form to display adequate competence in the assessment.
In great contrast with the general tone and substance of their reflective letters, most students interviewed for the study also indicated
that they were not invested in writing the reflective letters for their
own benefit. Showing an often surprisingly detailed knowledge of
the state’s assessment and accountability practices, students thought
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
24
WRITTEN COMMUNICATION / January 2005
that their portfolios primarily served the state’s need to evaluate its
own curricular effectiveness and growth and that their own learning
and agency was secondary. They also often expressed considerable
anxiety about the state’s requirements and recognized that their
teachers and schools needed their portfolios to score well to receive
rewards and avoid sanctions. For instance, when asked to discuss the
purpose of her portfolio, Mary, Katrina’s classmate, said,
It is meant to help the school board to evaluate the way that teachers
teach their approach and the programs that [the] county uses; I would
like to think that I own my own writing, but as far as the portfolio, once I
sign off on it, it is theirs to do what they want with it. . . . It is definitely
more [the] county’s than mine. (1.10 15-16; 1.2 6-8)
Predictably, Mary’s reflective letter to the reviewer also expresses a
positive orientation toward the portfolio, and does not mention the
state or its assessment.
Jim, another classmate, said,
Well . . . I think in the short term, it serves the purposes of the state, but
in the long term it serves the purposes of the kids. Not necessarily the
kids who write it, but I know the state looks at it and that is how they
plan what they are going to focus on in curriculum and I like to think
that gets back to the future students in the cycle somewhere. . . . I don’t
think I am going to reap any long-term benefits from it. (1.5 8-14)
Again, these feelings rarely get expressed in the reflective letters that
are written as a part of the state’s large-scale assessment—this type of
critique is outside of the bounds established by the genre. In generic
reflective letters, students are self-assessing for their own benefit:
There is no institutional framework. To critique, or even explicitly
acknowledge, the sponsoring bureaucracy would not be consistent
with the subject position established by the genre.
CONCLUSION
I found that the genre of the reflective letter to the reviewer serves
multiple functions within the context of the Kentucky writing curriculum, and this has made it a highly intractable generic mode. Indeed,
Kentucky curriculum administrators expressed some frustration
with the pedestrian adoption of the genre by students throughout the
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
25
system and indicated that they would like to see much more innovation and diversity in the letters. They attributed this lack of investment among students to a continued lack of understanding and support of portfolio pedagogy among too many teachers in the system.
Although this may be generally true, the teachers in the two classes I
observed embraced the Kentucky curriculum, and they exhibited a
sophisticated understanding of portfolio pedagogy. Nevertheless,
they taught, and most of their students adopted, a standard form for
their reflective letters. Rather than a result of a lack of engagement,
understanding, or investment in the curriculum on the part of teachers, the genre is so intractable only because it serves many pragmatic
functions:
It sets a benchmark for teachers—helping them to understand what the state
expects. The reflective letter to the reviewer, and indeed reflection generally as an aspect of writing pedagogy, are vague. Scholars who theorize reflection, and most teachers who assign it, may even make it
vague intentionally—the less prescribed the form, the more opportunity for creativity and innovation. However, in this system, the reflective letter to the reviewer is a required form in a portfolio that is a part of
a statewide system of accountability. When careers, funding, and salary
bonuses are at stake, vagueness is not likely to be seen as an opportunity for creativity and innovation: It is rather a source of anxiety and
frustration. Simply put, the genre of the reflective letter to the reviewer
helps to clarify for teachers and students what the state expects of the
text, and meeting expectations leads to higher scores. The Kentucky
portfolio requirements were initially the cause for considerable anxiety
among English teachers throughout the state (see Callahan, 1997, 1999).
Over the years, the state has responded to this anxiety through providing models and other support materials and making reflective writing a
regular part of its professional development efforts. The result of this
clarification is a standard genre that gives teachers a benchmark and a
level of comfort.
It offers students a form that enables them to fulfill the requirement with a
limited amount of investment and exposure. Students understand the
stakes of the Kentucky assessment for their teachers and schools. Moreover, the pressure that teachers are under to have their students’ portfolios score well gets transferred to students in various ways. Students
spent considerable time in the writing classes I observed learning and
meeting the Kentucky requirements. They adopted the genre with the
assurance that they would fulfill the “reflective letter to the reviewer”
requirement to the satisfaction of their teachers. These students were
busy and anxious to graduate, and faithful adoption of the genre is the
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
26
WRITTEN COMMUNICATION / January 2005
path of least resistance—it is the easiest, most efficient, course. In interviews, student participants indicated that they did not put much effort
into their reflective letters at all. Indeed, several just modified the letters
they had completed in their junior years. One student even admitted
that she wrote her reflective letter the morning it was due.
It validates the curriculum through producing the subjects that the curriculum envisions. The reflecting students who show up in reflective letters
to the reviewer are the students the curriculum is designed to produce.
The genre encourages students to become the “Kentucky reflecting student” in their work. Seeing positive, independent students praising
various aspects of portfolio pedagogy in letter after letter helps to create the progressive culture the portfolio was intended to promote.
The Costs
Adopting a genre for reflection is not necessarily bad. As much
prior research has shown, sophisticated writers learn to recognize
and adapt genres to achieve their goals in particular situations. In
Kentucky, students learn to adapt genres to achieve favorable scores.
There is something nevertheless troubling, if not cynical, in this particular instance of adaptation. In this case, the genre masked students’
more negative feelings about the requirements of the curriculum and
likely contributed to students’ lack of investment in the reflective process. There is an unmistakable systemic logic at work here that exerts
considerable influence on students’ thinking and writing. The curriculum seeks to foster a sense of ownership among students. Therefore,
the reflecting student, encouraged by the genre, presents her work as
though she wrote it for herself and then included it in the portfolio
because it happens to be her best work. The overall impression students leave in their reflective letters is that they are using the portfolio
exclusively as a technology to improve themselves—this is not something that they have to do, and the state assessment and the accountability ramifications for their teachers and schools don’t significantly
affect the composing process.
It may be, however, that ownership and empowerment are more
contextually prescribed rhetorical stances than actual dispositions.
Most student participants saw the composition of the reflective letter
primarily as a bureaucratic exercise rather than as an empowering or
even worthwhile learning event. Unfortunately, when students
adopted the subject position that they are compelled to adopt in
reflective letters, too many were unengaged and resentful. A few even
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
27
admitted to being consciously dishonest in their letters to adopt the
tone that they felt would effect the most favorable score. The invested
student subjects one encounters in the reflective letters collected for
this study are largely an institutional creation, a carefully crafted
generic entity that continually validates a large-scale portfolio
initiative.
A serious concern raised by the study is that this practice encourages students to view literacy in terms of bald, if sophisticated, compliance to bureaucratic prerogatives. The assessment exerts a powerful influence on students’ work. When compiling materials for their
portfolios—a year-long process—students conceive and shape texts
based on state-generated descriptions of the required modes, and
they critique and revise based on the state’s holistic scoring rubric.
Even reflective writing, ostensibly a form of self-analysis, takes place
in an institutional forum and is scrutinized according to institutional
means and standards. Participating students felt compelled to create
plausible narratives of curricular success. What is lost in the process is
the kind of growth that can come from a more critical, more ambivalent, perhaps even oppositional stance. What are the possibilities for
growth outside of the modes and standards imposed by this system?
In what ways are the creative and intellectual possibilities of teaching
and writing limited by the system’s need to have students produce
texts that are standardized enough to yield reliable numbers in the
large-scale assessment? As accountability measures and standardized curriculums become more pervasive, there is an urgent need for
more systematic investigation of how students’ writing and general
dispositions toward literacy are affected. More studies should examine everyday pedagogy and students’ composition processes in light
of the bureaucratic cultures created by accountability measures.
In terms of reflective writing, it clearly does not, of itself, make
classrooms less authoritarian and assessments more dialogic. Indeed,
because the practice tends to conflate the personal with the social,
reflection can be a particularly invasive means of reinforcing institutional authority. Before teachers and curriculum designers incorincorporate reflection into a standardized assessment, they should
carefully examine the broad social/political ramifications of asking
students to self-assess for the benefit of any evaluating authority.
Those who assign and assess reflective writing should be mindful of
the dispositions toward authority that this practice might foster in
time.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
28
WRITTEN COMMUNICATION / January 2005
NOTES
1. All names used are pseudonyms. All participants signed an informed consent
form. Those who were not 18 at the time of their participation were also required to get a
signature from a parent or guardian.
2. Other states have implemented limited portfolio curriculums, such as Vermont
(see Koretz et al., 1993) and California (see Freedman, 1993, and Calfee & Perfumo,
1996). Kentucky, however, has implemented a statewide writing portfolio curriculum
for more a decade now. It has, therefore, been the subject of a number of studies (see
Callahan, 1997, 1999; Guskey, 1994; Evaluation Center of Western Michigan University,
1995; Whitford & Jones, 2000).
3. This description of the Kentucky system refers to the system as it existed in the
2000-2001 school year.
4. All interview citations are referenced according to the interview title and number,
page number and line number from the study transcripts.
APPENDIX A
Questions for State Education Officials
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Why do you advocate the use of portfolios in writing instruction?
How do they benefit students?
How do they benefit the school system?
Do you think portfolios are generally functioning in the way that was
envisioned by those who initiated the curriculum?
Can you describe the role you think reflective writing plays in portfoliocentered pedagogy?
How do you think reflective writing should be assessed? What does good
reflective writing look like? What does poor reflective writing look like?
What makes reflective writing distinct from other types of writing—like
narrative or expository writing?
When students write reflective letters for their portfolios, who is their
audience?
Do you think that reflection is generally understood and practiced in the
same way by teachers throughout the state?
How has the general perception of reflection within the system changed
since the implementation of KERA.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
29
APPENDIX B
Questions for First Student Interviews
1. What are your goals for your portfolio?
2. If forced to define a portfolio, what would you say it is?
3. How do you think your portfolio will be evaluated—what are the specific
qualities you think it should have?
4. Who owns the writing in your portfolio?
5. Do you think that using writing portfolios has helped you to become a
better writer? Why or why not?
6. If forced to define reflection as it relates to your development as a writer,
what would you say it is?
7. What is the purpose of the reflective cover letter you will be submitting
with your portfolio?
8. Who do you imagine will be the audience of the cover letter?
9. How do you think it will be evaluated?
10. What qualities does a good reflective letter have?
11. What qualities would a bad reflective letter have?
12. What specific things has your teacher done thus far to prepare you to
write your reflective letter?
APPENDIX C
Sample Interview Excerpt From
Second Round of Student Interviews
Why do you only discuss three of the five required pieces in your reflective
letter?
Um, honestly, I really wanted to, a lot of what I had included in this was
in my letter to the reviewer last year, because I was using the same
pieces in my portfolio. I wanted to go a little further into what, like
being honest, more so than it was last year—such as like talking about
my memoir. I included this time that a lot of my memoir was fiction. So I
was very truthful—which I was trying to go for. I was trying to articulate myself well. And I was trying to include the things that she had
gone over and what we had been learning in class as far as not only
explaining yourself, but getting both sides, you know the negative and
the positive, and I didn’t want to talk about all of my pieces so I just
used three of my favorite ones. And when I look back it is all of the ones
that I had the most flexibility with—like um, that I could use my creativity in. Whereas with my social studies piece it was all like an editorial. And with these I could have used my imagination more. I wish that
I could have maybe covered one of these fictional pieces and then
maybe the social studies piece. But, these were my three favorites and
the ones that I knew what was going on in my head when I was writing
those.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
30
WRITTEN COMMUNICATION / January 2005
APPENDIX D
Table 1A
Code List
Code
Portfolios
P:
P:
Goals
Definition
Language
Genres
Assessment
Perception of values
Influence
Contrast with 10th
Authority
Accountability
Ownership
Usefulness
Pressure (obligation)
Agency
PG
PD
PD—L
PD—G
PA
PA—V
PA—I
PA—C
PAUTH
PACC
PO
PU
PP
PAG
RD
RD—S
RD—UNS
RD—P
RR
RR—AUD
RR—INTENT
RR—TON
RG
RG—E
RG—F
RAUTH
RP
RPROCC
RA
R:
Definition
Successful
Unsuccessful
Purpose
Rhetoric
Audience
Intention
Tone
Genre
As an evolving genre
Elements that form genre
Authority
Reflective practice
Process
Assessment
Reflection as a tool for assessing programs and
teachers
And Portfolios
KERA
K:
K:
K:
K:
K:
K:
K:
K:
Individual perception
General perception
Early history
Failings
Successes
Goals
Present situation
Politics
KIP
KGP
KEH
KF
KS
KG
KPS
KPOL
P:
P:
P:
P:
P:
P:
P:
Reflection
R:
R:
R:
R:
R:
R:
R:
R:
RAPT
RPORT
(continued)
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
31
Table 1A (continued)
Code
Assessment
A:
Assessment as a tool for curriculum formation
A:
Holistic Scoring/Rubrics
A:
Development of system of assessment (historical)
ACF
AHS/R
AHD
Table 2A
Definitions of Codes
Portfolios (P)
PG
PD
PD—L
PD—G
PA
PA—V
PA—I
PA—C
PAUTH
PACC
PO
PP
PAG
Reflection (R)
RD
RD—S
RD—UNS
RD—P
RR
RR—AUD
RR—INTENT
Definition
Discussions and descriptions of goals for portfolios
Definitions of portfolios as an aspect of writing pedagogy,
emphasis on pedagogical and learning goals
Definitions of the language of the Kentucky portfolio system
Definitions of the genres created by the Kentucky portfolio
system
Discussions of portfolio assessment
Discussions and descriptions of what is valued in the portfolio
assessment
Influence of assessment on writing and pedagogy
Discussions and descriptions of contrast between 10th and
12th-grade portfolios
Discussions and descriptions of portfolios and student authority
Discussions and descriptions of portfolios and teacher and
school accountability
Discussions of portfolios and student ownership
Instances and discussions of pressure on students to compile
high-scoring portfolios
Discussions and descriptions of portfolios and student agency
Definition
Definitions of reflection as an aspect of writing pedagogy,
emphasis on pedagogical and learning goals
Definitions and descriptions of successful reflection
Definitions and descriptions of unsuccessful reflection
Definitions and descriptions of the pedagogical purpose of
reflection
Reflection as rhetoric, emphasis on rhetorical aspects of
reflective texts
Definitions and descriptions of the audience of reflection
Definitions and descriptions of the intentions of reflection—what
the texts do for their audience
(continued)
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
32
WRITTEN COMMUNICATION / January 2005
Table 2A (continued)
Reflection (R)
Definition
RR—TON
Definitions and descriptions of the desired tone of reflective
writing
Definitions and descriptions of reflection as a genre
Descriptions of how the genre has evolved
Elements that form the genre (models, descriptions, exercises,
etc.)
Discussions of authority (student independence and agency) as
an aspect of reflective practice and writing
Descriptions of students engaging in reflective practice
Descriptions and definitions of the reflective process
Discussions and descriptions of the assessment of reflection
Discussions and descriptions of how reflection functions in the
assessment of teachers and programs
Discussions and descriptions of how reflection relates to
portfolios
RG
RG—E
RG—F
RAUTH
RP
RPROCC
RA
RAPT
RPORT
KERA (K)
Definition
KIP
KGP
Individual perceptions of the KERA system
Discussions and descriptions of the general perception of the
KERA system
Discussions and descriptions of KERA’s early history
Discussions and descriptions of KERA failures
Discussions and descriptions of KERA successes
Discussions and descriptions of the goals of the KERA system
KERA’s present situation
KERA and local and state politics
KEH
KF
KS
KG
KPS
KPOL
Assessment (A)
ACF
AHS/R
AHD
Definition
Discussions and descriptions of assessment as a tool for
curriculum formation
Discussions and descriptions of assessment and holistic
scoring and rubrics
Discussions and descriptions of the historical development of
the Kentucky assessment
NOTE: KERA = Kentucky Education Reform Act.
REFERENCES
Atkinson, P. (1990). The ethnographic imagination: Textual constructions of reality. New
York: Routledge.
Bakhtin, M. M. (1986). Speech genres and other late essays (V. W. McGee, Trans., C. Emerson & M. Holquist, Eds.). Austin: University of Texas Press.
Bawarshi, A. (2000). The genre function. College English, 62(3), 335-360.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
33
Bazerman, C. (1997). The life of genre, the life of the classroom. In W. Bishop & H.
Ostrom (Eds.), Genre and writing: issues, arguments, alternatives (pp. 19-26).
Portsmouth, NH: Boynton/Cook.
Beebee, T. O. (1994). The ideology of genre. Pennsylvania: Pennsylvania State University
Press.
Belanoff, P. (1991). Addendum to “Using portfolios to increase community in a writing
program.” In P. Belanoff & M. Dickson (Eds.), Portfolios: Process and product (pp. 3036). Portsmouth, NH: Boynton/Cook.
Belanoff, P. (1994). Portfolios and literacy: Why? In L. Black, D. Daiker, J. Sommers, & G.
Stygall (Eds.), New directions in portfolio assessment: Reflective practice, critical theory,
and large-scale scoring (pp. 13-24). Portsmouth, NH: Boynton/Cook.
Berkenkotter, C., & Huckin, T. (1993). Rethinking genre from a sociocognitive perspective. Written Communication 10(4), 475-509.
Berkenkotter, C., & Huckin, T. (1995). Genre knowledge in disciplinary communication: Cognition, culture, and power. Hillsdale, NJ: L. Erlbaum Assoc.
Callahan, S. (1997). Tests worth taking? Using portfolios for accountability in Kentucky.
Research in the Teaching of English, 31, 295-336.
Callahan, S. (1999). All done with the best of intentions: One Kentucky high school after
six years of state portfolio tests. Assessing Writing, 6(1), 5-40.
Calfee, R., & Perfumo, P. (Eds.). (1996). Writing portfolios in the classroom: Policy and practice, promise and peril. Mahwah, NJ: Erlbaum.
Camp, R. (1985). The writing folder in post-secondary assessment. In P. Evans (Ed.),
Directions and misdirections in English evaluations (pp. 91-99). Ottawa: Canadian
Council of Teachers of English.
Camp, R. (1993). The place of portfolios in our changing views of writing assessment. In
R. E. Bennett & W. C. Ward (Eds.), Construction versus choice in cognitive measurement:
Issues in constructed response, performance testing, and portfolio assessment (pp. 183212). Hillsdale, NJ: Lawrence Erlbaum & Associates.
Camp, R. (1998). Portfolio reflection: The basis for dialogue. The Clearing House,
72(1),10-12.
Camp, R., & Levine, D. (1991). Background and variations in sixth through twelfth
grade classrooms. In P. Belanoff & M. Dickson (Eds.), Portfolios: Process and product
(pp. 194-205). Portsmouth, NH: Boynton/Cook.
Dias, P., Freedman, A. Medway, P., & Paré, A. (1999). Worlds apart: Acting and writing in
academic and workplace contexts. New York: Lawrence Erlbaum and Associates.
Emerson, R., Fretz, R., & Shaw, L. (1995). Writing ethnographic fieldnotes. Chicago: University of Chicago Press.
Evaluation Center of Western Michigan University. (1995). An independent evaluation of
the Kentucky Instructional Results Information System (KIRIS). Frankfort, KY: Kentucky Institute of Educational Research.
Freedman, S. (1993). Linking large-scale testing and classroom portfolio assessments of
student writing. Educational Assessment, 1(1), 27-52.
Freedman, A., & Medway, P. (1994). Learning and teaching genre. Portsmouth, NH:
Boynton/Cook Heinemann.
Guskey, T. R. (1994). High stakes performance assessment: Perspectives on Kentucky’s educational reform. Thousand Oaks, CA: Corwin Press.
Hansen, J. (1992). Literacy portfolios: helping students know themselves. Educational
Leadership, 49(8), 66-68.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
34
WRITTEN COMMUNICATION / January 2005
Hillocks, G., Jr. (2002). The testing trap: How state writing assessments control learning. New
York: Teachers College Press.
Koretz, D. et al. (1993). Can portfolios assess student performance and influence instruction?
The 1991-92 Vermont experience. CSE Tech. Rep. 371. Los Angeles: CRESST/RAND
Institute on Education and Training, University of California.
Lewiecki-Wilson, C. (1994). Teaching in the contact zone of the two-year college classroom: Multiple literacies/deep portfolio. Teaching English in the Two-Year College,
21(4), 267-276.
nd
Miles, M. B., & Huberman, M. (1994). Qualitative data analysis (2 ed.). Thousand Oaks,
CA: Sage Publications.
Miller, C. (1984). Genre as social action. Quarterly Journal of Speech, 70(2), 151-167.
Mishler, E. G. (1990). Validation in inquiry-guided research: the role of exemplars in
narrative studies. Harvard Educational Review, 60, 415-442.
Mitchell, R. (1992). Testing for learning: How new approaches to evaluation can improve
American schools. New York: Macmillan.
Munger, R. (2000). Evolution of the emergency medical services profession: A case
study of EMS run reports. Technical Communication Quarterly 9(3), 329-347.
Murphy, S. M. (1998). Introduction: Reflection—In portfolios and beyond. The Clearing
House, 72(1), 7-9.
Murphy, S. M. Bergami, J., & Rooney, P. (1997). The impact of large-scale portfolio
assessment programs on classroom practice: Case studies of the new standards
field-trial portfolio. Educational Assessment 4(4), 297-333.
Murphy, S. M., & Camp, R. (1996). Moving toward systemic coherence: A discussion of
conflicting perspectives on portfolio assessment. In R. Calfee & P. Perfumo (Eds.),
Writing portfolios in the classroom: Policy and practice, promise and peril (pp. 103-148).
Mahwah, NJ: Lawrence Erlbaum & Associates.
nd
Patton, M. Q. (1990). Qualitative research and evaluation methods (2 ed.). Newbury Park,
CA: Sage.
Rubin, H. J., & Rubin, I. S. (1995). Qualitative interviewing: The art of hearing data. Thousand Oaks, CA: Sage Productions.
Russell, D. (1997). Rethinking genre in school and society: an activity theory analysis.
Written Communication, 14, 504-554.
Sauer, B. (1998). Embodied knowledge: The textual representation of embodied sensory information in a dynamic and uncertain material environment. Written Communication, 15(2), 131-69.
Schryer, C. F. (1993). Records as genre. Written Communication, 10(2), 200-34.
Simmons, W., & Resnick, L. (1993). Assessment as a catalyst of school reform. Educational Leadership, 50(5), 11-15.
Seale Swain, S. (1998). Studying teachers’ transformations: Reflection as methodology.
The Clearing House, 72(1), 28-34.
Sunstein, B. S. (1998). Searching under the surfaces: Reflection as an antidote for forgery. The Clearing House, 72(1), 39-43.
Underwood, T. (1999). The portfolio project: A study of assessment, instruction and middle
school reform. Urbana, IL: NCTE.
Valencia, S., & Calfee, R. (1991). The development and use of literacy portfolios for students, classes, and teachers. Applied Measurement in Education, 4(4), 333-345.
Van Maanen, J. (1988). Tales of the field: On writing ethnography. Chicago: University of
Chicago Press.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015
Tony Scott
35
Weinbaum, K. (1991). Portfolios as a vehicle for student empowerment and teacher
change. In P. Belanoff & M. Dickson (Eds.), Portfolios: Process and product (pp. 206214). Portsmouth, NH: Boynton/Cook.
Whitford, B., & Jones, K. (Eds). (2000). Accountability, assessment, and teacher commitment:
Lessons from Kentucky’s reform. Albany, NY: State University of New York Press.
Wiggins, G. (1989). A true test: Toward more authentic and equitable assessment. Phi
Delta Kappan, 70, 703-713.
Winsor, D. (2000). Ordering work: Blue-collar literacy and the political nature of genre.
Written Communication, 17(2), 155-84.
Writing portfolio development teachers handbook. (1999). CITY, KY: Kentucky Department
of Education.
Yancey, K. B. (1996). Dialogue, interplay, and discovery: Mapping the role and the rhetoric of reflection in portfolio assessment. In R. Calfee & P. Perfumo (Eds.), Writing
portfolios in the classroom: Policy and practice, promise and peril (pp. 83-102). Mahwah,
NJ: Lawrence Erlbaum & Associates.
Yancey, K. B. (1998a). Reflection in the writing classroom. Logan, Utah: Utah State University Press.
Yancey, K. B. (1998b). Reflection, self-assessment, and learning. The Clearing House,
72(1), 13-17.
Yancey, K.B., & Weiser, I. (1997). Teacher portfolios: Lessons on resistance, readiness,
and reflection. In K. B. Yancey & I. Weiser, (Eds.), Situating portfolios: Four perspectives
(pp. 244-263). Logan: Utah State University Press.
Tony Scott is an assistant professor at The University of North Carolina at Charlotte,
where he teaches courses in writing, technical writing, and theory. He also works with
the National Writing Project. A book he coedited on labor issues in Composition, Tenured Bosses and Disposable Teachers: Writing Instruction in the Managed University,
was recently published by Southern Illinois University Press.
Downloaded from wcx.sagepub.com at UNIV CALIFORNIA IRVINE on November 2, 2015