FROM BLACK BOX TO
PANDORA'S BOX:
EVALUATING REMEDIAL/
DEVELOPMENTAL EDUCATION
W. Norton Grubb
David Gardner Chair in Higher Education
School of Education
University of California, Berkeley
February 2001
Supported by the Alfred T. Sloan Foundation
For additional copies please contactCommunity College Research Center
Teachers College, Columbia University
439 Thorndike Building
525 W. 120th Street, Box 174
New York, New York 10027
212-678-3091 (telephone)
212-678-3699 (fax)
Abstract
Our conceptions of institutional responsibility for learning have been
changing. An older conception is that educational institutions provide a
curriculum imparted by teachers through traditional didactic methods monitored
by conventional assessmentsfor example, multiple-choice tests covering the
facts and skills of a body of content. Students are responsible for learning this
content, and too bad if they fail. A high rate of non-completion is if anything a
badge of the institution’s honor.
Over time, this conception has been modified by one that places greater
responsibility for student success on the institution itself. Accepting this
responsibility means identifying and then correcting the many possible reasons for
non-completion or failure to learnin short, providing remedial/developmental
education. No longer is it possible to be complacent about high non-completion
rates, particularly since open access in community colleges has brought to
postsecondary education more under-prepared studentsand more lower-income,
minority, and immigrant studentswhose high dropout rates are both personal
tragedies and institutional embarrassments. Thus, the roster of student services
has expanded, especially remedial education.
Relatively few evaluations of remedial programs have been conducted,
and many existing evaluations are useless because, failing to recognize what the
program does, they provide little information about what should be changed to
make it more effective. In place of this kind of “black box” evaluation, I
recommend a variety of evaluation approaches that can improve information
about many different aspects of remediation, including not only its effects but also
the instructional methods used, the progress of students, and the ways students are
assigned to remedial programs. I call this a “Pandora’s box” approach because it
is designed to open up the black box, to reveal the problems with existing
programs, including the potential reasons for their effectiveness or
ineffectivenessand then to improve them.
Adult education is such a vast and varied worldadult basic ed, job
training, welfare-to-work, community collegethat it is difficult to characterize
what happens in this sphere. Although some teachers have developed studentcentered approaches to teaching and although some community colleges have
established "learning communities" that show considerable promise, by far the
most common approach to remedial/developmental education is the approach I
have labeled “skills and drills.” This tends to focus on arithmetic procedures,
punctuation and vocabulary, math problems of the most contrived sort, and
passages from texts that have been simplified for low reading levels. This
approach takes place not only in classes identified as remedial; it also emerges in
college-level classes that become remedial if the majority of students are not
ready for what the instructor considers college-level work. Conventional skills
and drills approaches violate all the maxims for good teaching in adult education.
Does remedial/developmental education work? The evidence is sparse:
most states and colleges have not yet evaluated their remedial programs. No one
knows much about what works and what does notor why. In this vacuum, it is
not helpful to recommend one particular approach to evaluation over others. Too
many dimensions of remedial education are poorly understood; investigating them
requires several different methods. Each of the following evaluation approaches
has the potential to illuminate a different aspect of this difficult problem.
1. Dropout rates from remedial courses need more investigation.
Complex combinations of reasons are responsible, and students
themselves cannot articulate why they stay with or leave a particular
program. A combination of qualitative, interview-based studies and
quantitative studies might begin to provide evidence for improving
remedial courses.
2. We need a more systematic collection of outcome measures, but these
measures need to include more than test scores of basic skills. Such
measures should include persistence in college and completion of
degrees, writing portfolios, and completion of occupational courses.
3. It is important in institution- or state- or national-level studies to have
control or comparison groups. One instance when this would be
especially useful is where some students are thought to need
remediation but do not take such courses.
4. Classroom practices in remedial courses must be observed and
described. Otherwise it is difficult to know what might have generated
a particular set of outcomesand therefore what might be changed.
5. If evaluation is to have any influence on classroom practice, it needs to
compare different approaches to teaching. Some successes may be
replicable and others may not, but understanding them better is a
necessary first step to improving the quality of instruction.
6. The “assignment” of students to remedial courses needs to be better
understood. The question is whether some students who might benefit
do not attend remedial courseseither because the assignment test
fails to identify those in need of remediation, or because enrolling in
such courses is voluntary. Some consideration of alternative
assignment procedures is appropriateeither different basic skills
tests, or procedures that incorporate other information and counseling
as well as testing.
The expansion of postsecondary education since the 1960s, especially the
growth of open-access community colleges, has provided opportunities for some
students where none existed before, and the dedication of many colleges and most
instructors to their non-traditional students is unmistakable. But dedication and
student-centeredness, while necessary, may not be sufficient, so a program of
evaluation and improvement is central to improving the performance of students.
TABLE OF CONTENTS
Introduction.............................................................................................................2
Multiple Approaches to Remedial/Developmental Education...........…….............5
Multiple Approaches to Evaluation.......................................................................17
An Eclectic Approach to Evaluation......................................................................30
Appendix: The "Assignment Problem" in Developmental Education..................36
Endnotes.................................................................................................................42
References..............................................................................................................46
1
INTRODUCTION
Our conceptions of institutional responsibility for learning have been
changing, albeit slowly and incompletely. An older conception is that educational
institutions provide a specified curriculum, imparted by teachers to students
through traditional didactic methods monitored by conventional assessments —
for example, multiple-choice tests covering the facts and skills of a body of
content. Students are responsible for learning this content, and too bad if they fail.
A high rate of non-completion is if anything a badge of the institution's honor, and
in any event not the responsibility of the institution or its instructors. Like the
caricature of the college with a dropout rate of two thirds ("Look to your right;
look to your left . . ."), dropout rates are expected to be high — and in many twoand four-year colleges they are, often atrociously so.
But over time this conception of student responsibility has been modified
by one that places greater responsibility for success on the institution itself.
Accepting such responsibility means identifying and then correcting the many
possible reasons for non-completion or failure to learn: providing
remedial/developmental education,i tutoring, counseling, and other forms of
student services; providing financial aid for low-income students who might
otherwise drop out for financial reasons; and providing child care, transportation,
and other social services as necessary.
The existence and growth of remedial/developmental education in both
two- and four-year colleges are testimony to a shift toward a greater institutional
responsibility for learning and completion. No longer is it possible to be
2
complacent about high rates of non-completion, particularly since open access (in
community colleges) and the expansion of higher education have brought to
postsecondary education more under-prepared students — and more lowerincome, minority, and immigrant students — whose high dropout rates are both
personal tragedies and institutional embarrassments. And so the roster of student
services has expanded, especially remedial education.
At the same time, colleges remain wedded to older ideals about their
responsibilities. Various factions still emphasize student responsibility —
sometimes older faculty, sometimes trustees and administrators wedded to
reputations gained through high standards, sometimes the forces urging
community colleges to remain "collegiate" rather than more varied in their
purposes (e.g., Eaton, 1994), certainly critics deploring the dreadful state of
higher education (e.g., Traub, 1994; McGrath and Spear, 1991), and certainly
policymakers wanting to reduce funding for remediation. These groups find the
expansion of remedial education somewhere between worrisome and abominable.
And so in practice most institutions are effectively hybrid in their approach,
taking greater responsibility for their students' success but often ambivalent about
their efforts to do so. Most postsecondary institutions now provide some form of
developmental education or basic skills, but remedial education itself has
remained marginal in most institutions, squeezed into the back pages of college
catalogues, usually under-funded, taught by part-timers, provided as an
afterthought, segregated from the "regular" offerings. In every way, colleges
signal that this is not real education.
3
But remedial/developmental education is real education, of the most
difficult sort. Under the best circumstances, it tries to do more than simply filter
accomplished students from the others; it tries to educate all of them, including
those who seem not to have learned much in ten or twelve years of conventional
schooling. It requires the most skilled instructors — not simply part-timers
pressed into service, nor individuals untrained in its special challenges.ii The task
is self-evidently difficult, and given a lack of certainty about "what works" in
teaching of any sort, the approaches to remedial/developmental education vary
enormously — as I illustrate in Section I. Remediation is not simple, and it
certainly is not a single kind of program.
Because remedial education has developed as a solution to a particular
problem — the lack of educational progress of many students — almost no one
views it as valuable in its own right. Instead, it is usually considered instrumental
to achieving other goals including increased learning (of basic reading, writing, or
math) increased retention, and other measures of educational progress.iii Therefore
remedial education should be easy to evaluate because — unlike other forms of
education freighted with multiple purposes (Labaree, 1997) — its goals are
relatively clear. But there have been relatively few evaluations of remedial
programs, and many existing evaluations are quite useless because, as I point out
in Section II, they fail to recognize what the program does — and therefore they
provide little information about what should be changed to make it more
effective. In place of this kind of "black box" evaluation, I recommend in Section
III a variety of evaluation approaches that can improve information about many
different aspects of remediation, including not only its effects but also the
4
instructional methods used, the progress of students, and the ways students find
themselves in remedial programs. We might call this a "Pandora's Box" approach
because it is designed to open up the black box, to reveal the problems with
existing programs including the potential reasons for their effectiveness or
ineffectiveness — and then to improve them.
In the end, debates about remedial/developmental education involve the
most central issues in American schooling, particularly those about equity and
about student versus institutional responsibility for outcomes. Is it possible to
have comprehensive institutions that include a broad variety of students, coming
with very different levels of preparation, and teach them all? Or are our schools
and colleges merely sorting mechanisms, sifting the competent from the
incompetent through batteries of tests and placements, coursework and
assignments, promoting one group and "cooling out" the other? The answers to
these questions may be partly ideological, but they are empirical as well — and
here lies a role for evaluation.
I. MULTIPLE APPROACHES TO
REMEDIAL/DEVELOPMENTAL EDUCATION
For all the debate over remedial education, there is almost no discussion
about what it looks like — what goes on in classrooms, whether it appears to be
educative in any sense of the word, whether it stands any chance of bringing
students up to "college level."iv It is important to understand the variety of
activities that march under the banner of remedial education. Otherwise it is easy
5
to fall into the trap of assuming that developmental education is well defined, and
can be readily evaluated like any other program.
There are different approaches to remediation in the various institutions
and programs of postsecondary education. These institutions are almost
completely independent of one another, with instructors in one unaware of what
their peers — who are often literally down the street — are doing, even though
their instructional tasks are similar. In community colleges, what is often termed
developmental education is usually placed in departments separate from English
(or reading and writing) departments. Although the variety of remedial or
developmental education is staggering, there is at least some sense of this as a
distinct field of instruction, with a journal, an association, conferences, and from
time to time some efforts at state-level reform.v In four-year colleges, the same
field is generally referred to as "basic English" or basic instruction, and usually
takes Shaughnessy's (1997) Errors and Expectations as its starting point; she
argued that the errors in the writing of basic writers follow patterns rather than
simply being random mistakes, and that a skilled instructor could use these errors
to reconstruct writing — an approach that requires a certain kind of studentcenteredness, rather than blindly plowing ahead with a standardized program.
Since then the field of compositional studies has further elaborated various
approaches to basic reading and writing courses (Hull, undated). However,
educators in two- and four-year colleges have virtually no contact with one
another; even though there are journals and associations to which the two groups
might contribute, like College Composition and Communication, in practice these
are dominated by four-year colleges.
6
Adult education is a vast chaotic world of programs funded with state and
federal money, as well as a bewildering array of local funds (for example, for
library-based programs) and charitable donations. It is very difficult to
characterize what happens in this sphere since there is so much variation. Because
there is so little institutional oversight, small innovative programs coexist with
much larger and more conventional programs. I have visited a number of
community-based programs that look promising, and it is possible that librarybased programs are more constructivist and student-centered in their teaching than
the rest of adult educationvi — but these cells of innovation often do not know
what the rest of adult education is like or how different they are, and there is not
enough contact among adult programs to facilitate discussion about different
approaches. Conventionally, however, adult education programs at several levels
(adult basic education distinct from adult secondary education) prepare students
to pass the GED, a credential of dubious valuevii that, because of its conventional
multiple-choice format, encourages a "skills and drills" approach to instruction.
Often, because of its belief in flexible enrollment and open-entry/open-exit,
students in remedial adult ed work independently on programmed texts, working
through vast arrays of homonyms and synonyms, of grammar exercises and
sentence completion exercises, of short passages read to answer simple factual
questions (Grubb and Kalman, 1994). In an era of insistent concern with the
higher-order skills for a flexible labor force, the complex area of literacy and
communication has been shriveled to grammar and punctuation.
Finally, the equally vast and complex area of job training programs —
including welfare-to-work programs — provides remedial education too,
7
sometimes as a prerequisite to vocational skills training. It is impossible to know
how much of this goes on, because providers are not required to distinguish
remedial education from other services and because remedial education is usually
a local option; but there is general agreement that programs are forced to provide
more remediation than they would like. Often, these programs subcontract with
adult education to provide remediation, so it is back to skills and drills and
programmed workbooks for such clients (Grubb and Kalman, 1994). Sometimes
job training programs create remedial labs with computer-based programs — but
these programs are invariably just skills and drills conveyed to the computer, with
even shorter reading passages because of the small size of the screen, a rigid
progression through topics, and a lab "manager" (rather than a teacher) whose
jobs is to turn the machines on and off and monitor progress but who is not
trained to provide any instruction. Finally, some adult education and job training
programs have become enamored with functional context literacy training (Sticht
et al., 1987), an approach that uses the materials from a "functional context" (like
employment or the military) to teach multiple literacy skills (e.g., 186 "reading to
do" skills and 143 "reading to learn" skills, in Sticht, 1979). While the functional
context approach can be used in constructivist and student-centered ways, it also
lends itself to the most didactic and skills-oriented teaching. Functional context
instruction has become orthodoxy in some circles, including workplace literacy
programs (Schultz, 1997; Gowan, 1992).
These areas of remediation are remarkably different from one another,
with different histories, different students, and different goals. Certain pedagogies
emerge in all of them, particularly the behaviorist, didactic, teacher-centered (or,
8
more often, text-centered) approach I often call skills and drills. But because
programs are not in communication with one another, examples of innovation and
good practice cannot readily spread from one area to another, so the prospects for
reform often look gloomy — particularly in the spheres of adult education and job
training.
But there is substantial variation within each of these areas as well, and I
will illustrate this with examples from community colleges.viii These institutions
have certain advantages over the others that provide remedial education. As open
access institutions, many of their students come unprepared for college-level
work, and so (unlike four-year colleges) the necessity for remedial education is
built into their basic structure. Community colleges also pride themselves on
being "teaching colleges"; even though this ideal is "honored more in the breech .
. . the tradition is there and can be called upon when warranted," as one English
instructor described it. Unlike adult education and job training, with their reliance
on untrained instructors hired in casual ways for part-time work, instructors in
community colleges generally have master's degrees and are hired through
painstaking procedures (even though these usually have little to do with the
quality of teaching). Although these colleges have come to rely too much on parttime instructors, there is still a commitment to teaching as a career. Of all the
postsecondary institutions that offer remedial education, community colleges may
have the greatest chance of doing it well.
By far the most common approach to remedial/developmental education
within community colleges is the approach I have labeled “skills and drills”
(Grubb and Associates, 1999, Ch. 5). This tends to focus on sub-skills — on
9
arithmetic procedures like multiplication and percentages, on grammar and
punctuation and vocabulary, on math "problems" of the most contrived sort and
reading passages from texts that have been simplified for low reading levels
("there is nothing to read in these texts" complained one instructor). Occasionally
instructors will bring in reading from outside the class — from newspapers, for
example — but in a typical heterogeneous remedial class there are few common
experiences to use as the basis of more contextualized instruction. Students rarely
know one another, because of the common pattern of taking courses almost
randomly, and therefore do not serve as resources for one another; mastering
"literacy" is an individual responsibility with the teacher as the sole authority,
rather than a collective and social activity (Worthen, 1997). This approach to
remediation takes place not only in classes identified as remedial, to which
students are referred if they score below a certain score on a basic skills test; it
also emerges in covert or hidden remediation, which takes place in some "collegelevel" classes — particularly in English, Business English, or Technical Math —
that are converted into remedial classes because the majority of students are not
ready for what the instructor considers "college level work" (Grubb and
Associates, 1999, Ch. 5). Thus the amount of remediation in most community
colleges almost surely exceeds the count of official remedial courses, and is
therefore difficult to estimate; according to conventional estimates, the proportion
of students needing remediation varies among colleges from 25 percent to 50
percent to 78 percent in Tennessee (Grubb and Kalman, 1994).
Often, remedial instructors use computer programs to supplement their
instruction. Invariably, the programs are simply drills transferred to the screen,
10
with short reading passages followed by questions of fact, fill-in-the-blank
exercises, arithmetic drills, conversions of fractions and percentages and simple
word problems.ix They typically allow students to move to the next level only
when they have passed a short "test" on one subject, so they manage the student's
progress carefully — and often create records for the instructor. Often, students
work on these programs in large labs  perhaps 60 students  overseen by an
instructor or "manager," but this individual typically has neither the time nor the
training for instruction: if a student gets stuck, he or she has to go back in the
computer program to try to work out the problem, but there is no teaching in the
conventional sense of the term. Some of these programs are quite elaborate,
covering many different topics, and some are quite expensive; they are often
promoted with elaborate claims about teaching "higher-order skills."
Unfortunately, the majority of these are simply repackaging and peddling the
"skills and drills" model.
The problem with this approach is not just that these classes are deadly,
with low levels of student engagement. They also violate all the maxims for good
teaching in adult education (Grubb and Kalman, 1994). And their tactic is simply
"more of the same": they take students who have not learned well in ten or twelve
years of standard didactic instruction, and then put them through an additional 15
weeks of similar instruction. There may be some success stories,x but overall there
is little chance that this dominant pedagogy can be very effective. It is foolish to
think that students who have never learned to read for meaning, or who have no
real understanding of numerals, can suddenly learn quickly from another round of
skills and drills.
11
In our observations, substantial numbers of community college instructors
have come to see didactic and behaviorist methods as unsuccessful, so they
develop approaches to teaching — largely through trial and error — that are more
constructivist, student-centered, and interpretive (Grubb and Associates, 1999,
Ch. 5). They are quite aware that community college students have suffered a
great deal of humiliation in their earlier education, as well as a remarkable amount
of poor teaching; they are likely to blame urban school districts for the low levels
of their students. These instructors are more likely to bring in reading materials
from work, or from newspapers and political debates; they tend to spend
considerable time probing the interests of their students and their purposes for
attending college, so they can mold reading and writing to these interests. Here is
a description from one such instructor:
I operate a student-centered classroom, so that means a kind of teaching
from the sidelines in the early part of the semester. I kind of try to have
people doing activities which I direct, but sort of low key, to give me a
chance to see how people perform in all different situations . . . Then
toward the middle third of the semester, I try to begin giving information
in whatever ways I've seen people's interest. Like this semester I have one
student who is particularly interested in myth, the Indian guy. So I did a
lot of stuff to kind of relate this European and Native American
tradition…Then by the last third of the semester I really move back into
the early format, but this time folks are much more independent. They
have much more — they have the tools and then it's mostly just finetuning . . .
Everybody is so different, you can't assume they're like you; you
can't assume they're like each other. So you really have to spend that first
month doing what I call four for nothing. You're just beating; you're just
finding the rhythm, one, two, three, four, go; one, two, three, seeing what
they do. Then after you kind of know how people do things, then you can
12
begin to teach what they are interested in, what they need, what makes
sense to them.
These instructors also foster work in groups rather than individual drill, partly in
recognition of language as a communicative and therefore social process rather
than an individual struggle with an unresponsive text. They do not avoid
conventional grammar and punctuation drills, but they stress above all having
students learn to create meaning from and with texts, and they subordinate drills
to that kind of reading. Here, for example, is one instructor talking about her
approach, where drill is subordinated to meaning-making:
It's very student-centered — it focuses on what students need to be able to
do to succeed . . . they need to be able to write in ways that let their papers
be read with respect . . . more bottom-up than top-down, because I'm
trying to get them to have the meaning — I try to have meaning drive
what they're doing. Although we may need to do a drill, time is so
precious that I'd rather that they do more writing and talking than doing
worksheets. And I expect them to take responsibility for a lot of it
themselves — I'm not the error police.
The topics these classes cover are idiosyncratic, because a student-centered class
invariably proceeds in different ways depending on the backgrounds and interests
of students. This creates problems for evaluation, because the outcomes are not
necessarily well defined — in fact, they are partially student-defined — but it
does mean that such classes are livelier than skills-oriented remediation. Students
are much more engaged, with each other as well as the instructor; the activities
and materials of the class are generally adult, rather than the childish drills of the
behaviorist classroom; and there seems much greater chance that this approach
13
can finally teach students about the complexities of language and mathematical
thinking.
In most colleges, the appearance of more student- and meaning-centered
teaching seems random and idiosyncratic, because the odyssey from didactic to
constructivist teaching is usually one that instructors make on their own, through
trial and error, with at best a little help from their friends. In most community
colleges, there are few institutional resources to help instructors make this
transition — though there are a few. We discovered a developmental studies
division, in an institution we call North County Community College, which
developed a coherent philosophy about remedial/developmental education,
codified in two enormous volumes referred to as the "basic writing curriculum
book." This is a self-consciously hybrid approach to instruction; the head of the
division complained that existing basal readers generally follow either a "phonics"
or a "comprehension" strategy, and that debates about remediation are similarly
polarized: "We're back to the same old thing — top-down or bottom-up; and that's
ridiculous." Instead, the philosophy of this department follows "transactional
theory," in which language including writing is a "dialectic or interchange among
writer, audience, and reality." Writing is a "recursive activity" incorporating
prewriting, rewriting, and revision, and includes "strategies for invention and
discovery whereby instructors help students to generate content and purpose." The
approach includes grammar, spelling, and other mechanics, but only in the final
stages of writing since its use early in writing has been so counter-productive.
The introduction to the "basic writing curriculum book" is an elegant
approach to meaning-centered teaching, replete with examples from the latest
14
research and practice. The rest of the volumes contain examples, applications, and
syllabuses in great profusion. The purpose of the "basic writing curriculum book"
was partly to prepare new and part-time instructors to the division's methods. The
division had developed a hiring process selecting individuals for their
commitment to constructivist practice, and then assigned them a mentor to help
them with their initial stages of teaching. In this division — and in a small number
of other community colleges that have developed institution-wide support for
teaching — the appearance of student-centered and constructivist approaches to
remediation is deliberate and planned, rather than idiosyncratic.
A third major approach to developmental education in community
colleges is the use of learning communities (LCs). Generically, LCs develop
when students take two or more classes jointly; then, if instructors spend
sufficient time planning together, each course can support and complement the
others. LCs are infinitely flexible and can be used for a variety of purposes
including multi-disciplinary approaches to general education, the integration of
academic and vocational education, and the presentation of complementary
subjects like science and math, or history and literature.xi
The use of learning communities for remedial purposes has several
distinctive features.xii Typically, a "lead" course — an occupational course, or a
central academic course — is matched with an English and/or a math course. For
example, one institution found that a particular biology course was blocking the
progress of students who wanted to go into health occupations, partly because of
their problems in reading and math. The biology course was then joined with
supportive math and English courses, which in turn modified their content to
15
provide the kinds of academic competencies necessary in biology. An automotive
instructor who discovered the problems his students had with reading devised a
learning community with an English instructor called "Reading, Writing, and
Wrenches." At LaGuardia Community College in New York, all programs for
welfare recipients are taught in learning communities. The Bridge program at
Laney College, Oakland, and the Puente program in numerous colleges in
California (specifically for Hispanic students, emphasizing bilingual education
and some multi-cultural subjects) are other examples of learning communities
devised specifically for remedial/developmental education.
The benefits of LCs are multiple, at least when they work well. Most
obviously, students find themselves making progress in subjects that they care
about — biology for health occupations, or an automotive program, for example
— rather than simply being in drill-oriented remedial classes with no apparent
relation to their future goals. The combination of classes allows instructors to
contextualize their teaching: examples and applications in English and math can
come from the "lead" course, and the lead instructor can develop writing exercises
and problem sets that are used in the other courses. Students within learning
communities get to know one another much better than most community college
students do, and they universally report forming study groups as a result.xiii And
instructors report benefits too, since learning communities break down the
isolation of instructors and allow them to create communities of like-minded
teachers. With all these apparent benefits, the evidence so far indicates that
students in learning communities tend to persist longer and earn higher grades
than do similar students in conventional classes.xiv
16
There are a number of other approaches to improving developmental
instruction that have been suggested, though their prevalence is unknown.
Weinstein and her colleagues (Weinstein et al., 1998) have suggested strategic
learning as one approach; Cross and Angelo's (1993) classroom assessment
techniques can be used in developmental education as well as other classes, and
are sometimes used as the basis for staff development for all faculty in a
community college.
By now my point should be clear: Remedial education in community
colleges can vary enormously. The student-centered teaching from constructivist
instructors (and typical of many LCs) looks completely different from
conventional skills and drills; the combinations of courses in learning
communities are vastly different from the conventional tendency to fragment the
curriculum into stand-alone courses. Some of these approaches to developmental
education stand virtually no chance, on a priori grounds, of helping students who
have come to college with many years of formal schooling but without adequate
command of language and math; others promise new approaches that might
correct these problems. Even within one institution like the community college,
remediation is not just one thing.
II. MULTIPLE APPROACHES TO EVALUATION
But does remedial/developmental education work? Do any of the
approaches described in the previous section enable students to make further
progress in their education, to complete community college programs and either
17
move out into the world of employment or onto further education? The evidence
is sparse, and partly it is for lack of trying: most states and most colleges that
provide remediation have not yet started to evaluate their programs in any way.xv
Evaluation serves multiple purposes, of course. At the level of policy,
decisions to expand or abolish programs might be based on such evidence: if, for
example, there were many years of evidence that no form of remediation benefits
students, then we might be tempted to eliminate all funding for such programs.
But given the educational imperatives behind remediation and the lack of
evidence, very few educators would be willing to take such a step — so I presume
the purpose of evaluation is both to learn more about the conditions of success
and to improve remedial programs.
One common form of evaluation is to examine the completion rates in
developmental courses. However, such an approach fails to see whether there are
any long-run effects from completion — for example, effects on subsequent
retention and completion of credentials. While completion of such courses may be
a good thing, it cannot by itself help people find jobs or help them vote, or
provide them transcendent experiences of literature or art; such information
provides no evidence about whether remedial programs have helped individuals
get along with their lives.
Another common way of evaluating remedial efforts has been the
comparison of pre-tests and post-tests, usually on some test of basic skills like the
18
TABE or the CASAS. For example, the Learning Assessment and Retention
Consortium (LARC) of the California community colleges used to publish
volumes of such figures (e.g., LARC, 1989a and b); they enable one to determine,
for example, that increases in reading scores were higher in one college than
another. But such results are almost useless, for a number of reasons. Most
obviously, pre-test/post-test comparisons are available only for students who have
stayed with a course until the end, when the post-test is given; if weaker students
drop out, or if only "brush-up" students survive until the end, then the test
increases will badly overstate the results for the average or random student. In
addition, without knowing about the backgrounds of students in different colleges,
the comparisons among institutions are impossible to interpret. At the extremes it
might be possible to draw some conclusions, either if gains were close to zero or
if they seem high, by some unarticulated and possibly idiosyncratic standard; but
even that is risky. The meaning of such results improves a little when they include
the proportion of students who complete the course. For example, recent figures
from the CUNY system reveal that in 1990, 64.7 percent of post-tested students
gained at least one year on the TABE; since only 61.7 percent of students were
post-tested, this means that — under certain shaky assumptions — perhaps as few
as 40 percent of students entering these adult education programs gained at least a
year.xvi
19
But other problems with pre-test/post-test results cannot be resolved
simply by collecting more data. From some perspectives, the tests themselves are
objectionable: they ask for the kinds of responses, about grammar and vocabulary,
arithmetic operations and simple word problems, that fuel skills and drills
approaches to remediation. No constructivist teacher, and few of those in learning
communities, would accept the results of such tests; the vitriolic debate between
McKenna, Miller, and Robinson (1990) and Edelsky (1990) about the evaluation
of whole language is an indication of the debates over alternative outcome
measures. In addition, like the completion of remedial courses, increasing test
scores may be better than the opposite, but it may still not lead to further progress,
the completion of meaningful degrees, or other outcomes. Finally, and worst of all
from the perspective of improving programs, these statistical results say nothing
about why test scores are what they are: they provide literally no information
about why students fail to complete courses in such large numbers, about whether
some approaches are better than others, about whether stand-alone remedial
courses are effective compared to those embedded in other programs (like
learning communities). Such figures, if they are discouraging enough, may lend
some urgency to the problem of reform, but they do not provide any guide about
what to do next.
Pre-test/post-test comparisons are close to the simplest and weakest
evaluation designs, but many of the problems with such results are repeated in the
20
most sophisticated approach to evaluation — exemplified by the randomassignment evaluation of the remedial program in the San Diego GAIN program
(Martinson and Friedlander, 1994). This evaluation assigned welfare recipients in
five California counties randomly to a welfare-to-work program (GAIN), which
included an initial assessment of whether individuals needed remedial education;
a control group of welfare recipients was not assigned to such a program, though
8.4 percent of them participated in adult basic education or some kind of GED
program on their own (compared to 43.6 percent of the experimental group
enrolled in GAIN). Participating in the GAIN program increased the rate at which
individuals received the GED — since 9.1 percent of the GAIN group but only
2.0 percent of the control group received a GED. However, the GAIN group as a
whole did not improve their scores on the Test of Adult Literacy Skills
(TALS),xvii though individuals who scored the highest on an initial screening test
(the CASAS) did increase their TALS scores overall.
Only in San Diego County was there a statistically significant increase —
from 454 among the experimental group to 488 among the GAIN group, on a
scale ranging up to 1,000. There are a few clues about the distinctiveness of the
San Diego program: it was designed specifically for the GAIN program, "built on
the premise that existing adult education services were not appropriate for the
GAIN population because of their previous negative experiences in school"; key
features included "up-to-date computer assisted learning combined with
21
classroom instruction, integrated academic and life-skills instruction," and "a new
teaching staff." The results are not particularly encouraging even for San Diego,
then, because even there the increase in TALS is trivial.xviii But the TALS is not a
self-evidently meaningful measure of outcomes (though elsewhere it has been
linked to higher earningsxix). Aside from an implication that effective remedial
problems need to be different from conventional "school-like" adult education
programs, there is not much guidance about how to reform the remedial programs
that were part of GAIN — and there is no information whatsoever on the nature
of the remedial programs that caused even more dismal results in other counties.
Another approach to evaluation fixes some of these problems but not
others. For a number of years Miami-Dade Community College has evaluated its
remedial programs in the format of Table 1: completion rates are calculated for
students who are judged "below standard" in one, two, or three subjects, and who
have successfully completed all appropriate remedial courses versus those who
have not.xx (Table 2 provides some earlier results, in a slightly more detailed
format.) The amount of information in this table is substantial. It indicates, for
example, that of the 6,324 students who entered, 59 percent were judged to need
some kind of remedial education. Students with three deficiencies had a much
harder time than students with one deficiency: only 42 percent of the former
group corrected all three deficiencies and only 9 percent of these students
graduated within three years, while 63 percent of students with one deficiency
22
corrected it and 28 percent of these graduated. (There are no surprises here, but
there is a substantial warning to high school students who think they can easily
make up during college the learning they have failed to do in high school.xxi) And
even students who take the full complement of remedial courses they presumably
need, graduated at much lower rates than those who entered needing no
remediation. Other conclusions are possible, of course, depending on the
outcomes for those students still enrolled after five years: some of these will
graduate, though the probability of doing so surely decreases with time. A
reasonable conclusion from these results is that remedial courses help a great deal,
but they cannot eliminate the gap between students with and without some need
for remedial education, and a substantial fraction of students judged to need
remediation fail to complete these courses.
It is easy to critique these results: they fail to control for variation in
academic achievement and other characteristics (like family background) among
groups; they neglect maturation effects, test effects, regression to the mean (which
describes some "brush-up" students), and selection effects — particularly greater
motivation among students who complete all remedial classes necessary. No
doubt these results overstate the effects of remedial courses, and more
sophisticated statistical analysis could improve the results. But they are a vast
improvement over some of the evaluation results already presented: the outcome
measure is one of intrinsic value,xxii they clarify that the amount of remedial
23
education completed matters a great deal, and they compare many different
groups of students with varying needs for remediation. However, they still fail to
investigate what remediation is: they provide no clue about why so many students
fail to complete remedial courses, they have not examined what about them
attracts and repels students, and they do not investigate what these courses are like
and whether some of them are more effective than others. Interesting as they are,
it is hard to know what to do next.
24
Table 1
Five-Year Outcomes (Graduated or Still Enrolled)
For First-Time Degree-seeking Students Entering Fall 1989
Miami-Dade Community College
Successfully Completed Remedial Courses in:
Below
standard
Completed all
Did not complete
in:
remedial courses
all remedial courses
__________________________________________________________________
No Subject
(N=2581)
N
2581 (these students did not need remediation)
Graduated
45%
Still enrolled 14%
Total
59%
One Subject N
1097
(N=1735)
Graduated
28%
Still enrolled
24%
Total
52%
638
7%
10%
17%
Two Subjects N
(N=1118)
Graduated
Still enrolled
Total
485
16%
34%
50%
633
5%
12%
17%
Three Subjects N=
(N=890)
Graduated
Still enrolled
Total
218
9%
40%
49%
672
2%
11%
13%
"Still enrolled" refers to those still enrolled with a GPA of 2.00 or better.
Source: Morris (1994), Table 6.
25
Table 2
Three-Year Persistence Rates (Graduated or Re-Enrolled)
For Tested First-Time Students Entering Fall Term 1982
Miami-Dade Community College
Successfully Completed Remedial Courses in:
Below
Standard
No
One
Two
Three
in:
Subject
Subject
Subjects
Subjects
__________________________________________________________________
No Subject
(N=2021)
N
Graduated
Still enrolled
Total
One Subject N
(N=1524)
Graduated
Still enrolled
Total
2021 (these students did not need remediation)
26%
21%
47%
873
11%
17%
28%
651
21%
25%
46%
Two Subjects N
530
(N=1360)
Graduated
5%
Still enrolled 9%
Total
14%
509
11%
26%
37%
321
15%
33%
48%
Three Subjects N=
641
(N=1457)
Graduated
1%
Still enrolled 9%
Total
10%
357
4%
19%
23%
303
8%
29%
37%
"Still enrolled" refers to those still enrolled with a GPA of 2.00 or better.
Source: Losak and Morris (1985), Table 1.
26
156
9%
37%
46%
I know of no outcome evaluations that try to compare different approaches
to remediation except for the investigations of learning communities cited above
(endnote 15). And a large testimonial literature indicates that students like
learning communities more than conventional classes (MacGregor, 1991) — a
result that is significant if only because of the evident dreariness of and
disengagement within most remedial classes.
A final issue important to remediation is rarely mentioned in program
evaluation, perhaps because the dominant focus has been on measuring the effects
of a program on those enrolled in it. But what I will call, for want of a better term,
the "assignment problem" arises because students are typically assigned to
remediation based on an assessment of some sort — usually a basic skills test like
the TABE or the CASAS, sometimes with a writing sample, sometimes with some
counseling. Then the question becomes whether students enrolled in remediation
fare better than those who did not. It is possible that some students assigned to
remediation do not need it — for example, students only needing brush up — and
of course if remediation programs are completely ineffective then everyone
assigned to them is misassigned. But a different question is what happens to those
students who did not "fail" the assignment test — who scored just above the cutoff point, and did not enroll in remedial programs. It is quite possible that these
students would still benefit from remediation, or that some students score well on
basic skills tests but still cannot write or reason well — because these tests do not
measure such higher-level competencies. Any assignment procedure runs the risk
of false assignment — Type I errors, or assigning students to remediation who do
not need it — as well as allowing students who need some kind of remedial work
27
to progress to college-level work (Type II errors). Both Type I and Type II errors
may reduce rates of completing programs, and Type II errors have the added cost
of putting unprepared students in regular classes — a process that ends up
generating a good deal of hidden remediation as "college-level" courses are
converted into remedial courses. The assignment problem is a difficult one, and I
have further explored its difficulties in an appendix.
These issues about errors in assignment are at the heart of debates about
community colleges and whether they are egalitarian, advancing students who
otherwise would have no access to postsecondary education, or whether instead
they "cool out" students who otherwise would go further in four-year colleges.xxiii
A particular incident illustrates this problem clearly. California instituted a
process known as "matriculation," intended to help entering students be placed
correctly in regular and remedial/developmental classes. However, some colleges
implemented matriculation poorly, using tests that had not been validated for the
purposes used, preventing many individuals from enrolling on the basis of
irrelevant tests, and using test results by themselves whereas the education code
stated that they were to be advisory only. To improve the quality of matriculation,
MALDEF (the Mexican-American Legal Defense and Education Fund)
successfully sued a college on equal protection grounds, claiming that the
requirement discriminated against Mexican-American students. In effect,
MALDEF claimed that there were too many Type I errors, of Mexican-American
students incorrectly assigned to remedial education whose progress through
college was thereby impeded. As a result, enrolling in remedial courses became
voluntary; what's more, the Office of the Chancellor in Sacramento imposed
28
regulations requiring that any prerequisites for any courses be justified through a
validation study, a burdensome procedure that has all but eliminated prerequisites
in California community colleges. Now there may be fewer Type I errors — and
if there are, they are presumably the choice of students rather than the result of
college assignment — but there may be more Type II errors, of students who need
more remedial/developmental education (or other prerequisites) than they get.
Indeed, the lawyer involved in the case admitted that the remedy was imprecise
and "broad brush"; MALDEF intended to improve the sensitivity of the
matriculation process, but instead got a series of crude rulings and bureaucratic
procedures, as is typical in legal cases.xxiv But the result of the lawsuit is unclear:
no one knows whether the MALDEF case helped Mexican-American students or
hurt them, or whether it benefited or harmed other students who have been
equally affected by the ruling and the response of the Chancellor's Office. Indeed,
no one has even thought to ask the question, never mind answer it.
It is clear, then, that the evaluation of remedial education is still in its
infancy, and no one knows much about what works and what does not.xxv What
little evidence there is indicates that completion rates in remedial courses are low,
that the amount of remediation does matter to important outcomes like persistence
in and completion of college programs, and that learning communities are
probably more effective than stand-alone classes. There is some suggestion (from
the GAIN evaluation) that "school-like" programs are less effective, but it is not
entirely clear what this means. The observational evidence (such as that in Section
I) indicates how much remedial courses vary, though my interpretation of these
classes — that many of them provide virtually no possibility for significant
29
learning — might not be widely accepted and might not even be correct; there is
no particular reason to think that the remedial courses at Miami-Dade are
particularly innovative and yet they have substantial effects on graduation and
retention (Table 1). But, aside from the possible recommendation to teach all
remediation in LC formats, there is not much evidence to suggest how to improve
the state of remedial/developmental education.
III. AN ECLECTIC APPROACH TO EVALUATION
In this vacuum, it is not helpful to recommend one particular approach to
evaluation over others. The orthodoxy in the evaluation literature — randomassignment studies, to rule out as many selection and self-selection effects as
possible — is no more useful than the most basic pre-test/post-test designs, as the
GAIN results illustrate. There are too many dimensions of remedial education that
are poorly understood, and investigating them requires several different methods.
Furthermore, there are at least two levels at which evaluation is useful.
One is the program level, where information about a particular course and a
specific instructor would be useful in diagnosing what is going well and badly.
This kind of information might include completion rates, assessments of academic
progress including locally developed measures (writing portfolios, for example),
and subsequent progress through the college, but it should also include peer
observation so that instructors can begin to create communities of discussion and
practice around remedial education.xxvi If such program-level evaluations are to be
30
useful to instructors themselves, they cannot be particularly complex — they
surely cannot use comparison groups, or follow students over long periods of
time, or introduce special assessments unrelated to normal teaching, for example.
A second level includes more formal evaluations carried out at the institution
level — for example, like the evaluations of Miami-Dade's programs in Table 1
— or at the state or national level. These can be more complex, with control or
comparison groups, and can follow students over longer periods of time; their
purpose is not to improve the practice of specific instructors, but rather to assess
institutional and state policy, the overall effects of remediation, and the
effectiveness of different approaches.
Instead of a single approach, therefore, I recommend a number of different
approaches, each of which has the potential for illuminating a different aspect of
this difficult problem:
1. The dropout rates from remedial courses need more investigation.
While it is plausible that dreary teaching is the reason, the difficult lives of many
community college students — including financial problems, child care problems,
transportation problems, other family problems including abusive spouses and
boyfriends, and the pervasive indecision of experimenting and uncommitted
students — surely play important roles. It is also possible that complex
combinations of reasons are responsible, and that even students themselves cannot
articulate why they stay with or leave a particular program. As one student
commented on his leaving the community college,xxvii
It was not even a decision. I just did not go. Sometimes you decide on
certain things. It was not a decision at all. Just like you go home, tired
31
from work, you don't decide about "Oh, I'm just going to go to sleep now."
You just doze off and go to sleep. It wasn't a plan. That's the way
[dropping] the class was: it wasn't a plan.
A combination of qualitative, interview-based studies and quantitative studies
might begin to provide evidence for improving remedial courses.xxviii
2. Outcome measures need to include more than test scores of basic skills.
(Indeed, it is an open question whether such test scores mean anything at all.)
Persistence in college and completion of degrees are obvious measures, because
completion is particularly important to the economic benefits of community
colleges.xxix However, constructivist teachers have their own measures of success
like writing portfolios, and other measures emerge from a college's intentions for
remedial programs — for example, completion of occupational programs may be
the most valuable outcomes in some cases. More systematic collection of outcome
measures could build up better understanding of the different outcomes that
remedial courses can achieve. Some students may declare other purposes —
political or familial goals, for example — to be more important, and qualitative
studies can clarify these goals and the contribution of courses toward achieving
them.
3. It is important, at least in institution- or state- or national-level studies,
to have some kinds of comparison or control groups, to see if completing remedial
courses produces benefits compared to students who do not take such courses.xxx
But while it might be possible to design a random-assignment study under certain
32
conditions — for example, comparing the effects of learning communities to
conventional formats, or comparing one pedagogical approach with anotherxxxi —
it would not be ethical or feasible to compare the effects of remediation to its
absence through random assignment. Instead, the kinds of comparison groups in
Table 1 — where some students are thought to need remediation, but do not take
such classesxxxii — is the only feasible way to construct comparison groups.
Evaluation studies can still collect other information like prior test scores and
grades to use as regression controls, to improve comparability somewhat.
4. No outcome evaluation should ever fail to understand the program it is
evaluating — and this means observing and describing the classroom practices in
remedial courses. The conventional "black box" evaluation, in which the nature of
the program being evaluated is never described, should be replaced with a
"Pandora's box" approach that clarifies both the triumphs and the troubles of
classroom practices.xxxiii Otherwise it becomes difficult to know what might have
generated a particular set of outcomes, and therefore what might be changed.
5. If evaluation is ever to have any influence on classroom practice, it
needs to compare different approaches to teaching. This in turn requires some
conceptualization of different approaches. The differences between behaviorist
and didactic practices on the one hand, and constructivist and student-centered
practices on the other, are dimensions of teaching that emerge over and over
again, both in the comments of instructors themselves and in various theories of
teaching and learning. Although there is endless debate about what dimensions of
teaching are important, these two polar approaches can be operationalized for
purposes of evaluationxxxiv and compared along their many dimensions.
33
However, there may be other ways to think about the power of different
approaches, particularly since student motivation arises for many complex reasons
external as well as internal to classrooms. While skeptical about "skills and
drills," we have observed drill-oriented remedial classes where students seemed to
be attentive and engaged, possibly because the class was followed by an
occupational class where the academic material would be applied. Some teachers
following behaviorist approaches develop idiosyncratic methods,xxxv or a special
rapport with students, that overcome the limits of drill, and some students —
particularly ESL students, who seem to be able to sit through anything, and older
students with clear and passionate goals — are able to learn from even the most
dreary teaching. Some of these possible successes may be replicable and others
may not, but understanding them better is a necessary first step to improving the
quality of instruction.
6. The "assignment" problem needs to be better understood. (See the
appendix for an initial effort.) Understanding this issue depends first on
ascertaining whether remedial programs themselves are effective: if they are
ineffective, then every student assigned to them is misassigned. However, in the
case of programs judged to be effective, the question of Type II errors is whether
some students who might benefit do not attend them — either because the
assignment test fails to identify those in need of remediation, or because enrolling
in such courses is voluntary (as in post-MALDEF California). Examining this
problem requires looking at the subsequent experiences of several groups: (a)
students judged in need of remediation who did not enroll in such courses — like
some of the groups whose progress is measured in Table 1; (b) the "near misses,"
34
or those who barely passed the assignment test, compared both to those who
enrolled in remediation and those who clearly do not need remediation, at least
based on the initial basic skills test.xxxvi Finally, some consideration of alternative
assignment procedures is appropriate — either different basic skills tests, or
procedures that incorporate other information and counseling as well as testing.
In the end, many questions about remedial/developmental education are
empirical issues of this kind. The expansion of postsecondary education since the
1960s, and especially the expansion of open-access community colleges, has
provided opportunities for some students where none existed before, and the
dedication of many colleges and most instructors to their non-traditional students
is unmistakable. The shift toward viewing institutions as responsible for learning
and advancement is a move in the right direction, certainly for proponents of
equity. But dedication and student-centeredness, while necessary, may not be
sufficient, so a program of evaluation and improvement is central to improving
the performance of students.
35
APPENDIX:
The "Assignment Problem" in Developmental Education
The "assignment problem" arises in education whenever a student is
assigned to one form of education rather than another, based on an assessment of
some kind. In remedial education, the assessment is usually a basic skills exam,
though a few colleges add a holistically graded writing sample; the assessment
process could be a more complex procedure in which multiple tests (and more
sophisticated tests) are used along with interviews, an examination of prior
education, and the like. Then a person is assigned to remedial education based on
this assessment; the assignment may be mandatory or voluntary, and in either
event some students enroll in the remedial course and others do not. Of those who
enroll, some fraction complete and others (often a very large percent) do not.
Sometimes there is an exit exam to move to the next level of education (for
example, the first college-level course, or the next remedial course in a sequence),
and sometimes course completion is sufficient. While this is a relatively familiar
sequence in remedial education, the assignment problem arises in many other
contexts including the assignment of students to special education and to various
tracks in K-12 education, admission to college, and to various majors within a
college.
The question is whether assignment to a remedial program benefits the
student or not, or conversely whether those not assigned to remediation would
have benefited had they been assigned (Type II errors). To see the complexity of
the issue, it is helpful to describe a simplified world. Imagine that facility in
36
reading, writing, math, or any other subject can be measured on a 100-point scale,
ascertained by a conventional test, and that (arbitrarily, perhaps, but with the
weight of tradition) a certain point on this scale (say 70) is considered necessary
for college-level work. Then anyone with a score below 70 is assigned to
remediation; otherwise the college-level course would have to go over sub-college
material, which is the ubiquitous problem of "hidden" remediation. Students stay
in remediation until they achieve 70 on an exit exam; there may be different levels
of remedial courses depending on the scores students attain (e.g., one for those
scoring 50-60, another for those scoring 60-70). The effectiveness of remedial
programs is simply measured by the absolute increase in the score for the average
student in the class, which is the conventional pre-test/post-test comparison.
Finally, there might be a different standard (say 85) for graduation from this
institution, perhaps a standard established by demands in employment; an exit
exam for graduation (as Florida has in its rising junior exam, or high schools now
have in many states) is sufficient to prevent under-prepared students from
graduating.
In this simple world, there are not any special problems once the test has
been created and the various cut-points established: the test used for initial
assignment is highly valid, pre-test/post-test evaluations of remedial courses are
adequate, and the required exit exam is obvious. Indeed, the conventional pattern
of developmental education seems to assume that the world of education follows
this simple model. But of course facility in various subjects does not follow this
model at all because there are multiple dimensions to reading, writing, or math.
The specific dimensions of reading necessary in an automotive course are
37
different from those in a standard English course; the dimensions tested in a
diagnostic basic skills test may be different from those required in any subsequent
college-level courses; and the content of any specific remedial course may be
different from both the diagnostic exam and subsequent courses. Thus the initial
assessment may not be aligned with the remedial program (which is a problem of
predictive validity); the remedial program may not be aligned with the exit exam,
if the exit exam is established independently of the program (though in this case
one might assume that instructors would start teaching to the test); and neither the
program nor the exit exam may be aligned with subsequent "college-level"
education, in which case successful completion of developmental courses may not
enhance subsequent outcomes. Of course, the college program as a whole may not
be aligned with the competencies necessary for employment, generating
complaints from employers about under-prepared employees. So there are many
points at which the multiple dimensions of any particular subject can create
problems, all of them leading to students assigned to remediation who do not
benefit (Type I errors). When a college has a series of remedial courses, the
problems of alignment are simply compounded.
A further problem, of course, is that attending remedial courses requires
time and (usually) money. The additional time required to complete a remedial
sequence may itself lead to non-completion. Distinguishing the time and money
dimensions of remediation from the alignment issues is important because the
remedies are different.
By the same token, students may not enroll in remedial courses even if
they would benefit — the problem of Type II errors. This may happen (as in
38
California and some other states) because a placement test is advisory only,
because students dislike the additional time and money costs of remediation, or
because college-level courses stress competencies that are not measured by the
simple initial assessment. If, for example, developmental instructors understand
that subsequent college-level work requires facility with analytic thought and try
to teach that, while the initial basic skills assessment measures facility with
grammar and vocabulary, then students will pass the initial assessment but may
still lack the analytic abilities necessary to succeed subsequently.
Now we can see a little more clearly where conventional evaluations fail
to incorporate the complexities of the assignment problem. In the first place, it is
necessary to have some intrinsically valuable outcome measure, like the
graduation and re-enrollments rates for Miami-Dade in Table 1. Then equations
describing the probability of graduation as a function of initial assessment scores,
completion (or non-completion) of remedial courses, plus the many other
variables that help explain graduation (gender, family background, race/ethnicity,
family support or family responsibilities, etc.) can identify the increase in the
probability of graduation due to completing remedial courses for those judged in
need of them. Then one minus this probability is the probability of Type I error;
for example, using the simplified figures in Table 1 (instead of the logit or probit
equations that could be estimated), 55 percent of those below standard in one
subject benefited from completing the appropriate remedial course, in the sense
that they graduated rather than not graduating, but 45 percent did not.xxxvii Using
the figures on those who graduated or were still enrolled, the results are much
better: 83 percent benefited and only 17 percent did not. However, the likelihood
39
of finally graduating surely decreases as the period of time enrolled increases, and
so the equations predicting graduation should incorporate a measure of time, or
should use event history methods to examine the probability of completion as a
function of time, where the time necessary to graduate is increased (and the
likelihood of graduation reduced) by the need to take remedial courses as well as
many other variables including the demands on students' time. So it is necessary
to estimate a system of equations, some describing outcomes and others
describing the time enrolled, with remediation affecting both of these.
However, when there is a statistical finding of large Type I errors, this
kind of statistical analysis cannot distinguish among potential explanations. While
the nature of alignment or misalignment among the different aspects of the overall
assignment problem could in theory be resolved by having sub-tests for different
dimensions of reading, writing, and math, the difficulty of doing this is
overwhelming. The alternative is a careful content analysis of the diagnostic
exam, the various remedial courses, exit exams (if any) and the courses students
subsequently enroll in. However, while community college instructors often
complain about misalignment (Grubb and Associates, 1999, Ch. 5), careful
analysis of the problem is rare.
The examination of Type II errors is made virtually impossible because of
the lack of a particular kind of information. When students are judged in need of
remediation according to some assessment procedure, some of them fail to enroll
in (or complete) remedial courses, so the effectiveness of completing remediation
can be determined by comparing the two groups (as in Tables 1 and 2). But if
students are judged not in need of remediation, then none of these students enroll
40
in remedial courses — so it is impossible to tell if some of them would have
benefited from remediation. It might be possible to simulate the experience of this
group: if we assume that students within one standard deviation of the critical cut
score are statistically indistinguishable from one another, then analysis of this
restricted group and the effects of remediation might provide an estimate of Type
II errors. Some understanding of Type II errors now comes from instructors
complaining that students are not prepared for the specific uses of reading,
writing, or math in their courses, as occupational instructors often do; the
solutions sometimes include learning communities or applied academics courses
with the basic skills necessary to particular occupational areas. But in the absence
of such solutions, the consequence is a large number of under-prepared students
in conventional classes, leading either to hidden remediation, to non-completion,
or to some of both.
My contention is that the "assignment problem" needs to be much better
understood before there can be much progress on the quality of remediation.
Many developmental instructors are quite aware of many of these problems, and
they complain about several dimensions of misalignment (see Grubb and
Associates, 1999, Ch. 5, especially section 1). But most writing about
remedial/developmental education fails to address this question, and the policy
debates (e.g., at CUNY) have failed to address any dimensions of quality
whatsoever. Until these issues can be more carefully examined and understood,
the effectiveness of remedial education will continue to be haphazard and its
evaluation be incomplete.
41
ENDNOTES
i
I will not repeat the various debates over the terms "remedial", "developmental", and "basic
skills" education; see Goto (1995) for an excellent review of these issues and the compromise of
"remedial/developmental" education.
ii
A full appreciation of the difficulty of remedial/developmental teaching can be found in Goto
(1998), who followed a number of students in two community college classrooms. Such
understanding can come only from examining the lives of students as well as activities within the
classroom and pedagogical strategies.
iii
In reality, most instructors and institutions interpret remediation in instrumental terms, but
students may not. A more student-centered conception could accept education that looks to be
relatively basic to be valuable in its own right, for students whose purposes may not include
completing advanced degrees. See Goto's (1998) description of basic writing students in a
community college, some of whom view it as valuable regardless of its instrumental purposes.
This way of looking at remedial education is more common in certain branches of adult
education; see for example, Gowen and Bartlett's (1997) description of several women able to
confront domestic abuse through a remedial writing program.
iv
Traub (1994) includes some descriptions of one College Skills class at CCNY with many
comments from the instructor about the lack of academic preparation among his students. These
descriptions convey an example of a disequilibrium between instructor and students — where the
instructor has an expectation of what students should be able to do that is not matched by their
preparation. For other descriptions of remedial/developmental classes in community colleges, see
Grubb and Associates (1999), Ch. 5, and Goto (1998). But most examinations of remedial or
developmental education contain no analysis of classrooms whatsoever; see, for example,
Roueche and Roueche (1993) and McCabe and Day (1998).
v
See the National Center for Developmental Education at Appalachian State University in
Boone, NC, which publishes the Journal of Developmental Education and a newsletter, the
Review of Research in Developmental Education. The national association is the National
Association for Developmental Education.
vi
I based this statement on recent observations in several library and adult programs by Caleb
Paull.
vii
Cameron and Heckman (1993) found no employment value to the GED, using sophisticated
statistical techniques; reworking the same data, Murnane, Willett, and Boudett (1995) found a
small effect, though they noted that it might not be enough to overcome the pedagogical
disadvantages of the test. Nor does the GED appear to enhance subsequent education attainment;
see Quinn and Haberman (1986).
viii
This section draws heavily on a book about teaching in community colleges (Grubb and
Associates, 1999) based on observations of and interviews with about 280 instructors (including
27 English instructors and 36 remedial/developmental instructors) and about 60 administrators.
See also Worthen (1997), drawn from the same data. This is, amazing to say, almost the only
empirical work on teaching in community colleges since Richardson, Fisk, and Okun (1983).
42
ix
See Grubb and Kalman (1994), and the earlier review of computer programs for job training
programs by Weisberg (1988). While the latter review is by now several computer generations
old, we saw only drill-oriented computer programs in our observations during 1993 - 1997.
There are some interesting constructivist uses of computers by a few community college
instructors, but they are all individual efforts by instructors developing computer applications on
their own; see Grubb and Associates (1999), Ch. 7.
x
Community college and adult instructors sometimes tell stories of students, invariably older,
who breeze through a programmed text or workbook. I interpret some of these as "brush-up"
students, who have been out of school for a decade or more and have failed an initial placement
exam because they have forgotten the trivia involved in such tests. If they have learned basic
English and math in their earlier schooling, one additional exposure is sufficient to brush up on
these skills.
xi
Learning communities have generated a great deal of interest; see especially Matthews (1994a
and 1994b).
xii
Learning communities have also been used for ESL classes — for example, by pairing a
computer class with an ESL class concentrating on computer-related literature and vocabulary —
but I will not discuss these here.
xiii
Such study groups are reminiscent of those that are at the heart of Uri Treisman's approach to
teaching math, though they are more informal.
xiv
Gudan, Clack, Tang, and Dixon (1991); Tokina (1993); Tinto, Goodsell-Love, and Russo
(1994); MacGregor (1991).
xv
Boylan, Bliss, and Bonham (1997) found that only 14 percent of community colleges had any
systematic evaluation. A recent SHEEO survey found very few states able to comment on the
impact of remedial policies on student success, and most studies seem to be somewhere in the
planning stages (Russell, 1998, p. 26 and Appendix G). Most studies that purport to describe
effective programs rely on nominations of programs by various observers, not on outcome
measures; see, for example, the programs profiled in McCabe and Day (1998) or in Roueche and
Roueche (1993).
xvi
This is true if none of the students who failed to survive to the post-test gained at least a year.
Since some of these students may have benefited from their period in adult ed, one might say that
the results support gains among 40 percent to 64.7 percent of students — too broad a range to
have much confidence about the outcomes. Other results indicate that students with less than 21
contact hours gained an average of .76 years, while those with more than 120 hours gained .96
years — a difference that, whether it is statistically significant or not, strikes me as being trivial
in practical terms. See Student Outcomes Research Project (1996), Tables 2 and 4. This report,
to its great credit, spends a great deal of time clarifying the limitations of pre-test/post-test
comparisons and uses various other measures of success including interviews with teachers and
students.
xvii
The TALS, developed by the Educational Testing Service, includes document, quantitative,
and prose literacy components; only the first two were used.
xviii
The gain from 454 to 488 in San Diego is equivalent of a gain from 227 to 244 on each
component. ETS divides the TALS scores into five levels of proficiency: scores below 225 are
considered the most basic; then levels 2, 3, and 4 are distinguished by 50-point increases. An
43
average increase of 17 points is therefore only one-third of the variation within these levels, and
is unlikely to move an individual from one level to another.
xix
This finding is based on an article sent to me for refereeing, and therefore anonymous.
xx
Similar results are available for the CUNY system. For example, they indicate that 36.5
percent of Associate degree students who passed all the basic skills tests they needed had
graduated eight years later, compared to 14.9 percent of those who did not pass all of them and
33.9 percent of those who took no remediation (CUNY Office of Institutional Research 1998,
Table 12).
xxi
See also Rosenbaum (1998), who clarifies the much slower progress through postsecondary
education for students who have done poorly in high school.
xxii
The results in Table 2 are also available for CLAS test scores. The CLAS test is a "rising
junior" exam required of all students in Florida before they start their junior year; while it is only
a test score, it is one of great importance to students who want to transfer to four-year colleges.
xxiii
I have summarized these debates and the evidence in Grubb (1996), Ch. 2. See also Lavin and
Hyllegard (1996) and Rouse (1995). The empirical evidence on balance is against the hypothesis
of cooling out, since most community college students would not otherwise have gone to
postsecondary education at all. In addition, the critics of community colleges tend to rely on
ancient "evidence" about the role of counselors in Clark (1960), although approaches to
counseling have changed dramatically since the 1950s. If there is any truth to the charge of
"cooling out," my argument is that it occurs by accepting "non-traditional" students and then
teaching them in traditional ways; see Grubb and Associates (1999), especially Ch. 10.
xxiv
Oral communication, Susan Brown, Council for Latino Issues Forum, San Francisco,
December 1997.
xxv
There is, to be sure, a large advice literature about how best to teach adults, but this is based
largely on experience rather than empirical evidence of any sort. Similarly, the synthetic lists of
recommendations about good teaching, like the widely cited "seven principles for good practice"
(Chickering and Gamson, 1991), are based on a mixture of evidence from the K-12 literature,
experience, and student ratings.
xxvi
For a more extended argument about the creation of such communities of practice through
peer observation, see Grubb and Associates (1999).
xxvii
This is taken from interviews with community college students, in Grubb (1996), Ch. 2.
xxviii
The dominant approach to dropouts in higher education has followed Tinto's (1987) model,
which assumes that the extent of academic and social integration into a college explains dropping
out. But this model is much too restrictive for community colleges since it fails to include the
many external factors — fiscal reasons, complex lives, issues of identity and commitment — that
affect community college students. Therefore quantitative analysis should look for causes
beyond Tinto's model. Some of these, like the reasons for experimenting, are difficult to quantify
and are probably best examined through interviews.
xxix
See the battle between Kane and Rouse (1995a, 1995b) and me (Grubb, 1993, 1995) about
whether program completion is necessary for economic benefits to materialize. Other work with
the SIPP data (Grubb, 1997) and a survey of the available literature (Grubb, 1998) indicates that
the benefits of taking courses without completing credentials is on the average quite low and
quite variable.
44
xxx
For an example of a large study without any comparison group, the National Study of
Developmental Education surveyed results for developmental students only in a variety of
postsecondary institutions. The findings — e.g., that 24 percent of developmental students in
community colleges persisted until graduation — are impossible to interpret without knowing
more about persistence of other students at the same institutions. See Boylan and Bonham
(1992); other results from this study came out in subsequent issues of Research in
Developmental Education.
xxxi
Community college students often take courses according to the time of day they are taught,
to fit into their complex schedules. One possible design, therefore, would be to randomly assign
different pedagogical approaches to different times of day. Then students could either choose a
course or be assigned randomly according to the time of day they prefer; any particular class
would have a combination of self-selected and randomly assigned students, and these two groups
could be compared to see if the students who select a particular course are different on any
dimensions from randomly assigned students.
xxxii
In some states, placement in remedial courses is mandatory if students score below certain
levels on diagnostic tests; in other states remediation is voluntary.
xxxiii
For a similar recommendation in the context of job training programs see Friedlander,
Greenberg, and Robins (1997).
xxxiv
See, for example, the work of Knapp & Turnbull (1990) and Knapp & Associates (1995).
They defined "the conventional wisdom" and "alternatives to conventional practice" almost
precisely as I have defined skills and drills versus meaning-centered approaches. They then
compared the effects of classes with different numbers of practices drawn from the list of
"alternatives."
xxxv
One instructor we observed had devised a way of reinforcing material in four different ways,
and was highly conscious of using different materials — written materials, oral instruction, films,
computer/based materials, etc. — to fit different "learning styles." While the skills presented in
her class were quite basic, students seemed more engaged than in most remedial classes.
xxxvi
Where individuals are assigned to remediation based on a basic skills tests with continuous
results, information is available on the subsequent education experiences of those just above and
just below the cut-off score for assignment to remediation. Alternatively, analysis of completion
as a function of scores at entry plus remediation could reveal whether remediation is effective for
different groups of entering students. Most community colleges have these data in their files,
though they are often scattered in different data systems and research of this kind seems always
to be a low institutional priority.
xxxvii
From those below standard in one subject, 7 percent who did not complete the required
course graduated, while 28 percent who passed the required course graduated. If we assume that
remedial courses ought to bring students to the level of those who did not need remediation, then
45 percent should have graduated; therefore (28-7)/(45-7)=55 percent benefited. Other
assumptions of what remediation can hope to achieve obviously generate different conclusions.
45
REFERENCES
Boylan, H., Bliss, L., & Bonham, B. (1997). Program components and their relationship to
student success. Journal of Developmental Education, 20, 2-9.
Boylan, H., & Bonham, B. (1992). The impact of developmental education programs. Review of
Research in Developmental Education, 9.
Cameron, S., & Heckman, J. (1993, January). The Non-equivalence of High School Equivalents.
Journal of Labor Economics, 11.
Chickering, A.W., & Gamson, Z.F. (1991). Applying the seven principles for good practice in
undergraduate education. San Francisco, CA: Jossey-Bass Publishers.
Clark, B. (1960). The open door college: A case study. New York: McGraw-Hill.
Cross, K.P. & Angelo, T. (1993). Classroom assessment techniques: A handbook for college
teachers, (2nd ed.). Ann Arbor: National Center for the Improvement of Postsecondary
Teaching and Learning.
CUNY Office of Institute Research and Analysis (1998, February). Basic skills and ESL at the
City University of New York: An overview. New York: CUNY.
Eaton, Judith S. (1994). Strengthening collegiate education in community colleges. San
Francisco: Jossey-Bass.
Edelsky, C. (1990, November). Whose agenda is this anyway ? A response to McKenna.
Educational Researcher, 9,7-11.
Friedlander, D., Greenberg, D., & Robins, P. (1997). Evaluating government training programs
for the economically disadvantaged. Journal of Economic Literature, 35,1809-1855.
Goto, S. (1995, October). The evolving paradigms of developmental/remedial instruction in the
community college. Berkeley: Graduate School of Education, University of California.
Goto, S. (1998, October). The threshold: Basic writers and the open door college. Unpublished
doctoral dissertation. Berkeley School of Education, University of California.
Gowan, S. (1992). The politics of workplace literacy: A case study. New York: Teachers College
Press.
46
Gowen, S., & Bartlett, C. (1997). Friends in the kitchen: Lessons from survivors. In G. Hull
(Ed.), Changing Work, Changing Workers: Critical Perspectives on Language, Literacy,
and Skills. (pp. 36-51) Albany: State University of New York Press.
Grubb, W. N. (1993). The varied economic returns to postsecondary education: New evidence
from the class of 1972. Journal of Human Resources, 28, 365-382.
Grubb, W.N. (1995). Response to comment. Journal of Human Resources, 30, 222-228.
Grubb, W. N. (1996). Working in the middle: Strengthening education and training for the midskilled labor force. San Francisco: Jossey-Bass.
Grubb, W.N. (1997). The returns to education and training in the sub-baccalaureate labor market,
1984-1990. Economics of Education Review, 16,231-246.
Grubb., W.N. (1998, August). Learning and earning in the middle: The economic benefits of
sub-baccalaureate education. Occasional Paper, Community College Research Center,
Teachers College, Columbia University.
Grubb, W.N., & Associates (1999). Honored but invisible: An inside look at teaching in
community colleges. New York and London: Routledge.
Grubb, W.N., & Kalman, J. (1994). Relearning to earn: The role of remediation in vocational
education and job training. American Journal of Education, 103, 54-93.
Gudan, S., Clack, D., Tang, K., & Dixon, S. (1991). Paired classes for success. Livonia, MI:
Schoolcraft College.
Hull, G. (undated). Alternatives to remedial writing: Lessons from theory, from history, and a
case in point. Paper prepared for the Conference on Replacing Remediation in Higher
Education, National Center for Postsecondary Improvement, Stanford University.
Berkeley: School of Education, University of California.
Kane, T., & Rouse, C. (1995a). Comment on W. Norton Grubb, The varied economic returns to
postsecondary education: New evidence from the class of 1972. Journal of Human
Resources, 30, 205-221.
Kane, T., & Rouse, C. (1995b). Labor market returns to two- and four-year colleges. American
Economic Review, 85, 600-614.
Knapp, M. S., & Turnbull, B. J. (1990, January). Better schooling for the children of poverty:
Alternatives to conventional wisdom. Volume I: Summary. Washington, DC: U.S.
Department of Education, 1990. (ERIC Document Reproduction Service No. ED 314
548)
47
Knapp, M., & Associates. (1995). Teaching for meaning in high-poverty classrooms. New York:
Teachers College Press.
Knapp, M., Shields, P., & Turnbull, B. (1993). Academic challenge for the children of poverty.
Volume 1: Findings and conclusions. Washington, DC: U.S. Department of Education.
Labaree, D. (1997, Spring). Private goods, public goods: The American struggle over
educational goals. American Educational Research Journal, 34, 39-81.
Learning Assessment and Retention Consortium (LARC). (1989a, August). Student outcomes
study: Fall 1988 follow-up study of students enrolled in remedial writing courses in fall
1986 and remedial reading courses in fall 1987. Santa Ana, CA: Rancho Santiago
College.
Learning Assessment and Retention Consortium (LARC). (1989b, October). Student outcomes
study: Mathematics, year 3. Santa Ana, CA: Rancho Santiago College.
Lavin, D., & Hyllegard, D. (1996). Changing the odds: Open admissions and the life chances of
the disadvantaged. New Haven: Yale University Press.
Losak, J., & Morris, C. (1985). Comparing treatment effects for students who successfully
complete college preparatory work (Research Report No. 85-45). Miami, FL: MiamiDade Community College, Office of Institutional Research.
MacGregor, J. (1991, Fall). What differences do learning communities make? Washington
Center News, 6, 4-9.
Martinson, K., & Friedlander, D. (1994, January). GAIN: Basic education in a welfare-to-work
program. New York: Manpower Demonstration Research Corporation.
Matthews, R.S. (1994a, Spring). Notes from the field: Reflections on collaborative learning at
LaGuardia. Long Island City, NY: Office of the Associate Dean for Academic Affairs,
LaGuardia Community College.
Matthews. R.S. (1994b). Enriching teaching and learning through learning communities. In T.
O’Banion & Associates (Eds.), Teaching and learning in the community college. (pp.
179-200). Washington DC: American Association of Community Colleges.
McCabe, R., & Day, P. (1998, June). Developmental education: A twenty-first century social and
economic imperative. Mission Viejo, CA: League for Innovation in the Community
College and The College Board.
48
McGrath, D., & Spear, M.B. (1991). The academic crisis of the community college. New York:
State University of New York Press.
McKenna, M.C., Miller, J.W., & Robinson, R.D. (1990). Whole language: A research agenda for
the nineties. Educational Researcher, 19, 3-6.
Morris, C. (1994, November). Success of students who needed and completed college
preparatory instruction. (Research Report No. 94-19R). Miami: Institutional Research,
Miami-Dade Community College.
Murnane, R., Willett, J., & Boudett, K.P. (1995). Do high school dropouts benefit from obtaining
a GED? Educational Evaluation and Policy Analysis, 17, 133-148.
Quinn, L., & Haberman, M. (1986, Fall). Are GED certificate holders ready for postsecondary
education? Metropolitan Education, 2, 72-82.
Richardson, R.C., Jr., Fisk, E.C., & Okun, M.A. (1983). Literacy in the open-access college.
San Francisco: Jossey-Bass.
Romberg, T., & Carpenter, T. (1986). Research on teaching and learning mathematics: Two
disciplines of scientific inquiry. In M.C. Wittrock, (Ed.), Handbook of Research on
Teaching, (Third edition, pp.850-873). New York: McMillan.
Rosenbaum, J. (1998, October). Unrealistic plans and misdirected efforts: Are community
colleges getting the right message to high school students? Occasional Paper. New York:
Community College Research Center, Teachers College, Columbia University.
Roueche, J. E., & Roueche, S. D. (1993). Between a rock and a hard place: The at-risk student in the
open-door college. Washington, DC: American Association of Community Colleges.
Rouse, C. (1995). Democratization or diversion: The effect of community colleges on
educational attainment. Journal of Business and Economic Statistics, 13, 217-224.
Russell, A. (1998, January). Statewide college admissions, student preparation, and remediation
policies and programs. Denver: State Higher Education Executive Officers.
Schultz, K. (1997). Discourses of workplace education: A challenge to the orthodoxy. In G. Hull,
(Ed.), Changing Work, Changing Workers: Critical Perspectives on Language, Literacy,
and Skills. Albany: State University of New York Press.
Shaughnessy, M. (1977). Errors and expectations : A guide for the teacher of basic writing.
New York: Oxford University Press.
49
Sticht, T. (1979). Developing literacy and learning strategies in organizational settings. In H.
O'Neill, (Ed.), Cognitive and Affective Learning Strategies. New York: Academic Press.
Sticht, T., Armstrong, W. B., Caylor, J. S., & Hickey, D. T. (1987). Cast-off youth: Policy and
training methods from the military experience. New York, NY: Praeger Publishers.
Student outcomes research project, final report. (1996, December). Division of Adult and
Continuing Education, City University of New York.
Tinto, V. (1987). Leaving college: Rethinking the causes and cures of student attrition. Chicago:
University of Chicago Press.
Tinto, V. & Goodsell-Love, (1995). A longitudinal study of learning communities at LaGuardia
Community College. Washington, DC: National Center on Postsecondary Teaching,
Learning, and Assessment, Office of Educational Research and Improvement, U.S.
Department of Education. (ERIC Document Reproduction Service No. ED 380 178)
Tinto, V., Goodsell-Love, A., & Russo, P. (1994). Building learning communities for new
college students: A summary of research findings of the Collaborative Learning Project.
Washington D.C.: National Center on Postsecondary Teaching, Learning, and
Assessment, Office of Educational Research and Improvement, U.S. Department of
Education.
Tinto, V., Russo, P., & Kadel, S. (1994). Constructing educational communities: Increasing
retention in challenging circumstances. AACC Journal, 64, 26-29.
Tokina, K. (1993). Long-term and recent student outcomes of freshman interest groups. Journal
of the Freshman Year Experience, 5, 7-28.
Tokina, K., & Campbell. F. (1992). Freshman interest groups at the University of Washington:
Effects on retention and scholarship. Journal of the Freshman Year Experience, 4, 7-22
Traub, J. (1994, Sept. 19). Class struggle. New Yorker, 70, 76-90.
Weisberg, A. (1988). Computers, basic skills, and job training programs: Advice for
policymakers and practitioners. New York, NY: Manpower Demonstration Research
Corporation.
Weinstein, C, Dierking, D., Husman, J., Roska, L., & Powdrill, L. (1998). The impact of a course
on strategic earning on the long-term retention of college students. In J. Higbee and P.
Dwinnel (Eds.), Developmental Education: Preparing Successful College Students, pp.
85-96. Columbia, SC: National Research Center for the First-Year Experience and
Students in Transition.
50
Worthen, H. (1997). Signs and wonders: The negotiation of literacy in community college
classrooms. Unpublished doctoral dissertation, School of Education, University of
California, Berkeley.
51