Some Evidence on Charter School Performance
An Addendum to Collaborative Governance: Private Roles for Public Goals
John D. Donahue and Richard J. Zeckhauser
As we were writing Collaborative Governance: Private Roles for Public Goals, we
found the charter-school movement to be a particularly rich example of collaboration—its
advantages, to be sure, but also its problems and, in particular, the chronic failure to
recognize the very special governance challenges that the collaborative model implies. We
perused a great deal of the literature on the performance of charters, and cast it in terms of
our framework of production discretion versus payoff and preference discretion.
Eventually we realized that not all readers would appreciate 20 or so pages summarizing
detailed empirical studies, and pared down by rather a lot the material in the book itself.
But for those who do share our enthusiasm for diving into the evidence, we provide this
electronic lagniappe. We start with some brief discussion of the evidence concerning
potential losses from payoff and preference discretion, and then turn to the more extensive
and far more interesting evidence on the gains from production discretion.
Payoff Discretion
Inappropriate benefits flowing from the abuse of payoff
discretion can come in multiple forms, ranging from the ambiguous (a well-appointed
teachers’ lounge for a non-profit, healthy net revenues for a for-profit) to the egregious
(fraud, embezzlement, or payments wholly out of line with value delivered.) Given the
number of charter schools, the diversity of both operators and authorizers, and the
inevitability that any large group will include some bad apples and some dim bulbs, it
would be astonishing if no such instances came to light, and unsurprising to discover a
great many.
Concrete illustrations can be found across the spectrum of payoff discretion. One
case involved an international school-management firm enlisted to run a troubled public
school, newly converted to charter status, in Springfield, Massachusetts. At charterrenewal time the state inspector general questioned the deal’s financial terms, including a
large “management fee” earmarked for by the corporate parent in addition to the salaries of
the people actually managing the school. 1 The firm characterized the fees as fair payment
for sharing its proprietary curricular expertise with Springfield, and results made this seem
potentially reasonable. In light of the school’s respectable student test performance and its
popularity with parents, the for-profit retained the charter even after it rather
undiplomatically transferred an accumulated surplus of nearly $3 million to the corporate
parent just as the school district that had funded the surplus underwent a traumatic wave of
teacher layoffs. 2
Other examples of payoff discretion are less ambiguous. A couple running a
“virtual” charter for home-schooled California children allocated more than a third of the
funding as a management fee for themselves in consideration of fairly minimal services.
Other charters were found to have diverted their operating funds to private schools with
which they were allied, or to have collected payments to educate students whose
1
Office of the Inspector General, “SABIS International Charter School: Management Issues and
Recommendations,” Executive Summary, Commonwealth of Massachusetts, November 2000
2
Molnar et al, Seventh Annual Report, page 6
2
attendance, or even existence, could not be demonstrated. 3 In one of the more spectacular
such instances a private management organization running sixty schools collapsed just at
the start of the 2004 school year amid charges of financial misdeeds, leaving over ten
thousand student stranded. 4
A massive report on the charter movement prepared for the U.S. Education
Department found that financial irregularities were the most common reason for state and
local officials to invoke informal sanctions against charters, and the second-most-common
reason (after regulatory compliance) for formal sanctions. 5 Yet a closer look at the record
calls into question the supposition that the charter movement is rife with opportunism and
profiteering. A large fraction of the financial irregularities cites in formal reports and press
accounts have more to do with sloppiness than with greed. The potential for the abuse of
payoff discretion is clear enough, and charter opponents numerous enough, that a pattern of
blatant self-dealing would be vividly publicized. Yet the reports are isolated, not
systematic. Unless most cases are too sophisticated for anyone to spot, only a small
fraction of charter schools seem to display egregious accountability breakdowns of this
sort.
The for-profit sector has been gradually increasing its involvement in charter-school
management, but too slowly and selectively to indicate a rush to claim easy money. Since
all the major education-management companies are privately held and release little or no
3
Sherry Posnick-Goodwin, “Unchartered Waters” California Educator Volume 8, Issue 2, October 2003,
http://www.cta.org/CaliforniaEducator/v8i2/Feature_1.htm, accessed September 2005
4
Joetta Sack, “California Charter Failure Affects 10,000 Students,” Education Week, September 1, 2004
5
SRI International Final Report, Exhibit 4-7, p. 49
3
financial data, so it is impossible to say anything definitive about the industry’s
profitability. 6 It is true that the late 1990s saw a surge of corporate enthusiasm and
predictions of rich profits in the education sector. Public schools were perceived to be so
ridden with waste and inefficiency that private management could deliver superior school
performance and handsome dividends to investors at the same time. But experience
quickly tempered entrepreneurs’ zeal. Several pioneering firms were shut down or
acquired as running schools turned out to be harder than predicted. The Edison
Corporation, probably the best-known such firm, proudly announced its initial public
offering, at $18 a share, in late 1999. Over the next few years Edison’s education
performance was generally respectable and sometimes impressive, but its financial
performance rather less so. Barely four years after the initial public offering Edison
retreated from the public equity market in a deal that gave shareholders less than a dime on
the dollar, relative to the initial offering. 7
While further evidence or future developments may force revisions to this
generalization, the charter movement appears to be less plagued by payoff discretion than
one might have predicted. This may be because idealists are relatively common, and
opportunists relatively rare, among the pioneers of such a movement. It may also reflect
the focus on financial matters of most state and local accountability systems, and their
relative success at spotting, and deterring, financial improprieties. And it may simply be
6
Molnar et al 2005 report (page 3) that of the 59 EMO in operation in the 2004-05 school year, only two
small ones were publicly held.
7
“Edison Schools Completes Initial Public Offering,” November 11, 1999 press release; “Merger to Take
Edison Private Compete,” November 14, 2003 press release; both from Edison Schools news archives at
http://www.edisonschools.com/news/newsarchs.cfm accessed September, 2005
4
that if your goal is to extract unearned benefits from the government, there are plenty of
schemes that involve more money and less work than starting a charter school.
Preference Discretion
Payoff discretion is usually hidden but, once flushed out
into the daylight, as Justice Potter Stewart famously said of obscenity, “I know it when I
see it.” 8 Preference discretion has the opposite pattern. Charter schools are often candid,
even boastful, about using their discretion to advance distinctive educational agendas. The
difficulty lies in determining which uses of discretion are appropriate, in light of the criteria
for legitimate use of public funds, and which are not.
Sometimes these criteria are explicit enough for bright lines to be drawn. This is
often the case with respect to that perennial flashpoint of American education policy,
religion. California shut down the 14 schools run by Gateway Charter Academy when they
were found to be using public money to teach Islam. 9 One of the largest for-profit charter
franchise operations has repeatedly been required to stop teaching creationism in its biology
classes. 10 Secular liberals and Christian conservatives alike have objected to the 30-some
charters awarded to Waldorf schools on the grounds that Waldorf pedagogy is permeated
8
U.S. Supreme Court, Jacobellis v. Ohio, 378 U.S. 184, 197 (1964)
Sherry Posnick-Goodwin, “Unchartered Waters” California Educator Volume 8, Issue 2, October 2003,
http://www.cta.org/CaliforniaEducator/v8i2/Feature_1.htm, accessed September 2005.
10
Amy Franklin, “Charter school is sued over religion,” Detroit Free Press, March 21, 2000 Such a
curriculum is illegal but quite well aligned with the preferences of the target market for this firm, National
Heritage Academies.
9
5
with “anthroposophy,” a rather wooly brand of cosmic consciousness cherished by Rudolf
Steiner, the Weimar-Germany founder of Waldorf. 11
But in other domains it can be far from obvious whether a charter school is abusing
its preference discretion. The Education Department report found that 40 percent of
charters disproportionately enroll “students of a particular cultural orientation,” while
roughly the same share focus on “students with specific academic interests.” 12 A narrower
but more focused study of a single state, Michigan, found a widespread tendency for
charters to offer customized curricula catering to particular priorities and tastes. 13 It is
nearly impossible, at this point, to say how much of this represents healthy diversity, and
how much represents regrettable departures from authoritative public priorities. This
reflects less a lack of data than the absence, in most cases, of any specification of the public
interest in education precise enough to determine which preferences it can accommodate.
(One of the authors is involved with a charter devoted to the “essential school” concept, an
updated version of progressivism developed by the reformer Ted Sizer. Students and
teachers share a cheerful contempt for much of the education policy enacted by the state
government that pays our bills, with no consequences so long as we meet the quite
unconstraining state requirements for test scores, financial stewardship, and the like.)
Charges and counter-charges concerning cultural and curricular deviance on the part of
charter schools probably reflect less disagreement about the facts than tacit differences of
opinion about what constitutes public value in education. So assessing charters’ use of
11
David Mirhadi, “Waldorf Opponents Lose in Court Decision,” The Union (Nevada Valley, CA) September
15, 2005 at http://www.theunion.com/article/20050915/NEWS/109150094, accessed September, 2005
12
SRI International, “Evaluation of the Public Charter Schools Program: Final Report,” Report prepared for
the Office of the Deputy Secretary, U.S. Department of Education, November 2004 Appendix C-3, p. 84.
13
Jerry Horn and Gary Miron, An Evaluation of the Michigan Charter School Experience, The Evaluation
Center, Western Michigan University, Kalamazoo, November 2000
6
preference discretion—within any jurisdiction, or in the aggregate—requires more
normative clarity than we typically have, or are likely to get, about the rationale for public
spending on education.
Production Discretion
Managing production discretion—or, more precisely,
maximizing the net public benefit achieved through private collaborators’ deployment of
this discretion—is the whole point of the charter movement. What justifies all the political
and managerial effort of getting a movement underway, the expense of layering one
educational system on top of another, and the risks associated with shared discretion is the
prospect that charters will be able to out-perform traditional public schools. Here,
fortunately, we have a wealth of data to work with—not enough for any final judgment on
the merits of the charter movement, but enough to support some significant observations on
collaborative governance for educational improvements.
Some cautionary notes are in order. The studies of charter performance considered
here keep score primarily or exclusively in terms of standardized test results. Nobody
believes test scores are perfect measures of educational value, and some people are
decidedly skeptical about these tests. To the extent charter schools pursue educational
missions that are at odds with what the tests aim to measure, comparisons based on average
scores will be biased. A charter school might be creating abundant value but register
7
mediocre scores if much of what it teaches isn’t covered by the standardized tests. 14 Or,
conceivably, a charter could unintentionally equip its students to master the tests when it
was actually trying for a quite different emphasis. It seems safe to predict, however, that
scores will more often under-estimate than over-estimate the degree to which a school
accomplishes its goals.
There is surely a degree of such measurement problems. But charter schools, as a
group, are no more resistant to standardized testing, and probably less so, than traditional
schools. The charter movement has largely coincided with the high-stakes testing
movement, so that most charter-school teachers and administrators (unlike many of their
counterparts in other schools) entered the game once the rules had been set. While the
principle of fair comparison can be undermined by other factors to be addressed shortly, it
is unlikely that using test scores as the metric tilts the terrain against charter schools.
A further caveat: The charter-school debate is less debased by intellectual trench
warfare than the parallel debate over school vouchers, but there are certainly battle lines.
Some studies are more geared to advancing an agenda than to advancing inquiry, and many
players in the debate would reject the notion of an “objective” study (unless that is defined
as one whose findings comport well with the observer’s pre-existing position.) At points
below we characterize a researcher or research team as generally supportive of the charter
movement, or generally skeptical, but without implying that he, she, or they have shaded
their conclusions to fit their preferences.
14
At the school where one of the authors is a trustee students and teachers grudgingly submit to the tests
required by state law, but it is considered bad form to take them very seriously or to allocate to them more
than the minimum effort required for acceptable average scores.
8
Consider first a set of state-specific studies conducted by the Evaluation Center at
Michigan State University. The first looked close to home—an effort by the Center’s
Jeffrey Horn and Gary Miron to gauge the performance of charter schools in Michigan (one
of the first states to build up a sizeable network of charters). 15 Horn and Miron—who have
no discernible axe to grind—found that on average, conventional school students
“outperform charter school students” on Michigan’s standardizes tests. 16 Test results for
charter school students tended to be lower, and year-to-year gain in average levels of
student scores somewhat smaller, than for other schools. Two years later Miron and
another team of researchers conducted a comparable study in Pennsylvania, with results
that were mixed but less discouraging than the earlier Michigan study. Here, too, charterschool students tended to score lower than other students—considerably lower when
compared to all students, and slightly lower than students with comparable student
populations. But the gap seemed to be narrowing over time, however, suggesting that
charters would catch up with “demographically similar public schools” within a few
years. 17 Horn and Miron teamed up again to examine charters in a third state, Connecticut.
They found that Connecticut charters started out under-performing both their host districts
and the average for all state schools, but within five years had caught up with their districts,
and narrowed their deficit relative to the state average. In terms of improvement from one
15
Jerry Horn and Gary Miron, An Evaluation of the Michigan Charter School Experience, The Evaluation
Center, Western Michigan University, Kalamazoo, November 2000
16
Horn and Miron, 2000, page 7. They also find that charters tended to have lower proportions of specialneeds students than other Michigan schools.
17
Gary Miron, Christopher Nelson, and John Risely, “Strengthening Pennsylvania’s Charter School Reform:
Findings From the Statewide Evaluation and Discussion of Relevant Policy Issues” The Evaluation Center,
Western Michigan University, October 2002, executive summary, page 2
9
cohort to the next, charters were “outperforming their host districts.” 18 This set of cases
hint at how much the basis of comparison can matter—which tests, at which grades, in
which subjects; levels versus changes over time; compared to all public schools or a subset
of schools with supposedly similar students; and so on.
Another single-state study took this theme as a point of departure. A team from the
new “Charter College of Education” at California State University set out to compare
student test performance at charters and other schools in California. Raw data showed that
charters had lower scores on average, and also somewhat slower growth from 1999 to
2001. 19 But if charter schools tend to enroll different kinds of students, the California State
team observed, this could be a misleading comparison. So they narrowed their inquiry to
cover only those schools—both charter and non-charter—with large populations of lowincome students. (They used the standard measure of “low income”—eligibility for free or
subsidized school lunches under federal regulations.) For schools with at least 50 percent
low-income students, and for the still more focused set of schools with at least 75 percent
low-income students, average scores were still somewhat lower at charters. But, in line
with the Pennsylvania and Connecticut studies, at least for this subset of schools, scores at
charters were rising faster than elsewhere.
The RAND Corporation, an organization with a reputation for neutral competence,
covered much the same ground but from a different angle and with somewhat different
18
Gary Miron and Jerry Horn, “Evaluation of Connecticut Charter Schools and the Charter School Initiative:
Final Report” The Evaluation Center, Western Michigan University, September 2002, executive summary,
pages x-xii.
19
. Simeon Slovacek et al “California Charter Schools Serving Low-SES Students: An Analysis of the
Academic Performance Index¨, Charter College of Education, California State University, Los Angeles,
March 2002 Chart A page 3 and Table 1 page 5
10
results. 20 As a warm-up the RAND team simply compared average test scores of charters
and other public schools in California. They found no significant difference one way or the
other. Next, they statistically adjusted the data to control for disparities in the student
populations of charter and other schools—an alternative to the California State strategy of
restricting the comparison to supposedly similar schools. Here, too, they found little or no
difference between charters and conventional schools overall, and an inconclusive pattern
for particular kinds of charters with a mix of mostly-minor pluses and minuses, depending
on the type of school and the basis of comparison. The RAND group, though—unlike most
other studies—was able to find data that let them study individual students. Instead of
comparing the scores of one year’s third-graders and the next year’s third-graders, they
could track the test scores of the same students over time. But even with this less errorprone empirical strategy, the RAND group could not reach any clear conclusion supporting,
or undercutting, the claim of superior performance for charter schools.
These state-specific studies, with their messy mix of results, frustratingly failed to
settle the question of whether discretion let charters outperform conventional schools.
Another set of analysts cast a wider net. The Brookings Institution’s respected educationpolicy unit, led by Tom Loveless, assembled data on all ten states that had at least 30
charters in operation by 1999, and that tested students in three grades (4, 8, and 10) for
three years in a row using consistent tests. To boost the statistical reliability of the
comparison—at the expense of tracking changes over time—the Loveless team pooled all
20
Ron Zimmer et al, “Charter School Operations and Performance: Evidence from California” RAND
Corporation, Santa Monica, 2003, Summary and Chapter Three, “Academic Outcomes”
11
three years of data. 21 Charter-school students, as a group, scored less well—as in other
studies, a small but significant gap—than their counterparts in traditional schools. The
results brought no joy to Loveless, a scrupulous analyst though a charter fan, but they held
up even when controlling for school size and (to the primitive degree the data permitted)
for demographics. 22 The Loveless team published an updated study, with another year of
data and refined methods. This one yielded basically the same finding for the short term
but a glimmer more optimism for the long term. The data suggested that “…about 62% of
schools with similar demographic characteristics have higher test scores than the average
charter school.” But the rate of improvement for charters—for example, from one year’s
crop of eight-graders to the next—was somewhat higher, on average, for charters. 23
Around the same time, a team led by the Manhattan Institute’s Jay Green—an
entirely respectable analyst with undisguised sympathies for the charter movement—
conducted a national study of charters serving “general student populations.” (In other
words, the Green team excluded charters targeted to disadvantaged groups “such as at-risk
students, pregnant or young-mother students, dropout recovery students, or disabled
students,” and also those targeted to gifted and talented or otherwise advantaged students—
presumably because score disparities in these cases would say more about the students than
21
Tom Loveless et al, Brown Center Report on American Education, Part III, “Charter Schools,” Volume 1,
No. 3, The Brookings Institution, 2002. The methodology is described on pages 17-18
22
Loveless 2002, Table 1, page 18. A detail of the study illustrates the difficulty of extracting solid
conclusions from messy and incomplete data. While the ten-state pooled data was sturdy enough for
statistical significance, for only four individual states—Massachusetts, Texas, Michigan, and Minnesota—
could a clear gap be found. For half of the states the standard error of the difference between charters and
other schools were larger than the average difference.
23
Tom Loveless et al, Brown Center Report on American Education, Part III, “Charter Schools:
Achievement, Accountability, and the Role of Expertise,” Volume 1, No. 4, The Brookings Institution, 2003,
especially pages 30-31
12
about the schools. 24) They found comfortably large samples of such “untargeted” charters
in eleven states, and compared each one to the nearest conventional public school. The
basis of comparison was not absolute scores, but rather changes from one year to the next.
For example, they calculated the difference in average math scores for third-graders in 2001
and third-graders in 2002 at a charter school, then compared this to difference between
cohorts over the same time period at the nearest public school. The good news for charter
supporters is that Green’s team found statistically significant evidence that charters as a
group saw faster improvement from one cohort to the next than nearby public schools. The
bad news is that the average charter effect was paltry—just eight hundredths of a standard
deviation in math, and half that in reading. The team concluded that “untargeted charter
schools are somewhat better than regular public schools serving similar populations, but not
a great deal better.” 25
So far the empirical aspect of the charter-school conversation was playing out with
a conspicuous lack of drama. But this was about to change. The first strike came from an
unlikely source—the American Federation of Teachers. The AFT has long been considered
a progressive and open-minded union, particularly in comparison to the rival National
Education Association. Albert Shanker, who led the AFT for decades, had been a pioneer
of the charter movement, and the union’s official policy was pro-charter, with only mild
24
Jay Green, Greg Forster and Marcus Winters, “Apples to Apples: An Evaluation of Charter Schools
Serving General Student Populations” The Manhattan Institute, July 2003 accessed at http://www.manhattaninstitute.org/html/ewp_01.htm#12 in September, 2005
25
ibid, executive summary and Appendix Table 2
13
qualifications. But in mid-2004 the AFT released a generally downbeat review of the
record on charter performance. 26
The point of reference was the National Assessment of Educational Progress, or
NAEP—a system of periodic national tests sometimes called “the nation’s report card.”
The U.S. Department of Education had been keeping records on NAEP performance by
charter schools and other schools for several years, but had repeatedly delayed the release
of comparative data. Researchers at the AFT—suspecting that the Education Department
had something to hide—took matters into their own hands, sifting and sorting public data to
reverse-engineer the data sets for charter and non-charter student performance on the 2003
NAEP. The results were less than conclusive, and in line with state-by-state studies, but
generally discouraging to charter enthusiasts. Charter students at both the 4th and 8th grade
levels, and in both reading and math, seemed less likely than students elsewhere to attain
“basic” competency. (The differences were big enough to be statistically significant for
math, but not for reading.) 27
Charter schools had significantly fewer students reaching
“proficient” or “advanced” levels in 4th-grade reading and math. For the 8th-grade tests the
comparisons were too close to be statistically meaningful, with a tiny charter edge in
“proficient” reading and “advanced” math and an equally tiny charter deficit in “proficient”
math and “advanced” reading. Important subgroups of students—those with low incomes,
and those in urban areas—also seemed to do worse in charter schools. 28 Nor did the NAEP
26
F. Howard Nelson et al, “Charter School Achievement on the 2003 National Assessment of Educational
Progress,” American Federation of Teachers, Washington, D.C., August 2004
27
ibid., Table 1, page 4.
28
Table 2, page 5; Table 4, page 9
14
data indicate that charter schools, as a group, were producing better results for black or
Hispanic students. 29
The AFT study was presented in measured tones, with the usual caveats and
qualifications, and did not claim to prove that charters were a bad idea. But it inspired
prompt and vigorous rebuttals. The Center for Education reform took the unusual step of
buying a full-page add in the New York Times for a statement, signed by 31 academic and
think-tank supporters of the charter movement, denouncing the study. 30 The public
statement castigated the AFT researchers for focusing on scores from a single year instead
of changes over time, and failing to control for all of the student characteristics that can
confound comparisons of schools. These objections were reasonable enough. But they
reflected limitations in the data—which the authors had acknowledged—more than
analytical mendacity. Given the paucity of gold-standard experimental data, every study of
comparative charter performance had its limitations.
A more systematic analytical counterpunch quickly came from Caroline Hoxby,
widely considered both a masterful empiricist and a diehard charter supporter. Like the
AFT study it examined only a single year and controlled rather crudely (albeit as well as the
data permitted) for student characteristics. But Hoxby assembled a massive data set,
covering nearly all elementary students in charter schools. Instead of the NAEP—
administered only periodically, and only to a subset of students—she used individual states’
reading and math exams, and her measure was the share of students at a school ranked as
29
Tables 6 and 7, pages 11-12
“Charter School Evaluation Reported by the New York Times Fails to Meet Professional Standards,” Paid
advertisement in the NYT, August 25, 2004
30
15
“proficient” by whatever threshold each state applied. Instead of comparing charters
overall with non-charters overall, she compared charter schools with the nearest public
school that had a similar racial composition. Hoxby found that charter-school students
were slightly, but significantly, more likely to be rated proficient in both math and English
by the standards set by their states, with a higher edge for charters that had been in
operation for several years. 31
To deepen the theme of conflicting and inconclusive results, shortly thereafter SRI
International released the final version of a sprawling study the Education Department had
commissioned. 32 Like Hoxby and Loveless (and unlike the AFT) it used state rather than
national tests as the basis of comparison. Like the AFT and Loveless (and unlike Hoxby) it
found that charter schools significantly underperformed other schools. Like all analyses—
though in its own way—it found that it mattered greatly whether and how differences in
student populations were handled. For two of the states studied in detail (Colorado and
Texas) the charter-school deficit remained even after attempts to control for differences in
students’ ethnic and economic characteristics. In three others (Illinois, Massachusetts, and
North Carolina) the gap shrank sharply or disappeared once differences in student
31
Caroline M. Hoxby, “Achievement in Charter Schools and Regular Public Schools in the U.S.:
Understanding the Differences,” December 2004, manuscript at
http://post.economics.harvard.edu/faculty/hoxby/papers/hoxbycharter_dec.pdf
accessed September 2005
32
SRI International, “Evaluation of the Public Charter Schools Program: Final Report,” Report prepared for
the Office of the Deputy Secretary, U.S. Department of Education, November 2004 The SRI study echoed
earlier previous assessments in finding that charters enrolled a more than proportionate share of minority and
low-income students but a less than proportionate share of special-education students, compared to traditional
public schools as a group. (Table ES-2, Page xiii; detail on pages 23-28.) A separate study—this one by a
group of analysts somewhat skeptical about charters but, more importantly, upset by the heavy-handed
reactions to the AFT study—used National Center for Education Statistics data to question the notion that
charters served the “disadvantaged of the disadvantaged.” This team found little evidence that charter
students were systematically more likely to be low-income. Martin Carnoy et al, The Charter School DustUp: Examining the Evidence on Enrollment and Achievement, Economic Policy Institute and Teachers’
College Press, 2005 ATable 1, p. 34 and Tables B-1 and B-2, pages 142-43
16
populations were considered. 33 But even when attempting to control for student
characteristics, the SRI team could not find the pattern of charter-school superiority that
advocates expected and hoped for.
A few years later Stanford University’s Center for Research on Educational
Outcomes—known as CREDO—released yet another ambitious analysis of charter-school
effects. 34 This study examined fifteen states, plus the District of Columbia, that accounted
for about 70 percent of all charter-school students. Its basic method was to create a “virtual
twin” for each charter-school student—a child who matched the charter student in age,
residence, ethnicity, economic status, English proficiency, and prior participation in
special-ed programs—but who attended a traditional public school. The researchers would
then compare gains on state math and reading tests between charter students and their
“twins.”
The Stanford team found that in terms of gains in mathematics scores about half of
the charter schools performed about the same as traditional public schools, 17 percent
performed significantly better, and 37 percent performed significantly worse. In five states
(Arkansas, Colorado, Illinois, Louisiana, and Missouri) charter-school students did
measurably better than their counterparts, on average. In six states (Arizona, Florida,
Minnesota, New Mexico, Ohio, and Texas) charter students fared worse, on average. States
with multiple chartering authorities—rather than a single specialize agency responsible for
selecting and overseeing charter operators—tended to do worse. Interestingly, states with
33
SRI International 2004, Pages 55-57 and Appendices E-2 through E-4
34
Multiple Choice: Charter School Performance in 16 States Center for Research on Educational Outcomes,
Stanford University, June 2009
17
caps on the number of charters—rather than leaving open-ended how many charter schools
there would be—also tended to have disappointing charter-school performance.
The 2009 CREDO study also found that charters seem to do relatively well for
elementary school students (as opposed to high school); for white students; and for lowincome students. Students tend to do worst in their first year in a charter, with an
improvement in relative performance over time. The researchers suggested that the
problem isn’t that the charter movement lacks potential—there were important areas of
clear-cut success—but that the public officials overseeing charters tended to be unable or
unwilling to sanction or shut down underperforming schools. “The charter school
movement to date has concentrated its formidable resources and energy on removing
barriers to charter school entry into the market. It is time to concentrate equally on
removing the barriers to exit.” 35
Part of the conflict among the national studies had to do with rather arcane
methodological wrangles. Part hinged on whether the state or national tests had the more
grievous flaws. But the key issue was how to surmount or work around a fundamental
barrier to fair comparisons—the difference between students at charter schools and regular
schools. Students at charters—who affirmatively chose their schools, or whose parents did
for them—are certainly not the same as students who stick with the local public school.
The question is how they differ, or rather the relative weight of the various ways they differ,
beyond objective measures such as race or income or locale. Some families opt for charters
out of desperation, when a child is performing abominably at her prior school. Some
families opt for charters out of ambition, since many charters (and not so many regular
35
CREDO report, page 8
18
schools, especially in poor neighborhoods) both require and reward active parental
involvement. If the former affect predominates, charters will have more than their share of
weak students, and simple comparisons will under-rate their ability to create educational
value. If the latter, charter students will be abler than average, and test scores will overstate
their success.
As the charter-school debate matured, several researchers recognized the possibility
to overcome this problem by exploiting the fact that many charters face excess demand and
select students by lottery. This presents a sort of natural experiment. If the students who
win the lottery are similar to those who lose—and there is no reason to suspect otherwise—
then tracking the subsequent experience of the two groups should give a good picture of the
true effect of charter schools.
One particularly interesting application of this method was a study of Boston charter
schools published in 2009 by a large team of scholars that included our ingenious colleague
Tom Kane. This group identified five Boston charters at the middle-school level, and four
at the high-school level, that have excess demand and select by lottery. Carefully
comparing the subsequent performance on state standardized test scores, they find that
lottery winners out-perform lottery losers in English and (especially) math. 36 These were
not statistically measurable but no-big-deal gains around the edges, but true game-changing
performance improvements. For a sense of scale, the authors compare the Boston charterschool effects to the infamous “black-white” test score gap that so chronically bedevils
American education. The average performance edge of these nine Boston charters found in
36
Atila Abdulkadiroglu et al., “Accountability and Flexibility in Public Schools: Evidence from Boston’s
Charters and Pilots,” National Bureau of Economic Research Working Paper 15549, November 2009, pages
14-15 and Table 4.
19
the study was more than big enough to wipe out the black-white gap in math, and to whittle
away the majority of it in English, if the effect could be scaled-up and replicated
nationwide.
As the authors note, though, the particular Boston schools in this study are, by
definition, the ones that are popular enough that they have to turn students away. They rely
on lotteries to ration scarce slots. So there is little reason for confidence that all charter
schools in Boston—let alone nationally—can offer such spectacular results. This
establishes a theme that recurs in the evidence and is stressed in our book. The best charter
schools are very, very good. But as a nation we are not yet doing a very effective job at
making sure that all charter schools are up to the standards of the best.
The respected research firm Mathematica, at the request of the Department of
Education, used a similar lottery-based method in a large-scale 2010 study of charter
schools. It compared lottery winners and losers at 36 middle-school level charters in 15
states. The results, alas, were less upbeat than the Boston study. Parents whose kids won
the lottery reported being more satisfied with their schools, on average, than lottery losers.
But the hard, cold numbers gave little support to the warm feelings. Lottery winners did no
better educationally than lottery losers. Indeed, what difference there was suggested that
lottery winners did a bit worse, though this result wasn’t quite statistically significant. 37
As we have seen before, though, a disappointing average result obscures a more
complex and, on balance, more hopeful pattern. The average conflates a mix of charter
37
Phillip Gleason et al., The Evaluation of Charter School Impacts, Institute for Educational Sciences, U.S.
Department of Education, June 2010
20
schools than do very well, and those that do very badly. The big Mathematica study found
tantalizing evidence that some groups—lower-income children, in particular, and those who
had been having problems in traditional public schools—fared especially well in charters.
Readers may be frustrated that even in this supplement there’s no crisp, conclusive
answer as to the payoff from this big national experiment in educational collaborative
governance. Be assured that we would like more certainty, as well. If there were a national
system for tracking every student’s test scores over time—as there is a national system for
tracking every worker’s income—then jumps or slumps in performance as a student moves
from one school to another would provide a lot of leverage for isolating the “school effect”
from the student effect. Or if an experiment could randomly assign large groups of students
to either charters or regular schools, we could get a fix on charter performance unpolluted
by hidden differences in student populations. But neither remedy is on the horizon, forcing
researchers to improvise ways to “control for” student characteristics using coarse and
partial measures. These improvisations are always less than fully satisfying, ensuring that
anyone can find some basis for throwing rocks at a study with uncongenial results.
What inferences, if any, can we draw from the incomplete evidence on charter
schools’ performance? Can anything be said about the public benefits, or lack thereof,
delivered by this important category of private agents operating with production discretion?
Or is the evidence so messy and ambiguous, and analysts’ assessments so inconclusive or
conflicting, that the results of the grand charter experiment remain a mystery pending
further study?
21
Further study is surely an excellent idea. And both true experiments and long-term
tracking of individuals students are entirely feasible, in principle, and likely to clear away
much of the fog over charters’ comparative performance. But one thing that can be said at
this stage, with a fair degree of confidence, is that there is no large and systematic
difference between charters and regular schools in the educational results they deliver. If
charter schools as a group enjoyed a big performance edge over other schools, or suffered a
big performance deficit, we would know by now. There is already too much experience
with charters, and too many avid partisans anxious to prove the case either way, for a
dramatic average difference to remain hidden. Charter schools, endowed with production
discretion, have an average effect on student performance that may be negative and may be
positive but is surely small.
The key word in the prior sentence, though, is “average.” A negligible average
difference could mean that all or most charters are trivially better or trivially worse than
conventional schools. Or it could mean that some charters are a great deal better, and
others are a great deal worse. Common sense, casual empiricism, and the (limited)
systematic evidence 38 all suggest that the second interpretation is far more plausible than
the first. Some charter schools use their freedom to create curricular offerings that deliver
38
For example, Miron et al. find in Pennsylvania that “some schools posted very strong gains, others reported
steep losses. Thus, whether…charter schools are effective in leveraging achievement gains depends very
much on the particular charter school in question.” Gary Miron, Christopher Nelson, and John Risely,
“Strengthening Pennsylvania’s Charter School Reform: Findings from the Statewide Evaluation and
Discussion of Relevant Policy Issues” The Evaluation Center, Western Michigan University, October 2002,
executive summary, page 3. The Brookings study also finds results that hint at a wide range of charter
performance, relative to conventional schools, and Loveless directly addresses this theme in Brown Center
Report on American Education, Part III, “Charter Schools,” Volume 1, No. 3, The Brookings Institution,
2002, page 35.
22
tremendous performance by the metric of test scores, and even more public value that
eludes the coarse empirical nets of standardized testing. Some charters deploy their
discretion opportunistically or cynically, collecting private benefits at the expense of public
value. And some—no doubt many—sincerely intend to deliver high educational
performance but simply bungle the job.
The charter movement seems to be generating a blend of stellar and sub-par schools
with little net benefit, on balance, to American education. This interpretation—to the
extent further evidence supports it—is not a refutation of the notion that collaboration with
discretion-wielding private partners can create public value in the educational realm. Still
less is it an indictment of the private sector; of course education entrepreneurs cover a wide
range of motives and competence. Rather, it reflects regrettable shortfalls in governance.
If officials responsible for the public side of this collaboration did a better job on
their key governance tasks—selecting the right charter applicants and motivating them
appropriately—the range of charter performance would be narrower and higher. Suppose
the officials authorizing charter schools were more consistently successful at weeding out
the weak and shady applicants, and successful as well at granting charters to the good
bets—even diamonds-in-the-rough with solid fundamentals but shaky marketing skills.
Suppose the officials overseeing charter schools were able to loosen or tighten the reins of
discretion in light of performance, requiring laggards to mend their ways and letting the
stars innovate unfettered. In such a scenario—without assuming any changes whatsoever
in the private-sector population of potential charter operators—we could realistically expect
far clearer, and far more positive, evidence that the charter movement was delivering on its
23
promise. Whatever the current payoff from the charter experiment turns out to be, if and
when the fog of uncertainty lifts, it could be a lot better with excellent governance.
Sub-par governance, on the public side of this experiment in collaboration, is not in
the least surprising. Most states have failed to recognize the public-sector challenges that
charter schools entail, and have grievously under-invested in governance capacity. The
Government Accountability Office (GAO) found in 2005 that only eleven states provided
any funding whatsoever for oversight by local districts or other charter-school authorizers. 39
When states were surveyed in 2004 about the personnel dedicated to charter-school
oversight, the most common response was that there was a single full-time-equivalent
staffer on the job. (States with large oversight offices, such as Ohio, raised the average to
three.) Arizona had nearly 150 charters for each of its two staffers. 40 Among all public
entities entrusted with authorizing charters—including state boards, universities, and the
most common, local governments—just one-third had any identifiable staff unit devoted to
charter oversight, and four times as many of these authorizers attributed accountability
problems to staffing inadequacies as to political pressure. 41
39
U.S. Government Accountability Office, Report to the Secretary of Education, “Charter Schools: To
Enhance Education’s Monitoring and Research, More Charter-School Level Data Are Needed,” GAO-05-5,
January 2005, (Table 4, page 21)
40
SRI International, “Evaluation of the Public Charter Schools Program: Final Report,” Report prepared for
the Office of the Deputy Secretary, U.S. Department of Education, November 2004, page 18
41
Ibid, page 38 and Exhibit 4-8, p. 50 It is also worth noting that the American Federation of Teachers study
found that in California and Connecticut, states with generally more rigorous oversight, average charter
performance was better than in states with less oversight (Texas, Arizona, Michigan, DC, and Colorado).
Nelson et al, Tables 8 and 9, pages 13-15
24
An extensive study of “life-or-death” decision-making by authorizing agencies—
decisions to renew or revoke a charter—found that most of the public organizations
involved were well-intentioned, resistant to political pressure, and reasonably likely to end
up making defensible choices. But they were in over their heads, with few exceptions,
under-equipped for the subtle and sophisticated tasks their mandates entailed. Local school
boards, by far the biggest category of charter authorizers, were especially short on the
capacity required to gather, process, and analyze evidence on charter performance. Fewer
than half of the decisions were informed by “a merit-based comparison of evidence and
expectations.” 42 Again, this is not surprising. These sorts of high-level skills are
chronically in short supply in the government, especially scarce at the local level, and
perhaps rarer still in education agencies—often a distinctive enclave within local
government.
The charter movement has been widely misconstrued as lifting from government’s
shoulders, and shifting to private actors, the responsibility for creating public value. But it
really represents an alteration, more than a diminution, of government’s role. A shift from
conventional public schools to charter schools entails different tasks for government—and
perhaps a smaller public headcount, if charter staff are viewed as less “public” than their
counterparts at conventional schools—but those tasks are vital ones. Government must
choose, enable, motivate, and oversee its collaborators to maximize the odds that private
discretion works in the service of public value. If charter schools, as a group, are failing to
42
Bryan C. Hassel and Meagan Batdorff, “High Stakes: Findings from a Study of Life-or-Death Decisions by
Charter School Authorizers”, Public Impact and Smith Richardson Foundation, February 2004 See
especially pages ii, 20, and 29-30.
25
deliver the clear-cut performance gains that advocates anticipated, it may be less because
we have over-estimated the power of private-sector innovation than because we have
under-estimated the governance work to be done.
26