HOW PREDICTABLE ARE JUDICIAL DECISIONS IN NEW ZEALAND’S HIGHER COURTS?
Phil Creagh
Director, Anderson Creagh Lai Limited
This paper looks at a sample of decisions of the New Zealand Supreme Court, identifying and
following judicial decisions made on the ultimately determinative issues as they have progressed
through to our highest Court to see if there is any statistically significant pattern of predictability in
the decision making. The paper finds that there is a relative lack of ex ante predictability of judicial
outcomes for civil litigants taking appeals from the High Court to the Court of Appeal and, where
leave to appeal is given, on to the Supreme Court.
Throughout this paper the case categories of interest are confined to civil decisions of the three
higher Courts that are of relevance to and could inform and guide the business community, and
which are referred to as cases in the business interest category.
The paper finds that for such civil proceedings commenced in the High Court and taken as far as
the Supreme Court, outcomes for participants as a whole across the past 10 years are statistically
no different than if the outcomes had been decided by random coin toss rather, than engagement in
proceedings. The paper proposes that this lack of predictability in judicial outcomes provides
systemic incentives to parties involved in litigation and their advisers to game the civil justice
system by appealing decisions wherever and as far as possible.
Introduction
At the core of our system of civil justice is the proposition that Courts of law have as a primary
purpose the role of applying known and ascertainable principles of law to particular fact situations
so as to produce principled outcomes in civil dispute situations. Further, that this process applied
correctly sends important signals to firms and citizens as to how they should arrange their
commercial affairs, and as to their rights and remedies in commercial disputes. This proposition
necessarily implies that there should be a level of consistency in the judicial consideration of
discrete legal issues as they progress through our hierarchy of higher Courts1.
It seems to be generally accepted throughout the legal profession and judiciary that the ability for
citizens and firms to commence and prosecute a civil action is an essential ingredient of a
well-functioning state. For example former Chief High Court now Court of Appeal judge Justice
Winkelmann in a recent address commented2:
―The Courts’ decisions articulate clearly how the law applies to the citizen, and thereby allow others to
order their conduct and affairs so as to comply with the law. Through the independent operation of the
Courts, society also orders itself in the certain knowledge and belief that all can have a remedy for a
wrong, and that no one, no matter how rich or powerful, is above the law.‖
She goes on to affirm that access to the Courts to challenge the wrongful acts of others is an
essential ingredient of civil society. Attorney General Chris Finlayson QC has also commented3
1
2
3
This paper considers the position only for the High Court and superior Courts, and references to the
Courts are to be taken accordingly.
Ethel Benjamin address Access to Justice – Who Needs Lawyers? (7 November 2014) pp. 3, 4.
stuff.co.nz article by Rob Stock Civil Justice in Tatters, 5 June 2011.
2
that there are now too few commercial cases making it to the High Court creating too little case law
to guide business4.
The Te Ara Encyclopaedia of New Zealand5, in explaining the New Zealand judicial system states:
―Common law adjudication is organised around the doctrines of precedent (like cases should be decided
alike) and stare decisis. Consistency of judicial decision making ensures the orderly development and
application of the law. Consistency and stability in the law are values intrinsic to the rule of law, allowing
citizens to predict how the law, when applied, will affect them.‖
As US Supreme Court Justice Antonin Scalia, a well-known proponent of principles of legal
formalism has said6:
―… [There is]...another obvious advantage of establishing as soon as possible a clear, general principle
of decision: predictability. Even in simpler times uncertainty has been regarded as incompatible with the
Rule of Law. Rudimentary justice requires that those subject to the law must have the means of
knowing what it prescribes.‖
Finally the OECD, in a recent economic policy paper7 (OECD Paper) notes:
―As emphasised by a large body of empirical evidence, well-functioning judiciaries are a crucial
determinant of economic performance. They promote the efficient production and distribution of goods
and services by securing two essential prerequisites of market economies: security of property rights
and enforcement of contracts. Security of property rights gives agents incentives to save and invest by
protecting returns from these activities. A good enforcement of contracts stimulates agents to enter into
economic transactions, by dissuading opportunistic behaviour and reducing transaction costs.‖
The OECD Paper goes on to note in relation to OECD countries generally that predictability of
decisions is one of the keys to guarantee the certainty of rules, and that judicial decisions are
sometimes too uncertain, inducing litigants to undergo a long process of appeals before the higher
Courts. The predictability of judicial decisions is identified as being one of the three main
dimensions by which performance of judicial systems in the OECD area can be assessed8.
So a well understood and accepted role of the New Zealand Courts should be to apply a known
body of ascertainable law relevant to commercial and business matters to the disputes that come
before them in a consistent and relatively predictable way. This at least is the formalist ideal. An
alternative view might see a dominant role of the Courts as being to apply case by case justice,
relatively unconstrained by common law conventions of precedent9
Appeal rates, predictability and outcomes
The OECD Paper seeks to assess the predictability of judicial decisions. Here however the OECD
Paper notes that there is a great scarcity of comparable data, which limits the scope of its analysis
4
5
6
7
8
9
High Court new business statistics bear this concern out, with the figures for new civil proceedings
commenced across the past 3 reported years being: for 2011: 3005, for 2012: 2827, for 2013: 2669 and
for 2014: 2526.
A project of the Ministry for Culture and Heritage.
Antonin Scalia The Rule of Law as a Law of Rules 56 U.Chi.L.Rev.1175.
OECD Economic Policy Paper No. 5 Judicial Performance and its Determinants, a Cross Country
Perspective (June 2013) pg. 8. See also the related OECD Economics Department Policy Note No. 19
What Makes Civil Justice Effective? (18 June 2013)
The OECD Paper notes, however, that comparative information and data is extremely difficult to source
and that caution must be applied in the interpretation of its results.
This school of thought draws from work of the legal realists who seek to describe the process by which
law is made or applied in a broader than just legal social, psychological, political and economic context.
3
and is a major obstacle to empirical analysis of judicial systems10. There is some research looking
at aspects of the predictability of judges’ votes on the US Supreme Court11. However, this research
tends to look at voting patterns or voting predictors of individual justices, given that there are nine
sitting Supreme Court justices and that many decisions of the US Supreme Court are split
decisions. Research data on the predictability of identified and determinative legal issues that find
their way through to the higher appellate Courts is difficult to find.
In relation to the important third factor (predictability of Court decisions) the OECD Paper notes that
the predictability of Court decisions, meaning the possibility to predict ex ante how the law will be
applied by the Court, is extremely important from an economic perspective, as it guarantees the
certainty of law and enables ―economic agents‖ to anticipate the potential legal consequences of
their actions. This, says the OECD, is key to making correct decisions. The predictability of Court
rulings is influenced by the uniformity in the application of the law, which in turn requires the equal
treatment of similar disputes as one major factor. The OECD Paper goes on to note that
measuring predictability per se is difficult but that information on this can be inferred by proxy from
appeal rates before higher Courts. It will be obvious that this is an intuitive inference drawn by the
OECD Paper and no solid evidence is cited to establish the basic proposition i.e. that poor
predictability leads to higher appeal rates.
So the proposition here is that there is a relationship between appeal and reversal rates according
to which the likelihood that an appeal is filed will be maximised when the expectation of the litigant
parties about the probability of a reversal by the court of next instance is close to 50%, and that the
likelihood of an appeal will be lower when the probability of reversal is either very low or very high.
In short, there will be a higher rate of appeals when there is a higher rate of uncertainty (i.e. lack of
predictor information) about outcomes of litigation. This is at least partly regulated by the costs of
litigation as litigation is costly and if parties can predict with sufficient certainty the likely outcome of
litigation it will be in their interest to reach settlement and save on litigation costs12.
This paper looks at a selection of Supreme Court decisions of interest to business across the past
10 years and then tests the data obtained against the (no doubt provocative) proposition that the
decisions of the three higher Courts in these business interest categories are, on average,
effectively random, which is to say that on average, across all cases, there is no statistically
significant pattern of predictability in judicial decisions on cases going all the way to the Supreme
Court on what prove to be the determinative legal issues in those cases. The paper, testing the
proposition in the OECD Paper, then considers whether these findings mean there are
opportunities for litigants to game the Court system by maximising available appeal processes.
Available data
The OECD figures for New Zealand appeal rates only deal with appeals to the second instance,
and the reported rate for New Zealand is very low, indeed the third lowest in the reported OECD
countries. However, the data used in preparation of the OECD Paper should be viewed cautiously
in the New Zealand civil jurisdiction in. The OECD data is primarily sourced from questionnaires, is
very broad ranging and seems to include District Court civil cases in the data set, with the result
10
11
12
The OECD Paper gives the New Zealand Courts a relatively clean bill of health on a number of
measures. For example in trial length and cost of civil trials, New Zealand ranks towards the top end of
OECD performers, with low trial lengths and low costs of trial. New Zealand also ranks towards the
upper end of OECD countries in terms of the budget allocated to its Courts as a percentage of GDP.
See for example Guimera and Sales-Pardo Justice Blocks and predictability of US Supreme Court Votes
PLOS ONE 6 (9 November 2011), and Bommarito and Blackman Predicting the Behaviour of the
Supreme Court of the United States, A General Approach (21 July 2014) Social Science Research
Network.
OECD Paper pg15.
4
that the OECD Paper data could be skewed by including a large number of smaller civil claims that
are then appealed at first instance to the High Court. As will be seen, when appeal rates from High
Court decisions in civil matters are scrutinised they seem in fact to be quite high.
For the New Zealand Courts there is no easy way of obtaining data on reversal rates on appeals. It
seems curious that the High Court, for example, is subject to numerous useful measures of
performance13 on matters such as clearance rates, waiting time to trial, earliest available trial dates
and time to judgment, but does not report on reversal rates at all, or even identify reversal rates as
a relevant judicial performance issue. For example the 2013 year report from the High Court14
includes information relating to the disposal of trial adjudications, judgment timelines and
performance standards for civil proceedings, including waiting time to trial, available dates for trial
etc., but nowhere deals with that third factor identified as very important by the OECD, namely
providing any measurement of predictability in relation to decisions made by High Court judges in
civil cases.
This may be because there is no easy way in to currently source the necessary data based on the
information that is currently available. However it is possible to adopt a data analysis approach to a
selection of judicial decisions in a way that gives some ability to measure the consistency of
decision making, and therefore predictability. In future, providing information on the various Courts
or even individual judges reversal rates could be something that needs to be considered further,
because whether the judge or Court ―got it right‖ is something that is always top of mind for litigants
and counsel when a judgment issues. A comprehensive civil justice system should arguably
include mechanisms to improve judicial decision making through providing feedback on appeal
outcomes.
High Court data
Since 2013 the High Court has been classifying general proceedings filed with it into 32 ―nature of
claim‖ categories.15 For the purposes of this paper the 20 subcategories it is proposed are of
interest to the business community in terms of actually guiding decision making are16: Natural
disasters – Christchurch earthquakes; contractual disputes; Building defects – Weathertight claims;
Other trust litigation; Other real property disputes; professional negligence; agreements for sale and
purchase of land; other company law disputes; intellectual property; insurance claims; building
defects-other; landlord-tenant; agreements for sale and purchase of business; tax; agreements for
the sales of goods; finance companies; personal property claims; Securities Act; Commerce Act;
and Marine claims.
For the period from 1 January 2013 to 30 November 2014 (i.e. 23 months, the latest period for
which information was available when this paper was written) these 20 categories accounted for
1,633 (or approximately 60%) out of the 2,713 general proceedings filed across the country.17
Annualised, this is approximately 850 proceedings per annum in the business interest categories.
13
14
15
16
17
High Court National Performance Measures published by Courts of New Zealand.
Report from the High Court 2013- The Year in Review (9 April 2014).
Each case is classified into one category only. The classifications are made by Associate Judges and
Judges of the High Court.
It is appreciated that this is a subjective categorisation. For e.g. it may be that the category ―other tort
claims‖ could be included, or other trust litigation excluded.
2,713 general proceedings were filed in the High Court over this period. 320 had not been classified at
the time the data was compiled. Of the 2,393 general proceedings classified 1,440 (60.18%) fall into the
identified categories. The 60.18% figure has been applied to the total number of proceedings filed to
arrive at the 1,633 figure across the 23 month period.
5
Actions in other general proceedings categories e.g. debt recovery, insolvency-related claims and
actions on guarantees, would seem to be procedural or enforcement issues of little business
informative value, while categories such as estate litigation, claims against public authorities and
other tort claims appear to be of no great relevance to business generally.
New claims that relate to the selected business interest categories therefore appear to make up
around 60% of the annual new business arriving in the High Court under the broad head of civil
proceedings in each year.18 Over this specific period 15% of general proceedings were reported as
resolved by trial, which means that 85% or more were disposed of in other ways, including by
settlement. However reports for prior High Court years show that the resolution by actual trial
figures generally average around 9%. So this paper adopts a resolution by trial rate averaging
10%. Applying this rate it may be that as few as 85 cases in the identified business interest
categories actually make it to trial in the High Court each year across all the High Court registries
over the whole country19. This is likely to represent only a small fraction of the commercial disputes
that are resolved by means other than concluded High court litigation each year.
The threshold issue
This then raises a very important threshold issue, which is whether the cases that are commenced
by the filing of a High Court proceeding but which do not make it to trial are those involving
relatively clear cut legal issues and the background of certainty in the applicable law, so that those
cases which progress are mainly the finely balanced cases where reversal on appeal is not
surprising. The argument here is that the cases that ―enter the gate‖ in the High Court and go all
the way to trial can be expected to be the cases involving the more contentious legal issues, so that
uncertainty in the decision making is to be expected. This view holds that the fact that some 90%
of commenced High Court cases do not go to trial is an indicator of high predictability in the system,
as it suggests litigants, despite commencing cases, quite early after discovery and other preliminary
processes are completed, are informed enough as to their likely prospects to see economic
benefits in settlement and do in fact settle.
The counter to this is the proposition that litigation is commenced and pursued for any number of
reasons that are not tied to a careful and dispassionate analysis by litigants of their prospective
outcomes, so that the cases that go the distance are not necessarily the ones that involve and
require judicial clarification of generally unresolved legal issues. This view holds that judges
usually have a pick list of potentially determinative points to choose from in shaping their decisions,
that their decision making is often on quite subjective factors and supported by the flexibility and
opportunities afforded them by the pick list, and that as a result the likely outcomes in most cases
will always be 50/50 or thereabouts.
There is simply no way of which view is to be preferred at present. However after the first instance
High Court hearing cases on appeal will almost always involve points of law rather than contested
facts and it is possible to then empirically study the subsequent decisions on appeal to get some
direct measure of predictability.
Court of Appeal data
At the next level up information from the Court of Appeal is scarce. The most recent published
information available seems to be a report of the Honourable Grant Hammond J for the Court of
Appeal for 200920. Somewhat ironically the report has as its focus the discontinuing of detailed
18
19
20
https://www.courtsofnz.govt.nz/from/statistics/annual-statistics/latest-June-2014/high-court/high-courtcivil-proceedings-case-volumes-for-the-12-months-ending-30-June-2014.
There is however no information available breaking down the types of matters that make it to trial.
Court of Appeal Publication 2009: Discontinuance of Court of Appeal Annual Reports.
6
Court of Appeal annual reports as put out in previous years, but the report does include a useful
summary of the position in relation to civil appeals for the years 2005 to 200921. From 2005 to 2012
the Court of Appeal decided between 124 and 152 civil appeals in all categories at substantive
trial22 annually of which on average, 62% were dismissed and 38% allowed. The data for these
years is reproduced in table 1:
Table 1- Court of Appeal civil appeal decisions 2005-2009
Appeals allowed
Appeals dismissed
Total decided
2005
53
43%
71
57%
124
2006
57
41%
81
59%
138
2007
43
30%
101
70%
144
2008
51
38%
84
62%
135
2009
54
40%
81
60%
135
2010
58
41%
84
59%
142
2011
56
42%
78
58%
134
2012
43
28%
109
72%
152
415
38%
689
62%
1104
Note however that these civil appeals decided by the court of Appeal are for all civil categories and
are not broken into categories that align with the High Court’s ―nature of claim‖ categories
subsequently adopted for general proceedings. If as a rule of thumb the same 60% cases relevant
rate adopted for High Court proceedings is applied to appeals heard, and the volume of appeals
remains similar, then this implies that around 80 Court of Appeal decisions in the business interest
categories are decided each year. However as this paper adopts a running average of 85 such
cases starting in the High Court annually as the ―feeder‖ cases, either the numbers of feeder cases
reaching trial in the High Court must be considerably higher than 85 (possibly substantive business
interest disputes reach trial far more often than other disputes), or the 60% cases relevance rate for
Court of Appeal civil matters is much too low. It could be for example that these kinds of cases are
appealed more often. However even if the rate were higher, for example 80%, there would only be
just over 100 cases in the business interest categories tried in the Court of Appeal each year.
Regrettably there is no current way of obtaining better data with any level of reliability.
Supreme Court decisions
For the commercial community the ultimate arbiter of right and wrong in commercial causes since
2004 has been the Supreme Court of New Zealand. By definition decisions of the Supreme Court
are always ultimately correct, there being no further overturn possibilities beyond that Court. So the
decisions on the determinative issues on the Courts below in such cases are either correct or
incorrect viewed with hindsight.
Method
The number of cases in the business interest categories that have been decided by the Supreme
Court over the first 10 full years is not large. Obviously a large part of the Supreme Court’s
business deals with matters other than commercial causes. A large part of that substantive
business is dealing with criminal appeals, immigration appeals, procedural disputes and the like.
This paper looks at decisions commencing in 2005, when substantive civil decisions of the Court
21
22
OIA requests were necessary to source data for the later years.
As opposed to procedural leave to appeal applications.
7
began to issue, through to and including 2014, a 10 year sample. Selections of relevant business
interest decisions were then made to as closely as possible align with the High Court nature of
claim categories now being reported. Included are all decisions in general commercial cases, cases
involving competition law, tax law and employment law. Excluded are all criminal, immigration and
similar cases, family law cases, and cases which relate to procedural matters. This produces a set
of cases this paper proposes are the suite of relevant rulings issued in the past 10 years that would
be of interest or helpful to the general business community. Surprisingly there are fewer than 100
such decisions in total across the 10 year period, an average of around 10 per year.
Supreme Court decisions have a particular advantage statistically which is that they are almost
always decided on one or only a very few limited and approved legal issues that are easily
identified. Supreme Court practice in allowing leave to appeal is to tightly define the approved
issue or issues on appeal. Those issues will almost always have received specific consideration
below in the Court of Appeal, and before that below again in the High Court, and will have been
decided on a binary basis i.e. win/ lose. So it is possible to look at the decision in a High Court
case where there may have been a broad range of issues in argument, and the generally fewer
issues that have ultimately made it through to the Court of Appeal and then the one or two issues
approved for appeal by the Supreme Court.
Each discrete issue was followed through the court hierarchy, starting with the first instance High
Court decision, in a binary way i.e. plaintiff or appellant won or lost on that issue. Where multiple
issues featured in a final Supreme Court decision the focus was on the principle determinative
issue underpinning the decision. As the determinative issues were able to be extracted and
identified from the lower court judgements, information was extracted as to how, at each level, the
issue was decided. The ability to do this means that an issues based ex post facto statistical
review could be carried out that gives a direct measure of decision predictability, or the lack of it, in
relation to those civil matters that have proceeded right through the civil justice system.
This of course is by no means a perfect methodology, but as the OECD Report makes clear, the
OECD has been forced into the position of looking just at rates of appeal as a proxy for
predictability absent any direct measures of predictability per se, the OECD proposition being that
in systems where decision making is unpredictable appeal rates will be higher than would be the
case if litigants had better ex ante information about their appeal prospects. So while imperfect,
even an ex post facto review of a very limited set of High Court, Court of Appeal and then Supreme
Court decisions provides data that gives a direct issues and outcomes based measure of
predictability than indirect appeal rates.
8
Findings
The raw figures are produced in table 2:
Table 2- Supreme Court business interest decisions 2005-2014
year
Plaintiff
won HC
Plaintiff
lost HC
HC
upheld
by CA
HC
overturned
by CA
CA
upheld
by SC
CA
overturned
by SC
HC upheld
by SC
HC
overturned
by SC
2005
3
2
4
1
3
2
4
1
2006
6
6
5
7
6
6
5
7
2007
3
3
1
5
3
3
3
3
2008
5
7
9
3
7
5
4
8
2009
7
5
4
8
5
7
8
4
2010
10
6
6
10
8
8
8
8
2011
4
3
3.5
3.5
5.5
1.5
4
3
2012
7
4
3
8
7
4
3
8
2013
4
4
3
5
3
5
6
2
2014
3
4
3
4
3
4
5
2
Total
52
44
41.5
54.5
50.5
45.5
50
46
Average by year
5.2
4.4
4.15
5.45
5.05
4.55
5
4.6
54%
46%
43%
57%
53%
47%
52%
48%
As percentage
Analysis
A Chi-square test was carried on this data out to investigate the difference between the observed
numbers of wins/losses across all cases and the number of wins/losses that would be expected if
the outcome across all cases was random (i.e. as random as a fair coin toss). The ―null hypothesis‖
baseline against which these statistical tests were carried out was that for each of the three Court
levels, while there are variances one way or the other, decision outcomes on the same issues
across all cases are not significantly different from what would be observed from just tossing a coin
to decide the result. In short we find that the outcomes are no better than random23.
The ―alternative hypothesis‖ therefore was that there are statistically significant differences shown
and statistically significant consistency in the decision making on issues across the cases. In short
that the outcomes are produced by some non-random factor, in this case consistency of judicial
decision making on legal issues.
The analysis showed that of High Court cases that made it through to the Supreme Court, the
plaintiff who commenced the action on average won slightly more than 50% of the time at first
instance. There was no statistically significant difference between outcomes where the plaintiff won
and the defendant won (p-value > 0.05).
Table 2 also shows that the Court of Appeal overturned the first instance decision somewhat more
than 50% of the time. Again however on the data available, this is not a statistically significant
difference24 (p-value > 0.05).
23
24
To a p-value of 5%. Statistically the p-value is the probability of obtaining the observed sample results (or
a more extreme result) when the null hypothesis is actually true.
To take into account the fact that a very few cases progressed directly from the High Court to the
Supreme Court, where this has occurred the result at a notional Court of Appeal level has been
arithmetically declared a draw.
9
At Supreme Court level the data shows that the Court of Appeal was itself upheld or overturned by
the Supreme Court broadly equally often, and yet again the difference is not statistically significant
(p-value > 0.05). Finally for completeness looking back at the original High Court decision, the
Supreme Court ultimately upheld or overturned the original High Court decision broadly 50% of the
time.
Recently the New Zealand Law Society has published figures on the upholding/ overturn rates for
Supreme Courts for five common law jurisdictions plus the Privy Council for all appeals brought25.
Of interest is that there are two outlier jurisdictions, while the others including New Zealand have
broadly a 50% or just slightly higher overturn rate. The outliers are Canada where the overturn rate
is low at less than 36% and the USA where it is high at 75%. So the New Zealand Supreme Court
operates at a similar overturn rate to its counterparts in the UK and Australia in particular.
Notes of caution
These findings need to be looked at with caution for a number of reasons. Firstly the sample is
extremely selective and small. Only a proportion of cases commenced and then actually heard in
the High Court will ever make it through to the Supreme Court, but this may be a surprisingly high
proportion. As analysed above the High Court may, on average, be deciding as few as 85 general
proceedings cases per annum in the business interest categories. The Court of Appeal in turn may
hear on appeal a reasonably large proportion of first instance decisions in our business interest
categories each year, although currently there is no data from which to extract any meaningful
average figure for the appeal rate26.
As better figures can be extracted from the High Court reports it seems that there are 85 or so
business interest category decisions of the High Court annually, and in turn each year around 10
such category cases are ultimately decided by the Supreme Court. However given also that there
is currently a 20% approval rate generally for appeals to the Supreme Court, intuitively this final
decision rate seems too high, so significant timing factors and mismatches must be in play, and this
very high final decision rate must be viewed with caution. The Supreme Court may hear a final
appeal two or even more years after the first instance decision, and the recent consistent decline in
High Court civil proceedings commenced would suggest the 10 cases per year of business interest
average will fall as the number of High Court feeder case falls. More data would need to be
collected over a longer period before any conclusions on the appeal rates could be safely drawn.
The relevant High Court decisions in our categories of interest may also in the nature of things
relate to difficult or more complex matters and to disputes where the legal issues are uncertain. If
this sort of pre-selection is involved that to this degree ―errors‖ at the level of the High Court
identified with the benefit of hindsight, given the human imperfections inherent in judicial decision
making, would be understandable. These may well be a pre-selection of the sorts of difficult cases
which are in the grey zone and not black and white.
However when the Court of Appeal sits on a case, it will tend to hear an extraction of the issues
considered at first instance, as outlined in the case on appeal. Generally the Court of Appeal will
leave undisturbed findings of fact made by the Judge of first instance. So it is reasonable to expect
that the High Court is upheld markedly better than 50% of the time. As table 1 shows at first look
this is in fact what the available figures on their face imply i.e. that for appeals generally to the
Court of Appeal in all civil matters the High Court is upheld around 62% of the time. However, the
Court of Appeal civil appeal figures are for a case selection that is much broader than the business
25
26
https://my.lawsociety.org.nz/news/supreme-courts-allow-53-of-appeals.
This could of course be done by a similar case by case analysis of Court of Appeal decisions over a
suitable period as this paper has done for Supreme Court decisions, but a research project of this scale
was beyond the scope of this paper.
10
interest categories, and for example includes appeals on originating applications, judicial review
applications and procedural matters. There is no data available, but it may be likely if an analysis
was carried out on High Court decisions in the relevant categories that are appealed only one
instance to the Court of Appeal, that the upholding rate on determinative issues could be lower than
the 62% figure that is the average for the years for which figures are available. As we see from
table 2, the reported 62% Court of Appeal upholding rate (for all appeal categories) changes
markedly when decisions in the categories of interest are appealed further to the Supreme Court27.
Another distorting factor could be the statutory barrier requiring leave to appeal from the Court of
Appeal to the Supreme Court. In broad terms over recent years, around 80% of all applications for
leave to appeal to the Supreme Court are dismissed, which leaves around 20% proceeding through
to hearing. There is be the possibility that of that 20%, to some degree, and despite the statutory
threshold tests28 the Supreme Court is letting ―through the gate‖ cases that seem to be meritorious
and therefore likely to be overturned. A sufficiently meritorious case may well involve a substantial
miscarriage of justice in any event.
However, the pattern we actually see again is broadly 50/50, which suggests encouragingly that the
Supreme Court judges are following the statutory criteria well in their decisions to allow cases
through to final appeal. If there was a pre-selection of deserving cases explicitly or implicitly
involved we would expect to see a much higher rate of reversal of the Court of Appeal. In fact as
we see, the actual 10 year running average slightly favours the Court of Appeal, with 53% of
relevant decisions being upheld (although again this is not a statistically significant difference).
Discussion
If you are a believer that principles of legal formalism guide predictable decisions in the new
Zealand higher courts you may be disappointed by these results, since they reveal no pattern of
consistency in judicial decisions on determinative legal issues that have found their way to the
Supreme Court that is statistically relevant at any level.
Across the group of all civil appeals in the business interest categories, the data does not show that
there is any determinative pattern that is being applied. To this extent, the actual outcomes on
average across the set of all appeals at all levels are seen to be not different from a random
pattern. What this means is that for civil cases in the business interest categories that have found
their way through to a Supreme Court decision, at all levels the win/lose outcomes for the litigants
on both sides are not in the end statistically different from the outcomes that would have applied
had those litigants, at the two appellate levels, instead of litigating, simply agreed to be bound by
the outcome and tossed a coin to decide the result.
Taking the coin toss analogy further, if the coin is fair a sufficient sequence of tosses will show
random heads and tails results, because there is nothing determining the results other than chance.
To make this deterministic the coin can be loaded so it is no longer and fair heads comes up say
65% of the time. As a result the toss sequence will no longer be random when statistically tested
and there will be a visible pattern to the outcomes. To expand this to a model to mimic decision
making by multiple judges the hypothetical tester can then take 10 coins which may be fair or
loaded. Individually some may be fair and some loaded, but if across the whole set the results are
still statistically random then any loaded coin results are being diluted into statistical insignificance
by the fair (i.e. random) coins.
27
28
Looking just at the equivalent Supreme Court report years 2001- 2009 the upholding rate is 51%.
Section 13(2), Supreme Court Act: It is necessary in the interests of justice for the Supreme Court to
hear and determine a proposed appeal if—(a) the appeal involves a matter of general or public
importance; or (b) a substantial miscarriage of justice may have occurred, or may occur unless the
appeal is heard; or (c) the appeal involves a matter of general commercial significance.
11
Reverting to the table 2 data and the results found in this paper, as the outcomes in the sample of
cases studies are not different from random, this means there is no statistically relevant pattern of
consistency in the decision making on average throughout the appeal chain across the multiple
judges involved in the judicial decision making on ultimately determinative issues decided by the
Supreme Court. As this random pattern is consistent across the 10 year sample of cases studied,
looking ahead there is no reason to suppose that in future, on average, litigation prospects for any
party will be any better than the odds on a simple coin toss, provided the litigation progresses far
enough.
This in turn suggests that the belief that judges in the higher Courts are applying known principles
of law to fact situations so as to produce principled outcomes on a consistent basis may not be
valid. Table 1 of course shows a slightly different position for High Court decisions that are taken
on just one appeal (i.e. appeals dismissed 62% of the time) and where the Court of Appeal decision
is final, but the reported data is for all civil appeals, not just in the categories this paper proposes
are of business interest, and the upholding rate is likely to be skewed as a result.
If we believe that our highest Courts should be providing the business community with predictability
and some level of certainty in the application of well-known and ascertainable legal principles to
commercial disputes, then we would expect better than random results throughout. We would
expect, looking at the particular determinative issues that are consistent through the progress of a
case, that the Court of Appeal would more often than not uphold the High Court, that the
Supreme Court would more often than not in turn uphold the Court of Appeal, and that the scope of
―more often than not‖ would be statistically relevant according to norms applicable to this type of
statistical testing. But as observed for the case type selection reviewed in this paper this is not the
case.
Gaming the system- use all appeals!
Even though reliable numbers cannot be extracted, it seems that there is a high appeal rate for civil
cases in our selected categories, which implies that litigants and their counsel are explicitly or
implicitly aware that they have a broadly 40/60 (against) prospect on appeal as of right to the Court
of Appeal which then improves to a further broadly 50/50 prospect of success if leave can be
obtained to go to the Supreme Court. The OECD’s ―economic agents‖ may therefore be seeing
opportunities and incentives to ―game‖ litigation results through appeals rather than settlements.
Determined and well-resourced losing parties may therefore have systemic incentives to appeal
repeatedly if necessary.
How the statistical patterns reveal opportunities for gaming adverse litigation results through
initiating appeals can be seen in the models in tables 3A and 3B below. These have been
produced from an amalgam of the last available Court of Appeal data (i.e. table 1) and the figures in
table 2, and neither the sample periods nor sampled civil cases in the two tables are identical, so
the model is not claimed to be highly accurate. However it serves to illustrate the arithmetic trend
of regression to the mean in the parties’ converging outcome odds over repeated appeal decisions.
With increasing layers of appeal there are more binary win/ lose decisions on issues made by the
courts and therefore more coin toss like chances. Over time, if decision outcomes are in fact
random, these will tend to trend towards 50/50 outcomes and this is what we see in these tables. In
table 3A an initial 54% win rate for the plaintiff trends down to 51%. When a second layer of appeal
is added in table 3B, even with the hurdle of leave to appeal being relatively unlikely, the initial win
rate for the plaintiff trends even more towards 50%.
So if a litigant loses at any level there are good i.e. 50/50 odds of a successful appeal reversing the
decision. A factor in this is likely to be that the scale costs risk of losing an appeal are relatively
low, at least in comparison with jurisdictions which require losers to pay all the winners actual costs.
12
Also, second and third instance own party costs are progressively smaller by comparison with first
instance case costs as appeals progress up the tree and issues narrow, evidence is not required,
and hearing times reduce. So the probabilities often will favour ―having a go‖ on appeal.
Table 3A- Appeal decision tree- to Court of Appeal only
High Court odds
Overturn rate
Court of Appeal odds
One appeal
outcome
Plaintiff wins
win
win/win
Plaintiff wins
62%
lose
33%
win/lose
54%
38%
21%
51%
Plaintiff's
prospects
Plaintiff loses
win
lose/win
Plaintiff loses
38%
lose
17%
lose/ lose
46%
62%
29%
49%
Table 3B- Appeal decision tree- to Supreme Court
High
Court
odds
Overturn
rate
Court of
Appeal
odds
One
appeal
outcome
Leave to
appealSC
Overturn
rate
Supreme
Court odds
Two
appeals
outcome
Plaintiff
wins
win
win/win
Plaintiff
wins
yes
win
win/win/win
Plaintiff
wins
43%
lose
54%
23%
25%
win/lose
57%
31%
no
49%
53%
lose
75%
win/win/lose
47%
win
3%
50%
win/lose/win
53%
lose
Plaintiff's
prospects
3%
4%
win/lose/lose
47%
4%
win/win/stop
17%
win/lose/stop
23%
Plaintiff
loses
win
lose/win
57%
lose
46%
Plaintiff
loses
26%
20%
win
25%
lose/ lose
43%
yes
no
51%
lose/win/win
53%
lose
75%
Plaintiff
loses
3%
lose/win/lose
47%
win
3%
lose/lose/win
53%
lose
3%
lose/lose/lose
47%
2%
lose/win/stop
20%
lose/lose/stop
15%
50%
13
Conclusions - Lies, Damn Lies and Statistics29?
Our civil justice system assumes that a role of the Courts is to take a known body of ascertainable
law relevant to commercial and business matters, and apply this to such disputes as come before
them in a consistent and relatively predictable way. By this means the Courts are expected to
provide guidance in the ordering of commercial affairs based on relative ex ante predictability.
Relative predictability here means that commercial arrangements can be entered into or legally
significant decisions made by parties with an acceptable level of advance information as to the
legal nature and enforceability of those arrangements, and as to the parties’ likely rights and
remedies in the event of breaches or disputes.
With the availability of new nature of claim information, High Court and higher decisions of interest
to the business community can be more easily identified and studied. In broad terms, based on the
2013 High Court Report, there seem to be in the vicinity of around 850 civil actions in the business
interest categories filed across New Zealand annually at the present time.
Of all civil proceedings filed somewhere on average 10% currently make it to actual trial30, with the
vast majority settling or being dismissed. It is possible that as few as 85 cases in the business
interest categories are therefore heard in the High Court each year across the country, likely to be a
small proportion of the commercial disputes that are resolved by judgment or other means each
year31.
On average civil appeals to the Court of Appeal run at around 135 or so per annum of which, in raw
terms for the years studied, there is a 62% upholding rate of the lower, i.e., High Court, decisions.
However, the categories of civil cases heard by the Court of Appeal are broader than the categories
that are the focus of this paper, i.e., decisions of general relevance or importance to the business
community, so the upholding or reversal rates for appeals which stop at the Court of Appeal and go
no further, are not presently known. They may be lower than the reported 62%.
Of those Court of Appeal cases decided it is not possible to currently determine how many cases of
relevance are decided annually by that court. It could be though that the appeal rate in the
business interest categories is relatively or even very high, and that this reflects losing litigation
participants’ calculating that the odds generally favour an appeal. As the numbers of Court of
Appeal cases in these categories is not known the appeal rate for cases to the Supreme Court
(which necessarily have received leave) is also unknown but again may be relatively high. High
appeal rates imply a lack of predictability in judicial outcomes in civil litigation in the higher courts.
For those cases that make it right through to the Supreme Court, the determinative issues were
extracted and related back through the lower court decisions. The data shows that, based on the
determinative issues that underpinned the ultimate Supreme Court judgment, in statistical terms the
predictability of decisions made at second instance by the Court of Appeal and at third instance by
the Supreme Court are not different from the random results that could be expected from a
sequence of fair coin tosses. In short, that for cases in this category, outcomes for participants in
the litigation process at both appellate levels are not in a macro sense statistically different from
outcomes that could be found if the participants agreed to toss a coin to resolve their dispute, rather
than proceed through litigation.
29
30
31
With apologies to Mark Twain.
See note 16 above.
Although as pointed put above, the number of civil appeals heard annually by the Court of Appeal
implies that business interest disputes make it to trial more often than this.
14
As the proportion of originating High Court cases in these categories that are appealed, ultimately,
to the Supreme Court, seems significantly higher than the proportion reported in the OECD Paper
as the general civil appeal rate, this casts doubt on the relevance of the OECD data in this area for
New Zealand. The appeal rate for civil cases in the business interest categories as calculated and
assumed from the available data is in fact very high.
Quite apart from the findings in this paper, these high appeal rates themselves imply a high level of
unpredictability around judicial outcomes and that participants in the litigation process, including
legal counsel, are explicitly or implicitly aware that the prospects of overturning of unfavourable first
instance or even second instance decisions are, in broad terms 50/ 50, and therefore that there are
systematic incentives on parties to civil litigation to commence and pursue all available appeals32.
November 2015
32
The risk management decision here of course is influenced by the fact that, in general terms, costs
awards against unsuccessful parties in the higher Courts go nowhere near covering actual solicitor/client
costs for the victorious party, thus decreasing the costs of unsuccessful participation in litigation.