REGULATING INSCRUTABLE SYSTEMS
Andrew D. Selbst*
Solon Barocas†
From scholars seeking to “unlock[] the black box” to regulations requiring
“meaningful information about the logic” of automated decisions, recent discussions of
machine learning—and algorithms more generally—have turned toward a call for
explanation. Champions of explanation charge that algorithms must reveal their basis for
decision-making and account for their determinations. But by focusing on explanation as
an end in itself, rather than a means to a particular end, critics risk demanding the
wrong thing. What one wants to understand when dealing with algorithms—and why
one would want to understand it—can vary widely. Often, the goals that motivate calls for
explanation could be better served by other means. Worse, ensuring that algorithms can be
explained in ways that are understandable to humans may come at the cost of another
value that scholars, regulators, and critics hold dear: accuracy. Reducing the complexity of
a model, for example, may render it more readily intelligible, but also hamper its
performance. For explanation to serve its intended purpose and to find its appropriate
place among a number of competing values, its champions need to consider what they hope
it to achieve and what explanations actually offer
Existing law is replete with demands for explanation in service of other normative
goals. The requirement that those affected by decisions be able to make sense of them is a
central tenet of due process and administrative law. It is also embedded in specific laws
like the Equal Credit Opportunity Act (ECOA), which requires that lenders give
“statement[s] of reasons” for adverse determinations, and the General Data Protection
Regulation (GDPR)—the recent data protection legislation in the European Union—that
grants data subjects the right to seek “meaningful information about the logic” of
automated decisions. Importantly, the type of explanation required differs between ECOA
(specific reasons for a decision) and GDPR (the decision-making process itself), and that
reflects the different purposes underlying each law. Similarly, those calling for explanations
seek different results, whether a justification for those affected by a decision, means to assess
the validity of decision-making procedures, ways to evaluate the fairness of processes, or
even just mechanisms to anticipate an algorithm’s behavior and prepare accordingly.
Those concerned with machine learning have many of these same concerns in
mind, but new calls for explanation take issue not only with secrecy, opacity, or validity,
but with inscrutability as well. Increasingly, demands for explanation are also demands for
interpretability, reflecting a growing realization that certain algorithms may resist
*
†
[insert author’s note]
[insert author’s note]
2
Regulating Inscrutable Systems – DRAFT 3/20/2017
meaningful interrogation even when they are accessible, transparent, and perform
well. Indeed, machine learning may produce models that differ so radically from the
way humans make decisions that they resist sense-making. Older demands for explanation
presupposed the possibility of a human who could provide an account of her decisions, even
where those explanations amount to legal fictions. In this case, even when machines are
able to explain themselves, they may not do so in ways that satisfy humans’ curiosity or
concerns.
This Article takes seriously the calls for regulation via explanation to investigate
how existing laws implementing such calls fare, and whether interpretability research can
fix the flaws. Ultimately, it argues that while machine interpretability may make
compliance with existing legal regimes easier, or possible in the first instance, a focus on
explanation alone fails to fulfill the overarching normative purpose of the law, even when
compliance can be achieved. The paper concludes with a call to consider where such goals
would be better served by other means, including mechanisms to directly assess whether
models are fair and just.
Introduction ................................................................................................... 2
I. What-How-Why and The Problem of Inscrutability ................................. 6
A. The What-How-Why Model ......................................................................... 7
B. The Effects of Inscrutability ......................................................................... 12
II. Existing Regulation of Inscrutable Systems ............................................ 16
A. Credit Scoring Regulations ......................................................................... 17
1. Introduction to FCRA, ECOA, and Regulation B ............................. 17
2. The Use of Explanation in Credit Scoring Regulation ....................... 21
B. General Data Protection Regulation ........................................................... 27
III. Interpretability in Machine Learning Systems ...................................... 32
A. Favoring Interpretable Methods .................................................................. 33
B. Global Explanations..................................................................................... 34
C. Specific Decisions ........................................................................................ 36
D. Task-Specific Techniques............................................................................ 38
IV. How Interpretability Can Assist Regulation ......................................... 38
A. Meeting Legal Requirements ...................................................................... 38
B. Vindicating Normative Principles ................................................................ 38
Conclusion ................................................................................................... 39
INTRODUCTION
In Dan Solove’s 2001 Article, Privacy and Power, he argued that Franz
DRAFT – Please ask permission before citing or quoting
3
Kafka’s The Trial serves as a better metaphor for society’s anxieties about
today’s information culture than does George Orwell’s 1984.1 Though
surveillance was and remains important, he wrote, Kafka offers “a more
thoughtless process of bureaucratic indifference, arbitrary errors, and
dehumanization, a world where people feel powerless and vulnerable,
without any meaningful form of participation in the collection and use of
their information.”2 As much prescient prediction as analysis, Solove’s
replacement metaphor perfectly describes the state of anxiety over artificial
intelligence and machine learning today.
Complex, inscrutable algorithms regularly make consequential
decisions about all our lives, with only minimal input from the people they
affect and little to no explanation as to how they work. This worries people,
and rightly so. The results of these algorithms could be unnerving,3 unfair,4
unsafe,5 unpredictable,6 and unaccountable.7 How can inscrutable
algorithms be held accountable for bad results? As one commentator has
observed, “we cannot effectively regulate what we do not understand.”8
This notion has led a number of scholars and policymakers to ask for
Daniel J. Solove, Privacy and Power: Computer Databases and Metaphors for Information
Privacy, 53 STAN. L. REV. 1393 (2001).
2 Id. at 1398.
3 See, e.g., Sara M. Watson, Data Doppelgängers and the Uncanny Valley of Personalization, THE
ATLANTIC
(June
16,
2014),
https://www.theatlantic.com/technology/archive/2014/06/data-doppelgangers-and-theuncanny-valley-of-personalization/372780/; Omer Tene & Jules Polonetsky, A Theory of
Creepy: Technology: Privacy and Shifting Social Norms 16 YALE J.L. & TECH. 59 (2013).
4 See, e.g., Solon Barocas & Andrew D. Selbst, Big Data’s Disparate Impact, 104 CALIF L.
REV. 671, 677-692 (2016); Pauline T. Kim, Data-Driven Discrimination at Work, __ WM. &
MARY
L.
REV.
__
,
(forthcoming
2017),
available
at
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2801251; Andrew D. Selbst,
Disparate Impact in Big Data Policing __ (forthcoming 2017), available at
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2819182
5 See, e.g., David Lazer, et al., The Parable of Google Flu: Traps in Big Data Analysis, 343 SCI.
1203 (2014).
6 See, e.g., Jamie Condliffe, Algorithms Probably Caused a Flash Crash of the British Pound,
MIT
TECHNOLOGY
REV.
(Oct.
7,
2016),
https://www.technologyreview.com/s/602586/algorithms-probably-caused-a-flash-crashof-the-british-pound/; Curtis E.A. Karnow, The Application of Traditional Tort Theory to
Embodied Machine Intelligence, in ROBOT LAW 51, 57-58 (Ryan Calo, A. Michael Froomkin &
Ian Kerr, eds. 2016) (discussing unpredictability in robotics).
7 Joshua A. Kroll, et al., Accountable Algorithms, 165 U. PA. L. REV. __ (forthcoming
2017), available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2765268; Danielle
Keats Citron & Frank Pasquale, The Scored Society: Due Process for Automated Predictions, 89
WASH. L. REV. 1, 18-27 (2014).
8 Kiel Brennan-Marquez, Plausible Cause 70 VAND. L. REV __, AT *8 (forthcoming
2017), available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2827733
1
4
Regulating Inscrutable Systems – DRAFT 3/20/2017
inscrutable algorithms to explain themselves.9 In a way, this makes sense.
Explanation is at the core of law. Judges write opinions to explain
themselves. Litigators explain their theories of the case. Administrative
agencies write long explanations to accompany new rules and regulations.
Explanations are the difference between authority vested in a decision and
authority vested in a person or system.10 But the actual role of explanation
in law is complicated, and without more specificity, the call can do more
harm than good.
Machines and humans have different strengths. People have
“common sense” and evolutionary heuristics while machines lack human
biases and can handle much more complexity.11 It is precisely the
improvement in accuracy, often derived from complexity, that leads
decision-makers to use machine learning systems in the first place.12 But
accuracy often comes at a price. There is a commonly accepted tradeoff
between the degree of interpretability and accuracy in machine learning
systems.13 As a result, there will be cases in which asking for an explanation
inherently makes predictions worse. And given what we understand about
the flawed nature of human reasoning,14 we should be pretty clear on the
normative rationales for any requirement that we reduce machine accuracy.
At the same time, there is a vast and growing area of research on
interpretability in machine learning, and the tradeoff is not necessarily a
hard and fast rule. Therefore machine explanation might be useful to
regulation.
To be useful, machine explanation must correspond to the types of
9 General Data Protection Regulation (GDPR) (Regulation (EU) 2016/679) art 13(f)(2),
14(g)(2), 15 (1)(h) (requiring access to “meaningful information about the logic” of
automated decisions); Brennan-Marquez, supra note 8, at 18-24 (drawing contrast between
explanation and prediction); Kim, supra note 4, at (arguing for an employment
discrimination standard of whether an adverse action was “because of” protected class
membership, at 48-49 (arguing that employers should bear the burden of proving nondiscrimination based on the “meaning” of their models);
10 Frederick Schauer, Giving Reasons, 47 STAN. L. REV. 633, 636-37 (1995).
11 Andrew D. Selbst, A Mild Defense of Our New Machine Overlords, 70 VAND. L. REV. EN
BANC __, at *14 (forthcoming 2017) (responding to Brennan-Marquez, supra note 8).
12 [cite]
13 Leo Breiman, Statistical Modeling: The Two Cultures (with comments and a rejoinder by the
author) 16 STAT. SCI. 16, 199, 206-13 (2001).
14 DANIEL KAHNEMAN, THINKING FAST AND SLOW 199-200 (2011) (discussing the
“narrative fallacy”), 224-25 (discussing the inconsistency of human judgment); Jerry Kang
& Kristin Lane, Seeing Through Colorblindness: Implicit Bias and the Law, 58 UCLA L. REV. 465,
473 (2010) (discussing implicit bias).
DRAFT – Please ask permission before citing or quoting
5
explanation that are useful for regulation. There can be infinitely many
layered explanations for the same event. For example: a) The glass shattered
because it hit the ground; b) It hit the ground because it was dropped; c) It
was dropped because the holder was startled; d) It hit the ground because of
gravity; e) It hit the ground because the ground is solid; f) It also shattered
because glass is brittle; g) Glass is brittle because of its chemical composition,
and so on. If someone asks for an “explanation” of why a glass shattered, it
is necessary to infer the context of the question (e.g., someone is upset about
their broken barware versus being in chemistry class) in order to give the
correct answer. For the same reason, asking for machine explanation is
incomplete without considering the specific role of explanation in
regulation. If we understand that role and the emerging technology, then we
might be able to design regulation to hold inscrutable systems accountable.
This Article brings together the legal calls for explanation with the research
on interpretability to examine how they align.
The Article proceeds in four parts. Part I proposes a new model to
understand the role of explanation in regulating decision-making systems.
Called the What-How-Why model, it categorizes potential explanations for
a decision into three levels: what happened, how it happened, and why the
decision was made the way it was. In order to evaluate a decision-making
system under a particular legal regime, it is necessary to access reasons from
all three layers, but only those parts of the layers relevant to the particular
normative concern. After explaining the What-How-Why model, the Part
explains the role of inscrutability as breaking the model, and the role of
regulation-by-explanation as seeking to repair it.
Part II examines two existing legal frameworks of regulation by
explanation. The first is credit scoring. Machine learning systems are not
the first inscrutable systems. Statistical credit-scoring is similarly difficult to
understand, and was regulated in the 1970s to require adverse action notices
with “statement[s] of reasons” for the adverse action. The second is the
portions of the General Data Protection Regulation (GDPR)—the recent
data protection legislation in the European Union—that address automated
decisionmaking. Unlike credit scoring law, the GDPR seeks “meaningful
information about the logic” of automated decisions rather than what
happened in an individual case. The Part argues that depending on
interpretation, “meaningful information” could correspond perfectly to the
failure of the What-How-Why model caused by inscrutability.
Part III canvasses the computer science literature on interpretability.
There are essentially three types of approaches: (1) purposefully selecting a
sufficiently small number of variables and machine learning method to
ensure that the resulting model is simple enough for a human to interpret;
(2) adopting special methods that manage the trade-off between model
6
Regulating Inscrutable Systems – DRAFT 3/20/2017
complexity and performance or attempt reproduce complex models in
simpler terms, and (3) forgoing any attempt a constraining the complexity of
the model, but producing post hoc explanations for specific decisions by
examining how changes to the input affect the model’s output.
Part IV brings it all together. Individual technical approaches
discussed in Part III and combinations of them may aid in compliance with
the legal regimes of explanation. But there will be some types of algorithms
that are unavoidably complex. If we require explanation, those algorithms
will be entirely off limits. The Article concludes with a call for exploring
types of regulation that do not require explanation in parallel to those that
do. Without such regulation, we risk losing the societal benefits of some of
the most advanced and valuable technology research to date. We should
explore whether it is possible to avoid such a fate.
I. WHAT-HOW-WHY AND THE PROBLEM OF INSCRUTABILITY
Scholarly and policy debates over how to regulate a world controlled
by algorithms have been mired in difficult questions about how to observe,
audit, access, or understand those algorithms. This is at least partially
because writing and speaking about this subject is tricky. Various terms have
been used in the midst of describing the difficulties faced in regulating
algorithms specifically: “black box,”15 “transparent/transparency,”16
“opaque/opacity,”17 “secret/secrecy,”18 “cause/causality/causation,”19
“foreseeability,”20
“explanation,”21
“intelligible/intelligibility,”22
See. e.g., FRANK PASQUALE, The BLACK BOX SOCIETY (2015).
See, e.g., Citron & Pasquale, supra note 7, passim; Tal Z. Zarsky, Transparent Predictions,
2013 U. ILL. L. REV. 1503 (2013).
17 Jenna Burrell, How the Machine “Thinks”: Understanding Opacity in Machine Learning
Algorithms, Big Data & Soc. 1, Jan.-Jun. 2016; Roger Allan Ford & W. Nicholson Price II,
Privacy and Accountability in Black-Box Medicine, 23 MICH. TELECOMM. & TECH. L. REV. 1, 1112 (2016); Zarsky, supra note 16, passim.
18 See, e.g., Brenda Reddix-Smalls, Credit Scoring and Trade Secrecy: An Algorithmic Quagmire
or How the Lack of Transparency in Complex Financial Models Scuttled the Finance Market, 12 U.C.
DAVIS BUS. L. J. 87 (2011); Frank Pasquale, Restoring Transparency to Automated Authority, 9 J.
TELECOMM. & HIGH TECH. L. 235, 237 (2011).
19 See, e.g., Kim, supra note 4, at *18.
20 See, e.g., Karnow, supra note 6, passim.
21 See, e.g., Bryce Goodman & Seth Flaxman, European Union Regulations On Algorithmic
Decision-Making And A “Right To Explanation” (2016), https://arxiv.org/abs/1606.08813.
22 See, e.g., Brennan-Marquez, supra note 8, at *5.
15
16
DRAFT – Please ask permission before citing or quoting
7
“information/being
informed,”23
“knowledge/knowing,”24
and
“prediction/predictive” (used in opposition to explanation)25 among others.
These terms all have overlapping meanings, whether described as synonyms
or antonyms. For example, opacity is often seen as a synonym for secrecy,26
antonym for transparency,27 or antonym for understanding.28 The overlap
in all these terms obscures the differences between the concepts, which need
disentangling before the topic of regulation can even be addressed.29
In this Part, we attempt to avoid the confusion by introducing a new
term30 and new abstraction to the debate. The new term is “inscrutability”
and the new abstraction we call the What-How-Why model of
understanding systems. The What-How-Why model is a way of thinking
about the different layers of understanding necessary to regulate a decision
system: what is does in the individual case, how it works generally, and why
it works that way. Within that framework, we are able explain why the true
difficulty in regulating machine learning algorithms comes from their
inscrutability, and how that interferes with our normal regulatory processes.
A. The What-How-Why Model
Explanation is at the heart of the legal system. Judges write opinions.
Litigators explain the facts. Administrative agencies give reasons for their
rules. The concept of due process, complicated at it is, is essentially all about
23 See, e.g., Sandra Wachter, et al., Why a Right to Explanation of Automated Decision-Making
Does
Not
Exist
in
the
General
Data
Protection
Regulation,
at
26,
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2903469.
24 Mike Ananny & Kate Crawford, Seeing Without Knowing: Limitations of the Transparency
Ideal and Its Application to Algorithmic Accountability, NEW MEDIA & SOCIETY 1 (2016); Mireille
Hildebrandt, Profiling: From Data to Knowledge, 30 DATENSCHUTZ UND DATENSICHERHEIT
548 (2006).
25 Brennan-Marquez, supra note 8, at *18.
26 Burrell, supra note 17, at 3-4.
27 Zarsky, supra note 16; Ford & Price, supra note 17.
28 Burrell, supra note 17, at 4-5.
29 Moreover, most of these terms may be permanently confusing because they are
tethered to the general meanings of the words. Some of this may be the ubiquity of sight
metaphors in our vocabulary for knowledge and understanding. Even aside from
“transparent” and “opaque,” some examples are “double blind study,” “in the dark,” “do
you see what I mean?,” and even the cartoon image of a light bulb over someone’s head.
The list goes on – virtually every way we describe knowledge or understanding turns out to
be a sight metaphor.
30 While if the problem is that no one can agree on the right terminology, it is not clear
that the solution is to provide a whole new term, cf. Randall Munroe, Standards, XKCD,
https://xkcd.com/927/, we think it is important anyway.
8
Regulating Inscrutable Systems – DRAFT 3/20/2017
explanations, from predictability and vagueness doctrine,31 to reviewability
of agency orders.32 Almost universally in regulating decisionmaking, and
especially in liability regimes, the core questions turn on “why” an outcome
occurred.
In typical, human-driven decisions, a person makes a decision, then
describes the one or more motives behind that decision, at which point
another person evaluates the legality of that decision by testing it against the
rationales permitted by statute.33 That chain of events depends on the
second person understanding the first. If the first refuses to or cannot
explain, then the ability of the second person to test for normative concerns
is diminished.
But asking for explanation alone is underspecified. An explanation is
an infinitely varied concept, and there can be any number of valid
explanations for a given phenomenon. In the famous story, a priest asked
the notorious bank robber Willie Sutton: “Why do you rob banks?”, to
which he replied “Because that’s where the money is.”34 The example is an
illustration of the context-specific nature of explanation.35 Sutton’s
explanation is in fact a valid explanation, but only to a question of the form
“Why do you rob banks, not laundromats?” Surely the priest was asking
something more like “Why do you rob banks, rather than obey the law?”
This example demonstrates that explanations are tied to specific contexts.
Law seeks not just an explanation, but the best explanation for its
purposes.36 When the legal system seeks explanation of facts, those
explanations are “inherently contrastive.”37 “Why X, rather than Y?” In
casual conversation, this is true as well, but people often leave Y unspecified
because it can be inferred from social context, or corrected after a
misunderstanding.
The specific explanations the law seeks are tied to the normative
[cite]
[cite]
33 See Andrew Verstein, The Jurisprudence of Mixed Motives, 126 YALE L.J. __, at App’x B
(forthcoming 2017) (listing the vast breadth of mixed motive cases).
34 Philip Kitcher, Explanatory Unification and the Causal Structure of the World, in SCIENTIFIC
EXPLANATION 410, 414 (Philip Kitcher, Wesley C. Salmon, eds. 1989)
35 W. Bradley Wendel, Explanation in Legal Scholarship: The Inferential Structure of Doctrinal
Legal Analysis, 96 CORNELL L. REV. 1035, 1059 (2011).
36 Ronald J. Allen & Michael Pardo, Juridical Proof and the Best Explanation, 27 L. &
PHIL. 223, 229-33 (2008); John Josephson, On the Proof Dynamics of Inference to the Best
Explanation, 22 CARDOZO L. REV.. 1621, 1622-24 (2001).
37 Wendel, supra note 35, at 1049.
31
32
DRAFT – Please ask permission before citing or quoting
9
goals of the law. If a person is on trial for murder, the relevant question
might be “Why did you, the defendant, kill the victim?,” by which the court
means “…instead of not killing someone.” There, affirmative defenses can
all be used as explanations that will satisfy the law. But if the defendant
answers “because his brother was not available,” and that he has a vendetta
against the family, it would be a valid explanation, but not a valid legal
explanation. The availability or unavailability of legal explanation then, is
tied directly to the normative goals enshrined in law.
Now consider an arbitrary system that makes consequential
decisions about people. The system could involve humans or machines. It
could make decisions about credit scoring, lending, hiring, renting, policing,
or any other item. At this high level, the important question for regulating it
is whether it can offer an explanation of what it is doing and why that
matches up with the normative concerns that the law in question seeks to
vindicate. If the system were a black box, what would the legal system have
to answer in order to regulate it? We argue that the questions one might ask
of a black box system can be separated into three groups that we call the
What-How-Why model of explanation: 1) What happened in an individual
case? 2) How do the decisions get made generally? 3) Why are the decisions
made that way?
To answer the What question, the system would have to provide
reasons for a specific outcome. For a given input, what led to the output?
The kinds of explanations given in response to this question will vary. The
simplest is to explain what features were examined. This helps a person
verify that the input data is correct at least. For example, a person could
verify her credit history after being denied credit. Another possibility is a list
of factors that factored into the decision, or the most important factors by
some measure. Such an explanation could be further elaborated by
explaining a minimum standard or how the requirements were not met, if
applicable. To continue with a credit example, possible explanations for one
of the factors could be “length of residence” or “not in residence long
enough” or “not in residence one year.” If no factor was clearly more
important than the others, a possible explanation is that the applicant “just
missed,” and an improvement of any of the variables taken into account
would help. Overall, the important thing to consider about the What is that
it is an accounting of a single result.
The How level sits above the What. It is the set of rules that govern
the decision process. Like at the What level, explanations at the How level
are descriptive and take several forms. They could be descriptions of the
logic of the decisionmaking process: filters, decision trees, regressions, or
other decision logic. Another could be an explanation of how all or some
the inputs map to their respective outputs, or how changes in inputs change
10
Regulating Inscrutable Systems – DRAFT 3/20/2017
the outputs; such as can be achieved through input/output testing. How
level descriptions need not be full descriptions or mappings; partial ones can
be useful as well. What is important about the How level is that it is
functional information about more than the single instance. The What and
How levels are related. Specifically, if someone knows the complete How
level description, he will be able to determine all the individual outcomes on
the What level. And the number of instances of individual explanations
required to get from the What level to the How level is not fixed; the
important point is whether there are enough instances to find a pattern or
say something useful about the operation of the system.
The top-most level of the abstraction is the Why. Explanations here
may take the form of the ultimate goals of the decisionmaking process,
decisions about what technology to implement, if any, intentional or
unintentional assumptions made by the modelers, how input data to the
decision are collected or chosen, and whether the designers of the black box
considered certain normative goals laid out in law or policy, and how they
addressed them.38 The Why level comprises everything outside the black box,
about the black box. The Why level is often normative, but need not be so.
For a machine vision system involved in facial recognition, the
programmers will want to know a great deal of Why level information—
goals and assumptions for example—just to test out and improve different
detection methods. But the Why level also functions at the level of policy.
The explanations provided here can test for society’s various normative
requirements of decision systems: fairness, privacy, predictability, safety, and
reviewability, among other concerns. As described above, though, to access
the Why level, there much be a question in mind. Only once the normative
concern is identified, then, can explanations be useful.
Importantly, because the Why level is entirely outside the box
looking in, nothing can be explained by reference to things going on inside
the box. In a sense, that is the division implied by the very concept of the
black box.39 There is a set of descriptive explanations internal to the
38 See, e.g., Solon Barocas & Andrew D. Selbst, Big Data’s Disparate Impact, 104 CALIF L.
REV. 671, 677-692 (2016);
39 There is an irony to this use of the term “black box.” Engineering disciplines
generally use the concept of a black box for almost exactly the opposite meaning. Whereas
when referring to an algorithm as a black box implies mystery about its operation, see, e.g.
PASQUALE, supra note 15, at 4, the other meaning is an abstraction of a system we so
thoroughly understand that we do not care what is happening inside.
As an example, consider the function “square(x).” In English, it means for every input
x, the output is x2. In code, we do not know how square(x) is defined. It could be “square(x)
DRAFT – Please ask permission before citing or quoting
11
operation of the box itself, and a set of (mostly) normative explanations
about the design of the box, that by necessity are external.
To illustrate with a simple example, imagine a black box, inside of
which is an algorithm that filters résumés by whether or not the applicant
went to Harvard Law School,40 or messes up her application. If the black
box was only open at the What level, it means that a person who applies for
a job would be able to tell that she was rejected because either “she went to
Columbia” or “did not go to Harvard” Alternatively, she might be told she
had an incomplete application or was rejected over a typo in the cover
letter. All these reasons, singularly or in combination, give the applicant
reasons for not getting the job, but they tell her very little of import about
the system. This is true even in the case where she is told both “not
Harvard” and “typo”, because she does not have reason to know that those
are the entire basis of the decision.
The How level explanations look different. In this case, the options
are offering one or both of the minimum rules: Harvard only, and no
sloppiness on the application. Someone learning one of those rules will both
know and understand how the system works, at least partially, and someone
knowing both knows and understands everything about the internal
workings of the system.
Why level explanations cannot be provided by the black box, but
= x * x” or “square(x) = exp(double(log(x)))” When computer scientists say that “square(x)”
is a black box, they mean that they are disciplining themselves not to ask what is inside the
function, but instead just to take the function as it is. See HAROLD ABELSON & GERALD
SUSSMAN WITH JULIE SUSSMAN, STRUCTURE AND INTERPRETATION OF COMPUTER
PROGRAMS. Another example is treating a light bulb as a resistor inside a circuit. As long as
the light bulb has a resistance R when running current through it, it will obey Ohm’s Law:
V=IR, where the voltage V is the input, and the current I is the output, and the constant R
is the function of interest. This is a black box, because it enables an engineer to entirely
ignore the complicated physics and chemistry of a light bulb, as the function R describes
everything we need to know. ANANT AGARWAL & JEFFREY H. LANG, FOUNDATIONS OF
ANALOG AND DIGITAL ELECTRONIC CIRCUITS 6-7 (2005).
In this meaning, we only open the black box if what is coming out is different than our
supposedly complete understanding, and we need to troubleshoot. The Science,
Technology, and Society literature generally understands “black box” this way too. See, e.g.,
WIEBE E. BIJKER, OF BICYCLES, BAKELITES, AND BULBS: TOWARD A THEORY OF
SOCIOTECHNICAL CHANGE 77 (1997). But the meaning of “black box” in law, policy, and
popular consciousness means a decisionmaking engine that is entirely mysterious. The
differences between the two concepts are the same secrecy and inscrutability that are
driving the scholarly and policy discussion about algorithms. See Part I.B, infra.
40 This is the “algorithm” deployed by the fictional law firm Pearson Specter Litt
(f/k/a Pearson Hardman) in the TV show Suits. Suits: Pilot, (USA television broadcast June
23, 2011) (Harvey Spector to Mike Ross: “Unfortunately, we only hire from Harvard.”)
References to this filter are made through the show’s run.
12
Regulating Inscrutable Systems – DRAFT 3/20/2017
they are about the black box. As discussed above, though “filter for Harvard
and careful application” is the complete description of the system, knowing
that fact offers no insight about why the system was designed that way or
whether that is an acceptable rule. For that, it might need to know “why
Harvard” or “why a single school” or “why put so much weight on typos to
the exclusion of merits?”
The Why layer connects the description of the black box from the
How layer to the outside world. Suppose one wanted to test for
discrimination, and had the full description. “Harvard only” will likely have
a disparate impact, so an important explanation to get from the Why layer
is whether the filter was chosen for a reason consistently with a business
necessity defense.41 Or alternatively, to test for intentional discrimination,
one would need to know whether someone chose the Harvard filter as a
proxy for a discriminatory purpose. Suppose that from the How layer, one
does not have the rule, but only has access to the inputs and outputs. Well,
because the example is a simple one, it is possible to detect the rule. But
even without that intermediate step, given enough input-output pairs, there
would be enough information to discover whether there is prima facie
disparate impact, and then from there, the problem moves on to business
necessity as above. The Why reasons allow regulators to evaluate the
legality and propriety of the box, and the How reasons give the tools to
measure. One important thing to note is that there are an infinite number of
possible explanations in this, and all, layers. The only way to determine the
appropriate explanation is to start with the particular question that requires
an answer and work from there.
The relative simplicity of this example likely obscures the value of
the What-How-Why model. We intuitively know how to ask all these
questions and perform evaluations about a Harvard-only filter. But the
purpose of the abstraction is to allow us to think specifically about the
contributions of each layer of explanation, so when they break down, we
can better understand why. It might not be surprising, then, that machine
learning algorithms break this model.
B. The Effects of Inscrutability
Machine learning is a process of learning by example. Rather than
programming computers by hand with explicit rules, describing, machine
learning relies on pattern recognition algorithms and a large set of examples
41
See 42 U.S.C. 2000e-2(k).
DRAFT – Please ask permission before citing or quoting
13
to independently uncover relationships. For example, when learning how to
distinguish between spam and legitimate email, machine learning relies on
pattern recognition algorithms and a large set of examples to independently
uncover relationships between properties of email and whether these emails
are spam. Machine learning is crucial to spam detection because the full set
of rules that would be necessary to effectively instruct a computer to
recognize spam would be impossible for a human to enumerate manually.
And spam detection is among the simpler common tasks to rely on machine
learning. Machine learning can learn many more—and far more
complex—rules than humans would be able to work on our their own. The
power of machine learning lies not only in its ability to relieve programmers
of the difficult task of producing explicit instructions for computers, but it its
capacity to learn subtle relationships in data that humans might overlook or
cannot recognize.
The property of machine learning algorithms that makes them
difficult to regulate is that they are inscrutable. An inscrutable system is one
in which a person could look at the code, or the entire corpus of the decision
rules, and still not understand what the system is doing. What actually is
inscrutable will be different for different people because of differences in
knowledge specialization,42 but for the purposes of this discussion, we can
focus on that which is inscrutable to the technically competent reasonable
person.
An inscrutable system defies human capabilities of reasoning and
pattern matching. This inscrutability is a natural fact of machine learning
systems derived from their inherent complexity.43 As Jenna Burrell has
noted:
Though a machine learning algorithm can be implemented simply in such a
way that its logic is almost fully comprehensible, in practice, such an instance
is unlikely to be particularly useful. Machine learning models that prove useful
(specifically, in terms of the ‘accuracy’ of classification) possess a degree of
Burrell, supra note 17, at 4 (discussing “opacity as technical illiteracy”). We mostly
agree with Burrell’s way of thinking through the problem, but of her three definitions of
opacity—secrecy, technical illiteracy, and some inherent property of algorithms that relates
to complexity—we believe that secrecy is different in kind. What we call “inscrutability”
maps on to the latter two.
43 Id. at 5. (“[T]here are certain challenges of scale and complexity that are distinctive
to machine learning algorithms. These challenges relate not simply to total number of lines
or pages of code, the number of team members on the engineering team, and the multitude
of interlinkages between modules or subroutines. These are challenges not just of reading
and comprehending code, but being able to understand the algorithm in action, operating
on data.”)
42
14
Regulating Inscrutable Systems – DRAFT 3/20/2017
unavoidable complexity.44
Inscrutable systems are difficult to regulate because they break the
What-How-Why model. In order for the model to work, the How layer
must provide some functional way of understanding the system, either
wholly or partially. This can be enough of a data mapping for humans to
fashion a rough rule, or it could be a complete description of the system that
is simple enough that humans can understand. If it is a partial mapping or
description, it is only useful for regulation if it happens to answer a question
that corresponds with what one seeks from the Why layer, but in those
cases, it will work. By definition, though, an inscrutable system confounds
description. While input-output testing is possible, and a complete list of the
rules could even be available, humans cannot process them meaningfully,
and thus cannot connect them to the normative concerns that motivate
regulation.
Compare the Harvard Law hypothetical to an inscrutable system. In
the Harvard Law hypothetical, the How layer provided either a functional
description of the logic or enough data for an observer to get the gist of what
was happening. As a result, it was trivial to know what questions to ask
regarding discrimination or other values. But if instead the results seemed
incomprehensible, almost random, then it would be impossible to figure out
which of the Why level explanations to even seek. How would one
determine the propriety of the specific assumptions that went into the
model, if one cannot understand the effects of those assumptions?
Note, though, that the issue is not that there is no information
available from an inscrutable system. For example, based on testing the data
alone, it is possible to assess prima facie disparate impact based on the fourfifths rule.45 But that is not enough to find liability, because we always to
want to know why the decisionmaker chose a specific rule. When faced with
an inscrutable system, one might think that the burden is on the
decisionmaker to usefully explain why, but that is often not the case.46 Thus,
the typical operation of the What-How-Why model is disrupted by
inscrutability. The How and the Why can no longer connect, and the
Id.
The “four-fifths rule” is, in most contexts, the showing required to demonstrate
prime facie disparate impact: “A selection rate for any race, sex, or ethnic group which is
less than four-fifths . . . of the rate for the group with the highest rate will generally be
regarded . . . as evidence of adverse impact.” Uniform Guidelines on Employment
Selection Procedures, 29 C.F.R. § 1607.4(D) (2015).
46 See Barocas & Selbst, supra note 38, at 701-12.
44
45
DRAFT – Please ask permission before citing or quoting
15
system cannot be tested against the normative concerns of the law.
In the past, scholars and policymakers have called for algorithmic
transparency to remedy the difficulties with regulating black-box algorithms,
arguing we need to see the algorithm so that we can judge its fairness or
safety.47 The What-How-Why model makes clear why transparency into
systems of decisionmaking is important, but incomplete. Transparency
requires information to be provided at the What level and How level. At the
What level, transparency might require access to the inputs to an algorithm
that bear on a specific decision. At the How level, transparency might make
require access to the algorithm for input-output testing48 or even access to
the code itself.49 But requirement of transparency assumes that information
will lead to accountability. As Mike Ananny and Kate Crawford observe:
“Transparency concerns are commonly driven by a certain chain of logic:
observation produces insights which create the knowledge required to
govern and hold systems accountable.”50 When the rules are inscrutable, the
insights are missing, and therefore so is accountability.
The problems with calling for algorithms to explain themselves is
similarly demonstrated by the model. When the system is simple, and the
explanation is therefore available but hidden, a call for machines to explain
themselves is little more than a call for transparency at the How level. But
with an inscrutable system instead, the call is either impossible by definition,
if a total explanation is sought, or is underspecified. At what level should the
system explain itself? At some level, asking the machine to “explain itself” is
impossible, because it cannot answer questions about its own design, and
without knowledge from the Why level, normative regulations cannot be
met. But which of the various explanations is relevant at the Why level, and
whether that can be satisfied by the available partial explanations of the
How is determined by the particulars of the normative concern.
Explanations in the abstract do no good.
There are now two realities to reconcile. One is the explanatory
What-How-Why model of regulating systems, and the other is inscrutability.
The next Part addresses two attempts within existing law to reconcile them.
See generally PASQUALE, supra note 15; see also Mikella Hurley & Julius Adebayo, Credit
Scoring in the Age of Big Data, 18 YALE J.L. & TECH. 148, 196-98 (2016); Reddix-Smalls, supra
note 18.
48 Citron & Pasquale, supra note 7, at 24.
49 Zarsky, supra note 16, at 1524, 1539.
50 Ananny & Crawford, supra note 24, at 2.
47
16
Regulating Inscrutable Systems – DRAFT 3/20/2017
II. EXISTING REGULATION OF INSCRUTABLE SYSTEMS
The difficulties that arise in regulating automated decisionmaking
comes from the systems’ complexity and resulting inscrutability, the fact of
automation. And though much of the current concern over inscrutable
systems stems from the growing importance of machine learning,
inscrutable systems predate the technique. As a result, legislation already
exists that seeks to regulate inscrutable systems by having the systems
explain themselves.
Credit scoring is an example of an inscrutable system that predates
machine learning. Statistical credit scoring systems take information about
consumers as inputs, give the inputs certain point values, to obtain a total
score, and then make decisions based on that score. Credit scores and
decisions made based on credit scores are regulated primarily by two
statutes: the Fair Credit Reporting Act (FCRA)51 and the Equal Credit
Opportunity Act (ECOA).52 While they have historically had different
approaches, both statutes currently require adverse action notices that spell
out the factors that lead to a credit denial or other adverse action. An
adverse action notice prompts a consumer to check her credit report for
errors,53 and a “[s]tatement of specific reasons”54 aims to prevent intentional
discrimination and serves consumer education about how the processes
work so they can better plan future behavior.
Regarding automated decisions, the European Union tried a
different approach. The General Data Protection Regulation (GDPR)
requires data subjects to have access to “meaningful information about the
logic involved” in any automated decisionmaking that significantly affects
them.55 Yet, as the law has not yet taken effect, the import and proper
interpretation of the requirement remain unclear. The Part argues that if
interpreted correctly, the GDPR could mostly get it right. It goes farther
than FCRA and ECOA do as far as using explanation itself as a tool for
accountability, but the focus on “meaningful information” could place the
15 U.S.C. § 1681, et seq.
Id. § 1691, et seq.
53 [cite]
54 12 C.F.R. § 202.9 (b)(2).
55 General Data Protection Regulation (GDPR) (Regulation (EU) 2016/679) art. 15
(1)(h) (GDPR); see also id. art. 13(e)(2), 14(f)(2). The Data Protection Directive, passed in
1995 also requires access to information about automated decisions, but the word
“meaningful” was added to the GDPR. See [Data Protection Directive art. 12].
51
52
DRAFT – Please ask permission before citing or quoting
17
onus on the exact kind of functional description missing from the How layer
of inscrutable systems. But the GDPR also shares common features with the
prior approaches that ay limit its effectiveness—specifically access only to
the How layer, and not necessarily the Why.
A. Credit Scoring Regulations
1. Introduction to FCRA, ECOA, and Regulation B
Before the 1950s, credit was not a large part of daily life in the
United States.56 During the 1950s and 1960s, as credit became more
popular, small credit bureaus had begun to arise, amassing large quantities
of information about prospective credit applicants.57 These bureaus would
both “track peoples’ names, addresses, and loan information” and “scour
newspapers for notices of arrests, promotions, and marriages.”58 Once
credit became more common, credit decisions were guided largely by the
“three C’s of credit”: capacity, character, and collateral.59
By the late 1960s, the modern credit industry had begun to emerge,
relying on amassed information and statistical models to predict
creditworthiness.60 While credit scoring was seen as a fairer, more objective
way to make credit determinations, consumers were nonetheless worried.
Credit reports often contained incorrect or outdated information that credit
reporting agencies (CRAs; e.g., Experian, Transunion, and Equifax) had no
incentive to correct.61 The industry was “secretive and enigmatic,”62 and
consumers had no idea who had access to their information or to what uses
it was being put.63
Thus, in 1970, Congress passed FCRA64 to begin to rein in the
unregulated credit industry. The official purpose of FCRA is “to require
[cite]
ROBINSON + YU, KNOWING THE SCORE: NEW DATA, UNDERWRITING, AND
MARKETING
IN
THE
CONSUMER
CREDIT
MARKETPLACE
26
(2014),
https://www.teamupturn.com/static/files/Knowing_the_Score_Oct_2014_v1_1.pdf.
58 Id.
59 Winnie F. Taylor, Meeting the Equal Credit Opportunity Act’s Specificity Requirement:
Judgmental and Statistical Scoring Systems, 29 BUFF. L. REV. 73, 74 (1980).
60 ROBINSON + YU, supra note 57, at 26.
61 NATIONAL CONSUMER LAW CENTER, FAIR CREDIT REPORTING § 1.4.3 (8th ed.
2013) [hereinafter NCLC, FAIR CREDIT REPORTING].
62 Lea Shepard, Toward A Stronger Financial History Antidiscrimination Norm, 53 B.C. L.
REV. 1695, 1745 (2012).
63 NCLC, FAIR CREDIT REPORTING, supra note 61, §§ 1.4.2-1.4.4.
64 Pub. L. No. 91-508, tit. VI, 84 Stat. 1127 (1970) (codified as amended at 15 U.S.C.
§§ 1681-1681x).
56
57
18
Regulating Inscrutable Systems – DRAFT 3/20/2017
that consumer reporting agencies adopt reasonable procedures for meeting
the needs of commerce for consumer credit, personnel, insurance, and other
information in a manner which is fair and equitable to the consumer, with
regard to the confidentiality, accuracy, relevancy, and proper utilization of
such information.”65 To accomplish this, FCRA requires that CRAs enlist
“reasonable procedures to assure maximum possible accuracy” of the
information it holds,66 allow consumers to access to their credit reports67
and investigate and resolve disputes with either a deletion, correction or
annotation.68 FCRA also limits to whom and for what credit reports can be
disclosed.69 Finally, FCRA requires that any person who takes an adverse
action relying on information obtained from a CRA must issue the
consumer an adverse action notice listing information about, among other
things, the decision itself and her rights under the chapter.70
By 1974, credit was becoming more systematized and less characterdriven, but women were still being denied credit on the basis of prejudices
related to a woman’s perceived role in the home and society. For example,
creditors would often deny credit to a married woman in her own name.71
Thus, pushed by women’s groups, Congress passed ECOA in 1974.72
Coverage was expanded soon after, with the Equal Credit Opportunity Act
Amendments of 1976,73 when it became immediately apparent that groups
other than women also faced credit discrimination. ECOA now prohibits
discrimination in credit decisions on the basis of race, color, religion,
national origin, sex, marital status, age (for adults), receipt of public
assistance income, or exercise in good faith of the rights guaranteed under
the Consumer Credit Protection Act.74
As both a consumer protection and anti-discrimination statue,
ECOA has two purposes.75 It was designed primarily to root out
15 U.S.C. § 1681(b).
Id. § 1681e(b)(2).
67 Id. § 1681g.
68 Id. § 1681i.
69 Id. § 1681b.
70 Id. § 1681m.
71 Id. at 74 n.7 (citing NATIONAL COMM’N ON CONSUMER FINANCE, CONSUMER
CREDIT IN THE UNITED STATES 152-53 (1972)).
72 Pub. L. No. 93-495, §§ 501-503 88 Stat. 1521 (1974) (codified at 15 U.S.C. § 1691(e))
73 15 U.S.C. §§ 1691-1691f
74 15 U.S.C. § 1691.
75 John H. Matheson, The Equal Credit Opportunity Act: A Functional Failure, 21 HARV. J.
LEG. 371, 372 (1984); Taylor, supra note 59, at 83.
65
66
DRAFT – Please ask permission before citing or quoting
19
discriminatory credit decisions,76 which should only be based on predictions
of an applicant’s ability and willingness to repay a loan.77 Thus, ECOA is
more like other anti-discrimination legislation than other credit laws than
FCRA or the Truth-in-Lending Act.78 The second purpose was to provide a
“valuable educational benefit.”79 “For the first time, federal legislation
afforded rejected credit applicants an automatic right to discover why
adverse action was taken.”80 That is, for the first time, rejected credit
applicants were given a right to explanation.81
ECOA’s specific innovation was the requirement that a creditor that
takes an adverse action against an applicant give a statement of “specific
reasons” for the adverse action.82 As stated in the Senate Report
accompanying the 1976 amendments,
[t]he requirement that creditors give reasons for adverse action is . . . a strong
and necessary adjunct to the antidiscrimination purpose of the legislation, for
only if creditors know they must explain their decisions will they effectively be
discouraged from discriminatory practices.83
The Senate Report further explains:
Yet this requirement fulfills a broader need: rejected credit applicants will now
be able to learn where and how their credit status is deficient and this
information should have a pervasive and valuable educational benefit. Instead
of being told only that they do not meet a particular creditor’s standards,
consumers particularly should benefit from knowing, for example, that the
reason for the denial is their short residence in the area, or their recent change
of employment, or their already over-extended financial situation.84
Taylor, supra note 59, at 83.
Id. at 76.
78 Susan Smith Blakely, Credit Opportunity for Women: The Ecoa and Its Effects, 1981 WIS. L.
REV. 655, 662 (1981).
79 Id.
80 Taylor, supra note 59, at 82.
81 FCRA did not include anything like a statement of reasons until 2010, when the
Dodd-Frank Act amended it to include up to four key factors that make up the credit score
with serves as the basis for an adverse action. See Pub. Law 111-203, 124 Stat. 1376 (2010)
(Sec. 1100F “Use of Consumer Reports”)(adding a requirement to disclose “the
information set forth in subparagraphs (B) through (E) of section 609(f)(1)”); 15 U.S.C. §
1681g(f)(1)(C). (including among the disclosures “all of the key factors that adversely
affected the credit score of the consumer in the model used, the total number of which shall
not exceed 4”).
82 15 U.S.C. § 1691(d)(3).
83 S. REP. 94-589, 4, 1976 U.S.C.C.A.N. 403, 406.
84 Id.
76
77
20
Regulating Inscrutable Systems – DRAFT 3/20/2017
Thus, the statement of reasons links the explanation to a consumer’s future
action, so that she will know how to improve her credit.
This legislative requirement was subsequently implemented by the
Federal Reserve Board in “Regulation B,”85 which states that the
“statement of reasons . . . must be specific and indicate the principal
reason(s) for the adverse action.” The regulation also notes that
“[s]tatements that the adverse action was based on the creditor’s internal
standards or policies or that the applicant . . . failed to achieve a qualifying
score on the creditors credit scoring system are insufficient.”86 The Federal
Reserve Board offered further guidance in an official interpretive document
that describes how creditors should aim to comply.87 This document hints at
some of the difficulties with requiring explanation, and is discussed further
below. Thus, after ECOA, when a consumer is denied credit, she gets some
form of “explanation” that is designed both to prove the reasons were not
discriminatory and to educate her as to the reasons themselves.88 Following
the ECOA model, in 2010, the Dodd-Frank Act updated FCRA to require
adverse action notices to include “all of the key factors that adversely
affected the credit score of the consumer in the model used,” for an adverse
determination involving a credit score.89
An appendix to Regulation B offers a sample notification forms
designed to satisfy both the rule’s and FCRA’s notification requirements.
The statement of reasons section of Sample From 1 offers twenty-four
different options.90 Though it is not necessary to use the form, most
12 C.F.R. § 1002.1, et seq. (“Regulation B”).
Id. § 1002.9 (b)(2).
87 Id. Pt. 1002, Supp. I Para. 9(b)(2) (“Official Interpretations”)
88 The Senate Report gives one further reason, though it treats this third reason as an
extension of the second: “In those cases where the creditor may have acted on
misinformation or inadequate information, the statement of reasons gives the applicant a
chance to rectify the mistake.” S. REP. 94-589, 4, 1976 U.S.C.C.A.N. 403, 406. This
ability to correct the input data is a central motivation behind FCRA, passed several years
earlier. See 15 U.S.C. § 1681. A statement of reasons is not actually required to accomplish
this goal. FCRA, unlike ECOA, does not require any statement of reasons for adverse
actions. It only requires notice that the decision was based in part on the credit report, id. §
1681m, and assorted other information designed to help the consumer correct errors
according to procedures in another part of the statute. See id. § 1681i. FCRA is different
from ECOA in that it never requires credit reporting agencies to reveal how they use the
information they have. Citron & Pasquale, supra note 7, at 17.
89 15 U.S.C. § 1681m(a)(2)(B) (requiring disclosure “of the information set forth in
subparagraphs (B) through (E) of section 1681g(f)(1) of this title); id. § 1681g(f)(1)(C).
90 12 C.F.R. Pt. 1002, App. C (“Sample Form”). The options are:
__Credit application incomplete
85
86
DRAFT – Please ask permission before citing or quoting
21
creditors tend to report reasons contained on that form believing it to be a
safe harbor.91
2. The Use of Explanation in Credit Scoring Regulation
The original version of FCRA did not seek explanations for
inscrutable credit scores. This is likely because FCRA was not concerned
with how decisions were being made, but rather with the relatively new
phenomenon of amassing large quantities of information in the first place.
FCRA was, after all, “the first information privacy legislation in the United
States.”92 Moreover, inscrutable credit scoring systems were not as
dominant in 1970, with the majority of credit decisions still being made by
human credit managers.93 Thus, the original goals of FCRA were more
modest: ensuring that the information collected would only be used for
permitted purposes, and that the information was accurate and up to date.94
__Insufficient number of credit references provided
__Unacceptable type of credit references provided
__Unable to verify credit references
__Temporary or irregular employment
__Unable to verify employment
__Length of employment
__Income insufficient for amount of credit requested
__Excessive obligations in relation to income
__Unable to verify income
__Length of residence
__Temporary residence
__Unable to verify residence
__No credit file
__Limited credit experience
__Poor credit performance with us
__Delinquent past or present credit obligations with others
__Collection action or judgment
__Garnishment or attachment
__Foreclosure or repossession
__Bankruptcy
__Number of recent inquiries on credit bureau report
__Value or type of collateral not sufficient
__Other, specify: ___
91 [From conversations with CFPB; need source]. This is particularly ironic because it
can sometimes lead to violations. For example, a creditor might use education information
as a tool to infer income. The creditor will then report income, rather than education,
because it is listed.
92 PRISCILLA M. REGAN, LEGISLATING PRIVACY: TECHNOLOGY, SOCIAL VALUES,
AND PUBLIC POLICY 101 (1995).
93 [cite]
94 NCLC, FAIR CREDIT REPORTING, supra note 61, § 1.4.3.
22
Regulating Inscrutable Systems – DRAFT 3/20/2017
But four years later, ECOA started to aim at the question of how
credit decisions were actually made. By 1974, forcing hidden intentions into
the open was a common technique for addressing discrimination. Just one
year before ECOA’s passage, McDonnell Douglas Corp. v. Green laid out the
canonical Title VII burden-shifting framework, which requires a defendant
to rebut a prima facie case of employment discrimination with a nondiscriminatory reason, and allows a plaintiff a chance to prove that the
proffered reason is pretextual.95 Just two years before that, the Court in
Griggs v. Duke Power Co.96 invented disparate impact doctrine, the purpose of
which was arguably also to smoke out intentional discrimination where
intent was hidden.97 Thus, ECOA sought the same goal—to force
decisionmaking into the open in order to prevent discrimination.
This approach can work for the most flagrant forms of intentional
discrimination but less well in other cases. Forcing stated reasons into the
open captures the most egregious form of intentional discrimination. Thus,
no longer can creditors intentionally discriminate based on protected class
directly, but they can use known proxies to get at the same result, if
desired.98 And although Regulation B bars collection of protected class
information,99 for credit systems that rely on face-to-face contact, that
information is readily available.100 Even without the face-to-face contact, it
can often be inferred from other data, such as a person’s name.101 If a
creditor wants to discriminate, stating a made up reason is not that difficult
for “judgmental” systems that rely on human credit managers rather than
scoring.102 But that also makes judgmental credit systems easily subject to
95 411 U.S. 792, 805 (1973). The Supreme Court later found that a jury may presume
that if all the employer had was pretext, that itself is evidence of discrimination. St. Mary’s
Honor Ctr. v. Hicks, 509 U.S. 502, 511 (1993) (“The factfinder’s disbelief of the reasons
put forward by the defendant (particularly if disbelief is accompanied by a suspicion of
mendacity) may, together with the elements of the prima facie case, suffice to show
intentional discrimination.”).
96 Griggs v. Duke Power Co., 401 U.S. 424, 431 (1971).
97 Richard A. Primus, Equal Protection and Disparate Impact: Round Three, 117 HARV. L.
REV. 494, 520-21 (2003) (discussing the “evidentiary dragnet” theory of disparate impact).
98 Matheson, supra note 75, at 388.
99 12 C.F.R. § 1002.5
100 Matheson, supra note 75, at 387.
101 See Marianne Bertrand & Sendhil Mullainathan, Are Emily and Greg More Employable
than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination, 94 AMER. ECON.
REV. 991 (2004).
102 “Judgmental” credit systems are older, human-driven, and subjective. Regulation B
defines them as anything that is not a “statistical” system. 12 C.F.R. § 1002.2.
DRAFT – Please ask permission before citing or quoting
23
lawsuits. Over the years, scoring systems have become more of the norm,
partially because they are seen as more objective.103
Creditors using quantitative methods that do not expressly consider
protected class membership are likely not engaged in intentional
discrimination,104 thus satisfying the most basic purpose of ECOA. But the
scoring systems might very well evince a disparate impact. Disparate impact
doctrine, as developed under Title VII, states that if a facially neutral
decision has a disproportionate adverse effect on a protected class, liability
can be found for that reason, unless the decisionmaker can provide a
legitimate business reason for the scoring system and no equally effective,
but less discriminatory alternative exists.105 While ECOA does not expressly
provide for a disparate impact theory of discrimination, it is very likely
available.106 What is interesting, then, about ECOA’s affirmative statement
of reasons, is that while for intentional discrimination, a consumer only
needs to know that the decision was not made for an improper reason,
knowing specifically which reasons it was made for becomes important for a
disparate impact case. Thus, in a world of credit scoring, the demand for
explanation can serve three different purposes. 1) preventing intentional
discrimination, 2) permitting testing for disparate impact and subsequent
correction, and 3) enabling consumers to determine how to improve their
credit.107
So how does the statement of reasons work in practice? The advent
of credit scoring systems introduced an element of inscrutability that was not
present in the judgmental systems before it. Credit scoring can almost be
thought of as proto-machine learning,108 in that the systems exhibit the
See Taylor, supra note 59, at 119.
Though they could indeed still use proxies.
105 [cite] [[This ignores the word “refuse,” but is probably the more common reading.]
106 The Supreme Court has not ruled that it is available, but most circuit courts that
have considered it have permitted it. Hurley & Adebayo, supra note 47, at 193 (citing
Golden v. City of Columbus, 404 F.3d 950, 963 (6th Cir. 2005)). In addition, the Supreme
Court ruled in 2015 that disparate impact theory was cognizable in the Fair Housing Act,
which also does not expressly provide for it. Texas Dep’t of Housing & Cmty. Affairs v.
Inclusive Communities Project, Inc., 135 S. Ct. 2507, 2518 (2015).
107 Error-checking could be considered a fourth purpose. If the consumers knew which
factors mattered, they could check that the records were correct. But because a credit
history has only a relatively small number of variables, a consumer only needs notice that
the credit history was accessed for error-checking purposes, not notice about which factors
mattered, because she can check the whole report. See supra note 88. In the machine
learning context, error-checking/validation becomes another reason for explanation. See
Part III, infra.
108 This is just as inversion of the argument that data mining “scores” everything. See
generally Citron & Pasquale, supra note 7.
103
104
24
Regulating Inscrutable Systems – DRAFT 3/20/2017
same reliance on statistics and sufficient complexity that they are hard to
explain usefully. If an explanation given by the statement of reasons is too
complex, the average credit applicant will not understand it, and the
explanation’s purpose is defeated. The Federal Reserve Board recognized
this problem, observing in its official interpretation that although all the
principal reasons must be disclosed, “disclosure of more than four reasons is
not likely to be helpful to the applicant.”109 The difficulty is that there will
be situations where the complexity cannot be avoided in a faithful
representation of the scoring system.110
Winnie Taylor described the problem well.111 She imagined a
hypothetical credit scoring system with eight different variables, each
assigned different point values, including whether an applicant owns or
rents, whether he has a home phone, and what type of occupation he has,
among other things. In a system like that, someone who comes up one point
short will find himself with every factor listed as a “principal reason” for the
denial. In one sense this has to be right, because a positive change in any
factor at all would change the outcome. But in another sense, “can it be said
that credit is denied because the applicant is a 48 year old farm worker who
rents? Probably not,” even though the scoring system arrives at that
output.112
Complexity comes from more than just the number of variables,
however. Currently, creditors “may say ‘length of residence’ rather than
“too short a period of residence.”113 That does not do much for consumer
education. But the disclosure has to be structured this way because creditors
cannot easily disclose a minimum standard. Some variables are nonmonotonic or otherwise fail to conform to clear patterns. In a real system
Taylor described,
applicants who have lived at their present address for less than six months are
awarded 39 points, a level which they could not reach again until they had
maintained the same residence for seven and one-half years. Furthermore,
109 12 C.F.R. Pt. 1002, Supp. I Para. 9(b)(2) (“Official Interpretations”). FCRA codified
the same limitation. 15 U.S.C. 1681g(f)(1)(C).
110 The document also states that the “specific reasons . . . must relate to and accurately
describe the factors actually considered or scored by a creditor,” “[a] creditor need not
describe how or why a factor adversely affected an applicant, and “[i]f a creditor bases the
. . . adverse action on a credit scoring system, the reasons disclosed must relate only to those
factors actually scored in the system.” Id.
111 Taylor, supra note 59, at 105-107.
112 Id. at 107 (emphasis added).
113 12 C.F.R. Pt. 1002, Supp. I Para. 9(b)(2).
DRAFT – Please ask permission before citing or quoting
25
applicants who have been residents for between six months and 1 year 5
months (30 points) are considered more creditworthy than those who have
been residents for between 1 and 1/2 years and 3 years 5 months (27
points).114
If the creditor tried to explain simply, it would leave information out, but if
the creditor were to explain in complete detail how the system worked, it
would likely overwhelm a credit applicant. This is an equivalent problem to
just providing transparency into a machine learning algorithm;
transparency is not understanding.115 Thus, there are times where it is
impossible to truly comply using the type of checklist that Regulation B
models, based on complexity alone.
The checklist aims to disclose the reason for a specific decision
without explaining the system functionality, but the two are not always
easily separable, and a reason given for the former will often be unsatisfying
without the context provided by the latter. Suppose a credit scoring system
exists that weighs home ownership incredibly highly. Suppose also that an
applicant named Alice is denied credit, when she has above average point
values for every attribute, but does not own a home. If Alice is given a list of
reasons per ECOA, two possibilities exist. She could either be told that
home ownership was the principal reason, or be told that attributes that are
not home ownership are the principal reason, such as not being at her job
long enough, or needing to pay down some debt. In some sense, the latter
approach is just as correct. Tweaking all these levers could result in credit
approval, and they are probably easier to fix than buying a home, so they
serve the consumer education purpose. Both lists of reasons should therefore
satisfy ECOA. But the latter missing the point of the question. Here, the
primary factor was that the system was designed in such a way that home
ownership was incredibly heavily weighted. But weighting of variables and
other concerns of overall system functionality are not the kinds of
explanation that ECOA seeks, so the real explanation will not fit the
question asked.116
This is not to say that an explanation is always useless. As Taylor
points out, some “reasons, such as ‘no credit file,’ ‘unable to verify income,’
and ‘we do not grant credit to any applicant on the terms and conditions
you request’ are self explanatory, and appear to simultaneously provide the
threshold educational benefit Congress intended for consumers and the
Taylor, supra note 59, at 123.
See Ananny & Crawford, supra note 24, at 7 (“Transparency can intentionally
occlude.”)
116 [cite]
114
115
26
Regulating Inscrutable Systems – DRAFT 3/20/2017
brevity Congress intended to ease a creditor’s burdens.”117 There may be
ways to design scoring systems or explanations so that at least in certain
cases, the explanation will make sense to the applicant. Analogues to these
kinds of constrained systems and case-specific explanations are the subject of
Part III. But while some fixes are possible, it is doubtful that explanation will
ever work to solve the entire problem.
The discussion so far has addressed only descriptive concerns—
What or How level. But ECOA sets out some specific normative concerns as
well: anti-discrimination and predictability. ECOA envisions enforcement of
these normative concerns through litigation.118 With a good enough
understanding of the system, a person (in this case judge or jury) can test
whether it violates the normative concerns.
As discussed in Part I, relying on people to connect the normative
concerns is the only way to scrutinize a system normatively. No decision
system, whether machine- or human-driven, can self-examine; it cannot
know anything about the choices that went into its own design. It cannot
examine why certain features were chosen or emphasized, how the scoring
was determined, or if the results are truly bizarre. But reasons provided
under ECOA and FCRA are at the What level. The statement of reasons
alone cannot ensure the normative outcomes the statute seeks to vindicate.
But the complexity of this credit scoring system, even though it is low
when compared to machine learning, makes the How layer impenetrable.
Where the descriptive explanation is simple, it is easier for an external
observer to relate it to the normative concern. For example, if a single
variable is dominant in the outcome, rather than many interacting with
each other, it is easier to check whether that variable is also a proxy for a
protected class. Or if a variable is monotonic, rather than seemingly
arbitrary or random, it is easier to explain to an applicant how to improve
her credit score based on that variable in the future. But neither is true here,
and the increased complexity therefore creates a gap in understanding that
frustrates the ultimate purposes of the statute.
Returning to Alice’s credit application, imagine that the reason
home ownership is weighted so heavily was because in the 1970s, fewer
women owned homes, and a designer of the system wanted to deny credit to
women, who he thought should be getting their husbands’ approval for
loans. This gets to the core of the intentional discrimination ECOA sought
to curb. In this case, it would be correct to say that the reason for the
117
118
Taylor, supra note 59, at 97.
[cite]
DRAFT – Please ask permission before citing or quoting
27
ultimate result was discrimination, because it was the reason for the
weighting that ultimately cost Alice the loan. ECOA, by itself, can never get
this type of explanation from the system, because it only asks internal
questions. But the complexity further compounds the problem. If ECOA’s
statement of reasons does not even mention home ownership—recall that
both the version with and without probably satisfy ECOA—then there
would be no way for an outside observer to investigate whether home
ownership was included due to prejudice. The statement of reasons only
addresses one version of how the machinery got to its conclusion, not how
the machinery works generally, or why it works that way. Thus, the
statement of reasons could fail even for an intentionally discriminatory
design.119
B. General Data Protection Regulation
In 2016, the European Union passed the General Data Protection
Regulation (GDPR),120 which goes into effect in May 2018. The GDPR is
an EU-wide regulation that replaces the federalist system of data protection
governed by the 1995 Data Protection Directive (DPD).121 Both laws
regulate automated decisionmaking,122 but in the 22 years of the Directive’s
existence, little jurisprudence has developed around that particular aspect of
the law.123
The GDPR’s discussion of automated decisions is contained in
Article 22, Article 13(2)(f), Article 14(2)(g), and Article 15(1)(h). Article 22 is
the primary piece and states, in relevant part, as follows:
Article 22. Automated individual decision making, including profiling
1. The data subject shall have the right not to be subject to a decision based
solely on automated processing, including profiling, which produces legal
effects concerning him or her or similarly significantly affects him or her.
Note that ECOA does have a substantive nondiscrimination provision, separate
from the statement of reasons, which may allow it to get to these types of questions in
litigation, given proof. Thus, the law overall can capture intentional discrimination, but the
law’s approach of using explanation alone cannot.
120 Regulation (EU) 2016/679 on the protection of natural persons with regard to the
processing of personal data and on the free movement of such data, and repealing Directive
95/46/EC (General Data Protection Regulation) [2016] OJ L119/1.
121 1995 Directive (95/46/EC) (“Data Protection Directive”)
122 GDPR art. 22(1) (“The data subject shall have the right not to be subject to a
decision based solely on automated processing, including profiling, which produces legal
effects concerning him or her or similarly significantly affects him or her.”; Data Protection
Directive art.
123 Wachter, et al., supra note 23, at 19.
119
28
Regulating Inscrutable Systems – DRAFT 3/20/2017
2. Paragraph 1 shall not apply if [exceptions (a)-(c)].
3. In the cases referred to in points (a) and (c) of paragraph 2, the data
controller shall implement suitable measures to safeguard the data subjects
rights and freedoms and legitimate interests, at least the right to obtain
human intervention on the part of the controller, to express his or her
point of view and to contest the decision.
4. [omitted]
Articles 13-15 spell out a data subject’s right to be informed about the data
that data controllers have about them.124 Articles 13 and 14 describe the
obligations of data processors to affirmatively notify data subjects about the
uses of their information,125 and Article 15 delineates the affirmative access
rights that data subjects have to information about their own data is used.126
Under Articles 13 and 14, the data controller shall provide, and under
Article 15 the data subject has the right to access, “the existence of
automated decision-making, including profiling, referred to in Article 22(1)
and (4) and, at least in those cases, meaningful information about the logic
involved, as well as the significance and the envisaged consequences of such
processing for the data subject.”127
After the passage of the GDPR, scholars have begun to debate
whether these requirements amount to a “right to explanation.”128 Bryce
Goodman and Seth Flaxman have argued that by combining Article 22(3)’s
requirement that any automated decisionmaking include “safeguards for the
rights and freedoms of the data subject” with the Article 13-15 requirement
that data subjects have “meaningful information about the logic involved”
in automated decisions, one can infer a right to explanation.129 Sandra
Wachter, Brent Mittelstadt, and Luciano Floridi disagree. They note that
non-binding Recital 71, which explains the purpose of Article 22,130 repeats
the safeguard language of Article 22, but includes a “right . . . to obtain an
explanation of the decision reached,” which is missing in the operative text.
Wachter, et al., supra note 23, at 14.
See GPDR art.13-14.
126 See GDPR art. 15.
127 GPDR art.13(2)(f), art. 14(2)(g), art. 15(1)(h).
128 See Goodman & Flaxman, supra note 21. But see, Wachter, et al., supra note 123.
129 Goodman & Flaxman, supra note 21, at 6.
130 Recitals in EU law are explanations of the purpose of operative provisions and are
nonbinding. See Tadas Klimas & Jurate Vaiciukaite, The Law of Recitals in European Community
Legislation, 15 ILSA J. INT’L & COMP. L. 1, 32 (“Recitals have no positive operation of their
own.”)
124
125
DRAFT – Please ask permission before citing or quoting
29
The authors then look at prior drafts and surrounding commentary of the
GPDR to argue convincingly that the right to explanation was intentionally
removed from Article 22.131 Then they somewhat less convincingly argue
that the same right to explanation cannot be derived from Articles 13-15 or
anywhere else.132 If the right to explanation was removed from Article 22, it
could have been because it was already captured by “meaningful
information about the logic,” in Articles 13-15, for example.
Whether one calls it a right to explanation or not, requiring that data
subjects have meaningful information about the logic has to mean
something related to explanation. The phrase “meaningful information” is
the major difference between the DPD access right’s treatment of
automated decisions and the parallel language in the GDPR. Article 12 of
the DPD granted data subjects a right of access to “knowledge of the logic
involved” in automated decisions,133 whereas Article 15 of the GDPR grants
access to “meaningful information about the logic involved.”134 The word
“meaningful” is then central to this new right.
If one considers the possibility of information that is not
meaningful—that is, literally contains no meaning to the data subject—then
the addition of the word “meaningful” implies that data processors must go
beyond mere provision of the information to include something else to
provide meaning. That something else could be explanatory, as explanation
is a method of infusing otherwise unintelligible information with meaning.
But perhaps there was a different intent. Another possibility could be
simplification of the automated decision-making process itself such that the
available information would naturally be meaningful. But Wachter et al.’s
demonstration that Article 22 does not include the right among its
“safeguards” makes this interpretation unlikely. Article 22 is written as a
prohibition on automated decisions, except in certain circumstances.135 If
the drafters intended “meaningful information” to imply a limit on the
complexity of automated processing itself, why would it be part of the
Article describing a data subject’s right to access information rather than the
Article delineating the limitations on permissible automated decisions? As
Wachter, et al., supra note 23, at 9-11.
See Andrew D. Selbst, Calling Foul on the “Right to Explanation” Debate (draft on file with
authors). Wachter et al. argue that the GDPR creates a “right to be informed,” which they
put in opposition to a “right to explanation,” all without ever engaging with the word
“meaningful.” Wachter, et al., supra note 23, at 26-27 (misreading Article 15 to state that it
“requires only information about the ‘existence of’ automated decision-making.”).
133 DPD art. 12(a).
134 GPDR art. 15(1)(h).
135 It could also be a right of data subjects to object, see Wachter, et al., supra note 23, at
35-37, but this does not change the analysis.
131
132
30
Regulating Inscrutable Systems – DRAFT 3/20/2017
Wachter et al. point out, the drafters expressly removed such a right from the
safeguards in Article 22. Thinking about “meaningful information” as a
safeguard on the automated decision itself therefore does not comport with
the structure of the regulation. Articles 13-15 instead take the processing as
a given, and allow data subjects certain rights to be informed about it. In
this case that has to be some additional requirement to make the provided
information “meaningful.”
So whatever the right is called, what does it mean in practice?
According to the regulation, the “meaningful information” must be about
the “logic” of the decisions.136 The logic of the decisions must mean
something different than the outcomes of the decisions. In Wachter et al.’s
terminology, the difference is an explanation of the results in a specific
decision versus the overall system functionality, or what we have called the
What and the How. Whereas ECOA and FCRA seek the factors that lead
to a certain result, the “logic involved” in the “decision-making” seems to
refer directly to the How.
This is, in principle, a broader right than the one provided by
ECOA and FCRA, and likely more useful to data subjects. To demonstrate
this, return to Taylor’s hypothetical system where a person missed on her
credit application by one point, after the creditor totaled the point values
from eight factors. It might be impossible to point to four factors or fewer
that were “principal reasons,” but the explanation of what the eight factors
were, that they were all assigned point values, and that the hypothetical
applicant just missed by a point would be much more useful to the rejected
applicant. Or consider Alice in the heavily home-weighted credit system.
Telling her that home ownership was weighted heavily in this particular
system would be meaningful, even if other factors contributed more directly
to her rejection, in the sense that she could more easily improve those. The
logic of the system would allow her to do what is necessary to improve her
credit, but also, to potentially expose the home-ownership centered system
as discriminatory in litigation.
Within the What-How-Why framework, “meaningful information
about the logic” appears to precisely target inscrutability. As discussed
earlier, inscrutability destroys the ability of the How layer to offer a
functional description or pattern from which a rule can be inferred. Read in
what appears to be the most natural way, “meaningful information” seems
to require that for automated decisions, data processors need to find a way
to bring back the capability of the How layer to serve its function.
136
GPDR art.13(2)(f), art. 14(2)(g), art. 15(1)(h).
DRAFT – Please ask permission before citing or quoting
31
“Meaningful information” is a flexible standard, but that is because the
kinds of functional information required will be different for every problem.
This flexibility fits within the overall purposes of the GDPR, which
are much broader than FCRA and ECOA. The EU treats data protection
as a fundamental right,137 and Article 5 lists the following as principles the
GDPR seeks to vindicate with respect to personal data: lawfulness, fairness
and transparency, purpose limitation, data minimization, accuracy, storage
limitation, integrity and confidentiality, and accountability. Several of these
principles are a restatement of the familiar Fair Information Practice
Principles (FIPPs) that have shaped privacy policy for decades.138 But
considered as a whole, including “lawfulness” and “fairness,” they begin to
sound like the general purpose of due process in all its expansiveness. To
achieve such a result, the standard needs to target exactly the part of the
reasoning process that inscrutability interferes with. And it does just that.139
While requiring an explanation at the system functionality level will
be more useful, it still does not require that the data processors explain the
choices that led to the decision model. ECOA and FCRA ask for the What,
the GDPR asks for the How and neither one asks Why. Like ECOA and
FCRA, then, the GDPR relies on private rights of action140 and other
provisions141 to test the propriety of the algorithms from a normative
perspective. But if “meaningful information” does the work to repair the
What-How-Why model, then perhaps algorithms can be regulated.
The GDPR’s demand for meaningful information requires either
that systems be designed so that the algorithm is simple enough to
understand, or can provide enough functional information about the logic
of the system that it can be tested. Part III canvasses the computer science
GDPR art. 1.
Kate Crawford & Jason Schultz, Big Data and Due Process: Toward A Framework to
Redress Predictive Privacy Harms, 55 B.C. L. REV. 93, 106-7 (2014). While different lists of
FIPPs conflict, one prominent example is the OECD’s list of eight: Collection Limitation
Principle, Data Quality Principle, Purpose Specification Principle, Use Limitation
Principle, Security Safeguards Principle, Openness Principle, Individual Participation
Principle, and Accountability Principle. OECD, THE OECD PRIVACY FRAMEWORK 14-15
(2013), http://www.oecd.org/sti/ieconomy/oecd_privacy_framework.pdf
139 Here, we have only discussed the provisions in Articles 13-15 that require
meaningful information. But the GDPR has an enormous loophole that limits its
applicability: Article 22, and by extension Articles 13(f)(2), 14(g)(2), and 15(h)(1) only apply
to decisions “solely based on automated processing,” GPDR art. 22(1). The ultimate
meaning of the clause is unclear, as humans are always involved to a degree, but it could
turn out to eviscerate the right. See Wachter, et al., supra note 23, at 30-33. Nonetheless, that
limitation does not affect the substance of Articles 13-15, when they do apply.
140 See GDPR art. 79.
141 See GDPR art. 77, 83(5) (authorizing potentially enormous administrative fines).
137
138
32
Regulating Inscrutable Systems – DRAFT 3/20/2017
research on interpretability, and Part IV considers whether it might prove
helpful to regulation.
III. INTERPRETABILITY IN MACHINE LEARNING SYSTEMS
The demand for explanation in machine learning is not new. Early
research recognized and tried to grapple with the challenge of explaining
the decisions of machine learning models such that people using these
systems would feel comfortable acting upon them.142 Interpretability has
continued to receive considerable attention in the field, generating a
surprisingly vast body of scholarship, despite a lack of consensus regarding
the precise technical meaning of interpretability or the exact problem to
which improved interpretability actually responds.143
Much of this work has taken place in the context of an ongoing
debate about the value of interpretability, given the apparent trade-off
between performance and interpretability in many practical applications of
machine learning. Notably, Leo Breiman, a significant figure in the field,
argued in the early 2000s that there are entire classes of problems where the
accuracy of predictions is paramount and where the additional insights that
practitioners might draw from a more interpretable model will not
compensate for the loss of accuracy.144 The success of neural networks, for
example, seems to depend on a willingness to forgo interpretability, or an
acceptance that the improved accuracy of the resulting predictions provides
more information than might be gleaned by inspecting a simpler model. As
a result, much of the research on interpretability in machine learning has
focused on managing the trade-off between performance and
interpretability, with the goal of increasing interpretability while sacrificing
the minimum amount of performance.145
At the same time, however, the experience of applying machine
142 Bruce G. Buchanan & Edward H. Shortliffe. Rule-Based Expert System, in THE
MYCIN EXPERIMENT OF THE STANFORD HEURISTIC PROGRAMMING PROJECT __ (1984).
143
Zachary C Lipton, The Mythos of Model Interpretability, arXiv preprint
arXiv:1606.03490 (2016); Doshi-Velez, Finale, and Been Kim. "A Roadmap for a Rigorous
Science of Interpretability." arXiv preprint arXiv:1702.08608 (2017); David Bamman (2016),
"Interpretability in Human-Centered Data Science," CSCW Workshop on HumanCentered Data Science
144 Breiman, supra note 13, at __.
145 Michael Gleicher, A Framework For Considering Comprehensibility In Modeling. 4 BIG
DATA 4, 75 (2016).
DRAFT – Please ask permission before citing or quoting
33
learning to real-world problems has lead to widely held beliefs among
practitioners about the relative interpretability of the kind of models that
result from different learning methods. Henrik Brink and Joshua Bloom
capture much of this informal knowledge in a chart that plots learning
methods along two axes: interpretability and accuracy.146 In the top left
corner (highly interpretable; relatively low performance) are methods like
linear regression; in the bottom right corner (difficult to interpret; relatively
high performance) are methods like deep learning. Though this is a
common perception of the trade-offs involve with different methods,
researchers, including Brink and Bloom themselves, point out that such
contrasts do not rest on a rigorous definition of interpretability.147 And yet
such beliefs routinely guide practitioners’ decisions when applying machine
learning to different kinds of problems.148
Over the past decade, however, other methods have emerged that
attempt to side-step these difficult choices altogether. Rather than aiming to
build interpretable models, recent methods provide mechanisms to generate
explanations for specific decisions; they forgo any attempt at providing a
global description of the model in favor of furnishing explanations on a caseby-case basis.
The demand for explanations in machine learning can thus be met
with at least three different responses: (1) purposefully choosing a feature set,
learning method, or parameters that are likely to lead to an intelligible
model; (2) adopting specialized methods that attempt to minimize any loss
in performance when building interpretable models; and (3) applying post
hoc methods to account for individual decisions.
A. Favoring Interpretable Methods
Where complexity might cause a model to become uninterpretable,
practitioners have a number of different levers at their disposal to ensure
greater simplicity. First, they may choose to consider only a limited set of all
possible features. By limiting the analysis to a smaller set of variables, the
total number of relationships uncovered in the learning process might be
sufficiently limited to be intelligible to a human. A model with five features,
for example, is far more likely to be interpretable than a model with five
hundred. Such choices are sometimes forced on practitioners to avoid the
Henrik Brink & Joshua Bloom, Overcoming the Barriers to Production-Ready MachineLearning Workflows, STRATA (2014)
147 Alex A. Freitas, Comprehensible Classification Models: A Position Paper. 15 ACM
SIGKDD EXPLORATIONS NEWSLETTER 1 (2014).
148 [Cite]
146
34
Regulating Inscrutable Systems – DRAFT 3/20/2017
so-called “curse of dimensionality,” whereby exploring the total number of
potential relationships between variables is too large to be computationally
tractable.149 These natural limits on the complexity of the desired
computation may result in a less than ideal selection of variables and
discoveries with less predictive accuracy, but they may also improve the
interpretability of these discoveries.150 Second, practitioners might elect to
use a learning method that outputs a model that can be more easily parsed
than the output of other learning methods. For example, decision tree
algorithms learn nested rules that can be represented visually as a tree with
subdividing branches. To understand how the model would process any
particular case, practitioners need only walk through the relevant branches
of the tree; to understand the model overall, practitioners can explore all the
branches to develop a sense of how it would apply to all possible cases.
Finally, practitioners might set the parameters of the learning process to
ensure that the resulting model is not so complex that it defies human
comprehension. Even decision trees will become unwieldy for humans at
some point if they involve an exceedingly large number of branches and
terminal leaves. Practitioners routinely set an upper bound on the number
of terminal leaves to constrain the potential complexity of the model.
B. Global Explanations
A number of more specialized methods exist that allow practitioners
to achieve greater model interpretability while maintaining a level of
performance that would be impossible with less sophisticated learning
methods. These methods generally fall into two camps: regularization and
rule extraction. Much like setting an upper limit on the number of branches
in a decision tree, regularization methods allow model complexity to be
taken into account during the learning process by assigning a cost to excess
complexity. In other words, regularization allows practitioners to make
model simplicity an additional objective alongside model performance—and
the learning process can be set in such a way to find an optimal trade-off
between these sometimes competing objectives.151 It can help to automate
difficult choices about feature selection, reducing the overall complexity of
Breiman, supra note 13, at 208.
David Martens & Bart Baesens, Building Acceptable Classification Models, in 8 DATA
MINING AND INFORMATION SYSTEMS: QUO VADIS? 53, (Robert Stahlbock, Sven F Crone,
& Stefan Lessmann, eds. 2009).
151 Gleicher, supra note 145.
149
150
DRAFT – Please ask permission before citing or quoting
35
the model while still trying to maximize performance.
One such method is Lasso.152 Lasso was originally designed to
increase accuracy by avoiding overfitting, which occurs when a model
learns patterns that are peculiar to the training data and not representative
of the patterns the model will encounter when deployed in the real world.153
Lasso increases accuracy by forcing the learning process to ignore
relationships that are relatively weak, and therefore more likely to be
artifacts of the specific examples that happened to be in the training data,
rather than real relationships. Because regularization works by strategically
removing unnecessary features, in many real-world applications, the
technique can simultaneously improve interpretability (by reducing
complexity) and increase performance (by helping to avoid overfitting).
Where it applies, this demonstrates that improved interpretability might not
always come at the cost of performance. Where potential overfitting is not a
danger, however, regularization methods will indeed result in degradations
in performance, and these methods will attempt to keep those losses to a
minimum.
In contrast, rule extraction is a common technique that works by
allowing practitioners to learn a model using any method they like and then
replicating the discovered rules using simpler methods. Simpler models
cannot always reliably discover as many relationships. Take a decision tree,
for example. Once the computer makes a decision to take a particular
branch, the method does not permit walking back up the branch.
Therefore, if there is a relationship between items on two different branches,
it will not be discovered.154 More complex algorithms lack the same
limitation, but they do not result in models as interpretable as decision trees.
Rule extraction takes advantage of the fact that rules that can’t be
learned with simpler methods can still be represented effectively by simpler
models. Simpler models are often expressive enough to represent—with
high fidelity—the complex relationships captured by more sophisticated
learning methods like support vector machines or neural networks.155 The
Robert Tibshirani. Regression Shrinkage and Selection Via the Lasso, J. ROYAL STAT.
SOC’Y 267. Series B (Methodological) (1996).
153 Machine learning is only effective in practice when it successfully identifies robust
patterns in the training data while also ignoring patterns that are just artifacts of the
particular sample of cases assembled in the training data.
154 A decision tree is an example of a “greedy” algorithm. At each stage, it makes what
it perceives to be an optimal choice, but it will often not be a global optimum. STUART
RUSSELL & PETER NORVIG, ARTIFICIAL INTELLIGENCE: A MODERN APPROACH 92 (3d.
ed. 2014).
155 Bart Baesens, Rudy Setiono, Christophe Mues, & Jan Vanthienen. Using Neural
Network Rule Extraction and Decision Tables for Credit-Risk Evaluation, 49 MANAGEMENT
152
36
Regulating Inscrutable Systems – DRAFT 3/20/2017
problem with simpler learning methods is that they might miss certain
relationships. With the discoveries of complex learning methods behind
them, rule extraction methods allow simple models to “cheat” because the
answer is known ahead of time.
Where rule extraction works well, the resulting simpler model can be
as accurate or nearly as accurate as the more complex model it aims to
replicate. Because the goal of rule extraction is to generate models that are
interpretable to humans, they must sometimes sacrifice part of the
complexity that allowed other learning methods to achieve better
performance. But even when rule extraction involves a notable loss in
performance, the resulting models frequently perform far better than what
would have been learned with simple methods alone. This helps to explain
why such methods are widely used in practice.156
C. Specific Decisions
A new class of tools have emerged over the past decade that attack
the problem of interpretability from a different direction. Rather than
attempting to ensure that machine learning generates an intelligible model,
these new tools furnish more limited explanations that only account for the
relative importance of different features in making particular
determinations. At a high level, most of these methods adopt a similar
approach: they attempt to establish the importance of any feature to a
particular decision by iteratively varying the value of that feature while
holding the value of other features constant.157 In so doing, these methods
SCIENCE 3 (2003); David Martens, et al., Comprehensible Credit Scoring Models Using Rule
Extraction From Support Vector Machines, 183 EUROPEAN J. OPERATIONAL RESEARCH , 1466
(2007); David Martens, et al., Performance of Classification Models from a User Perspective. 51
DECISION SUPPORT SYSTEMS 782 (2011).
156 Johan Huysmans, Bart Baesens, & Jan Vanthienen. Using Rule Extraction To Improve
The Comprehensibility Of Predictive Models. (2006).
157 David Baehrens, et al., How to Explain Individual Classification Decisions.11 J. MACHINE
LEARNING RESEARCH 1803 (2010); Andreas Henelius, et al.. A Peek Into the Black Box:
Exploring Classifiers by Randomization, 28 DATA MINING AND KNOWLEDGE DISCOVERY 1503
(2014). Philip Adler, et al., Auditing Black-box Models for Indirect Influence, ___ ; Marco Tulio
Ribeiro, Sameer Singh, & Carlos Guestrin. "Why Should I Trust You?: Explaining the Predictions
of Any Classifier." In PROCEEDINGS OF THE 22ND ACM SIGKDD INTERNATIONAL
CONFERENCE ON KNOWLEDGE DISCOVERY AND DATA MINING, 1135. ACM, 2016;
Anupam Datta, Shayak Sen, & Yair Zick, Algorithmic Transparency Via Quantitative Input
Influence: Theory And Experiments With Learning Systems, in SECURITY AND PRIVACY (SP), 2016
IEEE SYMPOSIUM on, pp. 598-617. IEEE, 2016.
DRAFT – Please ask permission before citing or quoting
37
can uncover the relative contribution of different features to specific
determinations, but also the features whose values would have to change the
least or the most to change the determination.158
Such methods are particularly popular when trying to explain the
decisions of a deep learning model developed for object detection because
they can help to pinpoint what part of an image the model ascribes greatest
significance when making its determination (e.g., what in this image explains
why the model has identified a cat). Note that such explanations do not
provide a global description of the model, which, in the case of deep
learning, would almost certainly defy human interpretation; they instead
describe what gives the model greatest confidence about its classification or
what would have to be different in a specific image to trigger a different
classification, which can be easily rendered visually for a human observer.
Although these methods are especially well suited to models that
take raw signal as input (e.g., the pixel values in an image), they work
equally well when applied to models with a much more limited number of
semantically meaningful features. A credit scoring model that considers a
handful of discrete features can be subject to the same test,159 establishing
which features most influence a specific outcome. If practitioners know the
distribution of feature values that the model is likely to encounter in the real
world, they can also determine which features are most important on
average for a specific outcome, which are most important to different
outcomes, and which account for a disparity between groups.
Unfortunately, these methods will not work well in cases where, to
stick with the example, credit scoring takes a much larger set of features into
account. Should many features each contribute a small amount to a
particular determination, listing each of them in an explanation for a
particular decision is not likely to be terribly helpful. Indeed, the number of
features identified as influential might be sufficiently large that the
explanation would simply reproduce the problem of inscrutability that it
aims to address. This might come as a surprise, given how well this
approach works when applied to deep learning models, but recall that
explanations in the case of object recognition took the form of visualizations.
Unfortunately, post hoc explanations for credit and other similarly
important decisions are likely to be most attractive precisely when they do
These methods are generally sensitive to interactions among variables and are able
to measure indirect as well as direct influence. See, e.g., Adler, et al., supra note 157; Datta, et
al., supra note 157; JULIUS ADEBAYO, FAIRML: AUDITING BLACK-BOX PREDICTIVE
MODELS (2017), http://blog.fastforwardlabs.com/2017/03/09/fairml-auditing-black-boxpredictive-models.html.
159 Anupam Datta, Shayak Sen, & Yair Zick, supra note 158.
158
38
Regulating Inscrutable Systems – DRAFT 3/20/2017
not seem to work well—that is, when the only way to achieve a certain level
of performance is to vastly expand the range of features under
consideration.
D. Task-Specific Techniques
[still to come]
IV. HOW INTERPRETABILITY CAN ASSIST REGULATION
[What follows is an outline of how the research on interpretability
can assist with regulatory compliance, but why that might not go far
enough.]
A. Meeting Legal Requirements
•
•
•
Global descriptions can fulfill GDPR
Specific decisions can fulfill ECOA
o Existing practice of using rule extraction for ECOAcompliance
o But these will not work where we want them most (e.g., deep
learning)
Combining global descriptions with specific decisions might be a
way forward
o Simplify the model enough to be able to give a limited
number of reasons
B. Vindicating Normative Principles
•
•
But let’s imagine we can fulfill both ECOA and GDPR; would this
get us what we want?
The writing on this presupposes that having the explanation will
automatically get us to the goal of accountability.
o The what is not enough to answer the how and the how is
not enough to answer the why
o The concerns we have in mind when we ask for the why are
rarely generic questions about why; they are questions to
make sure the basis for decision is not something we would
find objectionable (e.g., faulty reasoning; weak proxies;
discriminatory intent; non-volitional factors, etc…)
DRAFT – Please ask permission before citing or quoting
•
•
•
•
39
o Explanation mismatch: is the explanation going to be the one
we need?
Interpretability is useful because we don’t know in advance what
bases for decisionmaking we’ll find objectionable in general, or in
the future.
o Interpretability gives us something to work with and makes
us ask the question.
o But still a weaker solution to things we do know we care
about than targeted ones (e.g. discrimination-aware data
mining).
The real problem is just category error; we need to think about why
we ask “why”?
o The honest answer to why would be “because these are the
patterns in the data”
§ On what basis should we assess whether these
patterns are a legitimate basis for decisions making?
o Perhaps we should look for regulation that does not ask
“why”
§ Environmental law, nuisance…
Machine explanation cannot address disagreement in the why layer
itself
o All the work is being done at the What and How
o If there’s disagreement about the normative goal (e.g., in
discrimination, what counts?), then fixing the What and How
layers to self-explain doesn’t solve the problem.
Other styles of explanation-regulation include asking at the Why
layer.
o Describing what has been set as the target variable and why
§ Credit: creditworthiness or maximum profit?
§ Employment: all the things in BDDI
o Can we get specific about asking for aspects of modeling as
justification?
CONCLUSION
[still to come]