Predictive Coding on Trial
Federal Courts Law Review
FEDERAL COURTS LAW REVIEW
Volume 7, Issue 1
2013
THE GROSSMAN-CORMACK GLOSSARY OF
TECHNOLOGY-ASSISTED REVIEW
with
Foreword by John M. Facciola, U.S. Magistrate Judge
FOREWORD
“When I use a word,” Humpty Dumpty said, in rather a scornful tone,
“it means just what I choose it to mean—neither more nor less.”
“The question is,” said Alice, “whether you can make words mean so many
different things.”
Lewis Carroll, Through the Looking Glass, What Alice Found There 1
“A word is not a crystal, transparent and unchanged, it is the skin of a
living thought and may vary greatly in color and content according to the
circumstances and the time in which it is used.”
Justice Oliver Wendell Holmes Jr., Towne v. Eisner, 245 U.S. 418,
425 (1918)
In the heels of the higgling lawyers, Bob,
Too many slippery ifs and buts and howevers,
Too much hereinbefore provided whereas,
Too many doors to go in and out of.
When the lawyers are through
What is there left, Bob?
Can a mouse nibble at it
And find enough to fasten a tooth in?
Carl Sandburg, The Lawyers Know Too Much 2
1.
2.
1970).
THE COLLECTED STORIES OF LEWIS CARROLL 238 (Citadel Press 1994).
THE COMPLETE POEMS OF CARL SANDBURG 189 (Harcourt, Brace and Co., rev. ed.
2
FEDERAL COURTS LAW REVIEW
[Vol. 7
It is always the words. Humpty Dumpty, Holmes, and Sandburg, who
sensed the power of language instinctively, knew how quickly their
meanings can slip away. What seemed clear when the contract or statute
was drafted is now recondite. When the parties contracted to buy and sell a
horse, did they mean a mare or a stallion? When the statute required that a
lawyer be disbarred upon conviction of a crime of moral turpitude, did the
legislature mean a lawyer who gets convicted of assault after a brawl in a
bar? Lawyers and judges are mocked for their continued use of Latin but
they know that it is so much easier to say “res judicata” and take advantage
of the encrusted meaning of those words than to start fresh and try to
improve on what they convey. Law books come and go but Black’s Law
Dictionary will always be around. The words, as Paul Simon might put it,
keep “slip, sliding away.” 3
It is hard enough when the world in which the words are used remains
static, like the farm on which the horse, be it mare or stallion, lives. But,
what happens when the movement of technology radically transforms what
a word might have once meant? What is the “original” of an e-mail? Is
another e-mail a copy of it when the visible text is the same but the
metadata created in its production by a computer, rather than a human
being, is entirely different? What happens when the meaning of the words
in a statute applied to a process that was in existence when the statute was
enacted but now no longer exists? Some of the definitions in the Stored
Communications Act, 4 enacted in 1986, may drive judges to distraction
since they were premised on technology used in 1986 but is no longer and
must be applied to new processes that no one knew would exist when the
statute was enacted. What kind of words can be used in a statute or a rule
that are capacious enough to hold their meaning despite unknown
technological change but precise enough to convey a definite meaning? It
may be hard to believe, but there was not even a clear indication in the
Federal Rules of Civil Procedure until 2006 that “electronically stored
information” was within the scope of what a party had to produce in
3. PAUL SIMON, Slip Slidin’ Away, on THE ESSENTIAL PAUL SIMON (Warner Bros. 2007).
4. See, e.g., The definition of “remote computing service” in 18 U.S.C. § 2711(2) (“the
provision to the public of computer storage or processing services by means of electronic
communications systems”). As Professor Orin Kerr explains, the statute “freez[es] into law the
understanding of computer network use as of 1986.” Orin S. Kerr, A User’s Guide to the Stored
Communications Act, and a Legislator’s Guide to Amending It, 72 GEO. WASH. L. REV. 1208,
1214 (2004). In 1986, users would use remote computing services to outsource computing tasks
such as storing extra files or processing data when doing so was beyond the capacity of their
computers. Id. Given the storage and processing capacities of new computers and tablets, this
kind of distant processing capacity has disappeared. In its wake, however, are the difficult
questions of the application of the definitions in the Stored Communications Act to “cloud
computing.” See William Jeremy Robison, Free at What Cost? Cloud Computing Privacy under
the Stored Communications Act, 98 GEO. L.J. 1195, 1210, 1212-13 (2010).
2013]
Glossary of Technology-Assisted Review
3
discovery. 5 Indeed, until 2006, the word “phonorecords” appeared in
Federal Rules of Civil Procedure 34(a) 6 which must have, at the time, 7
mystified anyone under 35.
The pace of technological change makes the situation worse. That
pace is astonishing. What was thought to be the impossible crossing of the
Atlantic Ocean in a plane and the landing of a man on the moon occurred in
my father’s lifetime. Yet, George Washington’s troops moved no faster
than Caesar’s, and it took thousands of years before human beings
discovered how to transmit messages by the telegraph using electricity. 8
Until then, a messenger, whether from Marathon to Athens, or through the
colonial towns of Massachusetts, had to deliver them by hand.
The legal system deals in words and the pace of technological process
is creating billions of them on a nearly daily basis, 9 creating a set of
problems that were unimaginable a few years ago.
The first problem is that this explosion of words has been matched by
the ever-increasing capacity of machines to capture and preserve them. It is
a simple fact that an iPod has much more memory, i.e., capability to store
information indefinitely, than the first computer10 which took up an entire
room. Additionally, the cost of storage is diminishing. A one-terabyte drive
can be purchased for about $100 11 and can hold what would otherwise be
hundreds of thousands of pages of paper. Indeed, now the information can
be kept on a distant server with the space rented from a vendor for that
purpose and retrieved by use of the Internet (“cloud computing”). For the
first time, it is cheaper for human beings to buy a new file cabinet and keep
more paper than to clean the useless clutter out of the old file cabinet to
make room for the new information.
5. Compare FED. R. CIV. P. 34(a)(1)(A) (West 2006 rev. ed.) (describing “any designated
documents or electronically stored information—including writings, drawings, graphs, charts,
photographs, sound recordings, images, and other data or data compilations—stored in any
medium from which information can be obtained either directly or, if necessary, after translation
by the responding party into a reasonably usable form” as discoverable) with FED. R. CIV. P.
34(a)(1)(A) (West 2006) (mentioning only “any designated documents (including writings,
drawings, graphs, charts, photographs, phonorecords, and other data compilations from which
information can be obtained, translated, if necessary, by the respondent through detection devices
into reasonably usable form”).
6. FED. R. CIV. P. 34(a)(1)(A) (West 2006).
7. Vinyl records are making a comeback; eight track tapes will not. See, e.g., Brian
Passey, Vinyl Records spin back into vogue, USATODAY.COM (Feb. 26, 2011).
8. See TOM STANDAGE, THE VICTORIAN INTERNET (Walker & Co. 1998).
9. See George L. Paul & Jason R. Baron, Information Inflation: Can the Legal System
Adapt?, 13 RICH. J.L. & TECH. 10, 14-23 (2007).
10. See, e.g., What is a Mainframe Operating System?, WISEGEEK, http://www.wisegeek.com/
what-is-a-mainframe-operating-system.htm (last visited Nov. 19, 2012).
11. See AMAZON, http://www.amazon.com/s/ref=nb_sb_ss_i_1_7?url=search-alias%3Daps&fieldkeywords=1+terabyte+hard+drive&sprefix=1+terab%2Caps%2C227 (listing numerous options under $100
for a one-terabyte external hard drive).
4
FEDERAL COURTS LAW REVIEW
[Vol. 7
This phenomenon has led to the consequence that litigants are
confronted with the often horrifying costs of searching through immense
amounts of data to find what they need. The Federal Rules of Civil
Procedure contemplate a system of demand and production; plaintiff asks
for all documents pertaining to the merger of companies A and B and
defendant either objects or produces them. But, when there are now
hundreds of thousands of documents that may meet the definition of
“pertinent,” how can defendant find them and not go bankrupt in the
process? Of course, defendant may be the victim of its own failure to
maintain a responsible record-keeping process in which a principled
decision-making process guides what will be kept and what will be thrown
out. And, as anyone knows who has ever cleaned out a closet or an old
hard drive, keeping everything is no solution. It only increases the expense
and cost of finding what you want or need. Nevertheless, the affordability
of cheap storage has led too many entities in our society to be quickly
overwhelmed by their inability to search for what they need, whether
because they need it to run their business or because they must produce it in
discovery. They may find that the cost of searching and producing is so
great that settling the lawsuit may be the only way out of an otherwise
impossible situation.
It is understandable that, nature and technology abhorring a vacuum, a
new scientific methodology has emerged to aid in the collection and
searching process. “Technology-Assisted Review,” called by its nickname
“Predictive Coding,” describes a process whereby computers are
programmed to search a large amount of data to find quickly and efficiently
the data that meet a particular requirement. Computer science and the
sciences of statistics and psychology inform its use. While it bruises the
human ego, scientists, including the authors of this glossary we are
publishing, have determined that machines are better at the task of making
such discoveries than humans. 12 Lawyers love to think that there is no
substitute for their reviewing each document page by page. Not only is
there a substitute, but an improvement. It is now indubitable that
technology-assisted review is an appreciably better and more accurate
means of searching a set of data. 13 That is hardly surprising news to those
judges and lawyers who have experienced the mind-numbing tedium of
reviewing large data sets only to find that one is seeing the same e-mail
12. See Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in
E-Discovery Can Be More Effective and Efficient Than Exhaustive and Manual Review, XVII
RICH. J.L. & TECH. 11 (2011).
13. Id.
2013]
Glossary of Technology-Assisted Review
5
chain again and again, or worse, wading through mountains of data and
finding nothing of any pertinence to the case being litigated.
The great benefits of technology-assisted review, however, bring in
new concerns and questions for judges and lawyers. In a paper universe,
the manner in which a party searched through a file cabinet hardly raises
any significant issues. If what was produced appeared to be what was
demanded and there were no inexplicable gaps, that was that and the court
and parties moved on to other things. Now, the methodology of the use of
technology-assisted review may itself be in dispute, with the parties
controverted to each other’s use of a particular method or tool. Those
controversies have already lead to judicial decisions that have to grapple
with a wholly new way of searching and with scientific principles derived
from the science of statistics or other disciplines.14 Lawyers and judges
once again have to learn a whole new vocabulary to resolve the emerging
and inevitable battle of the “experts.”
To aid in the creation of that vocabulary, we publish with pride the
glossary created by Maura R. Grossman and Gordon V. Cormack who are
two of the most respected and acknowledged experts in this field. We agree
with them that the creation of a clear and common vocabulary is essential to
a comprehension of the legal issues at stake.
We are particularly gratified that the authors announced that they
intend the glossary to be interactive so that others can suggest additional
clarifications, revisions, and additions. We are certain that, if the
experience of the courts in the first few years of the information technology
revolution is any guide, the learning curve will be steep and that it must be
climbed quickly if courts are going to be able to resolve promptly the
controversies before them at the least expense. We are equally certain that
the bench and bar will find this glossary useful as this new science develops
and grows.
The Editors of the Federal Courts Law Review
By John M. Facciola, U.S. Magistrate Judge
14. See, e.g., EORHB, Inc. v. HOA Holdings, Civ. Ac. No. 7409-VCL (Del. Ch. Oct. 19,
2012); Kleen Prods. LLC v. Packaging Corp., Civ. No. 10C 5711, 2012 WL 4498465 at *84-85
(N.D. Ill. Sept. 28, 2012); In re Actos (Pioglitazone) Prods. Liab. Litig., MDL No. 6:11-md-2299
(W.D. La. July 27, 2012); Global Aerospace Inc. v. Landow Aviation, L.P., No. CL 61040 (Va.
Cir. Ct. Apr. 23, 2012); Moore v. Publicis Groupe & MSL Group, No. 11 Civ. 1279 (ACL) (AJP),
2012 WL 607412 (S.D.N.Y. Feb. 24, 2012).
6
FEDERAL COURTS LAW REVIEW
[Vol. 7
PREAMBLE
“Disruptive technology” is a term that was coined by Harvard
Business School Professor Clayton M. Christensen, in his 1997 book, The
Innovator’s Dilemma, to describe a new technology that unexpectedly
displaces an established technology. The term is used in business and
technology literature to describe innovations that improve a product or
service in ways that the market did not expect, typically by designing for a
different set of consumers in the new market and later, by lowering prices in
the existing market. Products based on disruptive technologies are typically
cheaper to produce, simpler, smaller, better performing, more reliable, and
often more convenient to use. Technology-Assisted Review (TAR) is such
a disruptive technology. Because disruptive technologies differ from
sustaining technologies – ones that rely on incremental improvements to
established technologies – they bring with them new features, new
vernaculars, and other challenges.
The introduction of TAR into the legal community has brought with it
much confusion because different terms are being used to refer to the same
thing (e.g., “technology-assisted review,” “computer-assisted review,”
“computer-aided review,” “predictive coding,” and “content-based
advanced analytics,” to name but a few), and the same terms are also being
used to refer to different things (e.g., “seed set” and “control sample”).
Moreover, the introduction of complex statistical concepts and terms of art
from the science of information retrieval have resulted in widespread
misunderstanding and sometimes perversion of their intended meanings.
This glossary is written in an effort to bring order to chaos by
introducing a common framework and set of definitions for use by the
bench, the bar, and service providers. This glossary endeavors to be
comprehensive, but its definitions are necessarily brief. Interested readers
may look elsewhere for detailed information concerning any of these topics.
The terms in this glossary are presented in alphabetical order, with defined
terms in capital letters.
We envision this glossary to be a living, breathing work that will
evolve over time. Towards that end, we invite our colleagues in the
industry to send us comments on our definitions, as well as any additional
terms they would like to see included in the glossary, so that we can reach a
consensus on a consistent, common language relating to TAR. Comments
can be sent to us at [email protected] and [email protected].
Subsequent versions of this glossary will be available online at
http://cormack.uwaterloo.ca/targlossary/.
The authors would like to acknowledge the helpful comments
provided by Craig Ball, Michael Levine, Ralph Losey, Amir Milo, and
2013]
Glossary of Technology-Assisted Review
7
Keith Roland on an earlier draft of this work. We are very grateful to
Magistrate Judge John M. Facciola for his enthusiastic support.
We hope that you will find this glossary useful.
Maura R. Grossman*
Wachtell, Lipton, Rosen & Katz
New York, New York
Gordon V. Cormack
University of Waterloo
Waterloo, Ontario
January 2013
* The views expressed herein are solely those of the author and should not be attributed to her
firm or its clients.
8
FEDERAL COURTS LAW REVIEW
[Vol. 7
THE GLOSSARY
Accept on Zero Error: A technique in which the training of a Machine
Learning method is gauged by taking a Sample after each training step, and
deeming the training process complete when the learning method codes a
Sample with 0% Error (i.e., 100% Accuracy).
Accuracy: The fraction of Documents that are correctly coded by a search
or review effort. Note that Accuracy + Error = 100%, and that Accuracy =
100% – Error. While high Accuracy is commonly advanced as evidence of
an effective search or review effort, its use can be misleading because it is
heavily influenced by Prevalence. Consider, for example, a Document
Population containing one million Documents, of which ten thousand (or
1%) are Relevant. A search or review effort that identified 100% of the
Documents as Not Relevant, and, therefore, found none of the Relevant
Documents, would have 99% Accuracy, belying the failure of that search or
review effort.
Active Learning: An Iterative Training regimen in which the Training Set
is repeatedly augmented by additional Documents chosen by the Machine
Learning Algorithm, and coded by one or more Subject Matter Expert(s).
Actos: See In Re: Actos.
Agreement: The fraction of all Documents that two reviewers code the
same way. While high Agreement is commonly advanced as evidence of an
effective review effort, its use can be misleading, for the same reason that
the use of Accuracy can be misleading. When the vast majority of
Documents in a Population are Not Relevant, a high level of Agreement
will be achieved when the reviewers agree that these Documents are Not
Relevant, irrespective of whether or not they agree that any of the Relevant
Documents are Relevant.
Algorithm: A formally specified series of computations that, when
executed, accomplishes a particular goal. The Algorithms used in EDiscovery are implemented as computer software.
Area Under the ROC Curve (AUC): From Signal Detection Theory, a
summary measure used to assess the quality of Prioritization. AUC is the
Probability that a randomly chosen Relevant Document is given a higher
priority than a randomly chosen Non-Relevant Document. An AUC score
of 100% indicates a perfect ranking, in which all Relevant Documents have
2013]
Glossary of Technology-Assisted Review
9
higher priority than all Non-Relevant Documents. An AUC score of 50%
means the Prioritization is no better than chance.
Artificial Intelligence: An umbrella term for computer methods that
emulate human judgment.
These include Machine Learning and
Knowledge Engineering, as well as Pattern Matching (e.g., voice, face, and
handwriting recognition), robotics, and game playing.
Bag of Words: A Feature Engineering method in which the Features of
each Document comprise the set of words contained in that Document.
Documents are determined to be Relevant or Not Relevant depending on
what words they contain. Elementary Keyword Search and Boolean Search
methods, as well as some Machine Learning methods, use the Bag of
Words model.
Bayes / Bayesian / Bayes’ Theorem: A general term used to describe
Algorithms and other methods that estimate the overall Probability of some
eventuality (e.g., that a Document is Relevant), based on the combination of
evidence gleaned from separate observations. In Electronic Discovery, the
most common evidence that is combined is the occurrence of particular
words in a Document. For example, a Bayesian Algorithm might combine
the evidence gleaned from the fact that a Document contains the words
“credit,” “default,” and “swap” to indicate that there is a 99% Probability
that the Document concerns financial derivatives, but only a 40%
Probability if the words “credit” and “default,” but not “swap,” are present.
The most elementary Bayesian Algorithm is Naïve Bayes; however, most
Algorithms dubbed “Bayesian” are more complex. Bayesian Algorithms
are named after Bayes’ Theorem, coined by the 18th century
mathematician, Thomas Bayes. Bayes’ Theorem derives the Probability of
an outcome, given the evidence, from: (i) the probability of the outcome,
independent of the evidence; (ii) the probability of the evidence, given the
outcome; and (iii) the probability of the evidence, independent of the
outcome.
Bayesian Classifier / Bayesian Filter / Bayesian Learning: A colloquial
term used to describe a Machine Learning Algorithm that uses a Bayesian
Algorithm resembling Naïve Bayes.
Bigram: An N-Gram where N = 2 (i.e., a 2-gram).
Binomial Calculator / Binomial Estimation: A statistical method used to
calculate Confidence Intervals, based on the Binomial Distribution, that
10
FEDERAL COURTS LAW REVIEW
[Vol. 7
models the random selection of Documents from a large Population.
Binomial Estimation is generally more accurate, but less well known, than
Gaussian Estimation. A Binomial Estimate is substantially better than a
Gaussian Estimate (which, in contrast, relies on the Gaussian or Normal
Distribution) when there are few (or no) Relevant Documents in the
Sample. When there are many Relevant and many Non-Relevant
Documents in the Sample, Binomial and Gaussian Estimates are nearly
identical.
Binomial Distribution: The Probability that a Random Sample from a
large Population will contain any particular number of Relevant
Documents, given the Prevalence of Relevant Documents in the Population.
Used as the basis for Binomial Estimation.
Binomial Estimate: A Statistical Estimate of a Population characteristic
using Binomial Estimation. It is generally expressed as a Point Estimate
accompanied by a Margin of Error and a Confidence Level, or as a
Confidence Interval accompanied by a Confidence Level.
Blair and Maron: Authors of an influential 1985 study (David C. Blair &
M.E. Maron, An Evaluation of Retrieval Effectiveness for a Full-Text
Document-Retrieval System, 28 COMMC’NS ACM 289 (1985)), showing that
attorneys supervising skilled paralegals believed they had found at least
75% of the Relevant Documents from a Document Collection, using search
terms and iterative search, when they had in fact found only 20%. That is,
the searchers believed they had achieved 75% Recall, but had achieved only
20% Recall. In the Blair and Maron study, the attorneys and paralegals used
an iterative approach, examining the retrieved Documents and refining their
search terms until they believed they were done. Many current
commentators incorrectly distinguish the Blair and Maron study from
current iterative approaches, failing to note that the Blair and Maron
searchers did in fact refine their search terms based on their review of the
Documents that were returned in response to their queries.
Boolean Search: A Keyword Search in which the Keywords are combined
using operators such as “AND,” “OR,” and “[BUT] NOT.” The result of a
Boolean Search is precisely determined by the words contained in the
Documents. (See also Bag of Words.)
Bulk Coding: The process of Coding all members of a group of Documents
(identified, for example, by Deduplication, Near-Deduplication, Email
2013]
Glossary of Technology-Assisted Review
11
Threading, or Clustering) based on the review of only one or a few
members of the group. Also referred to as Bulk Tagging.
Bulk Tagging: See Bulk Coding.
Classical, Gaussian, or Normal Calculator / Classical, Gaussian, or
Normal Estimation: A method of calculating Confidence Intervals based
on the assumption that the quantities to be measured follow a Gaussian
(Normal) Distribution.
This method is most commonly taught in
introductory statistics courses, but yields inaccurate Confidence Intervals
when the Prevalence of items with the characteristic being measured is low.
(Cf. Binomial Calculator / Binomial Estimation.)
Classifier / Classification / Classified / Classify: An Algorithm that
Labels items as to whether or not they have a particular property; the act of
Labeling items as to whether or not they have a particular property. In
Technology-Assisted Review, Classifiers are commonly used to Label
Documents as Responsive or Non-Responsive.
Clustering: An Unsupervised Learning method in which Documents are
segregated into categories or groups so that the Documents in any group are
more similar to one another than to those in other groups. Clustering
involves no human intervention, and the resulting categories may or may
not reflect distinctions that are valuable for the purpose of a search or
review effort.
Code / Coded / Coding: The action of Labeling a Document as Relevant or
Non-Relevant, or the set of Labels resulting from that action. Sometimes
interpreted narrowly to include only the result(s) of a Manual Review
effort; sometimes interpreted more broadly to include automated or semiautomated Labeling efforts. Coding is generally the term used in the legal
industry; Labeling is the equivalent term in Information Retrieval.
Collection: See Document Collection.
Computer-Aided Review: See Technology-Assisted Review.
Computer-Assisted Review (CAR): See Technology-Assisted Review.
Concept Search: An industry-specific term generally used to describe
Keyword Expansion techniques, which allow search methods to return
Documents beyond those that would be returned by a simple Keyword or
12
FEDERAL COURTS LAW REVIEW
[Vol. 7
Boolean Search. Methods range from simple techniques such as Stemming,
Thesaurus Expansion, and Ontology search, through statistical Algorithms
such as Latent Semantic Indexing.
Confidence Interval: As part of a Statistical Estimate, a range of values
estimated to contain the true value, with a particular Confidence Level.
Confidence Level: As part of a Statistical Estimate, the chance that a
Confidence Interval derived from a Random Sample will include the true
value. For example, “95% Confidence” means that if one were to draw 100
independent Random Samples of the same size, and compute the
Confidence Interval from each Sample, about 95 of the 100 Confidence
Intervals would contain the true value. It is important to note that the
Confidence Level is not the Probability that the true value is contained in
any particular Confidence Interval; it is the Probability that the method of
estimation will yield a Confidence Interval that contains the true value.
Confusion Matrix: A two-by-two table listing values for the number of
True Negatives (TN), False Negatives (FN), True Positives (TP), and False
Positives (FP) resulting from a search or review effort. As shown below, all
of the standard evaluation measures are algebraic combinations of the four
values in the Confusion Matrix. Also referred to as a Contingency Table.
An example of a Confusion Matrix (or Contingency Table) is provided
immediately below.
Truly Relevant
Truly Non-Relevant
Coded Relevant
True Positives (TP)
False Positives (FP)
Coded Non-Relevant
False Negatives (FN)
True Negatives (TN)
Accuracy = 100% – Error = (TP + TN) / (TP + TN + FP + FN)
Elusion = 100% – Negative Predictive Value = FN / (FN + TN)
Error = 100% – Accuracy = (FP + FN) / (TP + TN + FP + FN)
Fallout = False Positive Rate = 100% – True Negative Rate = FP / (FP+TN)
False Negative Rate = 100% ‒ True Positive Rate = FN / (FN+TP)
Negative Predictive Value = 100% – Elusion = TN / (TN + FN)
Precision = Positive Predictive Value = TP / (TP + FP)
Prevalence = Yield = Richness = (TP + FN) / (TP + TN + FP + FN)
Recall = True Positive Rate = Sensitivity = TP / (TP+FN)
True Negative Rate = Specificity = TN / (TN + FP)
2013]
Glossary of Technology-Assisted Review
13
Content-Based Advanced Analytics (CBAA): See Technology-Assisted
Review.
Contingency Table: See Confusion Matrix.
Control Set: A Random Sample of Documents coded at the outset of a
search or review process that is separate from and independent of the
Training Set. Control Sets are used in some Technology-Assisted Review
processes. They are typically used to measure the effectiveness of the
Machine Learning Algorithm at various stages of training, and to determine
when training may cease.
Crossover Trial: An Experimental Design for comparing two search or
review processes using the same Document Collection and Information
Need, in which one process is applied first, followed by the second, and
then the results of the two efforts are compared. (Cf. Parallel Trial.)
Culling: The practice of narrowing a larger data set to a smaller data set for
the purposes of review, based on objective criteria (such as file types or date
restrictors), or subjective criteria (such as Keyword Search Terms).
Documents that do not match the criteria are excluded from the search and
from further review.
Cutoff: A given score or rank in a Prioritized list, resulting from a
Relevance Ranking search or Machine Learning Algorithm, such that the
Documents above the Cutoff are deemed to be Relevant and Documents
below the Cutoff are deemed to be Non-Relevant. In general, a higher
Cutoff will yield higher Precision and lower Recall, while a lower Cutoff
will yield lower Precision and higher Recall. Also referred to as a
Threshold.
Da Silva Moore: Da Silva Moore v. Publicis Groupe, Case No. 11 Civ.
1279 (ALC) (AJP), 2012 WL 607412 (S.D.N.Y. Feb. 24, 2012), aff’d, 2012
WL 1446534 (S.D.N.Y. Apr. 26, 2012). The first federal case to recognize
Computer-Assisted Review as “an acceptable way to search for relevant
ESI in appropriate cases.” The opinion was written by Magistrate Judge
Andrew J. Peck and affirmed by District Judge Andrew L. Carter.
Decision Tree: A step-by-step method of distinguishing between Relevant
and Non-Relevant Documents, depending on what combination of words
(or other Features) they contain. A Decision Tree to identify Documents
pertaining to financial derivatives might first determine whether or not a
14
FEDERAL COURTS LAW REVIEW
[Vol. 7
Document contained the word “swap.” If it did, the Decision Tree might
then determine whether or not the Document contained the word “credit,”
and so on. A Decision Tree may be created through Knowledge
Engineering or Machine Learning.
Deduplication: A method of replacing multiple identical copies of a
Document by a single instance of that Document. Deduplication can occur
within the data of a single custodian (also referred to as Vertical
Deduplication), or across all custodians (also referred to as Horizontal
Deduplication).
Dimensionality Reduction: A Feature Engineering method used to reduce
the total number of Features considered by a Machine Learning Algorithm.
Simple Dimensionality Reduction methods include Stemming and Stop
Word elimination. More complex Dimensionality Reduction methods
include Latent Semantic Indexing and Hashing.
Document: In the context of Electronic Discovery, a discrete item of
Electronically Stored Information that may be the subject or result of a
search or review effort.
Document Collection: The process of gathering Electronically Stored
Information for search, review, and production; the set of Documents
resulting from such a process. In many cases, the Document Collection and
Document Population are the same; however, it is important to note that
Document Population refers to the set of Documents over which a particular
Statistical Estimate is calculated, which may be the entire Document
Collection, a subset of the Document Collection (e.g., the documents with a
particular file type or matching particular Search Terms), a superset of the
Document Collection (e.g., the universe from which the Document
Collection was gathered), or any combination thereof.
Document Population: The set of Electronically Stored Information or
Documents about which a Statistical Estimate may be made.
Early Case Assessment (ECA): An industry-specific term generally used
to describe a variety of tools or methods for investigating and quickly
learning about a Document Collection for the purposes of estimating the
risk(s) and cost(s) of pursuing a particular legal course of action.
EDI Study: See JASIST Study.
2013]
Glossary of Technology-Assisted Review
15
EDI-Oracle Study: An ongoing initiative (as of January 2013) of the
Electronic Discovery Institute to evaluate participating vendors’ search and
document review efforts using a Document Collection contributed by
Oracle America, Inc.
Electronic Discovery / E-Discovery: The process of identifying,
preserving, collecting, processing, searching, reviewing, and producing
Electronically Stored Information that may be Relevant to a civil, criminal,
or regulatory matter.
Electronically Stored Information (ESI): Used in Federal Rule of Civil
Procedure 34(a)(1)(A) to refer to discoverable information “stored in any
medium from which the information can be obtained either directly or, if
necessary, after translation by the responding party into a reasonably usable
form.” Although Rule 34(a)(1)(A) references “Documents or Electronically
Stored Information,” individual units of review and production are
commonly referred to as Documents, regardless of the medium.
Elusion: The fraction of Documents identified as Non-Relevant by a search
or review effort that are in fact Relevant. Elusion is estimated by taking a
Random Sample from the Null Set and determining how many (or what
Proportion of) Documents are actually Relevant. A low Elusion value has
commonly been advanced as evidence of an effective search or review
effort (see, e.g., Kleen), but that can be misleading because it quantifies
only those Relevant Documents that have been missed by the search or
review effort; it does not quantify the Relevant Documents found by the
search or review effort (i.e., Recall). Consider, for example, a Document
Population containing one million Documents, of which ten thousand (or
1%) are Relevant. A search or review effort that returned 1,000
Documents, none of which were Relevant, would have 1.001% Elusion,
belying the failure of the search. Elusion = 100% – Negative Predictive
Value.
Email Threading: Grouping together email messages that are part of the
same discourse, so that they may be understood, reviewed, and coded
consistently as a unit.
EORHB: EORHB v. HOA Holdings LLC, Civ. Action No. 7409-VCL, tr.
and slip op. (Del. Ch. Oct. 19, 2012). The first case in which a court sua
sponte directed the parties to use Predictive Coding as a replacement for
Manual Review (or to show cause why this was not an appropriate case for
Predictive Coding), absent either party’s request to employ Predictive
16
FEDERAL COURTS LAW REVIEW
[Vol. 7
Coding. Vice Chancellor J. Travis Laster also ordered the parties to use the
same E-Discovery vendor and to share a Document repository.
Error / Error Rate: The fraction of all Documents that are incorrectly
coded by a search or review effort. Note that Accuracy + Error = 100%,
and that 100% – Accuracy = Error. While a low Error Rate is commonly
advanced as evidence of an effective search or review effort, its use can be
misleading because it is heavily influenced by Prevalence. Consider, for
example, a Document Population containing one million Documents, of
which ten thousand (or 1%) are relevant. A search or review effort that
found none of the relevant Documents would have 1% Error, belying the
failure of the search or review effort.
ESI: See Electronically Stored Information.
Experimental Design: A standard procedure accepted in the scientific
community for the evaluation of competing hypotheses. There are many
valid experimental designs. Some that can be appropriate for evaluating
Technology-Assisted Review processes include Crossover Trials and
Parallel Trials.
F1: The Harmonic Mean of Recall and Precision, often used in Information
Retrieval studies as a measure of the effectiveness of a search or review
effort, which accounts for the tradeoff between Recall and Precision. In
order to achieve a high F1 score, a search or review effort must achieve both
high Recall and high Precision.
Fallout: See False Positive Rate.
False Negative (FN): A Relevant Document that is missed (i.e., incorrectly
identified as Non-Relevant) by a search or review effort. Also known as a
Miss.
False Negative Rate (FNR): The fraction (or Proportion) of Relevant
Documents that are Missed (i.e., incorrectly identified as Non-Relevant) by
a search or review effort. Note that False Negative Rate + Recall = 100%,
and that 100% – Recall = False Negative Rate.
False Positive (FP): A Non-Relevant Document that is incorrectly
identified as Relevant by a search or review effort.
2013]
Glossary of Technology-Assisted Review
17
False Positive Rate (FPR): The fraction (or Proportion) of Non-Relevant
Documents that are incorrectly identified as Relevant by a search or review
effort. Note that False Positive Rate + True Negative Rate = 100%, and
that 100% – True Negative Rate = False Positive Rate. In Information
Retrieval, also known as Fallout.
Feature Engineering: The process of identifying Features of a Document
that are used as input to a Machine Learning Algorithm. Typical Features
include words and phrases, as well as metadata such as subjects, dates, and
file types. One of the simplest and most common Feature Engineering
techniques is Bag of Words. More complex Feature Engineering
techniques include the use of Ontologies and Latent Semantic Indexing.
Features: The units of information used by a Machine Learning Algorithm
to Classify or Prioritize Documents. Typical Features include text
fragments, such as words or phrases, and metadata such as sender, recipient,
and sent date. See also Feature Engineering.
Find Similar: A search method that identifies Documents that are similar to
a particular exemplar. Find Similar is commonly misconstrued to be the
mechanism behind Technology-Assisted Review.
Gain Curve: A graph that shows the Recall that would be achieved for a
particular Cutoff. The Gain Curve directly relates the Recall that can be
achieved to the effort that must be expended to achieve it, as measured by
the number of Documents that must be reviewed and Coded.
Gaussian Calculator / Gaussian Estimation: See Classical, Gaussian, or
Normal Calculator / Classical, Gaussian, or Normal Estimation.
Gaussian Distribution: See Normal Distribution.
Gaussian Estimate: A Statistical Estimate of a Population characteristic
using Gaussian Estimation. It is generally expressed as a Point Estimate
accompanied by a Margin of Error and a Confidence Level, or as a
Confidence Interval accompanied by a Confidence Level.
Global Aerospace: Global Aerospace Inc. v. Landow Aviation, Consol.
Case No. CL 61040, 2012 WL 1431215 (Va. Cir. Ct. Apr. 23, 2012). The
first State Court Order approving the use of Predictive Coding by the
producing party, over the objection of the requesting party, without
prejudice to the requesting party raising an issue with the Court as to the
18
FEDERAL COURTS LAW REVIEW
[Vol. 7
completeness or the contents of the production, or the ongoing use of
Predictive Coding. The order was issued by Loudoun County Circuit Court
Judge James H. Chamblin.
Global Deduplication: Deduplication of Documents across multiple
custodians. Also referred to as Horizontal Deduplication. (Cf. Vertical
Deduplication.)
Gold Standard: The best available determination of the Relevance or NonRelevance of all (or a sample) of a Document Population, used as
benchmark to evaluate the effectiveness of a search and review effort. Also
referred to as Ground Truth.
Goodhart’s Law: An observation made in 1975 by Charles Goodhart,
Chief Adviser to the Bank of England, that statistical economic indicators,
when used for regulation, become unreliable. Restated and generalized in
1997 by University of Cambridge Professor Marilyn Strathern as “When a
measure becomes a target, it ceases to be a good measure.” Within the
context of Electronic Discovery, Goodhart’s Law suggests that the value of
Information Retrieval measures such as Recall and Precision may be
compromised if they are prescribed as the definition of the reasonableness
of a search or review effort.
Grossman and Cormack: Authors of the JOLT (a.k.a. Richmond Journal)
Study.
Ground Truth: See Gold Standard.
Harmonic Mean: The reciprocal of the average of the reciprocals of two or
more quantities. If the quantities are named a and b, their Harmonic Mean
is
2
1 1
+
𝑎 𝑏
. In Information Retrieval, F1 is the Harmonic Mean of Recall and
Precision. The Harmonic Mean, unlike the more common arithmetic mean
(i.e., average), falls closer to the lower of the two quantities. As a summary
measure, a Harmonic Mean may be preferable to an arithmetic mean
because a high Harmonic Mean depends on both high Recall and high
Precision, whereas a high arithmetic mean can be achieved with high Recall
at the expense of low Precision, or high Precision at the expense of low
Recall.
Hashing / Hash / Hash Value: A statistical method used to reduce the
contents of a Document to a single, fixed-size, alphanumeric value, which
2013]
Glossary of Technology-Assisted Review
19
is, for all intents and purposes, unique to a particular Document; the single,
fixed-size alphanumeric value resulting from Hashing a particular
Document. Common Hashing Algorithms include, but are not limited to,
MD5, SHA-1, and SHA-2. Hashing and Hash Values are typically used for
Document identification, Deduplication, or ensuring that Documents have
not been altered.
Horizontal Deduplication: See Global Deduplication.
Deduplication.)
(Cf. Vertical
Index: A list of Keywords in which each Keyword is accompanied by a list
of the Documents (and sometimes the positions within the Documents)
where it occurs. Manual indices have been used in books for centuries;
automatic indices are used in Information Retrieval systems to identify the
Documents that contain particular Search Terms.
Indexing: The manual or automatic process of creating an Index. In
Electronic Discovery, Indexing typically refers to the automatic
construction of an electronic Index for use in an Information Retrieval
system.
Information Need: In Information Retrieval, the information being sought
in a search or review effort. In E-Discovery, the Information Need is
typically to identify Documents responsive to a request for production, or to
identify Documents that are subject to privilege or work-product protection.
Information Retrieval: The science of how to find information to meet an
Information Need. While modern Information Retrieval relies heavily on
computers, the discipline predates the invention of computers.
In Re: Actos: In Re: Actos (Pioglitazone) Products Liability Litigation,
MDL No. 6:11-md-2299 (W.D. La. July 27, 2012). A product liability
action with a Case Management Order (CMO) that memorializes the
parties’ agreement on a “search methodology proof of concept to evaluate
the potential utility of advanced analytics as a Document identification
mechanism for the review and production” of Electronically Stored
Information. The search protocol provides for the use of a TechnologyAssisted Review tool on the email of four key custodians. The CMO was
issued by District Judge Rebecca F. Doherty.
Internal Response Curve: From Signal Detection Theory, a tool for
estimating the number of Relevant and Non-Relevant Documents in a
Population, or the number of Documents that fall above and below a
particular Cutoff. The use of Internal Response Curves for this purpose
20
FEDERAL COURTS LAW REVIEW
[Vol. 7
assumes that the scores yielded by a Machine Learning Algorithm for
Relevant Documents obey a Gaussian Distribution, and the scores for NonRelevant Documents obey a different Gaussian Distribution. These
distributions are then used to predict the number of Relevant and NonRelevant Documents in any given range of scores.
Interval Sample / Interval Sampling: See Systematic Sample / Systematic
Sampling.
Issue Code(s) / Issue Coding: One or more subcategories of the overall
Information Need to be identified in a search or review effort; the act of
generating such subcategories of the overall Information Need. Examples
include specification of the reason(s) for a determination of Relevance or
Non-Relevance, Coding of particular subcategories of interest, and Coding
of privileged, confidential, or significant (“hot”) Documents.
Iterative Training: The process of repeatedly augmenting the Training Set
with additional examples of Coded Documents until the effectiveness of the
Machine Learning Algorithm reaches an acceptable level. The additional
examples may be identified through Judgmental Sampling, Random
Sampling, or by the Machine Learning Algorithm, as in Active Learning.
Jaccard Index: A measure of the consistency between two sets (e.g.,
Documents Coded as Relevant by two different reviewers). Defined
mathematically as the size of the intersection of the two sets, divided by the
size of the union (e.g., the number of Documents coded as Relevant by both
reviewers, divided by the number of Documents identified as Relevant by
one or the other, or both reviewers). It is typically used as a measure of
consistency among review efforts, but also may be used as a measure of
similarity between two Documents represented as two Bags of Words.
Jaccard Index is also referred to as Overlap or Mutual F1. Empirical studies
have shown that expert reviewers commonly achieve Jaccard Index scores
of about 50%, and that scores exceeding 60% are rare.
JASIST Study: A 2009 study (Herbert L. Roitblat, Anne Kershaw &
Patrick Oot, Document Categorization in Legal Electronic Discovery:
Computer Classification vs. Manual Review, 61 J. AM. SOC’Y. FOR INFO.
SCI. & TECH. 70 (2010)), showing that the Positive Agreement between
each of two Technology-Assisted Review methods, and a prior production
to the Department of Justice, exceeded the Positive Agreement between
each of two Manual Review processes and the same production. Also
referred to as the EDI Study.
2013]
Glossary of Technology-Assisted Review
21
JOLT Study: A 2011 study (Maura R. Grossman & Gordon V. Cormack,
Technology-Assisted Review in E-Discovery Can Be More Effective and
More Efficient Than Exhaustive Manual Review, XVII RICH. J.L. & TECH.
11 (2011)), available at http://jolt.richmond.edu/v17i3/article11.pdf, that
used data from TREC 2009 to show that two Technology-Assisted Review
processes (one using Machine Learning and one using a Rule Base)
generally achieved better Recall, better Precision, and greater efficiency
than the TREC Manual Review process. Also known as the Richmond
Journal Study, or the Richmond Study.
Judgmental Sample / Judgmental Sampling: A method in which a
Sample of the Document Population is drawn, based at least in part on
subjective factors, so as to include the “most interesting” Documents by
some criterion; the Sample resulting from such method. Unlike a Random
Sample, the statistical properties of a Judgmental Sample may not be
extrapolated to the entire Population. However, an individual (such as a
quality assurance auditor or an adversary) may use Judgmental Sampling to
attempt to uncover defects. The failure to identify defects may be taken as
evidence (albeit not statistical evidence, and certainly not proof) of the
absence of defects.
Keyword: A word (or Search Term) that is used as part of a Query in a
Keyword Search.
Keyword Expansion: See Query Expansion.
Keyword Search: A search in which all Documents that contain one or
more specific Keywords are returned.
Kleen: Kleen Prods. LLC v. Packaging Corp. of Am., Case No. 1:10-cv05711, various Pleadings and Tr. (N.D. Ill. 2012). A federal case in which
plaintiffs sought to compel defendants to use Content-Based Advanced
Analytics (CBAA) for their production, after defendants had already
employed complex Boolean Searches to identify Responsive Documents.
Defendants advanced Elusion scores of 5%, based on a Judgmental Sample
of custodians, to defend the reasonableness of the Boolean Search. After
two days of evidentiary hearings before (and many conferences with)
Magistrate Judge Nan R. Nolan, plaintiffs withdrew their request for
CBAA, without prejudice.
Knowledge Engineering: The process of capturing the expertise of a
Subject Matter Expert in a form (typically a Rule Base) that can be
executed by a computer to emulate the human’s judgment.
22
FEDERAL COURTS LAW REVIEW
[Vol. 7
Label / Labeled / Labeling: See Code / Coded / Coding.
Latent Semantic Analysis (LSA): See Latent Semantic Indexing.
Latent Semantic Indexing (LSI): A Feature Engineering Algorithm that
uses linear algebra to group together correlated Features. For example,
“Windows, Gates, Ballmer” might be one group, while “Windows, Gates,
Doors” might be another. Latent Semantic Indexing underlies many
Concept Search tools. While Latent Semantic Indexing is used for Feature
Engineering in some Technology-Assisted Review tools, it is not, per se, a
Technology-Assisted Review method. Also referred to as Latent Semantic
Analysis.
Linear Review: A Document-by-Document Manual Review in which the
Documents are examined in a prescribed order, typically chronological
order.
Logistic Regression: A state-of-the-art Supervised Learning Algorithm that
estimates the Probability that a Document is Relevant, based on the
Features it contains.
Machine Learning: The use of a computer Algorithm to organize or
Classify Documents by analyzing their Features. In the context of
Technology-Assisted Review, Supervised Learning Algorithms (e.g.,
Support Vector Machines, Logistic Regression, Nearest Neighbor, and
Bayesian Classifiers) are used to infer Relevance or Non-Relevance of
Documents based on the Coding of Documents in a Training Set. In
Electronic Discovery generally, Unsupervised Learning Algorithms are
used for Clustering, Near-Duplicate Detection, and Concept Search.
Manual Review: The practice of having human reviewers individually read
and Code the Documents in a Collection for Responsiveness, particular
issues, privilege, and/or confidentiality.
Margin of Error: The maximum amount by which a Point Estimate might
likely deviate from the true value, typically expressed as “plus or minus” a
percentage, with a particular Confidence Level. For example, one might
express a Statistical Estimate as “30% of the Documents in the Population
are Relevant, plus or minus 3%, with 95% confidence.” This means that
the Point Estimate is 30%, the Margin of Error is 3%, the Confidence
Interval is 27% to 33%, and the Confidence Level is 95%. Using Gaussian
Estimation, the Margin of Error is one-half of the size of the Confidence
Interval. It is important to note that when the Margin of Error is expressed
as a percentage, it refers to a percentage of the Population, not to a
2013]
Glossary of Technology-Assisted Review
23
percentage of the Point Estimate. In the current example, if there are one
million Documents in the Document Population, the Statistical Estimate
may be restated as “300,000 Documents in the Population are Relevant,
plus or minus 30,000 Documents, with 95% confidence”; or, alternatively,
“between 270,000 and 330,000 Documents in the Population are Relevant,
with 95% confidence.” The Margin of Error is commonly misconstrued to
be a percentage of the Point Estimate. However, it would be incorrect to
interpret the Confidence Interval in this example to mean that “300,000
Documents in the Population are Relevant, plus or minus 9,000
Documents.” The fact that a Margin of Error of “plus or minus 3%” has
been achieved is not, by itself, evidence of a precise Statistical Estimate
when the Prevalence of Relevant Documents is low.
Miss / Missed: A Relevant Document that is not identified as Relevant by a
search or review effort. Also referred to as a False Negative.
Miss Rate: The fraction (or proportion) of truly Relevant Documents that
are not identified as Relevant by a search or review effort. Miss Rate =
100% – Recall. Also referred to as the False Negative Rate.
Model: See Statistical Model.
Mutual F1: See Jaccard Index.
Naïve Bayes: A Supervised Learning Algorithm in which the relative
frequency of words (or other Features) in Relevant and Non-Relevant
Training Examples is used to estimate the likelihood that a new Document
containing those words (or other Features) is Relevant. Naïve Bayes relies
on the simplistic assumption that the words in a Document occur with
independent Probabilities, with the consequence that it tends to yield
extremely low or extremely high estimates.
NDLON: National Day Laborer Organizing Network v. U.S. Immigration
and Customs Enforcement Agency, Case No. 10-Civ-3488 (SAS), 2012 WL
2878130 (S.D.N.Y. July 13, 2012), a Freedom of Information Act (FOIA)
case in which District Judge Shira A. Scheindlin held that “most custodians
cannot be ‘trusted’ to run effective searches because designing legally
sufficient electronic searches in the discovery or FOIA contexts is not part
of their daily responsibilities,” and stated (in dicta) that “beyond the use of
keyword search, parties can (and frequently should) rely on latent semantic
indexing, statistical probability models, and machine learning to find
responsive documents. Through iterative learning, these methods (known
as ‘computer-assisted’ or ‘predictive’ coding) allow humans to teach
computers what documents are and are not responsive to a particular FOIA
24
FEDERAL COURTS LAW REVIEW
[Vol. 7
or discovery request and they can significantly increase the effectiveness
and efficiency of searches.”
Near-Duplicate Detection: An industry-specific term generally used to
describe a method of grouping together “nearly identical” Documents.
Near-Duplicate Detection is a variant of Clustering in which the similarity
among Documents in the same group is very strong. It is typically used to
reduce review costs, and to ensure consistent Coding. Also referred to as
Near-Deduplication.
Near-Deduplication: See Near-Duplicate Detection.
Nearest Neighbor: A Supervised Learning Algorithm in which a new
Document is Classified by finding the most similar Document in the
Training Set, and assuming that the correct Coding for the new Document is
the same as the most similar one in the Training Set.
Negative Predictive Value (NPV): The fraction (Proportion) of
Documents that are identified as Non-Relevant by a search or review effort,
that are in fact Non-Relevant. The complement of Precision; that is,
Negative Predictive Value is computed the same way as Precision when the
definitions of Relevant and Non-Relevant are transposed.
N-Gram: N consecutive words or characters treated as a Feature. In the
phrase, “To be or not to be,” a word Bigram (i.e., 2-gram) would be “to be”;
a word Trigram (i.e., 3-gram) would be “to be or”; a Quad-Gram (i.e., 4gram) would be “to be or not”; and so on. See also Shingling.
Non-Relevant / Not Relevant: In Information Retrieval, a Document is
considered Non-Relevant (or Not Relevant) if it does not meet the
Information Need of the search or review effort. The synonym “irrelevant”
is rarely used in Information Retrieval.
Normal Distribution: The “bell curve” of classical statistics. The number
of Relevant Documents in a Sample tends to obey a Normal (Gaussian)
Distribution, provided the Sample size is large enough to capture a
substantial number of Relevant and Non-Relevant Documents. In this
situation, Gaussian Estimation is reasonably accurate. If the Sample size is
insufficiently large to capture a substantial number of both Relevant and
Non-Relevant Documents (as a rule of thumb, at least 12 of each), the
Binomial Distribution better characterizes the number of Relevant
Documents in the Sample, and Binomial Estimation is more appropriate.
Also referred to as a Gaussian Distribution.
2013]
Glossary of Technology-Assisted Review
25
Null Set: The set of Documents that are not returned by a search process, or
that are identified as Not Relevant by a review process.
Ontology: A representation of the relationships among words and their
meanings that is richer than a Taxonomy. For example, an Ontology can
represent the fact that a wheel is a part of a bicycle, that gold is yellow, and
so on.
Overlap: See Jaccard Index.
Parallel Trial: An Experimental Design for comparing two search or
review processes using the same Document Collection and Information
Need, in which both processes are applied concurrently but independently,
and then the results of the two efforts are compared. (Cf. Crossover Trial.)
Pattern Matching: The science of designing computer Algorithms to
recognize natural phenomena like parts of speech, faces, or spoken words.
Point Estimate: The most likely value for a Population characteristic.
When combined with a Margin of Error (or Confidence Interval) and a
Confidence Level, it reflects a Statistical Estimate.
Population: See Document Population.
Positive Agreement: The Probability that, if one reviewer Codes a
Document as Relevant, a second independent reviewer will also Code the
Document as Relevant. Empirical studies show that Positive Agreement
rates of 70% are typical, and Positive Agreement rates of 80% are rare.
Positive Agreement should not be confused with Agreement (which is a less
informative measure) or Overlap (which is a numerically smaller measure
that conveys similar information). Under the assumption that the two
reviewers are equally likely to err, Overlap is roughly equal to the square of
Positive Agreement. That is, if Positive Agreement is 70%, Overlap is
roughly 70% × 70% = 49%.
Positive Predictive Value (PPV): See Precision. Positive Predictive Value
is a term used in Signal Detection Theory; Precision is the equivalent term
in Information Retrieval.
Precision: The fraction of Documents identified as Relevant by a search or
review effort, that are in fact Relevant. Also referred to as Positive
Predictive Value.
26
FEDERAL COURTS LAW REVIEW
[Vol. 7
Precision-Recall Curve: The curve representing the tradeoff between
Precision and Recall for a given search or review effort, depending on the
chosen Cutoff value. See Recall-Precision Curve.
Precision-Recall Tradeoff: The notion that most search strategies can be
adjusted to increase Precision at the expense of Recall, or vice versa. At
one extreme, 100% Recall could be achieved by a search that returned the
entire Document Population, but Precision would be low (equal to
Prevalence). At the other extreme, 100% Precision could be achieved by a
search that returned a single Relevant Document, but Recall would be low
(equal to 1/N, where N is the number of Relevant Documents in the
Document Population). More generally, a broader search returning many
Documents will have higher Recall and lower Precision, while a narrower
search returning fewer Documents will have lower Recall and higher
Precision. A Precision-Recall Curve illustrates the Precision-Recall
Tradeoff for a particular search method.
Predictive Coding: An industry-specific term generally used to describe a
Technology-Assisted Review process involving the use of a Machine
Learning Algorithm to distinguish Relevant from Non-Relevant
Documents, based on Subject Matter Expert(s)’ Coding of a Training Set of
Documents. See Supervised Learning and Active Learning.
Prevalence: The fraction of Documents in a Population that are Relevant to
an Information Need. Also referred to as Richness or Yield.
Prioritization / Prioritized: See Relevance Ranking.
Probabilistic Latent Semantic Analysis: A variant of Latent Semantic
Analysis based on conditional Probability rather than on correlation.
Probability: The fraction (proportion) of times that a particular outcome
would occur, should the same action be repeated under the same conditions
an infinite number of times. For example, if one were to flip a fair coin, the
Probability of it landing “heads” is one-half, or 50%; as one repeats this
action indefinitely, the fraction of times that the coin lands “heads” will
become indistinguishable from 50%. If one were to flip two fair coins, the
Probability of both landing “heads” is one-quarter, or 25%.
Proportion: The fraction of a set of Documents having some particular
property (typically Relevance).
Proportionality: Pursuant to Federal Rules of Civil Procedure 26(b)(2)(B),
26(b)(2)(C), 26(g)(1)(B)(iii), and other federal and state procedural rules,
2013]
Glossary of Technology-Assisted Review
27
the legal doctrine that Electronically Stored Information may be withheld
from production if the cost and burden of producing it exceeds its potential
value to the resolution of the matter. Proportionality has been interpreted in
the case law to apply to preservation as well as production.
Quad-Gram: An N-Gram where N = 4 (i.e., a 4-gram).
Quality Assurance: A method to ensure, after the fact, that a search or
review effort has achieved reasonable results.
Quality Control: Ongoing methods to ensure, during a search or review
effort, that reasonable results are being achieved.
Query: A formal search command provided as input to a search tool.
Query Expansion: The process of adding Search Terms to a Query to
improve Recall, often at the expense of decreased Precision.
Random Sample / Random Sampling: A subset of the Document
Population selected by a method that is equally likely to select any
Document from the Document Population for inclusion in the Sample; the
Sample resulting from such action. Random Sampling is the basis of
Statistical Estimation.
RAND Study: A 2012 study (Nicholas M. Pace & Laura Zakaras, Where
the Money Goes: Understanding Litigant Expenditures for Producing
Electronic Discovery, RAND Institute for Civil Justice (2012)), indicating
that Document review accounts for 73% of Electronic Discovery costs, and
concluding that “[t]he exponential growth in digital information, which
shows no signs of slowing, makes a computer-categorized review strategy,
such as predictive coding, not only a cost-effective choice but perhaps the
only reasonable way to handle many large-scale productions.”
Ranking: See Relevance Ranking.
Recall: The fraction of Relevant Documents that are identified as Relevant
by a search or review effort.
Recall-Precision Curve: See Precision-Recall Curve.
Recall-Precision Tradeoff: See Precision-Recall Tradeoff.
28
FEDERAL COURTS LAW REVIEW
[Vol. 7
Receiver Operating Characteristic Curve (ROC): In Signal Detection
Theory, a graph of the tradeoff between True Positive Rate and False
Positive Rate, as the Cutoff is varied.
Relevance Feedback: An Active Learning process in which the Documents
with the highest likelihood of Relevance are coded by a human, and added
to the Training Set.
Relevance Ranking: A search method in which the results are ranked from
the most likely to the least likely to be Relevant to an Information Need; the
result of such ranking. Google Web Search is an example of Relevance
Ranking.
Relevance / Relevant: In Information Retrieval, a Document is considered
Relevant if it meets the Information Need of the search or review effort.
Responsiveness: A Document that is Relevant to an Information Need
expressed by a particular request for production or subpoena in a civil,
criminal, or regulatory matter.
Richness: See Prevalence or Yield.
Richmond Journal Study / Richmond Study: See JOLT Study.
Roitblat, Kershaw, and Oot: Authors of the JASIST (a.k.a. EDI) Study.
Rolling Collection / Rolling Ingestion: A process in which the Document
Collection is periodically augmented as new, potentially Relevant
Documents are identified and gathered.
Whenever the Document
Collection is augmented, the results of prior search or review efforts must
be supplemented to account for the new Documents.
Rolling Production: A process in which Responsive Documents are
delivered incrementally to a requesting party to provide timely, partial
satisfaction of a Document request.
Rule: A formal statement of one or more criteria used to determine a
particular outcome, e.g., whether to code a Document as Relevant or NonRelevant.
Rule Base: A set of Rules created by an expert to emulate the human
decision-making process for the purposes of Classifying Documents in the
context of Electronic Discovery.
2013]
Glossary of Technology-Assisted Review
29
Sample / Sampling: A subset of the Document Population used to assess
some characteristic of the Population; the act of generating such a subset of
the Document Population. See Interval Sample, Judgmental Sample,
Random Sample, Statistical Sample, or Systematic Sample.
Sample Size: The number of Documents drawn at random that are used to
calculate a Statistical Estimate.
Search Term: See Keyword.
Sedona / Sedona Conference: The Sedona Conference® (https://thesedona
conference.org) is a nonprofit, 501(c)(3) research and educational institute,
founded in 1997 by Richard G. Braman, dedicated to the advanced study of
law and policy in the areas of antitrust, complex litigation, and intellectual
property rights. Sedona sponsors a preeminent think-tank in the area of
Electronic Discovery known as Working Group 1 on Electronic Document
Retention and Production. Sedona is well known for its thoughtful,
balanced, and free publications, such as The Sedona Conference®
Glossary: E-Discovery & Digital Information Management (Third Edition,
Sept. 2010), The Sedona Principles Addressing Electronic Document
Production, Second Edition (June 2007), and The Sedona Conference®
Cooperation Proclamation (July 2008).
Seed Set: The initial Training Set provided to the learning Algorithm in an
Active Learning process. The Documents in the Seed Set may be selected
based on Random Sampling or Judgmental Sampling. Some commentators
use the term more restrictively to refer only to Documents chosen using
Judgmental Sampling. Other commentators use the term generally to mean
any Training Set, including the final Training Set in Iterative Training, or
the only Training Set in non-Iterative Training.
Sensitivity: See True Positive Rate.
Shingling: A Feature Engineering method in which the Features consist of
all N-Grams in a text, for some number N. For example, the Trigram
Shingling of the text “To be or not to be” consists of the features “to be or”;
“be or not”; “or not to”; and “not to be.” Note that the Features overlap one
another in the text, suggesting the metaphor of roof shingles.
Signal Detection Theory: Invented at the same time and in conjunction
with radar, the science of distinguishing true observations from spurious
ones. Signal Detection Theory is widely used in radio engineering and
medical diagnostic testing. The terms True Positive, True Negative, False
Positive, False Negative, Sensitivity, Specificity, Receiver Operating
30
FEDERAL COURTS LAW REVIEW
[Vol. 7
Characteristic Curve, Area Under the ROC Curve, and Internal Response
Curve, all arise from Signal Detection Theory.
Significance / Significant: The confirmation, with a given Confidence
Level, of a prior hypothesis, using a Statistical Estimate. The result is said
to be Statistically Significant if all values within the Confidence Interval for
the desired Confidence Level (typically 95%) are consistent with the
hypothesis being true, and inconsistent with it being false. For example, if
the hypothesis is that fewer than 300,000 Documents are Relevant, and a
Statistical Estimate shows that 290,000 Documents are Relevant, plus or
minus 5,000 Documents, we say that the result is Significant. On the other
hand, if the Statistical Estimate shows that 290,000 Documents are
Relevant, plus or minus 15,000 Documents, we say that the result is not
Significant, because the Confidence Interval includes values (i.e., the values
between 300,000 and 305,000) that contradict the hypothesis.
Specificity: See True Negative Rate.
Statistical Estimate: A quantitative estimate of a Population characteristic
using Statistical Estimation. It is generally expressed as a Point Estimate
accompanied by a Margin of Error and a Confidence Level, or as a
Confidence Interval accompanied by a Confidence Level.
Statistical Estimation: The act of estimating the Proportion of a Document
Population that has a particular characteristic, based on the Proportion of a
Random Sample that has the same characteristic. Methods of Statistical
Estimation include Binomial Estimation and Gaussian Estimation.
Statistically Significant / Statistical Significance: See Significance.
Statistical Model: A mathematical abstraction of the Document Population
that removes irrelevant characteristics while largely preserving those of
interest for a particular purpose. For the purpose of computing Recall, a
Statistical Model need only consider whether or not the Documents are
Relevant, and whether or not the Documents are Coded Relevant, not any
other characteristics of the Documents.
Statistical Sample / Statistical Sampling: A method in which a Sample of
the Document Population is drawn at random, so that statistical properties
of the Sample may be extrapolated to the entire Document Population; the
Sample resulting from such action.
Stemming: In Keyword or Boolean Search, or Feature Engineering, the
process of equating all forms of the same root word. For example, the
2013]
Glossary of Technology-Assisted Review
31
words “stem,” “stemming,” “stemmed,” and “stemmable” would all be
treated as equivalent, and would each yield the same result when used as
Search Terms in a Query. In some search systems, stemming is implicit,
and in others, it must be made explicit through particular Query syntax.
Stop Word: A common word that is eliminated from Indexing.
Eliminating Stop Words from Indexing dramatically reduces the size of the
Index, while only marginally affecting the search process in most
circumstances. Examples of Stop Words include “a,” “the,” “of,” “but,”
and “not.” Because phrases and names such as “To be or not to be,” and
“The Who,” contain exclusively Stop Words that would not be Indexed,
they would not be identified (or identifiable) through a Keyword Search.
Subject Matter Expert(s): One or more individuals (typically, but not
necessarily, attorneys) who are familiar with the Information Need and can
render an authoritative determination as to whether a Document is Relevant
or not.
Supervised Learning: A Machine Learning method in which the learning
Algorithm infers how to distinguish between Relevant and Non-Relevant
Documents using a Training Set. Supervised Learning can be a stand-alone
process, or used repeatedly in an Active Learning process.
Support Vector Machine: A state-of-the-art Supervised Learning
Algorithm that separates Relevant from Non-Relevant Documents using
geometric methods (i.e., geometry). Each Document is considered to be a
point in [hyper]space, whose coordinates are determined from the Features
contained in the Document. The Support Vector Machine finds a
[hyper]plane that best separates Relevant from Non-Relevant Training
Examples. Documents outside the Training Set (i.e., uncoded Documents
from the Document Collection) are then Classified as Relevant or not,
depending on which side of the [hyper]plane they fall on. Although a
Support Vector Machine does not calculate a Probability of Relevance, one
may infer that the Classification of Documents closer to the [hyper]plane is
less certain than for those that are far from the [hyper]plane.
Synthetic Document: An industry-specific term generally used to describe
an artificial Document created by either the requesting party or the
producing party, as part of a Technology-Assisted Review process, for use
as a Training Example for a Machine Learning Algorithm. Synthetic
Documents are contrived Documents in which one party imagines what the
evidence might look like and relies on the Machine Learning Algorithm to
find actual Documents that are similar to the artificial Document.
32
FEDERAL COURTS LAW REVIEW
[Vol. 7
Systematic Sample / Systematic Sampling: A Sampling method in which
every Nth Document (for some fixed number N) is selected, when the
Documents are considered in some prescribed order; the Sample resulting
from such action. A Systematic Sample is random (and hence a true
Statistical Sample) only when the prescribed order is itself random.
Sometimes referred to as an Interval Sample / Interval Sampling.
TAR: See Technology-Assisted Review.
Taxonomy: A hierarchical organizational scheme that arranges the
meanings of words into classes and subclasses. For example, vehicles,
aircraft, and ships are modes of transportation; cars, trucks, and bicycles are
vehicles, and Fords and Chryslers are cars.
Technology-Assisted Review (TAR): A process for Prioritizing or Coding
a Collection of Documents using a computerized system that harnesses
human judgments of one or more Subject Matter Expert(s) on a smaller set
of Documents and then extrapolates those judgments to the remaining
Document Collection. Some TAR methods use Machine Learning
Algorithms to distinguish Relevant from Non-Relevant Documents, based
on Training Examples Coded as Relevant or Non-Relevant by the Subject
Matter Experts(s), while other TAR methods derive systematic Rules that
emulate the expert(s)’ decision-making process. TAR processes generally
incorporate Statistical Models and/or Sampling techniques to guide the
process and to measure overall system effectiveness.
Term Expansion: See Query Expansion.
Term Frequency and Inverse Document Frequency (TF-IDF): An
enhancement to the Bag of Words method in which each word has a weight
based on Term Frequency – the number of times the word appears in the
Document – and Inverse Document Frequency – the reciprocal of the
number of Documents in which the word occurs.
Thesaurus Expansion: In Keyword or Boolean Search, replacing a single
Search Term by a list of its synonyms, as listed in a thesaurus.
Threshold: See Cutoff.
Training Example: One Document from a Training Set.
Training Set: A Sample of Documents coded by one or more Subject
Matter Expert(s) as Relevant or Non-Relevant, from which a Machine
2013]
Glossary of Technology-Assisted Review
33
Learning Algorithm then infers how to distinguish between Relevant and
Non-Relevant Documents beyond those in the Training Set.
TREC: The Text REtrieval Conference, sponsored by the National Institute
of Standards and Technology (NIST), which has run since 1992, to “support
research within the information retrieval community by providing the
infrastructure necessary for large-scale evaluation of text retrieval
methodologies. In particular, the TREC workshop series has the following
goals: to encourage research in information retrieval based on large test
Collections; to increase communication among industry, academia, and
government by creating an open forum for the exchange of research ideas;
to speed the transfer of technology from research labs into commercial
products by demonstrating substantial improvements in retrieval
methodologies on real-world problems; and to increase the availability of
appropriate evaluation techniques for use by industry and academia,
including development of new evaluation techniques more applicable to
current systems.”
TREC Legal Track: From 2006 through 2011, TREC included a Legal
Track, which sought “to assess the ability of information retrieval
techniques to meet the needs of the legal profession for tools and methods
capable of helping with the retrieval of electronic business records,
principally for use as evidence in civil litigation.”
Trigram: An N-Gram where N = 3 (i.e., a 3-gram).
True Negative (TN): A Non-Relevant Document that is correctly identified
as Non-Relevant by a search or review effort.
True Negative Rate (TNR): The fraction (or Proportion) of Non-Relevant
Documents that are correctly identified as Non-Relevant by a search or
review effort.
True Positive (TP): A Relevant Document that is correctly identified as
Relevant by a search or review effort.
True Positive Rate (TPR): The fraction (or Proportion) of Relevant
Documents that are correctly identified as Relevant by a search or review
effort. True Positive Rate is a term used in Signal Detection Theory; Recall
is the equivalent term in Information Retrieval.
Uncertainty Sampling: An Active Learning approach in which the
Machine Learning Algorithm selects the Documents as to which it is least
34
FEDERAL COURTS LAW REVIEW
[Vol. 7
certain about Relevance, for Coding by the Subject Matter Expert(s), and
addition to the Training Set.
Unsupervised Learning: A Machine Learning method in which the
learning Algorithm infers categories of similar Documents without any
training by a Subject Matter Expert. Examples of Unsupervised Learning
methods include Clustering and Near-Duplicate Detection.
Validation: The act of confirming that a process has achieved its intended
purpose. Validation may involve Statistical or Judgmental Sampling.
Vertical Deduplication: Deduplication within a custodian; identical copies
of a Document held by different custodians are not Deduplicated. (Cf.
Horizontal Deduplication.)
Yield: See Prevalence or Richness.
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 1 of 53 PageID #:10395
IN THE UNITED STATES DISTRICT COURT
FOR THE NORTHERN DISTRICT OF ILLINOIS
EASTERN DIVISION
KLEEN PRODUCTS LLC, et al.,
Plaintiffs,
v.
PACKAGING CORPORATION
OF AMERICA, et al.,
Defendants.
)
)
)
)
)
)
)
)
)
)
No. 10 C 5711
Magistrate Judge Nan R. Nolan
MEMORANDUM OPINION AND ORDER
Lawyers have twin duties of loyalty: While they are retained to be
zealous advocates for their clients, they bear a professional obligation
to conduct discovery in a diligent and candid manner. Their combined
duty is to strive in the best interests of their clients to achieve the best
results at a reasonable cost, with integrity and candor as officers of the
court. Cooperation does not conflict with the advancement of their clients’ interests—it enhances it. Only when lawyers confuse advocacy
with adversarial conduct are these twin duties in conflict.
The Sedona Conference, The Sedona Conference Cooperation Proclamation, 10 Sedona Conf. J. 331, 331 (2009).
This matter is before the Court on Plaintiffs’ Motion to Compel Defendants to
Produce Documents and Data from All Reasonable Accessible Sources [Doc. 347],
Plaintiffs’ Motion to Compel Temple-Inland to Include Additional Document Custodians [Doc. 366], Plaintiffs’ Motion to Compel International Paper Company to Include Additional Document Custodians [Doc. 382], and Defendant Georgia-Pacific
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 2 of 53 PageID #:10396
LLC’s Motion for Protective Order [Doc. 373].1 However, this is a story as much
about cooperation as dispute.
I. BACKGROUND
A. Procedural
This action is the outgrowth of a number of class actions that had been brought
against Defendants Georgia-Pacific LLC (“GP”), Packaging Corporation of America
(“PCA”), International Paper Company (“IP”), Cascades Canada, Inc./Norampac
Holdings U.S., Inc. (“Norampac”), Weyerhaeuser Company (“Weyerhaeuser”), Temple-Inland, Inc. (“TIN”) and Smurfit-Stone Container Corporation (“Smurfit”)2
charging violations of Sherman Act §1. The instant action, which is the first-filed
case in these consolidated actions, was filed on September 9, 2010, and the related
later-filed cases were subsequently reassigned to the District Judge. On November
8, 2010, Plaintiffs filed a Consolidated and Amended Complaint (“Complaint”).
Plaintiffs have also filed a Motion to Compel Defendants to Produce Documents and
Data for the Time Periods Proposed by Plaintiffs [Doc. 345], which the Court will address in
a separate order.
1
On May 27, 2011, Smurfit was acquired by Rock-Tenn Company. To effect the acquisition, Smurfit was merged into a subsidiary of Rock-Tenn Company. The surviving entity
from the merger became RockTenn CP, LLC (“RockTenn”), a limited liability company and
wholly owned subsidiary of Rock-Tenn Company. On June 16, 2011, the Court allowed
RockTenn to be substituted as a defendant in place of Smurfit.
2
No. 10 C 5711
2
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 3 of 53 PageID #:10397
B. The Complaint3
Countless industrial and consumer products are manufactured from containerboard, the principal raw material used to manufacture corrugated products such as
linerboard and corrugated boxes. (Compl. ¶ 36.) Thus, the prices of those corrugated
products are tied directly to the price of containerboard. (Id. ¶ 40.) From the 1930s
onward, the containerboard industry has been subject to extensive antitrust litigation and other charges of unfair competition. (Id. ¶¶ 57–62.) In this instance, Plaintiffs allege the existence of anticompetitive behavior beginning in August 2005 and
continuing through the present. (Id. ¶ 1.)
During the class period, the containerboard industry was heavily consolidated.
(Compl. ¶ 39.) Significant barriers to entry in the form of capital-intensive startup
costs and high transportation costs make that industry susceptible to an oligopolistic structure. (Id. ¶ 41.) Because of those barriers to entry and the degree of consolidation in the industry, containerboard industry firms share a similar cost structure.
(Id. ¶¶ 47–48.) While the containerboard industry is consolidated, no single firm has
sufficient market power to control the supply and price of the product. (Id. ¶ 52.)
Because there are no close substitutes for containerboard, the demand for the product is inelastic. (Id. ¶¶ 50–51.)
In August 2005, the containerboard industry faced complex environmental factors, including declining profit margins, rising demand and a promising economic
This summary of the Complaint is adopted from the Court’s opinion denying Defendants’ motions to dismiss.
3
No. 10 C 5711
3
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 4 of 53 PageID #:10398
environment. (Compl. ¶ 64.) Between 2003 and 2005, individual producers had tried
and failed to institute price increases at least twice. (Id.) In 2005, many of the Defendants, including Smurfit, PCA, IP, TIN, GP and Norampac, significantly reduced
their production capacity through plant closures, capacity idling or scheduled production downtime. (Id. ¶¶ 71–72, 75, 78–81, 84–87.) Those capacity reductions coincided in time with the existence of a high demand for containerboard. (Id. ¶¶ 73–
75.)
There are a number of important industry groups and trade associations in the
containerboard industry. (Compl. ¶¶ 53–56.) Most Defendants belong to one or both
of two prominent organizations: the Fibre Box Association (“FBA”) and the American Forest and Paper Association. (Id. ¶¶ 53–54.) In June 2005, industry leaders,
including many representatives of Defendants, attended an industry conference
where pricing strategies were discussed. (Id. ¶¶ 76–77.) Just over three months after the conference, Smurfit, PCA and GP each announced a $30 per ton price increase effective October 1, 2005. (Id. ¶¶ 83–84.) After an FBA conference on September 27, 2005, the remaining Defendants followed suit. (Id. ¶ 89.) Only a few
months later, on November 28, the FBA Board of Directors met. (Id. ¶ 91.) At that
meeting, both Weyerhaeuser and PCA announced $40 per ton price increases effective January 1, 2006, which was soon matched by all other Defendants. (Id. ¶¶ 91,
98.) Then on March 14, 2006, the FBA Executive Committee met again, and Defendants raised their prices by $50 per ton only a few weeks later. (Id. ¶ 104.)
No. 10 C 5711
4
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 5 of 53 PageID #:10399
Even though prices were increasing throughout 2005 and into 2006, during that
period many of the Defendants reduced capacity. (Compl. ¶¶ 115, 117–18.) Such capacity decreases continued through 2007. (Id. ¶¶123, 125, 132.) Shortly after a large
industry conference in June 2007, both PCA and Smurfit announced $40 per ton
price increases effective August 1. (Id. ¶¶ 126–27.) On or about August 1, all other
Defendants followed suit. (Id. ¶ 129.)
In late March 2008, as the economy was beginning to decline, the industry conducted a series of conferences. (Compl. ¶¶ 138–39.) Less than two months later, in
early May 2008, both GP and Smurfit announced $55 per ton price increases to take
effect July 1. (Id. ¶ 141.) Most of the remaining Defendants instituted identical
price increase in the same time period. (Id.) About a month later, IP announced its
intention to raise prices by an additional $60 per ton on October 1, 2008. (Id. ¶ 145.)
IP’s price increase was followed by the majority of the industry soon thereafter. (Id.)
Over the course of the remainder of the year, the industry saw a continued decrease
in production. (Id. ¶¶ 150, 152.)
Despite a drastic economic slowdown in 2009, containerboard prices were inelastic and, for the most part, remained at the levels achieved by earlier price increases.
(Compl. ¶¶ 154–58.) In the face of economic weakness and normal seasonal weakness, Defendants raised prices by $50 per ton on January 1, 2010. Defendants continued to raise prices through the summer of 2010, raising prices by $60 per ton on
April 1 and August 1, 2010. (Id. ¶¶ 163, 167.)
No. 10 C 5711
5
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 6 of 53 PageID #:10400
II. DISCOVERY PROCESS
A. Timeline
On January 14, 2011, Defendants each filed a motion to dismiss. On April 8,
2011, the Court entered a memorandum opinion and order denying Defendants’ motions. Kleen Prods., LLC v. Packaging Corp. of America, 775 F. Supp. 2d 1071 (N.D.
Ill. 2011). On May 2, 2011, Defendants filed their answers.
Following the Court’s denial of the motions to dismiss, the parties began their
initial discovery. On May 3, 2011, Plaintiffs served Defendants with their First Request for Production of Documents (“RPD”). The RPD had 94 requests, which Plaintiffs divided into three general categories: (1) conduct requests, (2) data requests,
and (3) inquiries, investigations and prior litigation requests. Defendants each filed
responses and objections to the RPD on June 6, 2011.
Defendants began producing responsive hard copy documents in August 2011.
Shortly thereafter, the parties began meet-and-confer discussions in an attempt to
resolve issues raised in Defendants’ responses and objections and to draft a protocol
for the production of ESI. On October 26, 2011, the Court entered a stipulated Order governing the production format of e-mail, e-files and paper documents and other related matters (“ESI Production Order”) [Doc. 245].
Defendants served their First Set of Requests to Plaintiffs for the Production of
Documents on May 6, 2011. Plaintiffs served their responses and objections on June
No. 10 C 5711
6
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 7 of 53 PageID #:10401
10, 2011. Plaintiffs began producing paper documents in August 2011 and completed their paper production in November 2011.
In November 2011, Plaintiffs served Rule 30(b)(6) notices on each Defendant relating to document preservation and information systems issues. After conducting
meet-and-confer sessions, the parties agreed that Defendants would provide written
responses to the information sought in the Rule 30(b)(6) notices and the depositions
would be deferred pending Plaintiffs’ review of the responses.4
While the parties were able to resolve many of their discovery issues through
meet-and-confer sessions, by December 2011, four issues remained unresolved: (1)
reconciling Defendants’ production to the individual RPD requests; (2) ESI search
methods, (3) scope of document searches, and (4) relevant time periods.
As to the first issue, Plaintiffs requested that each Defendant provide an index
associating its produced documents with the 29 categories Plaintiffs included in the
RPD. (Pls.’ Statement [Doc. 266] 1.) Defendants asserted that complying with Plaintiffs’ request would impose great expense and is not required by Rule 34. Defendants argued that the metadata required by the parties’ agreed ESI Production Order provides Plaintiffs with sufficient identifying information. (Defs.’ Statement
[Doc. 267] 10.) Plaintiffs responded that metadata is not the issue and that the coding fields they are requesting are not addressed by the ESI Production Order. (Pls.’
Statement 3.)
The parties ended up supplementing the written responses with Rule 30(b)(6) depositions. Nevertheless, the Court commends the parties’ efforts to address the 30(b)(6) issues
without lengthy depositions.
4
No. 10 C 5711
7
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 8 of 53 PageID #:10402
With regard to the ESI search dispute, Plaintiffs criticized Defendants’ use of a
Boolean search method to identify responsive documents. (Pls.’ Statement 4–16.)
Plaintiffs argued that the Boolean keyword process is per se “subject to the inadequacies and flaws inherent when keywords are used to identify responsive documents.” (Id. 8.) They requested that Defendants use “content-based advanced analytics (‘CBAA’) technology analytics to conduct natural language, subject matter
searches across corporate departments or corporate functions, using content-based
search technology rather than keywords.” (Id. 5.)
Defendants defended their use of Boolean keywords, arguing that their testing
and validations processes “will have a degree of accuracy that meets or surpasses
not only industry standards but also the likely accuracy of any other available
methodology.” (Defs.’ Statement 3.) Further, given that by December 2011, Defendants had already collected and produced a significant amount of responsive information, they contend that Plaintiffs’ approach would involve additional costs and
burdens not contemplated by the Federal Rules, the Seventh Circuit Electronic Discovery Pilot Program,5 or case law. (Id. 8–10.)
As to the third issue, Plaintiffs criticized Defendants for limiting their search to
“main active servers.” (Pls.’ Statement 16.) Defendants stated they were willing to
consider searching for responsive archived documents if Plaintiffs both identify a
“particular and compelling need” and agree to bear the costs of restoring the archived information. (Defs.’ Statement 12–13.)
5
See http://www.discoverypilot.com.
No. 10 C 5711
8
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 9 of 53 PageID #:10403
Finally, with regard to the temporal scope dispute, Plaintiffs broke down their
document requests into the following time periods:
Conduct Requests: January 1, 2002–December 31, 2010;
Data Requests: January 1, 2000–December 31, 2010;
Inquiries, Investigations and Prior Litigation Requests: January 1,
1996–December 31, 2010; and
The prior antitrust litigation identified in the Complaint: no time limitation.
(Pls.’ Statement 17.) In response, Defendants proposed to produce conduct request
documents only back to January 1, 2004, and data request documents back to January 1, 2003. (Defs.’ Statement 14.) They argued that those time periods were appropriate based on the Complaint’s allegations that the conspiracy began in mid-2005.
(Id.)
On January 10, 2012, the case was referred to the undersigned Magistrate Judge
for discovery supervision. In February and March 2012, the Court conducted two
full days of evidentiary hearings on the ESI search dispute issue. Over the next six
months, the Court held 11 status hearings and Rule 16 conferences with all parties,
many of which lasted a half day or longer. The Court also facilitated three Rule 16
conferences between Plaintiffs and individual Defendants. A client representative
attended many of these conferences. Due to counsel’s hard work, the parties and the
Court were able to resolve a number of the issues through meet-and-confer discussions. These cooperative endeavors are described in the next section.
No. 10 C 5711
9
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 10 of 53 PageID #:10404
B. Issues on Which Parties Have Reached Agreement
1. Search Methodology
As discussed above, the parties had a fundamental dispute over what search
methodology Defendants should utilize to identify documents responsive to Plaintiffs’ RPD. Defendants argued that in order to best identify potentially responsive
ESI, they engaged leading consulting companies to develop Boolean search terms.
(Pls.’ Evidentiary Hr’g Br. [Doc. 288] 3.) During an iterative process, Defendants
and their consultants revised and refined the search terms over the course of several months. (Id. 4.) Sampling procedures were used throughout the process to evaluate the effectiveness and reliability of the search terms. (Id.)
On the other hand, Plaintiffs argued that Defendants’ search methodology is
likely to find less than 25% of responsive documents. (Defs.’ Evidentiary Hr’g Mem.
[Doc. 290] 1.) They asserted that their proposed content-based advance analytics
(“CBAA”) search would find more than 70% of responsive documents at no greater
cost. (Id. 1–2.) Plaintiffs contended that because CBAA searches do not focus on
matching words but instead on identifying relevant concepts out of the documents,
CBAA searches provide a richer, substantially more accurate return than Boolean
searches. (Id. 5.) Thus, they concluded that “[a] CBAA search of documents collected
by corporate or department function is the best and most practical method for locating responsive ESO in this case.” (Id. 6.)
In an attempt to resolve this impasse, an evidentiary hearing was held on February 21 and March 28, 2012, with both sides presenting witnesses in support of
No. 10 C 5711
10
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 11 of 53 PageID #:10405
their positions. At the conclusion of the second day, the Court observed that under
Sedona Principle 6, “[r]esponding parties are best situated to evaluate the procedures, methodologies, and techniques appropriate for preserving and producing
their own electronically stored information.” See The Sedona Conference, The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery, 8 Sedona Conf. J. 189, 193 (Fall 2007). Accordingly,
the Court urged the parties to consider whether there was a method to refine or
supplement Defendants’ Boolean search so that Plaintiffs had a reasonable assurance that they were receiving a high percentage of responsive documents without
completely scrapping Defendants’ search methodology.
Over the course of the next five months, the parties and the Court continued to
meet and confer on this issue. Finally, in August 2012, the parties came to an understanding on the ESI search methodology issue, and on August 21, 2012, the
Court entered a stipulated order relating to ESI search (“ESI Search Order”) [Doc.
385].6 In the ESI Search Order, Plaintiffs agreed to withdraw their demand that
Defendants apply CBAA to documents that have been or will be collected in response to the RPD (“First Request Corpus”). As to any documents or ESI beyond the
First Request Corpus, Plaintiffs agreed not to argue or contend that Defendants
should be required to apply CBAA or “predictive coding” methodology with respect
to any requests for productions served on any Defendant prior to October 1, 2013.
With respect to any requests for production served on any Defendant on or after Oc-
6
A copy of the ESI Search Order is attached as Exhibit A.
No. 10 C 5711
11
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 12 of 53 PageID #:10406
tober 1, 2013 that requires the collection of documents beyond the First Request
Corpus, the parties agreed to meet and confer regarding the appropriate search
methodology to be used for such newly collected documents.
2. Custodians
After Defendants filed their responses and objections to the RPD in June 2011,
the parties began meet-and-confer sessions in an attempt to work out their differences. In August 2011, Defendants identified 109 individuals for which they intended to collect custodian-specific documents responsive to the RPD. Because the Complaint alleged that “Defendants and their co-conspirators conducted their conspiracy
in secret and kept it mostly within the confines of their higher-level executives”
(Compl. ¶ 192; see id. ¶ 6), Defendants included as custodians their higher-level executives with decision-making responsibility for the pricing and sale of containerboard and corrugated products during the relevant period, along with some of their
subordinates. Plaintiffs complained that more custodians were needed, especially
within the sales and marketing departments and at the plant level. Subsequent
meet-and-confer discussions, some of which the Court supervised, resulted in a few
additional custodians.
During the Court’s meet-and-confer sessions with the parties, the Court encouraged Defendants to expand their lists of custodians and include at least a sample of
lower-level and plant-level employees. At the Court’s urging, Defendants produced
No. 10 C 5711
12
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 13 of 53 PageID #:10407
litigation hold lists to aid Plaintiffs in identifying appropriate custodians.7 (See, e.g.,
Docs. 325, 350–52.) Thereafter, to move the process along, the Court urged Plaintiffs to make formal requests for specific custodians from each Defendant. On July
27, 2012, after reviewing the litigation hold lists and documents produced by Defendants to date, Plaintiffs made formal requests for additional custodians. Thereafter, the parties promptly engaged in meet-and-confer discussions. Those discussions
ultimately led to agreements with five of the seven Defendants.8 (See Docs. 409,
410.) In those agreements, the parties agreed on specific additional custodians and
a protocol for how they would handle future requests for additional custodians.
3. Document Requests
Plaintiffs have objected to the form and substance of many of Defendants’ objections and responses to the RPD. They contend that the responses make it difficult to
determine what was produced in response to each document request. During meetand-confer sessions facilitated by the Court, Defendant IP prepared a chart that described the specific ESI searches it made to respond to each of the separate RPD requests. Plaintiffs agreed that this chart was helpful and could alleviate some of
their concerns. Thereafter, IP agreed to revise its RPD responses to incorporate the
information in the chart. IP served its revised RPD responses on July 23, 2012.
The Court determined that the most efficient use of the Court’s and the parties’
time is to address the RFP issue in stages. During phase one, Plaintiffs are meeting
7
A copy of the order is attached as Exhibit B.
8
A copy of one such agreement is attached as Exhibit C.
No. 10 C 5711
13
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 14 of 53 PageID #:10408
and conferring solely with IP in an effort to address Plaintiffs’ objections to IP’s revised responses. After Plaintiffs finish their review of the revised responses, they
will meet and confer with IP to see if they can work out any remaining disputes. After the meet-and-confer process is completed, Plaintiffs may file a motion to compel
against IP regarding any unresolved RFP responses. Phase two will include the remaining Defendants and will commence after the Court has ruled on Plaintiffs’ motion to compel against IP. To narrow any remaining disputes, the Court expects the
parties to use the IP process as a guide.
4. Sources of Data
Plaintiffs have asserted that Defendants should search and produce responsive
information regardless of where it is stored. Defendants countered that they will
search all active files for each of their custodians. Defendants are also prepared to
search for categories of data that are stored in centralized corporate systems and
exist on media or servers which can be searched using Defendants’ current system
capabilities and without incurring undue costs. With regard to archived media, Defendants will consider searching for responsive documents where Plaintiffs identify
a specific need and agree to bear the costs of restoring the archived documents.
The parties and the Court have conducted meet-and-confer sessions in an effort
to find some common ground on these issues. While the parties continue to dispute
both the cost and the need to restore information from archived media, Defendants
have agreed to preserve all such media during the pendency of the case. Further,
each Defendant has agreed that to the extent it has not already done so, it will deNo. 10 C 5711
14
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 15 of 53 PageID #:10409
termine if any of its custodians maintained potentially responsive information on
any personal device not synced to its servers. If any potentially responsive information is identified, it will be reviewed and responsive, nonduplicative, nonprivileged information will be produced.9
III. DISCUSSION
The following issues remain in dispute.
A. Georgia-Pacific Motion for Protective Order
On August 14, 2012, Defendant Georgia-Pacific filed a Motion for Protective Order. In its Motion, GP requests that the Court quash Plaintiffs’ Sixth Interrogatory.
1. Background
The Court facilitated a Rule 16 Conference between GP and Plaintiffs on May
31, 2012. Prior to the Conference, the parties engaged in numerous discussions concerning GP’s responses to Plaintiffs’ initial document requests. (Resp. 6.) Plaintiffs
complained that “GP refuses to identify which of its personnel were involved in Containerboard Products business other than ‘primary decision makers’ or reveal its
corporate structure.” (Mot. Ex. 7 at 11.) During the Conference, GP explained that
since 2005, it has been a privately-held company and does not maintain organizational charts or job descriptions. (Hr’g Tr., May 31, 2012, at 36, 46.) However, in the
interests of cooperation and compromise, GP agreed to produce a list of the individ-
The parties also continue to meet and confer in an effort to agree on the information
Defendants will provide in response to Plaintiffs’ transactional data requests.
9
No. 10 C 5711
15
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 16 of 53 PageID #:10410
uals who received the litigation-hold notice in connection with this litigation, along
with their job titles. (Id. 36–41, 48–50.)
On June 26, 2012, GP produced the list of litigation-hold recipients. (Mot. Ex. 1,
Attach. A.) The list included not only the names and titles of the litigation-hold recipients, but also the GP division in which each recipient works. (Id.)
Three days later, on June 29, 2012, Plaintiffs served their Sixth Interrogatory
request, demanding various background information over an eight-year period for
each of the approximately 400 persons on the litigation-hold list. (Mot. Ex. 1.) GP
informally objected to the request, contending that it was unfair and burdensome.
(Mot. 5.) Thereafter, the parties met and conferred in an effort to address GP’s concerns. (Id. 5–6, Ex. 9.) During a telephonic status conference on July 25, 2012, GP
indicated to the Court that it may have to move for protection from the Sixth Interrogatory. (Id. Ex. 10 at 27–29.) In response, Plaintiffs stated that they expected the
upcoming Rule 30(b)(6) deposition, scheduled for August 1, to largely eliminate
Plaintiffs’ need for the Sixth Interrogatory. (Id. Ex. 10 at 29–30.)
Plaintiffs took the Rule 30(b)(6) deposition on August 1, 2012. GP’s designee,
George Ragsdale, was prepared to answer questions about the structure of, and personnel in, GP’s containerboard and packaging businesses, including specific questions about the individuals and job titles on GP’s list of litigation-hold recipients.
(Ragsdale Dep. 166–68, 192–99.)10 When Plaintiffs’ counsel announced, well before
The transcript of the Rule 30(b)(6) deposition is available at Exhibit 11 to GP’s Motion
and Exhibit C to Plaintiffs’ Response.
10
No. 10 C 5711
16
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 17 of 53 PageID #:10411
the expiration of seven hours, that he had completed questioning the witness, GP’s
counsel reminded Plaintiffs’ counsel that the witness was available to answer questions about the litigation-hold list. (Id. 205–08.) Thereafter, Plaintiffs continued the
deposition, asking the witness further questions, but ultimately ended the deposition before the expiration of seven hours. (Mot. 8.)
The next day, GP requested that Plaintiffs withdraw the Sixth Interrogatory, believing that Plaintiffs had obtained all of their needed information during the Rule
30(b)(6) deposition. (Mot. 8.) Plaintiffs refused, stating that the Sixth Interrogatory
“is hardly burdensome” and can be “answered by a small production of paper.” (Id.
Ex. 12 at 13.)
2. Analysis
GP contends that the Sixth Interrogatory (a) “imposes undue and disproportionate burdens, and constitutes an abusive response to GP’s agreement voluntarily to
provide its list of litigation hold recipients and their job titles;” (b) “with its multiple
subparts, both on its own and when combined with prior interrogatories, exceeds
the number of interrogatories permitted by the Federal Rules;” and (c) “violates
Plaintiffs’ express commitment not to seek further discovery about GP’s organizational structure after GP responded to previous interrogatories on these subjects.”
(Mot. 3.) Plaintiffs disagree, contending that the Sixth Interrogatory “can be answered through the production of documents pursuant to Fed. R. Civ. P. 33(d), if GP
would produce the job descriptions that it maintains in the ordinary course of its
No. 10 C 5711
17
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 18 of 53 PageID #:10412
business.” (Resp. 1–2.) After carefully reviewing the issue, the Court finds that a
protective order is warranted.
First, issuing the Sixth Interrogatory within days of receiving the list of litigation-hold recipients violated the spirit of cooperation that this Court has encouraged
the parties to pursue. The Rule 16 conference was held to facilitate cooperative discussions between the parties on issues related to Defendants’ document production.
During the conference, the discussion largely focused on Plaintiffs’ need to understand GP’s key personnel and who some of the individuals are whose names were
appearing in the document production. GP explained that it is a privately-held
company and does not maintain organizational charts or job descriptions. (Hr’g Tr.,
May 31, 2012, at 36, 46.) The Court, acting as a neutral, facilitated an informal
brainstorming discussion with the parties.
The Court observed that without a formal organizational chart, Plaintiffs need a
better understanding of GP’s organizational structure. In this context, the Court
suggested that one possible solution might be to provide Plaintiffs with a list of GP’s
litigation-hold recipients. (See, e.g., Hr’g Tr. 10, 11, 14, 20, 31, 33, 35, 36.) However,
the Court’s suggestion was never intended to generate additional discovery obligations. Accordingly, the Court expected that Plaintiffs would use the list of litigationhold recipients, which included the recipients’ job titles and the division in which
they worked, in conjunction with their review of GP’s documents to create their own
organizational charts. (Id. 50.) Instead, Plaintiffs violated the spirit of cooperation
and used the list of litigation-hold recipients to request the additional discovery.
No. 10 C 5711
18
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 19 of 53 PageID #:10413
Such a response could have a chilling effect on both litigants and courts to engage in
candid discussions.
Second, GP has established an undue burden in responding to the Sixth Interrogatory. “All discovery is subject to the limitations imposed by Rule 26(b)(2)(C).”
Fed. R. Civ. P. 26(b)(1). The Rule 26 proportionality test allows the Court to “limit
discovery if it determines that the burden of the discovery outweighs its benefit.” In
re IKB Deutsche Industriebank AG, No. 09 CV 7582, 2010 WL 1526070, at *5 (N.D.
Ill. Apr. 8, 2010). Rule 26(b)(2)(C)(iii) requires a court to limit discovery if it determines that “the burden or expense of the proposed discovery outweighs its likely
benefit, considering the needs of the case, the amount in controversy, the parties’
resources, the importance of the issues at stake in the action, and the importance of
the discovery in resolving the issues.” In other words, “Rule 26(b)(2)(C)(iii) empowers a court to limit the frequency or extent of discovery if it determines that the
burden or expense of the proposed discovery outweighs its likely benefit or that it is
unreasonably cumulative or duplicative.” Sommerfield v. City of Chicago, 613 F.
Supp. 2d 1004, 1017 (N.D. Ill. 2009) objections overruled, 06 C 3132, 2010 WL
780390 (N.D. Ill. Mar. 3, 2010). “The ‘metrics’ set forth in Rule 26(b)(2)(C)(iii) provide courts significant flexibility and discretion to assess the circumstances of the
case and limit discovery accordingly to ensure that the scope and duration of discovery is reasonably proportional to the value of the requested information, the needs
of the case, and the parties’ resources.” The Sedona Conference, The Sedona Conference Commentary on Proportionality in Electronic Discovery, 11 Sedona Conf. J.
No. 10 C 5711
19
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 20 of 53 PageID #:10414
289, 294 (2010); see Sommerfield, 613 F. Supp. 2d at 1017 (“The application of Rule
26(b)(2)(C)(iii) involves a highly discretionary determination based upon an assessment of a number of competing considerations.”).
The Sixth Interrogatory requests for each of the approximately 400 persons on
the litigation-hold list, and for an eight-year period, that GP:
(a) describe their job functions;
(b) identify their employer;
(c) identify all the persons to whom they reported, the period during
which they so reported, and the job titles, employer and division of the
person to whom they reported; and
(d) identify all the persons who reported to them, the period during
which they so reported, and the job titles, employer and division of the
reporting person.
(Mot. Ex. 1.) GP has demonstrated that it “does not maintain any single report in
the ordinary course of business from which this information could be readily obtained.” (Mary K. McLemore Decl. ¶ 4.) Indeed, given the temporal scope of the request, some of this information would be difficult, or perhaps impossible, to obtain.
(Id. ¶¶ 4–8.) To attempt to respond to the Sixth Interrogatory, GP would need to (a)
conduct a multi-step process to compile the information that is available in its SAP
Data Warehouse, and then (b) interview each of the litigation-hold recipients—
along with their supervisors and direct reports—to confirm the information. (Id. ¶¶
4–6.) GP estimates that the process could take as many as 800 hours to accomplish.
(Id. ¶ 7.) Even after these hours were devoted, the response would likely be incomplete and could be impossible to verify. (Id. ¶ 8.)
No. 10 C 5711
20
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 21 of 53 PageID #:10415
In responding to the Motion, Plaintiffs do not address the Rule 26 proportionality principle. In other words, they do not explain how the value of the proposed discovery outweighs its burden. Instead, Plaintiffs contend that GP can answer the
Sixth Interrogatory by producing the job descriptions that it maintains. (Resp. 13)
(“GP can readily answer Plaintiffs’ Sixth Interrogatory through the production of a
defined set of documents that GP can readily assemble and produce.”) However, GP
has consistently stated that it does not maintain formal organizational charts or job
descriptions. (See, e.g., Hr’g Tr. 36 (“Since 2005, [GP] has not been a public company
[and] there are not in existence organizational charts.”), 46 (“There are no written
job descriptions in [GP’s] system.”); Resp. Ex. E at 2 (GP “has searched and does not
believe such formal written job descriptions exist.”).)
Plaintiffs assert that at the Rule 30(b)(6) deposition, “Plaintiffs learned for the
first time that as an organization, GP’s practice was to generate a Roles, Responsibilities and Expectations (“RRE”) document for each employee that sets forth that
employee’s duties, as well as providing pertinent reporting information.” (Resp. Ex.
E at 2.) Thus, Plaintiffs assert that “[w]ere GP to produce its RREs for the individuals identifies on GP’s litigation hold list, Plaintiffs would accept their Sixth Interrogatory as answered.” (Id. 2–3.)
But Plaintiffs misapprehend the testimony of GP’s representative. The phrase
“job description” does not appear in the transcript, and Plaintiffs’ counsel did not
even inquire if RREs were job descriptions. On the contrary, the witness testified
that job descriptions would be meaningless because an individual employee’s reNo. 10 C 5711
21
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 22 of 53 PageID #:10416
sponsibilities are defined annually on an individual basis by the employee and his
or her supervisor and are subject to change at any time. (Ragsdale Dep. 144–45.)
Thus, the RRE is retained locally “as a living document between the boss and the
subordinate[;] it’s updated annually, or at least it’s reviewed annually for update,
and, theoretically, once it’s updated, the old one goes away.” (Id. 147–48.) Furthermore, the RREs do not necessarily provide all of the information required by the
Sixth Interrogatory. “For example, rather than providing a fixed job description for
a title, a RRE is intended to be a negotiated framework between each employee and
his or her supervisor that is used to focus an employee on planned outcomes for
each year.” (McLemore Decl. ¶ 11.) And, while the RREs identify the employee’s supervisor, they do not provide the additional reporting relationships sought by Plaintiffs. (Id.)
The Court agrees. The RRE is a document used as part of an individual employee’s job performance review, not a document used by GP to describe a particular job.
Significantly, job descriptions are generic documents meant to apply to all individuals at all times, while performance evaluations are highly personal documents that
apply to a single employee at a particular point in time. See, e.g., Hooper v. Total
System Services, Inc., 799 F. Supp. 2d 1350, 1362 (M.D. Ga. 2011) (describing “job
descriptions” as “generic and meant to be used across business units”); Loeb v. Best
Buy Co, Inc., No. 05-720, 2007 WL 2264729, at *1, *15 (D. Minn. Aug. 6, 2007) (describing “job descriptions” as “fairly generic and broad” and “performance reviews”
as means to determine if employee is meeting current expectations).
No. 10 C 5711
22
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 23 of 53 PageID #:10417
In any event, the Rule 26 proportionality test cautions against producing the
RREs. Indeed, producing the RREs as an alternative to responding to the Sixth Interrogatory would be no less burdensome. “RREs are not centrally maintained in
any location and can be updated more than once each year.” (McLemore Decl. ¶ 12.)
Thus, “GP would have to interview each of the [litigation-hold recipients] to determine whether they maintain any current or historical RREs.” (Id.) “GP would also
be required to interview each of their supervisors to determine if they maintain any
RREs for the identified individuals.” (Id.) Plaintiffs contend that GP could simply
request all RREs by emailing each of the litigation-hold recipients. (Resp. 13–14 &
n.8.) But Plaintiffs fail to discuss how this process would work with former employees. Furthermore, GP could not provide a complete, accurate and verified response
to the Sixth Interrogatory without also determining who the employees’ supervisors
were during the eight-year period and contacting them to collect any RREs in their
possession. (See McLemore Decl. ¶¶ 5, 12.) Moreover, Plaintiffs do not explain how
any benefit from securing more RREs outweighs the burden of producing them. Despite having identified some 25 RREs in GP’s production (Resp. 13), Plaintiffs do
not describe how these documents have aided their document review or could lead
to relevant information.
Finally, much of the information sought in the Sixth Interrogatory has already
been obtained through a more convenient, less burdensome, and less expensive
method. “[T]he court must limit the frequency or extent of discovery otherwise allowed by these rules or by local rule if it determines that . . . the discovery sought is
No. 10 C 5711
23
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 24 of 53 PageID #:10418
unreasonably cumulative or duplicative, or can be obtained from some other source
that is more convenient, less burdensome, or less expensive . . . .” Fed. R. Civ. P.
26(b)(2)(C)(i). “In other words, where relevant information is available from multiple sources, the Rules give courts the authority to limit discovery to the least burdensome source, thus empowering courts to control litigation costs and promote efficiency in accordance with Rule 1.” The Sedona Conference, The Sedona Conference
Commentary on Proportionality in Electronic Discovery, 11 Sedona Conf. J. 289,
296–97 (2010); see Fed. R. Civ. P. 1 (The Rules “should be construed and administered to secure the just, speedy, and inexpensive determination of every action and
proceeding.”).
Here, GP provided a Rule 30(b)(6) witness to provide testimony about the litigation-hold recipients, specifically “their responsibilities at GP, the identities of those
reporting to him or her, and the identities of those to whom he or she reported,
whether directly or in a ‘dotted line’ relationship.” (Resp. Ex. R.) The witness answered almost all of the questions asked of him regarding GP’s organizational
structure and job titles, and Plaintiffs concluded the deposition early rather than
asking any additional questions of the deponent. (Ragsdale Dep. 166–68, 192–99,
205–08.) In fact, in response to GP’s request that Plaintiffs use the remaining time
to ask additional questions, Plaintiffs’ counsel responded that while “[t]here may be
small areas that remain where we need some further information,” “I believe, on my
recollection without review of the transcript, that I have accomplished what Plaintiffs, at a minimum to satisfy [the Sixth Interrogatory], need to accomplish.” (Id.
No. 10 C 5711
24
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 25 of 53 PageID #:10419
206.) Thereafter, Plaintiffs continued the deposition, asking the witness further
questions, but ultimately ending the deposition before the expiration of seven hours.
(Id. 206–30.)
Plaintiffs do not dispute that GP’s corporate witness was qualified and able to
provide significant information on the above topic. Instead, they argue that “there
were more than a half dozen instances where Mr. Ragsdale was either unable to
testify concerning the nature of a position or had to speculate about job functions.”
(Resp. 12 & n.7.) In response, GP has provided verified written answers to each of
the specific questions referenced in footnote 7 to Plaintiffs’ Response. (Doc. 407.)
These additional answers, together with the Rule 30(b)(6) deposition, provide Plaintiffs with a proportional response to their Sixth Interrogatory.
In sum, the Court finds that the Sixth Interrogatory violated the spirit of cooperation. Further, the Sixth Interrogatory does not pass the Rule 26 proportionality
test. GP’s burden of responding to the Sixth Interrogatory outweighs any benefits,
and Plaintiffs were able to get much of the information in a less burdensome way.
Defendant Georgia-Pacific LLC’s Motion for Protective Order is granted. Plaintiffs’
Sixth Interrogatory is quashed.
B. Custodians
As discussed above, the Court facilitated meet-and-confer discussions with the
parties in an effort to find a cooperative solution to the parties’ disputes over who is
an appropriate custodian. Those discussions ultimately led to agreements with GP,
No. 10 C 5711
25
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 26 of 53 PageID #:10420
Norampac, Weyerhaeuser, PCA and RockTenn. (Docs. 409, 410.) However, Plaintiffs
were unable to come to agreements with IP and TIN and have filed motions to compel additional custodians from them.
Plaintiffs contend that “[i]n this price-fixing conspiracy, [IP and TIN] should not
have the unilateral ability to select their own ‘priority’ document custodians, limit
such custodians to high level executives with ‘primary decision making authority,’
and reject Plaintiffs’ reasonable requests for additional custodians that are believed
likely to possess responsive documents.” (Custodian Mem. 7.) In their motions,
Plaintiffs request 19 additional custodians from TIN (TIN Mem. 6) and 16 additional custodians from IP (IP Mem. 8).
IP and TIN oppose the motions. IP and TIN were each willing to include 16 additional custodians but only if Plaintiffs agreed to limit the document sources for the
additional custodians and a written assurance that any future custodian requests
be accompanied with substantive support. (IP Resp. 1–5, 15; TIN Resp. 1–4.) Plaintiffs are unwilling to make this compromise. They assert that limiting sources to a
subset of ESI documents would preclude paper documents, which may the only viable source of discovery for the earlier time periods and for former employees. (TIN
Reply [Doc. 392] 14; see IP Reply [Doc. 402] 6.) Plaintiffs also contend that IP’s and
TIN’s “efforts to tie any additional custodians to specific allegations in the [Complaint] or documents already produced” is inconsistent with the Federal Rules, especially Rule 26. (TIN Reply 4–5; see IP Reply 2–3.)
No. 10 C 5711
26
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 27 of 53 PageID #:10421
In antitrust cases, courts generally take an expansive view of relevance and
permit broad discovery. In re Auto. Refinishing Paint Antitrust Litig., MDL No.
1426, 2004 U.S. Dist. LEXIS 21960, at *7–8 (E.D. Penn. Oct. 29, 2004); see U.S. v.
Int’l Bus. Machs. Corp., 66 F.R.D. 186, 189 (S.D.N.Y. 1974) (observing that “discovery in antitrust litigation is most broadly permitted and the burden or cost of
providing the information sought is less weighty a consideration than in other cases”) (citation omitted). “Broad discovery is permitted because direct evidence of an
anticompetitive conspiracy is often difficult to obtain, and the existence of a conspiracy frequently can be established only through circumstantial evidence, such as
business documents and other records.” In re Auto. Refinishing Paint Antitrust
Litig., 2004 U.S. Dist. LEXIS 21960, at *8; see Hosp. Bldg. Co. v. Trs. of Rex Hosp.,
425 U.S. 738, 746 (1976) (In antitrust cases, relevant evidence is “largely in the
hands of the alleged conspirators.”) (citation omitted); Callahan v. A.E.V. Inc., 947
F. Supp. 175, 179 (W.D. Penn. 1996) (“Discovery in an antitrust case is necessarily
broad because allegations involve improper business conduct. Such conduct is generally covert and must be gleaned from records, conduct, and business relationships.”) (citation omitted). Courts also note “the public importance of the decision,
the need of large corporate defendants to know which of their many activities are
attacked, [and] the issue narrowing function of discovery.” Bass v. Gulf Oil Corp.,
304 F. Supp. 1041, 1046 (S.D. Miss. 1969). These factors create a predisposition
among courts to allow broad discovery of antitrust defendants.
No. 10 C 5711
27
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 28 of 53 PageID #:10422
However, “[a]ll discovery, even if otherwise permitted by the Federal Rules of
Civil Procedure because it is likely to yield relevant evidence, is subject to the
court’s obligation to balance its utility against its cost.” U.S. ex rel. McBride v. Halliburton Co., 272 F.R.D. 235, 240 (D.D.C. 2011) (Facciola, M.J.); see Fed. R. Civ. P.
26(b)(2)(C). More specifically, Rule 26(b)(2)(C) requires the court to consider whether “(1) the discovery sought is unreasonably cumulative or duplicative, or obtainable
from a cheaper and more convenient source; (2) the party seeking the discovery has
had ample opportunity to obtain the sought information by earlier discovery; or (3)
the burden of the discovery outweighs its utility.” McBride, 272 F.R.D. at 240–41;
see Willnerd v. Sybase, Inc., No. 09 C 500, 2010 WL 4736295, at *3 (D. Idaho Nov.
16, 2010) (“In employing the proportionality standard of Rule 26(b)(2)(C) . . . , the
Court balances [the requesting party’s] interest in the documents requested, against
the not-inconsequential burden of searching for and producing documents.”). The
third factor requires the court to consider (a) the needs of the case; (b) the amount
in controversy; (c) the parties’ resources; (d) the importance of the issues at stake in
the action; and (e) the importance of the discovery in resolving the issues. Fed. R.
Civ. P. 26(b)(2)(C)(iii). Nevertheless, “[t]he party opposing a motion to compel carries a ‘heavy’ burden of persuasion.” U.S. v. AT&T Inc., No. 11-1560, 2011 WL
534178, at *5 (D.D.C. Nov. 6, 2011).
While the record does not afford a precise calculation, the Court can presume,
given the nature of the antitrust claims and the size of the companies involved, that
the amount in controversy is very large and that Defendants’ resources are greater
No. 10 C 5711
28
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 29 of 53 PageID #:10423
than Plaintiffs’. Further, claims of collusion in the containerboard and corrugated
box industries raise important, vital issues of public importance. Thus, these factors
weigh in favor of the discovery sought by Plaintiffs. See McBride, 272 F.R.D. at 241.
On the other hand, Defendants protest that Plaintiffs have not demonstrated
that the proposed additional custodians will have important, nonduplicative information. (IP Resp. 9; TIN Resp. 11–14.) Indeed, Plaintiffs do not point to any specific,
noncumulative evidence they expect to find with the additional custodians. Instead,
they selected the additional custodians by examining each Defendant’s organizational charts and the list and titles of persons who received a litigation hold notice11
and using metadata to analyze which individuals were sending and receiving emails
from the sales and marketing people that were already identified as custodians.
(Hr’g Tr. 18–21, Aug. 21, 2012.) Thus, Plaintiffs contend that the proposed individuals should be included as custodians because they are senior executives with responsibilities in containerboard, boxes, pricing, strategic planning, marketing and
sales, and who “exchanged an unusually large” number of emails with top sales and
marketing executives already named as custodians. (IP Reply 3–4; see TIN Reply
13–15.)
But just because a proposed custodian exchanged a large number of emails with
a current custodian does not mean that the proposed custodians will have a significant number of important, non-cumulative information. Further, until Plaintiffs
have had an opportunity to review the huge quantity of information already proBecause Temple-Inland sent a litigation-hold notice to all employees, it did not produce a litigation-hold list.
11
No. 10 C 5711
29
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 30 of 53 PageID #:10424
duced from the existing custodians, it is difficult for the Court to determine the utility of the proposed discovery. See McBride, 272 F.R.D. at 241 (“Without any showing of the significance of the non-produced e-mails, let alone the likelihood of finding
the ‘smoking gun,’ the [party’s] demands [for additional custodians] cannot possibly
be justified when one balances its cost against its utility.”); Jones v. Nat’l Council of
Young Men’s Christian Ass’ns of the United States, No. 09 C 6437, 2011 WL
7568591, at *2 (N.D. Ill. Oct. 21, 2011) (“The Court finds that Plaintiffs’ untargeted,
all-encompassing request fails to focus on key individuals and the likelihood of receiving relevant information.”); Garcia v. Tyson Foods, Inc., No. 06-2198, 2010 WL
5392660, at *14 (D. Kan. Dec. 21, 2010) (Waxse, M.J.) (“Plaintiffs present no evidence that a search of e-mail repositories of the 11 employees at issue is likely to
reveal any additional responsive e-mails. . . . Plaintiffs must present something
more than mere speculation that responsive e-mails might exist in order for this
Court to compel the searches and productions requested.”).
The Court also notes that IP already has 75 custodians—by far the most of any
Defendant—and has engaged in good faith meet-and-confer discussions with Plaintiffs that enlarged the scope of document collection and production that IP initially
agreed to undertake. (See IP Resp. 1, 6–7.) Similarly, TIN already has 28 custodians, which is more than most of the other Defendants.
However, the selection of custodians is more than a mathematical count. The selection of custodians must be designed to respond fully to document requests and to
produce responsive, nonduplicative documents during the relevant period. See, genNo. 10 C 5711
30
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 31 of 53 PageID #:10425
erally, Eisai Inc. v. Sanofi-Aventis U.S., LLC, No. 08-4168, 2012 WL 1299379, at *9
(D.N.J. April 16, 2012). First of all, Plaintiffs are entitled to discovery about Defendants’ box businesses. The Complaint adequately alleges a conspiracy both of
containerboard and of corrugated products, including corrugated boxes. See Kleen
Prods., 775 F. Supp. 2d at 1082. Further, while the Complaint alleges a conspiracy
mostly among higher-level executives (Compl. ¶ 192), it does not exclude lower level
employees. More importantly, even if the conspiracy is among higher-level executives, lower-level employees may possess important, relevant information which
could reasonably lead to admissible evidence. Fed. R. Civ. P. 26(b)(1); see In re Coordinated Pretrial Proceedings in Petroleum Prods. Antitrust Litig., 906 F.2d 432,
453 (9th Cir. 1990) (“With regard to the appellees’ contention that Rogers was too
low-level an employee to be of significance, we see no reason for concluding that
such information gathering cannot be delegated to subordinates. Accordingly, the
fact that Rogers did not himself have authority to make ARCO pricing decisions is
not dispositive.”); In re High Fructose Corn Syrup Antitrust Litig., 295 F.3d 651, 662
(7th Cir. 2002) (“One of Staley’s HFCS plant managers was heard to say: ‘We have
an understanding within the industry not to undercut each other’s prices.’ (He was
commenting on a matter within the scope of his employment and his comment was
therefore admissible as an admission by a party. Fed. R. Evid. 801(d)(2)(D).)”); see
also In re SRAM Antitrust Litig., No. 07-1819, 2010 U.S. Dist. LEXIS 132171, at
*40, *45–48, *51 (N.D. Cal. Dec. 10, 2010). Thus, in an antitrust case such as this,
No. 10 C 5711
31
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 32 of 53 PageID #:10426
Plaintiffs are at least entitled to a sample of lower-level and plant-level employees
to determine if they possess significant and nonduplicative information.
Further, IP and TIN have not established an undue burden to producing information from the additional custodians. IP contends that its burden is undue because it has “already produced more than 4 million pages, with more to come based
upon prior agreements with Plaintiffs.” (IP Resp. 7 (emphasis added); see id. 11.)
Similarly, TIN requests “an acknowledgment of the considerable burdens it already
has been subjected to and some assistance . . . in controlling this burden going forward.” (TIN Resp. 11.) But a party must articulate and provide evidence of its burden. While a discovery request can be denied if the “burden or expense of the proposed discovery outweighs its likely benefit,” Fed. R. Civ. P. 26(b)(2)(C)(iii), a party
objecting to discovery must specifically demonstrate how the request is burdensome.
See Heraeus Kulzer, GmbH v. Biomet, Inc., 633 F.3d 591, 598 (7th Cir. 2011); Sauer
v. Exelon Generation Co., No. 10 C 3258, 2011 WL 3584780, at *5 (N.D. Ill. Aug. 15,
2011). This specific showing can include “an estimate of the number of documents
that it would be required to provide . . . , the number of hours of work by lawyers
and paralegals required, [or] the expense.” Heraeus Kulzer, 633 F.3d at 598. Here,
TIN’s and IP’s conclusory statements do not provide evidence in support of their
burdensome arguments.
In sum, in this situation, the Rule 26(b)(2)(C) factors do not overwhelmingly favor either Plaintiffs or Defendants. However, because Plaintiffs had no input on the
initial custodian determinations and the case is still in the early stages of discovery,
No. 10 C 5711
32
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 33 of 53 PageID #:10427
the Court finds that Plaintiffs should be allowed a small number of additional custodians. Accordingly, Plaintiffs may select eight additional custodians from the prioritized list it sent to IP. (IP Mot. Ex. 9.) Similarly, Plaintiffs may select eight additional custodians from the proposed list it sent to TIN, two of whom should be a
random sample of the individuals identified as mill managers. (See TIN Mot. Ex. A.)
TIN and IP argue that they should not have to search all sources of information
for any new custodians. (TIN Resp. 9–11; IP Resp. 10–11.) They generally propose
to search only ESI from particular servers. (Id.) They contend that “the burden and
expense of searching [other] sources . . . would certainly outweigh its likely benefit.”
(IP Resp. 11; see TIN Resp. 11.)
The Court disagrees. While Plaintiffs have focused their review on email and
other ESI documents, they have not asserted that non-ESI documents are unimportant. Given the dearth of emails produced in the early time periods, hard copy
documents for those periods may prove valuable. Similarly, for former employees,
hard copy documents may be the only information available.
TIN and IP also seek a written assurance that any future custodian request
would be subject to a good faith belief that the individual possesses information
tending to prove the alleged conspiracy. (See IP Resp. 12; TIN Resp. 4.) Specifically,
IP requests assurance that “[a]ny future request for custodians would be based on a
good faith belief, arising from a review of the documents produced in this case and
taking depositions, than an individual possesses information tending the prove the
allege conspiracy, and Plaintiffs agree to provide the basis for their good faith belief
No. 10 C 5711
33
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 34 of 53 PageID #:10428
to IP.” (IP Resp. 12) (emphasis omitted). Similarly, TIN seeks confirmation that “if
Plaintiffs seek to add any additional custodians, it would have to be based on specific evidence from the record indicating that the proposed custodian would have responsive evidence tending to prove the alleged conspiracy.” (TIN Resp. 4.) Plaintiffs
counter that the agreement they reached with PCA provides sufficient protection for
all parties and is consistent with Rule 26. (IP Reply 4–5.)
The Court declines to set any restrictions on future custodian requests. The
Court finds that IP’s and TIN’s requests are too restrictive and run contrary to the
Federal Rules. For example, discoverable information “need not be admissible at the
trial if the discovery appears reasonably calculated to lead to the discovery of admissible information.” Fed. R. Civ. P. 26(b)(1). Nevertheless, the parties are free to
enter into any stipulations or agreements that fit their needs. Absent any agreements, however, the Court will be guided in any future custodian disputes by the
Federal Rules and applicable case law.12 See, e.g., Fed. R. Civ. P. 1, 26(b)–(c).
Plaintiffs’ motions to compel IP and TIN to include additional custodians are
granted in part.
C. Data Sources
In Plaintiffs’ Motion to Compel Defendants to Produce Documents and Data
from All Reasonably Accessible Sources, they request that Defendants be compelled
Plaintiffs are cautioned, however, that the Court expects them to support any future
custodian requests with specific evidence of the expected utility of the additional custodians
derived from their review of existing documents. See Fed. R. Civ. P. 26(b)(2)(C)(iii).
12
No. 10 C 5711
34
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 35 of 53 PageID #:10429
to search all reasonably accessible sources that potentially contain nonduplicative
responsive documents or data, including backup tapes. (Mot. 10.) Defendants respond that their backup tapes are not reasonably accessible and Plaintiffs have not
shown good cause for requiring their production. (Resp. 16–27.)
The resolution of Plaintiffs’ motion is dependent on the application of Rule
26(b)(2)(B). This Rule provides:
Specific Limitations on Electronically Stored Information. A party need
not provide discovery of electronically stored information from sources
that the party identifies as not reasonably accessible because of undue
burden or cost. On motion to compel discovery or for a protective order,
the party from whom discovery is sought must show that the information is not reasonably accessible because of undue burden or cost. If
that showing is made, the court may nonetheless order discovery from
such sources if the requesting party shows good cause, considering the
limitations of Rule 26(b)(2)(C). The court may specify conditions for the
discovery.
“Pursuant to this Rule, defendants must produce electronically stored information
that is relevant, not privileged, and reasonably accessible, subject to the discovery
limitations in Rule 26(b)(2)(C).” Major Tours, Inc. v. Colorel, No. 05-3091, 2009 WL
3446761, at *2 (D.N.J. Oct. 20, 2009). If Defendants establish that the requested
backup tapes are “inaccessible” within the meaning of Rule 26(b)(2)(B), the information must still be produced if Plaintiffs establish good cause considering the limitations in Rule 26(b)(2)(C). “The decision whether to require a responding party to
search for and produce information that is not reasonably accessible depends not
only on the burdens and costs of doing so, but also on whether those burdens and
costs can be justified in the circumstances of the case.” Fed. R. Civ. P. 26(b)(2), advisory committee’s note (2006). Factors to examine in this analysis include:
No. 10 C 5711
35
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 36 of 53 PageID #:10430
(1) the specificity of the discovery request; (2) the quantity of information available from other and more easily accessed sources; (3) the
failure to produce relevant information that seems likely to have existed but is no longer available on more easily accessed sources; (4) the
likelihood of finding relevant, responsive information that cannot be
obtained from other, more easily accessed sources; (5) predictions as to
the importance and usefulness of further information; (6) the importance of the issues at stake in the litigation; and (7) the parties' resources.
Id.
“Under this framework, a court does not reach the two-fold question of whether
inaccessible sources of electronically stored information should be searched and, if
so, which party should bear the associated costs unless it is first satisfied that the
request seeks relevant information that is not available from accessible sources.”
Baker v. Gerould, 03-CV-6558L, 2008 WL 850236, at *2 (W.D.N.Y. Mar. 27, 2008)
This is because relevant considerations in determining whether to order a search of
inaccessible sources include “the quantity of information available from other and
more easily accessed sources” and “the likelihood of finding relevant information
that seems likely to have existed but is no longer available on more easily accessed
sources.” Fed. R. Civ. P. 26(b)(2), advisory committee’s note (2006); see Zubulake v.
UBS Warburg LLC, 217 F.R.D. 309, 323 (S.D.N.Y. 2003) (Scheindlin, J.) (one of the
two most important considerations is “the availability of such information from other sources”).
Courts generally agree that backup tapes are presumptively inaccessible. See,
e.g., Zubulake, 217 F.R.D. at 319–20 (“ ‘Inaccessible’ data . . . is not readily usable.
Backup tapes must be restored using a process similar to that previously described,
No. 10 C 5711
36
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 37 of 53 PageID #:10431
fragmented data must be de-fragmented, and erased data must be reconstructed, all
before the data is usable. That makes such data inaccessible.”); Major Tours, 2009
WL 3446761, at *3 (Backup tapes are “typically classified as inaccessible.”); Go v.
Rockefeller Univ., 280 F.R.D. 165, 175–76 (S.D.N.Y. 2012) (“Information stored on
backup tapes is generally considered ‘not reasonably accessible.’ ”) (citation omitted); Clean Harbors Envtl. Servs., Inc. v. ESIS, Inc., No. 09 CV 3789, 2011 WL
1897213, at *2 (N.D. Ill. May 17, 2011) (Courts have already agreed that when information is stored on backup tapes, it is ‘likened to paper records locked inside a
sophisticated safe to which no one has the key or combination.’ ESIS has given us
no reason to believe that the information on the backup tapes in this case would be
more easily accessible.”) (quoting Zubulake v. UBS Warburg LLC, 216 F.R.D. 280,
291 (S.D.N.Y. 2003)).
In addition, Defendants have demonstrated a cost burden to restoring the backup media. They provided affidavits indicating that to restore the backup tapes
would cost each Defendant at least $200,000, with some estimates well over
$1,000,000. (See Resp. Exs. D, H.) Plaintiffs dispute the cost to restore Defendants’
backup media. They contend that sampling the media to determine whether they
contain responsive nonduplicative information could reduce costs. (Reply 6–7; Hanners Decl. ¶ 5.)
In any event, the Court finds that Plaintiffs’ request to produce the backup tapes
is premature. There is no discovery cutoff date in this case, and Plaintiffs are only
20% complete with their first level review of Defendants’ documents. Thus, PlainNo. 10 C 5711
37
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 38 of 53 PageID #:10432
tiffs should complete their review of Defendants’ ESI, including the information
produced from the additional custodians, before seeking to have archived backup
tapes restored. “The volume of—and the ability to search—much electronically
stored information means that in many cases the responding party will be able to
produce information from reasonably accessible sources that will fully satisfy the
parties’ discovery needs.” Fed. R. Civ. P. 26(b)(2), advisory committee’s note (2006).
Accordingly, Plaintiffs “should obtain and evaluate the information from such
sources before insisting that the responding party search and produce information
contained on sources that are not reasonably accessible.” Id.
If, at the appropriate time, Plaintiffs decide to pursue the backup tapes, the parties and their experts are urged to work together in a cooperative manner to determine the actual cost of restoring the archived media. If feasible and cost efficient,
sampling methods should be pursued. However, because each Defendant’s ESI storage protocoal is unique, it may be difficult or impossible to extrapolate any sampling results from one Defendant to the others.
Plaintiffs’ Motion to Compel Defendants to Produce Documents and Data from
All Reasonably Accessible Sources is denied without prejudice.
IV. CONCLUSION
Since its publication in 2009, more than 100 federal judges have endorsed the
Cooperation Proclamation. In an effort to aid courts and counsel, The Sedona Conference has published guides and toolkits to facilitate proportionality and coopera-
No. 10 C 5711
38
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 39 of 53 PageID #:10433
tion in discovery.13 Moreover, a number of courts have instituted model orders to
assist counsel in transitioning to the cooperative discovery approach.14
In pursuing a collaborative approach, some lessons have been learned. First, the
approach should be started early in the case. It is difficult or impossible to unwind
procedures that have already been implemented. Second, in multiple party cases
represented by separate counsel, it may be beneficial for liaisons to be assigned to
each party. Finally, to the extent possible, discovery phases should be discussed and
agreed to at the onset of discovery.
The Cooperation Proclamation calls for a “paradigm shift” in how parties engage
in the discovery process. The Sedona Conference, The Sedona Conference Cooperation Proclamation, 10 Sedona Conf. J. 331, 332–33 (2009). In some small way, it is
hoped that this Opinion can be of some help to others interested in pursuing a cooperative approach.15 The Court commends the lawyers and their clients for conducting their discovery obligations in a collaborative manner.
See The Sedona Conference, The Sedona Conference Cooperation Proclamation: Resources for the Judiciary (2011), available at http://www.thesedonaconference.org; The Sedona Conference, The Sedona Conference Cooperation Guidance for Litigators & In-House
Counsel (2011), available at http://www.thesedonaconference.org.
13
See, e.g., Seventh Circuit Electronic Discovery Pilot Program, Model Standing Order,
available at http://www.discoverypilot.com; Southern District of New York Pilot Program,
available at http://www.nysd.uscourts.gov; District of Delaware, Default Standard for Discovery, Including Discovery of Electronically Stored Information (“ESI”), available at
http://www.ded.uscourts.gov; see also David J. Waxse, Cooperation—What Is It and Why Do
It?, XVIII Rich. J. L. & Tech. 8 (2012), available at jolt.richmond.edu/v18i3/article8.pdf.
14
The Court notes that there are very few model agreements available for parties and
courts to follow. Accordingly, the Court suggests that The Sedona Conference and the Seventh Circuit Electronic Discovery Pilot Program serve as repositories for gathering ESI discovery agreements.
15
No. 10 C 5711
39
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 40 of 53 PageID #:10434
For the foregoing reasons, Plaintiffs’ Motion to Compel Defendants to Produce
Documents and Data from All Reasonable Accessible Sources [347] is DENIED
WITHOUT PREJUDICE; Plaintiffs’ Motion to Compel Temple-Inland to Include
Additional Document Custodians [366] is GRANTED IN PART AND DENIED IN
PART; Plaintiffs’ Motion to Compel International Paper Company to Include Additional Document Custodians [382] is GRANTED IN PART AND DENIED IN
PART; and Defendant Georgia-Pacific LLC’s Motion for Protective Order [373] is
GRANTED.
E N T E R:
Dated: September 28, 2012
Nan R. Nolan
United States Magistrate Judge
No. 10 C 5711
40
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 41 of 53 PageID #:10435
EXHIBIT A
Case:
Case:
1:10-cv-05711
1:10-cv-05711
Document
Document
#:#:
412
385
Filed:
Filed:
09/28/12
08/21/12
Page
Page
421of
of53
6 PageID
PageID#:9594
#:10436
UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF ILLINOIS
EASTERN DIVISION
KLEEN PRODUCTS LLC, et al.
individually and on behalf of all those
similarly situated,
Civil Case No. 1:10-cv-05711
Hon. Milton I. Shadur
Plaintiff,
Hon. Nan R. Nolan
v.
PACKAGING CORPORATION OF
AMERICA, et al,
Defendants.
STIPULATION AND ORDER RELATING TO ESI SEARCH
WHEREAS, in response to plaintiffs’ May 3, 2011 Request for Production of Documents
(the “First Request”) in this matter, defendants have collected electronic and hard copy
documents;
WHEREAS, defendants will continue to collect certain documents in response to the First
Request, including without limitation such documents that may be collected for review in
response to discovery conferences or pursuant to judicial orders arising out of motions brought
on the First Request (e.g., any documents that the Court orders included for review and
production based on the motions to be filed in July and August, 2012) (collectively all of the
documents that have been or will be collected in response to the First Request shall be referred to
in this Stipulation as the “First Request Corpus”);
WHEREAS, defendants have employed ESI vendors to process the electronic documents
contained within the First Request Corpus, and those ESI vendors have done so and continue to
do so;
1
Case:
Case:
1:10-cv-05711
1:10-cv-05711
Document
Document
#:#:
412
385
Filed:
Filed:
09/28/12
08/21/12
Page
Page
432of
of53
6 PageID
PageID#:9595
#:10437
WHEREAS, defendants have applied and continue to apply their ESI search methodology
(hereafter “Defendants’ ESI Methodology”), which was described during the evidentiary
hearings conducted on February 21, 2012 and on March 28, 2012 (the “Evidentiary Hearings”),
to those processed electronic documents within the First Request Corpus;
WHEREAS, plaintiffs have challenged Defendants’ ESI Methodology for the
identification of documents responsive to the First Requests and asked the Court to order
defendants to employ a “Content Based Advanced Analytics” (“CBAA”) approach, as defined by
plaintiffs, instead of Defendants’ ESI Methodology;
WHEREAS, defendants have opposed this challenge;
WHEREAS, the parties have made a substantial number of written submissions and oral
presentations to the Court with their views on this issue, and the Court held the Evidentiary
Hearings to address this dispute; and
WHEREAS, the parties continue to have a number of disputes, but in order to narrow the
issues, the parties have reached an agreement that will obviate the need for additional evidentiary
hearings on the issue of the technology to be used to search for documents responsive to the First
Requests.
THEREFORE, based upon and incorporating the foregoing, the parties, through their
respective counsel of record, hereby stipulate to and the Court hereby orders:
1.
Plaintiffs withdraw their demand that defendants apply CBAA to documents
contained within the First Request Corpus. Plaintiffs will not claim that defendants must use an
electronic search process other than Defendants’ ESI Methodology to locate relevant documents
contained in the First Request Corpus.
2
Case:
Case:
1:10-cv-05711
1:10-cv-05711
Document
Document
#:#:
412
385
Filed:
Filed:
09/28/12
08/21/12
Page
Page
443of
of53
6 PageID
PageID#:9596
#:10438
2.
As to any documents or ESI beyond the First Request Corpus, plaintiffs will not
argue or contend that defendants should be required to use or apply the types of CBAA or
“predictive coding” methodology and technology that were proposed by plaintiffs in connection
with the Evidentiary Hearings with respect to any requests for production served on any
defendant prior to October 1, 2013. With respect to any requests for production served on any
defendant on or after October 1, 2013, that requires the collection of documents beyond the First
Request Corpus, the parties will meet and confer regarding the appropriate search methodology
to be used for such newly collected documents. If the parties fail to agree on a search
methodology, either party may file a motion with the Court seeking resolution.
3.
Plaintiffs do not waive any additional objections they may have to defendants’
search methodology for the First Requests, including any additional objections relating to
defendants’ identification, collection, custodians, data sources, search terms, statistical validation,
review or production of documents, and that defendants’ objections to the First Request unduly
narrowed the scope of responsive documents, and defendants will not argue or contend that
plaintiffs, in whole or in part, have waived or otherwise failed to fully reserve such additional
objections by entering into this Stipulation. The Court has established briefing schedules and
other processes to resolve some of these issues by the end of September 2012.
4.
Defendants reserve all rights they currently have with respect to their position that
their document collection and production efforts met or exceeded relevant legal standards.
5.
In light of this agreement by the parties, the Evidentiary Hearings are
discontinued.
3
Case:
Case:
1:10-cv-05711
1:10-cv-05711
Document
Document
#:#:
412
385
Filed:
Filed:
09/28/12
08/21/12
Page
Page
454of
of53
6 PageID
PageID#:9597
#:10439
Stipulated and agreed:
By:
Daniel J. Mogin
Matthew T. Sinnott
THE MOGIN LAW FIRM, P.C.
707 Broadway, Suite 1000
San Diego, CA 92101
(619) 687-6611
[email protected]
[email protected]
By:
Michael J. Freed
Steven A. Kanner
FREED KANNER LONDON
& MILLEN LLC
2201 Waukegan Road, Suite 130
Bannockburn, IL 60015
(224) 632-4500
[email protected]
[email protected]
INTERIM CO-LEAD COUNSEL
FOR THE PROPOSED CLASS
INTERIM CO-LEAD COUNSEL
FOR THE PROPOSED CLASS
4
Case:
Case:
1:10-cv-05711
1:10-cv-05711
Document
Document
#:#:
412
385
Filed:
Filed:
09/28/12
08/21/12
Page
Page
465of
of53
6 PageID
PageID#:9598
#:10440
By:
Nathan P. Eimer
EIMER STAHL LLP
224 South Michigan Avenue, Suite 1100
Chicago, IL 60604-2516
(312) 660-7600
[email protected]
By:
Stephen R. Neuwirth
Deborah Brown
Sami H. Rashid
QUINN EMANUEL URQUHART &
SULLIVAN, LLP
51 Madison Avenue, 22nd Floor
New York, NY 10010
(212) 849-7000
[email protected]
[email protected]
[email protected]
James T. McKeown
FOLEY & LARDNER LLP
777 East Wisconsin Avenue
Milwaukee, WI 53202-5306
(414) 297-5530
[email protected]
James R. Figliulo
Stephanie D. Jones
FIGLIULO & SILVERMAN, P.C.
10 South LaSalle Street, Suite 3600
Chicago, IL 60603
(312) 251-4600
[email protected]
[email protected]
COUNSEL FOR DEFENDANT
INTERNATIONAL PAPER COMPANY
COUNSEL FOR DEFENDANT
GEORGIA-PACIFIC LLC
By:
Douglas J. Kurtenbach, P.C.
Daniel E. Laytin
Barack S. Echols
Leonid Feller
KIRKLAND & ELLIS LLP
300 North LaSalle
Chicago, IL 60654
(312) 862-2000
[email protected]
[email protected]
[email protected]
[email protected]
By:
R. Mark McCareins
Michael P. Mayer
James F. Herbison
WINSTON & STRAWN LLP
35 West Wacker Drive
Chicago, IL 60601
(312) 558-5600
[email protected]
[email protected]
[email protected]
COUNSEL FOR DEFENDANT
ROCKTENN CP, LLC
COUNSEL FOR DEFENDANT
PACKAGING CORPORATION OF
AMERICA
5
Case:
Case:
1:10-cv-05711
1:10-cv-05711
Document
Document
#:#:
412
385
Filed:
Filed:
09/28/12
08/21/12
Page
Page
476of
of53
6 PageID
PageID#:9599
#:10441
By:
Andrew S. Marovitz
Britt M. Miller
MAYER BROWN LLP
71 S. Wacker Drive
Chicago, IL 60606
(312) 782-0600
[email protected]
[email protected]
By:
Scott M. Mendel
John E. Susoreny
Lauren N. Norris
K&L GATES LLP
70 W. MADISON ST.
SUITE 3100
CHICAGO, IL 60602
(312) 372-1121
[email protected]
[email protected]
[email protected]
COUNSEL FOR DEFENDANT
TEMPLE-INLAND INC.
COUNSEL FOR DEFENDANTS
CASCADES, INC. AND NORAMPAC
HOLDING U.S. INC.
By:
.
David Marx Jr.
Jennifer S. Diver
Rachel Lewis
McDERMOTT WILL & EMERY LLP
227 W. Monroe Street
Chicago, IL 60606
(312) 372-2000
[email protected]
[email protected]
[email protected]
COUNSEL FOR DEFENDANT
WEYERHAEUSER COMPANY
E N T E R:
Dated: August 21, 2012
NAN R. NOLAN
United States Magistrate Judge
6
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 48 of 53 PageID #:10442
EXHIBIT B
Case:
Case:
1:10-cv-05711
1:10-cv-05711
Document
Document
#:#:
412
334
Filed:
Filed:
09/28/12
06/25/12
Page
Page
491of
of53
1 PageID
PageID#:6832
#:10443
Order Form (01/2005)
United States District Court, Northern District of Illinois
Name of Assigned Judge
or Magistrate Judge
Nan R. Nolan
CASE NUMBER
10 C 5711
CASE
TITLE
Sitting Judge if Other
than Assigned Judge
DATE
6/25/2012
Kleen Products, et al. vs. Packaging Corporation of America, et al.
DOCKET ENTRY TEXT
Rule 16 conference between Plaintiffs and Defendant Georgia-Pacific held on 5/31/2012. As stated below and
on the record, various agreements were reached.
O[ For further details see text below.]
Notices mailed by Judicial staff.
STATEMENT
At the 5/31/2012 Rule 16 conference, Plaintiffs and Defendant Georgia-Pacific each agreed to produce
the names and titles of all persons who have received a litigation hold notice related to this action, along with
the date that the notice was made. If the exact date of the notice cannot be readily determined, an
approximate range of dates will be provided. The production of the names of persons receiving the litigation
hold notice shall not constitute a waiver of the work-product doctrine or any other privilege. In addition, the
inclusion of a person on the list of those receiving the litigation hold notice shall not create any presumption,
or change the applicable standards for determining, that the person is an appropriate document custodian for
purposes of ESI and document searches, or is otherwise subject to discovery.
Defendant Georgia-Pacific also agreed to include James Hannan, Chief Executive Officer and President,
as a document custodian. Defendant Georgia-Pacific’s willingness to compromise, in the context of a meet
and confer supervised by the Court, on the foregoing issues of litigation hold recipients and making Hannan a
document custodian shall not be a factor in determining whether Hannan shall be subject to any further
discovery in this litigation.
10C5711 Kleen Products, et al. vs. Packaging Corporation of America, et al.
Page 1 of 1
Case: 1:10-cv-05711 Document #: 412 Filed: 09/28/12 Page 50 of 53 PageID #:10444
EXHIBIT C
Case:
Case:1:10-cv-05711
1:10-cv-05711Document
Document#:#:412
409Filed:
Filed:09/28/12
09/17/12Page
Page51
1 of 3
53PageID
PageID#:10389
#:10445
UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF ILLINOIS
EASTERN DIVISION
KLEEN PRODUCTS LLC, et al.
individually and on behalf of all those
similarly situated,
Civil Case No. 1:10-cv-05711
Hon. Milton I. Shadur
Plaintiff,
Hon. Nan R. Nolan
v.
PACKAGING CORPORATION OF
AMERICA, et al,
Defendants.
STIPULATION AND ORDER RELATING TO CUSTODIANS
In order to fully and finally resolve the parties’ custodian disputes as outlined in their
submissions to the Court, Plaintiffs and the undersigned defendants (“Defendants”) agree as
follows:
1.
Defendants have agreed, in connection with this Stipulation and Order and their
respective agreements with Plaintiffs, to add certain additional custodians to the document
review. Plaintiffs have no present intention of seeking additional custodians from any of the
Defendants.
2.
Plaintiffs shall not make a future request for custodians from Defendants unless
Plaintiffs’ counsel have a good faith belief, arising from a review of the documents produced in
this case, from depositions, or from some other identifiable source, that each requested individual
custodian has information, not unreasonably cumulative or duplicative, that is relevant to the
conduct alleged in Plaintiffs’ Complaint.
1
Case:
Case:1:10-cv-05711
1:10-cv-05711Document
Document#:#:412
409Filed:
Filed:09/28/12
09/17/12Page
Page52
2 of 3
53PageID
PageID#:10390
#:10446
3.
Plaintiffs will provide the Defendant from whom they are seeking any additional
custodians with a written summary explanation of their basis for requesting each additional
custodian, including a specific articulation of the basis for the request, such as Plaintiffs’ review
of documents and deposition testimony. In providing this information, Plaintiffs do not waive
any work product protection that might apply.
4.
If, after receiving the written explanation described in paragraph 3, an agreement
cannot be reached among the relevant parties, then either party may take the issue up with the
Court.
5.
Defendants reserve the right to oppose the addition of future custodians.
6.
Nothing in this agreement is intended to modify the applicability of the standard
set forth in Fed. R. Civ. P. 26(b)(2)(C) in the event that a future dispute arises with respect to
custodians.
STIPULATED AND AGREED:
By: Michael J. Freed
Michael J. Freed
Steven A. Kanner
FREED KANNER LONDON
& MILLEN LLC
2201 Waukegan Road, Suite 130
Bannockburn, IL 60015
(224) 632-4500
[email protected]
[email protected]
By: Daniel J. Mogin
Daniel J. Mogin
Matthew T. Sinnott
THE MOGIN LAW FIRM, P.C.
707 Broadway, Suite 1000
San Diego, CA 92101
(619) 687-6611
[email protected]
[email protected]
INTERIM CO-LEAD COUNSEL
FOR THE PROPOSED CLASS
INTERIM CO-LEAD COUNSEL
FOR THE PROPOSED CLASS
2
Case:
Case:1:10-cv-05711
1:10-cv-05711Document
Document#:#:412
409Filed:
Filed:09/28/12
09/17/12Page
Page53
3 of 3
53PageID
PageID#:10391
#:10447
By: R. Mark McCareins
R. Mark McCareins
Michael P. Mayer
James F. Herbison
WINSTON & STRAWN LLP
35 West Wacker Drive
Chicago, IL 60601
(312) 558-5600
[email protected]
[email protected]
[email protected]
By: Scott M. Mendel
Scott M. Mendel
John E. Susoreny
Lauren N. Norris
K&L GATES LLP
70 W. MADISON ST.
SUITE 3100
CHICAGO, IL 60602
(312) 372-1121
[email protected]
[email protected]
[email protected]
COUNSEL FOR DEFENDANT
ROCKTENN CP, LLC
COUNSEL FOR DEFENDANTS
CASCADES, INC. AND NORAMPAC
HOLDING U.S. INC.
By: Jennifer S. Diver
David Marx Jr.
Jennifer S. Diver
Rachel Lewis
McDERMOTT WILL & EMERY LLP
227 W. Monroe Street
Chicago, IL 60606
(312) 372-2000
[email protected]
[email protected]
[email protected]
COUNSEL FOR DEFENDANT
WEYERHAEUSER COMPANY
E N T E R:
Dated: September 17, 2012
NAN R. NOLAN
United States Magistrate Judge
3
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 1 of 21
1
2
3
4
5
6
7
8
9
UNITED STATES DISTRICT COURT
10
SOUTHERN DISTRICT OF CALIFORNIA
11
12
13
GABRIEL TECHNOLOGIES
CORPORATION and TRACE
TECHNOLOGIES, LLC,
14
Plaintiff,
v.
15
16
QUALCOMM INCORPORATED,
SNAPTRACK, INC. and NORMAN
KRASNER,
17
Defendants.
18
)
)
)
)
)
)
)
)
)
)
)
)
)
Civil No. 08cv1992 AJB (MDD)
ORDER GRANTING IN PART AND
DENYING IN PART DEFENDANTS’
MOTION FOR ATTORNEYS’ FEES AS
TO PLAINTIFFS AND NON-PARTY
WANG, HARTMAN, GIBBS & CAULEY,
PLC
(Doc. No. 332)
19
Before the Court is Defendants’ Motion for Attorneys’ Fees as to Plaintiffs and Non-Party
20
Wang, Hartman, Gibbs & Cauley, PLC (“WHGC”), Plaintiffs’ local counsel.1 (Doc. No. 332.)
21
Plaintiffs and WHGC filed separate opposition briefs to Defendants’ motion. (Doc. No. 355 and 354,
22
respectively.) On January 17, 2013, the Court held a motion hearing and took the matter under
23
24
25
1
26
27
28
Defendants’ Motion for Attorneys’ Fees originally sought fees from Plaintiffs’ lead counsel,
Hughes, Hubbard & Reed (“Hughes Hubbard”) as well. However, Defendants and Hughes Hubbard
reached a settlement on the issue of attorneys’ fees prior to the hearing on this matter. Accordingly,
Defendants and Hughes Hubbard filed a joint request to withdraw Defendants’ Motion for Attorneys’
Fees as to Hughes Hubbard. (Doc. No. 366.) Pursuant to the settling parties’ agreement, the terms of
the settlement are confidential, and the Court declined the parties’ offer to divulge the settlement amount
for the limited purpose of in camera review.
1
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 2 of 21
1
submission for further review. (Doc. No. 367.) For the reasons set forth below, Defendants’ motion is
2
GRANTED IN PART and DENIED IN PART.
3
4
Procedural Background
On October 24, 2008, Plaintiffs filed this lawsuit, which initially contained eleven causes of
5
action against Defendants, including: (1) breach of the 1999 license agreement; (2) breach of the 2006
6
license agreement; (3) fraud/fraudulent inducement; (4) tortious interference with contract; (5)
7
correction of inventorship; (6) declaration of ownership; (7) equitable patent infringement; (8)
8
misappropriation of trade secrets pursuant to California's Uniform Trade Secrets Act (“CUTSA”); (9)
9
conversion; (10) unfair competition pursuant to Cal. Bus. & Prof. Code section 17200; and (11) unjust
10
11
enrichment. (Doc. No. 1.) Plaintiffs sought over $1 billion in damages. (Id.)
On September 3, 2009, (Doc. No. 35), the Court granted in part and denied in part the Defen-
12
dants' motion to dismiss, and ordered Plaintiffs to file an amended complaint by September 14, 2009. In
13
the September 3, 2009 Order, Judge Anello dismissed Plaintiffs' first, fourth, seventh, ninth, tenth, and
14
eleventh causes of action with prejudice and dismissed Plaintiffs' third cause of action with leave to
15
amend. On September 14, 2009, Plaintiffs filed a Second Amended Complaint, which contained an
16
amended fraud and fraudulent inducement claim and incorrectly re-alleged claims that were previously
17
dismissed with prejudice. (Doc. No. 36.) On October 8, 2009, the Court held a telephonic status
18
conference with the parties and granted Plaintiffs leave to file a Third Amended Complaint that did not
19
include the claims previously dismissed with prejudice. (Doc. No. 39.) On October 9, 2009, Plaintiffs
20
filed their Third Amended Complaint, which included the amended fraud and fraudulent inducement
21
claims. (Doc. No. 40.) Defendants again moved to dismiss Plaintiffs' fraud claim. (Doc. No. 41.) On
22
December 14, 2009, the Court granted Defendants' motion and dismissed Plaintiffs' fraud and fraudulent
23
inducement claim with prejudice. (Doc. No. 48.) On January 11, 2010, Plaintiffs filed their Fourth
24
Amended Complaint ("FAC"), which did not contain any causes of action alleging fraud. (Doc. No. 53.)
25
On August 13, 2010, the Court denied Plaintiffs' motion for leave to file a fifth amended complaint,
26
which sought to add a fraudulent concealment claim. (Doc. No. 104.)
27
28
On July 2, 2010, Defendants filed a motion for bond pursuant to California Code of Civil
Procedure Section 1030. (Doc. No. 81.) Having reviewed all of the relevant evidence and the parties’
2
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 3 of 21
1
arguments, the Court granted in part and denied in part Defendants’ motion for a bond on September 20,
2
2010. (Doc. No. 110.) The Court reduced the amount of the requested bond from $1,291,580
3
($1,000,000 in fees and $291,580 in costs) to $800,000 in light of Plaintiff’s financial condition at the
4
time. (Id. at 23.) Pursuant to this order, Plaintiffs posted the $800,000 bond in order to avoid dismissal
5
of their claims. (Doc. No. 121.)
6
On September 27, 2011, Defendants filed a partial motion for summary judgment with regard to
7
Plaintiffs' claims for misappropriation of trade secrets and breach of contract, arguing the claims were
8
barred by the statute of limitations. (Doc. No. 188.) On March 13, 2012, the Court granted Defendant's
9
motion for summary judgment as to Plaintiff's eighth claim for misappropriation of trade secrets and
10
second claim for breach of contract on ground one as to trade secrets only and grounds two, three, four,
11
and five. (Doc. No. 252.) The Order denied partial summary judgment as to ground one and ground six
12
of Plaintiffs' second claim for breach of contract. (Id.)
13
On August 10, 2012, Defendants filed a motion for summary judgment as to Plaintiffs' three
14
remaining claims, which were as follows: (1) Breach of the Amended and Restated License Agreement
15
in Ground One and Ground Six (COUNT TWO); (2) Correction of Inventorship (pursuant to 35 U.S.C.
16
§ 256) (COUNT FIVE); and (3) Declaratory Judgment of Ownership Interest in the Patents (pursuant to
17
28 U.S.C. § 2201) (COUNT SIX). (Doc. No. 296.) On October 1, 2012, the Court granted Defendants'
18
motion for summary judgment and directed the Clerk to enter judgment in favor of the Defendants.
19
(Doc. No. 328.) Plaintiffs are appealing the judgment to the Federal Circuit. (Doc. No. 340.)
20
Defendants filed the instant motion for attorneys' fees on October 12, 2012. (Doc. No. 351.)
21
I. Defendants’ Motion for Attorneys’ Fees as to Plaintiffs
22
As support for the requested attorneys’ fees award, Defendants contend that Plaintiffs pursued
23
objectively baseless patent and misappropriation claims in bad faith. Accordingly, Defendants seek
24
attorneys’ fees with regard to Plaintiffs’ patent claims under 35 U.S.C. § 285 and with regard to
25
Plaintiffs’ misappropriation claims under Section 3426.4 of CUTSA. The Court addresses each
26
statutory basis for awarding attorneys’ fees in turn.
27
28
3
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 4 of 21
1
A. Attorneys’ Fees Pursuant to 35 U.S.C. § 285
2
Under 35 U.S.C. § 285, a “court in exceptional cases may award reasonable attorney fees to the
3
prevailing party” in a patent case. Defendants argue that Plaintiffs pursued frivolous patent claims and
4
engaged in litigation misconduct such that attorneys’ fees are warranted under 35 U.S.C. § 285.
5
Once it is determined that the party seeking fees is a prevailing party, determining whether to
6
award attorneys' fees under 35 U.S.C. § 285 is a two-step process. Highmark, Inc. v. Allcare Health
7
Mgmt. Sys., Inc., 687 F.3d 1300, 1308 (Fed. Cir. 2012) (citing Forest Labs., Inc. v. Abbott Labs., 339
8
F.3d 1324, 1327–28 (Fed. Cir. 2003)). First, a prevailing party must establish by clear and convincing
9
evidence that the case is “exceptional.” Id. (citing Forest Labs, Inc., 339 F.3d at 1327). “A case is
10
exceptional under Section 285 if there has been some inappropriate conduct relating to the matter in
11
litigation.” Raylon, LLC v. Complus Data Innovations, Inc., 700 F.3d 1361, 1370 (citing Brooks
12
Furniture Mfg., Inc. v. Dutailier Int'l, Inc., 393 F.3d 1378, 1381 (Fed. Cir. 2005). Cases may be
13
considered “exceptional” based upon a party’s frivolous claims, inequitable conduct before the Patent
14
and Trademark Office, or misconduct during litigation. Id. (citing Beckman Instruments, Inc. v. LKB
15
Produkter AB, 892 F.2d 1547, 1551 (Fed. Cir. 1989)). Second, “if the case is deemed exceptional, a
16
court must determine whether an award of attorneys' fees is appropriate and, if so, the amount of the
17
award.” Id. (citing Forest Labs., 339 F.3d at 1328). “[T]he amount of the attorney fees [awarded]
18
depends on the extent to which the case is exceptional.” Special Devices, Inc. v. OEA, Inc., 269 F.3d
19
1340, 1344 (Fed. Cir. 2001). The purpose of section 285, unlike that of Rule 11, is not to control the
20
local bar's litigation practices—which the district court is better positioned to observe—but is remedial
21
and for the purpose of compensating the prevailing party for the costs it incurred in the prosecution or
22
defense of a case where it would be grossly unjust, based on the baselessness of the suit or because of
23
litigation or Patent Office misconduct, to require it to bear its own costs. See Badalamenti v. Dunham's,
24
Inc., 896 F.2d 1359, 1364 (Fed. Cir. 1990); Cent. Soya Co., Inc. v. Geo. A. Hormel & Co., 723 F.2d
25
1573, 1578 (Fed. Cir. 1983).
26
27
28
1. Exceptional Case
As noted above, the Court has either dismissed or granted summary judgment in Defendants’
favor as to all of Plaintiffs’ claims, and thus considers Defendants to be the prevailing party in this
4
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 5 of 21
1
action. Therefore, the first step when considering whether to award attorneys’ fees under Section 285 is
2
to determine whether the case is exceptional. Here, Defendants argue that this case is exceptional based
3
upon Plaintiffs’ frivolous claims and misconduct during litigation. Defendants offer the following as the
4
primary evidence of the frivolousness of Plaintiffs’ claims: (1) Plaintiffs originally alleged patent
5
violations of 92 patents and ultimately reduced their claims down to violations of only 16 patents; (2)
6
the majority of Plaintiffs’ claims were dismissed at the pleadings stage; (3) Plaintiffs’ remaining claims
7
did not survive summary judgment because the Court found that Plaintiffs did not have sufficient
8
evidence to create a triable issue of fact on the issue of inventorship; (4) emails between Plaintiffs’
9
former employees suggest that Plaintiffs knew they had no case and pursued the claims anyway; and (5)
10
Judge Anello required Plaintiffs’ to post an $800,000 in order to proceed with the case after finding “a
11
strong likelihood Defendants will ultimately prove this case is exceptional, and attorneys’ fees will be
12
warranted at the conclusion of the litigation.”
13
First, the Court will determine whether the case is exceptional based upon the alleged frivolous-
14
ness of Plaintiffs’ claims. Under Section 285, sanctions may be imposed for frivolous claims only if two
15
separate criteria are satisfied: (1) the litigation is brought in subjective bad faith, and (2) the litigation is
16
objectively baseless. Highmark, Inc. v. Allcare Health Mgmt. Sys., 687 F.3d 1300, 1308 (Fed. Cir.
17
2012) (citing Brooks Furniture Mfg., Inc. v. Dutailier Int'l, Inc., 393 F.3d 1378, 1381 (Fed. Cir. 2005)).
18
In Highmark, Inv. v. Allcare Health Mgmt. Sys., the Federal Circuit detailed the analysis afforded to
19
these two factors as follows:
20
21
22
23
24
25
26
27
The requirement that the litigation be objectively baseless “does not depend on the state of
mind of the [party] at the time the action was commenced, but rather requires an objective
assessment of the merits.” Brooks Furniture Mfg. 393 F.3d at 1382. “To be objectively
baseless, the infringement allegations must be such that no reasonable litigant could
reasonably expect success on the merits.” Dominant Semiconductors Sdn. Bhd. v. OSRAM
GmbH, 524 F.3d 1254, 1260 (Fed. Cir. 2008) (internal quotation marks omitted).
Furthermore, even if the claim is objectively baseless, it must be shown that lack of objective
foundation for the claim “was either known or so obvious that it should have been known”
by the party asserting the claim. In re Seagate Tech., LLC, 497 F.3d 1360, 1371 (Fed. Cir.
2007); see also iLOR, LLC v. Google, Inc., 631 F.3d 1372, 1377 (Fed. Cir. 2011). This is
known as the subjective prong of the inquiry. This same objective/subjective standard
applies for both patentees asserting claims of infringement and alleged infringers defending
against claims of infringement. See iLOR, 631 F.3d at 1377.
Id. at 1308-09.
28
5
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 6 of 21
1
Having reviewed the allegations of both parties and taking particular note of Judge Anello’s
2
analysis in his bond decision, the Court concludes that Plaintiffs’ claims in this case were objectively
3
baseless and brought in subjective bad faith. Here, Plaintiffs could not have reasonably expected
4
success on the merits of the patent claims without knowing the identify of the allegedly omitted
5
inventors. As this Court noted in its summary judgment order, there is a “ ‘presumption that the
6
inventors named on an issued patent are correct, so misjoinder of inventors must be proven by clear and
7
convincing evidence.’ ” (Doc. No. 328 at 6 (quoting Fina Oil and Chem. Co. v. Ewen, 123 F.3d 1466,
8
1472 (Fed. Cir. 1997)). At no point during this case has Plaintiff been able to offer the evidence
9
necessary to overcome this presumption. It is apparent that Plaintiffs did not know the identity of the
10
11
allegedly omitted inventors when they filed this action in 2008 or at any later point in the case.
Close to two years after the case’s initial filing, Judge Anello described the obvious lack of an
12
objective foundation for Plaintiffs’ claims in his bond order: “The parties began working together
13
approximately a decade ago, Plaintiffs assert they suspected wrongdoing in approximately 2007, and
14
they have been investigating their claims for several years. By now, Plaintiffs should have more than
15
mere allegations to support their theory.” (Doc. No. 110 at 22.) Despite this warning, Plaintiffs
16
prolonged the litigation for another two years. In order to survive summary judgment, Plaintiffs
17
attempted late in the litigation to finally identify the allegedly omitted inventors; however, they were
18
never able to find individuals that would take credit for inventing any of the relevant patents or profess
19
knowledge of specifics relating to the patents. Thus, the Court found in its summary judgment order
20
that Plaintiffs had failed to create a triable issue of fact regarding Plaintiffs’ inventorship claims, proving
21
Judge Anello’s earlier characterization of Plaintiffs’ patent claims to be correct. Throughout this case,
22
Plaintiffs pursued inventorship claims despite their inability to identity the allegedly omitted inventors.
23
Moreover, Plaintiffs continued to pursue these claims after Judge Anello pointed out the significant lack
24
of evidentiary support. Based on the totality of the record, Plaintiffs’ claims were objectively baseless.
25
For similar reasons, the Court finds that Plaintiffs’ brought these claims in subjective bad faith.
26
It was readily apparent to Judge Anello in September 2010 that Plaintiffs’ had maintained this action for
27
slightly less than two years with nothing more than “mere allegations” to support their claims. Several
28
emails between Gabriel employees and former employees confirm this conclusion and suggest that
6
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 7 of 21
1
Plaintiffs knew they lacked the requisite evidence and opted to pursue their claims nonetheless.2 As
2
they did at the bond hearing before Judge Anello, Plaintiffs attempt to dismiss these emails as nothing
3
more than disgruntled employees’ expressions of frustration with management and the inability to
4
recover necessary documents from previous counsel, Munck Carter PC (“Munck”). (See Doc. No. 110
5
at 22; Pls. Opp., Doc. No. at 9-11). However, Judge Anello disagreed with Plaintiffs interpretation and
6
determined that the emails in combination with the lack of evidence supporting Plaintiffs’ claims
7
suggested that the case would ultimately be considered exceptional. (Doc. No. 110 at 22.) This Court
8
agrees. While the emails certainly express frustration towards Gabriel and Munck, the repeated
9
references to the utter lack of a case suggest that, not only did Plaintiff’s not have the necessary
10
evidence to bring their claims against Defendants, they were aware of the evidentiary deficiencies
11
during the early stages of litigation. (Pls. Opp., Doc. No. 355 at 10.) Additionally, Plaintiffs’ assertion
12
that the documents retained by Munck provided the “evidentiary support for its claims” is belied by the
13
record. (Doc. No. 355 at 10.) At the hearing regarding the instant motion, counsel revealed that
14
Plaintiffs reacquired these documents from Munck in early 2010. Despite Plaintiffs’ recovery of these
15
purportedly crucial documents, Plaintiffs could only produce “mere allegations” to support their claims
16
at the time of the bond hearing in September 2010. (See Doc. No. 110 at 22.) This suggests that the
17
documents were not the evidentiary key to Plaintiffs’ claims that Plaintiffs represents them to be.
18
On the whole, the Court concludes that the lack of evidence supporting Plaintiffs’ patent claims
19
was so obvious that it should have been known by Plaintiffs. Significantly, the emails suggest that
20
Plaintiffs knew there was no evidentiary basis for their claims in early 2010, and Plaintiffs certainly had
21
notice that their claims lacked substantiation following Judge Anello’s bond order in September 2010.
22
Accordingly, the Court concludes that Plaintiffs brought objectively baseless patent claims in subjective
23
bad faith.
24
Based simply on the frivolous nature of Plaintiffs’ claims, the Court is inclined to classify the
25
case as exceptional under 35 U.S.C. § 285. The Court is further persuaded by Defendants’ arguments
26
regarding Plaintiffs’ litigation misconduct. Quite apart from a finding of frivolous claims, “an
27
2
28
Plaintiffs object to the Court’s consideration of these emails; however, the emails have been
considered by the Court previously with regard to the bond motion and the Court finds them relevant to
its consideration of the instant motion. Accordingly, Plaintiffs’ evidentiary objections are overruled.
7
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 8 of 21
1
exceptional case finding can also be supported by litigation misconduct.” Highmark, Inc., 687 F.3d at
2
1315 (citing MarcTec, LLC v. Johnson & Johnson, 664 F.3d 907, 919 (Fed. Cir. 2012)). Litigation
3
misconduct involves unethical or unprofessional conduct by a party or his attorneys, but may also
4
include “advancing frivolous arguments during the course of the litigation or otherwise prolonging
5
litigation in bad faith.” Id. The frivolous arguments must be shown to be objectively unreasonable at
6
the time they were made. Id. (citing DePuy Spine, Inc. v. Medtronic Sofamor Danek, Inc., 567 F.3d
7
1314, 1339 (Fed. Cir. 2009)).
8
As discussed above, Plaintiffs brought and maintained claims without knowing the identity of
9
the allegedly omitted inventors, the most basic prerequisite for a successful correction of inventorship
10
patent claim. Viewing the entirety of Plaintiffs’ allegations, all of Plaintiffs’ causes of action relate in
11
some way to the alleged theft of Plaintiffs’ patents; however, Plaintiffs cannot provide sufficient
12
evidence to establish the rightful inventors of the patents. The Court is particularly struck by Plaintiffs’
13
decision to pursue their claims further following Judge Anello’s warning that the case would likely be
14
found exceptional based on the evidence before him at the bond hearing. Plaintiffs nevertheless
15
continued with their claims despite Judge Anello’s clear indication that there was a serious lack of
16
evidence regarding Plaintiffs’ patent claims. Over time, Judge Anello’s assessment proved correct and
17
Plaintiffs’ claims failed for the reasons he anticipated. The Court finds that Plaintiffs’ decision to
18
pursue their patent claims without knowing the identity of the allegedly omitted inventor for such a long
19
period of time, and particularly after Judge Anello’s bond order, constitutes litigation misconduct.
20
Furthermore, Defendants’ make a persuasive case for finding litigation misconduct based upon
21
Plaintiffs’ less-than-honest actions throughout the case.3 Having already classified the case as
22
23
24
25
26
27
28
3
Defendants suggest that the following actions should be considered litigation misconduct: (1)
Plaintiffs’ filed sham affidavits in order to avoid summary judgment; (2) Plaintiffs served incomplete
and unverified interrogatory responses without basic investigation regarding the actual identity of the
allegedly omitted inventors; (3) Plaintiffs granted a contingent interest in the litigation to a percipient
witness; (4) Plaintiffs invited a biased amicus brief that was filed on their behalf; (5) Plaintiffs pursued
inventorship claims that had no evidentiary support; (6) Plaintiffs pursued claims against Defendants in
order to garner outside investments in the litigation; (7) Plaintiffs misled the Court with regard to the
location of their headquarters at the time of the bond hearing; (8) Plaintiffs attempted to reallege fraud
claims that had previously been dismissed with prejudice; and (9) Plaintiffs alleged 92 patent claims
initially but required extensive discovery before narrowing their patent claims to 16. While the Court
does not necessarily find that each of these items constitutes litigation misconduct of its own accord,
items 2, 5, 6, 7, 8, and 9 raise significant concerns when considering the nature of Plaintiffs’ conduct
8
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 9 of 21
1
exceptional based upon Plaintiffs’ bad faith in pursuing objectively baseless claims that had no
2
reasonable expectation of success, the Court need not address these individual allegations of litigation
3
misconduct, but notes that the nature of the referenced incidents does not weigh in Plaintiffs’ favor. In
4
sum, the Court finds that Plaintiffs brought frivolous claims and engaged in litigation misconduct such
5
that this case is exceptional within the meaning of 35 U.S.C. § 285.
6
7
2. Appropriate Amount of Attorneys’ Fees
Having deemed the case exceptional, the second step of the analysis requires the Court to
8
determine whether an award of attorneys’ fees is appropriate, and, if so, the amount of the award. See
9
Forest Labs., 339 F.3d at 1328. The authority to award attorneys’ fees found in 35 U.S.C. § 285 is
10
limited to the patent side of the case. Monolith Portland Midwest Co. v. Kaiser Aluminum & Chemical
11
Corp., 405 F.2d 288, 297-98 (9th Cir. 1969); see also Stickle v. Heublein, Inc., 716 F.2d 1550, (Fed. Cir.
12
1983). “Where an action combines both patent and nonpatent claims, the nonpatent issues may in some
13
instances be so intertwined with the patent issues that the evidence would, in large part, be material to
14
both types of issues.” Stickle, 716 F.2d at 1564.
15
In this instance, the Court finds that attorneys’ fees are warranted under Section 285. Plaintiffs
16
initially alleged 92 patent violations without the requisite evidence to support their claims, never
17
acquired the requisite evidence, and thus caused Defendants to incur several years worth of attorneys’
18
fees in order to oppose Plaintiffs’ unsubstantiated claims. For this reason, the Court finds that attorneys’
19
fees under 35 U.S.C. § 285 are appropriate in this instance and will now consider how to properly limit
20
the award of fees to those relating to the patent side of the case as required by the statute.
21
Plaintiffs’ First Amended Complaint (“FAC”) contains eleven causes of action. (Doc. No. 14.)
22
Plaintiffs’ fifth, sixth, and seventh causes of action for correction of inventorship, declaratory judgment
23
of ownership interest in the patents, and equitable patent infringement are patent claims for which
24
attorneys’ fees may be imposed under 35 U.S.C. § 285. (Doc. No. 14 at 30-32.) However, having
25
reviewed the allegations offered as support for the non-patent causes of action, each of Plaintiffs’ claims
26
depends upon Plaintiffs’ patent-related allegations. For reasons set forth in greater detail below, the
27
Court finds that Defendants are entitled to attorneys’ fees under CUTSA for Plaintiffs’ misappropriation
28
throughout this action.
9
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 10 of 21
1
claims, which means that the attorneys’ fees associated with Plaintiffs’ eighth cause of action need not
2
be separated from the patent fees as they are both independently recoverable.4 As to Plaintiffs’
3
remaining non-patent causes of action, the Court finds that these claims are so intertwined with
4
Plaintiffs’ patent claims and patent-related evidence that the same evidence is material to all of the
5
issues.5 Therefore, the fees associated with these claims may be properly awarded under 35 U.S.C. §
6
285 as they are virtually inseparable from those fees accrued with regard to the purely patent claims.
7
Accordingly, the Court awards attorneys’ fees associated with Plaintiffs’ patent and patent-
8
related causes of action pursuant to 35 U.S.C. § 285. The Court will consider the reasonable amount of
9
the award following the attorneys’ fees analysis under CUTSA.
10
B. Attorneys’ Fees Pursuant to CUTSA, Cal. Civ. Code § 3426.4
11
Under Section 3426.4 of the California Uniform Trade Secret Act (“CUTSA”), courts may
12
award reasonable attorneys’ fees to the prevailing party if a claim of misappropriation is made in bad
13
faith. Based upon much of the same evidence argued with regard to 35 U.S.C. § 285, Defendants
14
15
16
17
18
19
4
The Court also notes that the discovery related to Plaintiffs’ misappropriation claims overlaps
significantly with Plaintiffs’ patent claims. Plaintiffs misappropriation claims suggest, among other
things, that Defendants incorporated the misappropriated trade secrets into their patent applications. As
a result, the discovery related to Plaintiffs’ misappropriation claims required further investigation into
Defendants’ patents. (See Doc. No. 160 at 1-2 (directing parties to designate up to two of Defendants’
patents identified in the Fourth Amended Complaint that are allegedly based upon Plaintiffs’ trade
secrets and directing Plaintiffs to identify its trade secrets that are allegedly incorporated in the
designated patents).)
5
20
21
22
23
24
25
26
27
28
Plaintiffs’ first cause of action alleges that SnapTrack breached the 1999 License Agreement
when it, among other things “took ownership of Locate’s patents” and “filed patent applications and
patents without listing Locate as an assignee or Locate personnel as inventors.” (Doc. No. 14 at 27, ¶
132.) Similarly, Plaintiffs’ second cause of action makes identical allegations in support for Plaintiffs
alleged breach of the 2006 license agreement claim. (Id. at 28, ¶ 138.) Plaintiffs’ third cause of action
alleges fraud and fraudulent inducement based upon SnapTrack “misappropriating Locate's technology
and filing patents and patent applications incorporating jointly-owned Program Technology as well as
Locate's technology.” (Id. at 29, ¶ 142.) In Plaintiffs’ fourth cause of action, Plaintiffs allege tortious
interference with contract which alleges that Krasner and Qualcomm “intentionally caused SnapTrack to
breach the License Agreement” and, as discussed with the first and second cause of action, the breach of
the license agreement claims are based in part upon the alleged patent violations. (Id. at 30.) Plaintiffs’
conversion claim in the ninth cause of action alleges that Defendants “--through their patent filings and
other acts -- converted that technology, confidential information, and Program Technology for
themselves.” (Id. at 34, ¶ 180.) Plaintiffs’ tenth cause of action alleges unfair competition based upon
Defendants’ misappropriation of Locate’s technology and filing “patents and patent applications
incorporating Locate’s technology and jointly owned Program Technology.” (Id. at 35, ¶ 184.) Lastly,
Plaintiffs’ eleventh cause of action claims unjust enrichment based upon Defendants misappropriation of
Locate’s technology “by applying for and obtaining patents based on that technology.” (Id. at 35, ¶
191.)
10
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 11 of 21
1
contend that Plaintiffs pursued frivolous misappropriation claims and engaged in litigation misconduct
2
such that attorneys’ fees are warranted under Section 3426.4 of CUTSA.
3
4
1. Legal Standard
Under Section 3426.4, a prevailing party may recover reasonable attorneys' fees if “a claim of
5
misappropriation is made in bad faith.” Cal. Civ. Code § 3426.4. Because CUTSA does not provide a
6
definition of “bad faith” to be used in the context of trade secret misappropriation, California courts
7
have developed a two-pronged standard for the evaluation of such claims. See SASCO v. Rosendin
8
Elec., Inc., 207 Cal. App. 4th 837, 834 (Cal. Ct. App. 2012); Smith v. Selma Cmty. Hosp., 188 Cal. App.
9
4th 1, 34 (Cal. Ct. App. 2010); Gemini Aluminum Corp. v. CA Custom Shapes, Inc., 95 Cal. App. 4th
10
1249, 1261 (Cal. Ct. App. 2002). The party seeking an award of attorney's fees under Section 3426.4
11
must show (1) the objective speciousness of opposing party's claim, and (2) the subjective bad faith of
12
the opposing party in bringing or maintaining the action, that is, for an improper purpose. Gemini
13
Aluminum Corp., 95 Cal. App. 4th at 1261.
14
With regard to the first prong, objective speciousness “exists where the action superficially
15
appears to have merit but there is a complete lack of evidence to support the claim.” FLIR Sys., Inc. v.
16
Parrish, 174 Cal. App. 4th 1270, 1276 (Cal. Ct. App. 2009). Objective speciousness may be shown by,
17
among other factors, demonstrating that there was no misappropriation or threatened misappropriation
18
or that the opposing party could not have suffered any economic harm. Id. The second prong requiring
19
subjective bad faith is satisfied when it may be inferred from the evidence that a party “intended to
20
cause unnecessary delay, filed the action to harass [the opposing party], or harbored an improper
21
motive.” FLIR Sys., Inc., 174 Cal. App. 4th at 1278 (citing Gemini, 95 Cal. App. 4th at 1263). “Similar
22
inferences may be made where the plaintiff proceeds to trial after the action’s fatal shortcomings are
23
revealed by opposing counsel.” Id. (citing Gemini, 95 Cal. App. 4th at 1264).
24
25
2. Analysis
Based on its review of the record, the Court concludes Plaintiffs’ misappropriation claims were
26
objectively specious and Plaintiffs brought and maintained their claims in subjective bad faith. Here,
27
Plaintiffs did not simply fail to articulate the trade secrets with particularity as an initial matter. Rather,
28
Plaintiffs made seven failed attempts to articulate their trade secrets, requiring Defendants’ to participate
11
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 12 of 21
1
in extensive discovery and motion practice before Magistrate Judge Porter and then Magistrate Judge
2
Dembin. (See Doc. No. 229.) In Magistrate Judge Dembin’s final order describing the lengthy
3
procedural history regarding Plaintiffs’ trade secret designations, he notes that “[d]espite seven attempts,
4
Plaintiffs have failed to put forth a designation that provides reasonable notice of the issues for trial and
5
assist the Court in defining the scope of discovery.” (Id. at 8.) Magistrate Judge Dembin found that
6
Plaintiffs’ trade secret designation number one lacked “adequate specificity,” and this failure
7
“condemn[ed] the designation to intolerable vagueness.” (Id. at 8.) Trade secrets number two through
8
ten fared no better as they contained “even less information and more ambiguous terms.” (Id. at 9.)
9
Despite both Magistrate Judges concluding that “Plaintiffs had failed to designate their trade secrets
10
with sufficient particularity to justify compelling discovery from Defendants” and Judge Anello
11
predicting the misappropriation claims were likely barred by the statute of limitations in addition to
12
lacking the requisite evidentiary connection, Plaintiffs continued to pursue their misappropriation
13
claims.
14
Plaintiffs contend that failing to sufficiently articulate a trade secret for discovery purposes does
15
not constitute objective speciousness under CUTSA and cites Pixion, Inc. v. PlaceWare Inc. as support.
16
2005 WL 3955890, *2-3 (N.D. Cal. Apr. 22, 2005). In Pixion, the defendant argued that, among other
17
things, plaintiff’s failure to clearly articulate trade secrets at the pleading stage constituted subjective
18
misconduct. Id. The court in that case concluded that plaintiff’s decision to maintain its position
19
regarding the adequacy of its trade secret disclosures despite the defendants’ evidence otherwise was
20
“hardly evidence of improper motive.” Id. at *3.
21
record was “entirely devoid of evidence of subjective bad faith on plaintiff’s part.” Id. For example,
22
several of plaintiff’s trade secrets survived summary judgment, and the court noted that the trade secrets
23
claims arose out of the same set of facts as plaintiff’s breach of contract claim, which was clearly not
24
specious. Id. However, there are several reasons why the misappropriation claims considered in Pixion
25
are distinguishable from Plaintiff’s misappropriation claims here. As discussed above, Plaintiffs did not
26
simply fail to articulate trade secret claims at the pleading stage; rather, Plaintiffs unreasonably pursued
27
trade secret claims after having been notified of their evidentiary deficiencies by two magistrate judges
28
and one district court judge. Additionally, Defendants argued, and Judge Anello agreed, that Plaintiffs’
Notably, the court in Pixion also found that the
12
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 13 of 21
1
claims were likely barred by the statute of limitations. Despite these significant warnings, Plaintiffs
2
decided to go forward with their misappropriation claims without remedying the deficiencies identified
3
by Defendants, Judge Anello, and Magistrate Judge Dembin. As predicted by Judge Anello, the Court
4
ultimately found that the misappropriation claims were barred by the statute of limitations and granted
5
summary judgment in Defendants’ favor. Additionally, unlike the plaintiff in Pixion that ultimately had
6
some success on claims that were clearly brought in good faith, the Court has concluded that Plaintiffs’
7
patent claims were objectively baseless and brought in subjective bad faith, and Plaintiffs’ misappropria-
8
tion claims arise out of the same set of facts as the patent claims. For these reasons, Pixion does not
9
compare favorably to the facts of this case.
10
Based on the facts set forth above, the Court concludes that Plaintiffs’ misappropriation claims
11
were objectively specious and maintained in subjective bad faith. Plaintiffs received more than
12
sufficient notice regarding the flaws in their claims from Defendants, Magistrate Judge Dembin, and
13
Judge Anello. Knowing the specific shortcomings of their misappropriation claims, Plaintiffs made the
14
decision to go forward despite their inability to sufficiently articulate the alleged trade secrets or
15
overcome Defendants’ statute of limitations argument.6 Accordingly, the Court finds that attorneys’
16
fees are warranted under Section 3426.4 of CUTSA.
17
C. Reasonable Amount of Award as to Plaintiffs
18
As discussed above, attorneys’ fees are warranted with regard to Plaintiffs’ patent and patent-
19
related claims under 35 U.S.C. § 285 and with regard to Plaintiffs’ misappropriation claims under
20
Section 3426.4 of CUTSA. Insomuch as each one of Plaintiffs’ claims is covered under the ambit of one
21
of the statutes, the Court need not apportion Defendants’ attorneys’ fees to the particular cause of action
22
it addressed. As such, the Court will determine the reasonable amount of the award using the lodestar
23
determination.
1. Legal Standard
24
25
26
6
27
28
“ ‘Bad faith may be inferred where the specific shortcomings of the case are identified by
opposing counsel, and the decision is made to go forward despite the inability to respond to the
arguments raised.’ ” See Gemini Aluminum v. Cal. Custom Shapes, 95 Cal. App. 4th 1249, 1264 (Cal.
Ct. App. 2002) (quoting Alamar Biosciences, Inc. v. Difco Lab., Inc., 1996 U.S. Dist. Lexis 18239, *3
(E.D. Cal. Feb. 23, 1996)
13
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 14 of 21
1
The Supreme Court has developed a two-pronged approach to the calculation of a reasonable
2
attorney's fee. “ ‘The lodestar determination has emerged as the predominate element of the analysis’ in
3
determining a reasonable attorney's fee award.” Morales v. City of San Rafael, 96 F.3d 359, 363 (9th
4
Cir. 1996) (quoting Jordan v. Multnomah County, 815 F.2d 1258, 1262 (9th Cir. 1987)). Under this
5
approach, a court must first calculate a “lodestar” figure by “multiplying the number of hours reasonably
6
expended on the litigation times a reasonable hourly rate.” Blum v. Stenson, 465 U.S. 886, 888 (1984);
7
see also Cunningham v. County of Los Angeles, 879 F.2d 481, 484 (9th Cir. 1988), cert. denied 493 U.S.
8
1035 (1990). The lodestar figure is presumptively reasonable. City of Burlington v. Dague, 505 U.S.
9
557, 562 (1992). While this lodestar amount is presumed to represent an appropriate fee, under certain
10
circumstances a court may then adjust the award upward or downward to take into account special
11
factors. Blum, 465 U.S. at 897.
12
13
2. Analysis
Defendants request a total award of $13,465,331.01. This amount is reached by adding the
14
following amounts: $10,244,053 for attorneys fees attributable to lead counsel Cooley LLP (“Cooley”);
15
$391,928.91 for attorneys’ fees attributable to document review performed by Black Letter Discovery,
16
Inc. (“Black Letter”); and $2,829,349.10 for fees associated with a document review algorithm
17
generated by outside vendor H5. Neither Plaintiffs nor WHGC objected to the requested amount or the
18
reasonableness of the rates charged and hours spent on the case.7
19
With regard to the attorneys’ fees attributable to Cooley, counsel spent 22,921.5 hours on this
20
case in the four years from September 2008 through September 2012. (Karr Decl., Doc. No. 322-2 at ¶
21
11; Ex. E, Doc. No. 336-1 at 388.) Based upon the nature of Plaintiffs’ claims, the many patents
22
involved in the case, and the length and complexity of the litigation, the Court finds the requested
23
number of hours spent on this case by Cooley to be reasonable. As support for the reasonableness of the
24
rates charged by Cooley attorneys, counsel provided a PeerMonitor Standard Rate Analysis comparing
25
7
26
27
28
Hughes Hubbard offered generalized objections to the requested amount of attorneys’ fees in
their opposition to Defendants’ motion. However, Defendants withdrew their motion as to Hughes
Hubbard, and therefore Hughes Hubbard’s objections are moot. To the extent that WHGC filed a notice
of joinder in Hughes Hubbard’s opposition more than a month after the opposition was filed, the Court
finds this objection to be untimely. Regardless, Hughes Hubbard’s objections did not provide specific
objections to the requested amount such that Court finds that the presumptively reasonable lodestar
amount should be adjusted.
14
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 15 of 21
1
Cooley rates generally to those charged by their peers in Los Angeles, Silicon Valley, San Francisco,
2
and San Diego. (Doc. No. 336.) Overall, Cooley’s rates are lower than those charged by comparable
3
firms in California. (Id.) Insomuch as neither of the remaining parties objects to the reasonableness of
4
the rates charged by Cooley and the rates appear to be in line with that of the community as supported
5
by the PeerMonitor analysis, the Court finds the requested rates to be reasonable. Accordingly, the
6
requested amount of $10,244,053 represents an appropriate lodestar amount for Cooley’s fees.
7
As to Black Letter, Defendants request $391,928.91 in attorneys’ fees. The Black Letter
8
attorneys billed a total of 6,949.5 hours of document review at rates of $55 to $67 per hour. As
9
Defendants note in their motion, Plaintiffs’ claims involved 92 patents resulting in voluminous
10
document production. For this reason, Cooley reasonably decided to have Black Letter perform
11
document review in this matter. Had Cooley performed the document review themselves, the resulting
12
attorneys’ fees would have undoubtedly been exponentially higher than those charged on behalf of
13
Black Letter. In light of the circumstances and the amount of discovery required, the Court concludes
14
that the rates charged and hours spent by Black Letter are reasonable and, thus, finds the resulting
15
lodestar amount of $391,928.91 to be reasonable as well.
16
The third aspect of Defendants’ requested fees is $2,829,349.10 attributable to computer-
17
assisted, algorithm-driven document review. Defendants provide the following explanation for the
18
resulting fees: “Over the course of this litigation, Defendants collected almost 12,000,000 records --
19
mostly in the form of Electronically Stored Information (ESI). . . . Rather than manually reviewing the
20
huge volume of resultant records, Defendants paid H5 to employ its proprietary technology to sort these
21
records into responsive and non-responsive documents.” (Defs. Mot., Doc. No. 332-1 at 26). After the
22
algorithm determined whether documents were responsive or unresponsive to discovery requests, Black
23
Letter attorneys reviewed the responsive documents for confidentiality, privilege, and relevance issues.
24
(Id. at 26, n. 11.) For this reason, the review performed by H5 and Black Letter accomplished different
25
objectives with the H5 electronic process minimizing the overall work for Black Letter. Again, the
26
Court finds Cooley’s decision to undertake a more efficient and less time-consuming method of
27
document review to be reasonable under the circumstances. In this case, the nature of Plaintiffs’ claims
28
resulted in significant discovery and document production, and Cooley seemingly reduced the overall
15
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 16 of 21
1
fees and attorney hours required by performing electronic document review at the outset. Thus, the
2
Court finds the requested amount of $2,829,349.10 to be reasonable.
3
Although the Court finds that Defendants’ requested lodestar amount reasonably reflects
4
Defendants’ attorneys’ fees incurred throughout the litigation, the Court may still exercise its discretion
5
in adjusting the overall award upward or downward to take into account special factors. See Blum, 465
6
U.S. at 897. As discussed above, the Court has determined that Plaintiffs pursued objectively baseless
7
claims against Defendants in bad faith. In reaching this conclusion, the Court returned frequently to
8
Judge Anello’s bond order issued September 20, 2010, and questioned Plaintiffs’ inadvisable decision to
9
continue forward with their claims beyond that point. Following this clear warning that their claims
10
lacked merit, the Court finds it particularly troubling that Plaintiffs maintained their claims for an
11
extended period of time without being able to remedy the evidentiary deficiencies accompanying their
12
claims. Insomuch as the bond order weighed heavily in favor of the Court’s decision to impose
13
attorneys’ fees, the Court finds it appropriate to limit the imposition of fees to those incurred after the
14
entry of the bond order. At the time of the bond hearing, Defendants’ counsel had accrued $1,000,000
15
in attorneys’ fees. Accordingly, the Court reduces the requested amount of $13,465,331.01 by
16
$1,000,000. This results in a total award of attorneys fees in the amount of $12,465,331.01 as against
17
Plaintiffs.
18
II. Defendants’ Motion for Attorneys’ Fees as to WHGC
19
In addition to the statutory requests for attorneys’ fees as against Plaintiffs, Defendants seek to
20
hold Plaintiffs’ local counsel WHGC liable for Defendants’ attorneys’ fees as well. Defendants asks
21
that the Court utilize Rule 11 of the Federal Rules of Civil Procedure, or alternatively exercise its
22
inherent authority, to impose sanctions against WHGC for its participation in the case. Under this
23
authority, Defendants ask that WHGC be held jointly and severally liable for the total amount of
24
attorneys’ fees requested.
25
A. Legal Standard
26
Rule 11 of the Federal rules of Civil Procedure imposes a duty on attorneys to certify that all
27
pleadings are legally tenable and well-grounded in fact. See Fed. R .Civ. P. 11; Chambers v. NASCO,
28
Inc., 501 U.S. 32, 41 (1991). Pursuant to Rule 11(b), attorneys must perform “an inquiry reasonable
16
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 17 of 21
1
under the circumstances” to ensure that they have reason to believe that their legal contentions are
2
“warranted by existing law” and that their factual contentions either “have evidentiary support or, if
3
specifically so identified, will likely have evidentiary support after a reasonable opportunity for further
4
investigation.” Fed. R. Civ. P. 11(b). When considering alleged Rule 11 violations, the Court utilizes
5
an objective standard of reasonable inquiry which does not mandate a finding of bad faith. Chambers,
6
501 U.S. at 47. Significantly, Rule 11 governs only papers filed with the Court and not alleged litigation
7
misconduct outside of court filings. See Fed. R .Civ. P. 11.
8
9
However, the court may impose sanctions for general litigation conduct falling outside of the
specific prohibitions of Rule 11 through the courts’ inherent power. See Chambers, 501 U.S. at 51
10
(finding reliance on inherent powers appropriate where “the conduct sanctionable under the Rules was
11
intertwined within conduct that only the inherent power could address”); Primus Automotive Financial
12
Services, Inc. v. Batarse, 115 F.3d 644, 648 (9th Cir. 1997). “The inherent powers of federal courts are
13
those that ‘are necessary to the exercise of all others.’” Roadway Express, Inc. v. Piper, 447 U.S. 752,
14
764 (1980) (quoting United States v. Hudson, 7 Cranch 32, 34, 3 L.Ed. 259 (1812)). The most common
15
utilization of inherent powers is the contempt sanction which serves to protect the due and orderly
16
administration of justice and maintain the authority and dignity of the court. Id. (citing Cooke v. U.S.,
17
267 U.S. 517, 539 (1925). Courts may exercise their inherent power to impose sanctions in form of
18
attorneys’ fees when a losing party has acted “in bad faith, vexatiously, wantonly, or for oppressive
19
reasons.” Alyeska Pipeline Serv. Co. v. Wilderness Soc'y, 421 U.S. 240, 258-59 (1975) (quotation
20
omitted).
21
Before awarding sanctions under its inherent powers, the court must make an explicit finding
22
that counsel's conduct “constituted or was tantamount to bad faith.” Roadway Express, 447 U.S. at 767;
23
see also Primus, 115 F.3d at 648. “A finding of bad faith is warranted where an attorney ‘knowingly or
24
recklessly raises a frivolous argument, or argues a meritorious claim for the purpose of harassing an
25
opponent.’ ” Id. (quoting In re Keegan, 78 F.3d at 436 (citation omitted)). The bad faith requirement
26
sets a high threshold. Id.; see also Mendez v. County of San Bernardino, 540 F.3d 1109, 1131-32 (9th
27
Cir. 2008).
28
B. Analysis
17
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 18 of 21
1
As a general matter, the Court recognizes that local counsel plays a unique role in the litigation
2
process. The local rules require out-of-state attorneys to acquire local counsel, and often local counsel
3
serves primarily in an administrative capacity for the limited purpose of filing documents with the
4
Court. Thus, the reasonable inquiry required for local counsel under Rule 11 may not be the same as
5
that required for lead counsel in many situations. However, Rule 11 remains applicable and sanctions
6
may be imposed against local counsel when appropriate under the circumstances. While it may be
7
reasonable for attorneys to rely on the work conducted by other attorneys in some instances, that
8
determination depends upon whether its reasonable under the particular circumstances.8
9
Here, it is undisputed that WHGC served in a more limited capacity than either of the firms
10
serving as Plaintiffs’ lead counsel, Munck Carter PC (“Munck”) and subsequently Hughes Hubbard &
11
Reed LLP (“Hughes Hubbard”). WHGC offers the following description of its role in the case as
12
support for finding sanctions unwarranted. As local counsel, WHGC “was asked to, and did, ensure that
13
Plaintiffs’ papers met the requirements of the Court’s Local Rules and physically filed the papers (either
14
electronically with the Court’s ECF system or by hand when documents required under seal treatment).”
15
(WHGC Opp., Doc. No. 354 at 10.) During the initial intake of Plaintiffs’ case, WHGC performed legal
16
research to ensure that each of Plaintiffs’ claims were properly pled under applicable law and relied
17
upon the assertions of Plaintiffs and Munck that there was a good faith basis for all of Plaintiffs’
18
allegations. (Id.) Throughout the case, “WHGC never had access to Plaintiffs’ percipient witnesses or
19
to Plaintiffs’ documents at the time it was retained to act as Plaintiffs’ local counsel.” (Id. at 11.) Even
20
after Munck left the case and Hughes Hubbard took over as lead counsel, WHGC’s role remained
21
unchanged. (Id.) WHGC never participated in discovery or motion practice regarding Plaintiffs’
22
23
24
25
26
27
28
8
See Qualcomm Inc. v. Broadcom Corp., 2008 WL 66932, *16 n. 14 (S.D. Cal. Jan. 7, 2008),
vacated in part by 2008 WL 636108 (S.D. Cal. Mar. 5, 2008) (noting that sanctions may be imposed on
local counsel and that the reasonableness of local counsel’s reliance on other attorneys is dependent on
the circumstances of the case); see also Coburn v. Optical Indus., Inc. v. Cilco, Inc., 610 F. Supp. 656,
660 n. 7 (D.C. N.C. 1985) (recognizing that “local counsel must be able to rely to some extent on the
representations of reputable out of state attorneys, especially when local counsel has no independent
knowledge concerning the representations”). But c.f. Val-Land Farms, Inc. v. Third Nat. Bank in
Knoxville, 937 F.2d 1110, (6th Cir. 1991) (determining that the text of Rule 11 “does not provide a safe
harbor for lawyers who rely on the representations of outside counsel” as counsel’s obligations under
the rule are nondelegable).
18
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 19 of 21
1
claims. (Id. at 14.) Based on these facts indicating WHGC’s limited role in the proceedings, WHGC
2
contends that it performed an appropriate and reasonable inquiry under the circumstances.
3
In response, Defendants contend that WHGC unreasonably relied upon the representations of
4
lead counsel and Plaintiffs when determining whether Plaintiffs’ claims had merit and were brought in
5
good faith. Moreover, Defendants highlight the fact that WHGC is the sole signatory on the operative
6
Fourth Amended Complaint which WHGC filed to comply with the amendment deadline after Munck
7
left the case and prior to Hughes Hubbard taking over as lead counsel. (See Doc. No. 53.) Most
8
significantly, Defendants argue that WHGC should have reevaluated its role in the case after it
9
discovered the deficiencies in Plaintiffs’ case as set forth by Judge Anello in his bond order some
10
months before.
11
In this instance, although WHGC may have justifiably relied upon Plaintiffs’ and Munck’s
12
representations as support for its filings prior to Judge Anello’s bond order, WHGC has offered no
13
justification for its decision to continue filing documents on Plaintiffs’ behalf after Judge Anello warned
14
of the substantial deficiencies in Plaintiffs’ case. As noted above, the order clearly indicated that
15
Plaintiffs’ claims were likely unmeritorious, lacked any significant evidentiary support, and appeared to
16
be brought in bad faith. (See Doc. No. 110 at 22.) Once again, Judge Anello’s bond order weighs
17
significantly in favor of the Court’s decision to impose sanctions. WHGC was represented at the
18
hearing, and received electronic notice of the subsequent order. While WHGC may not have partici-
19
pated in discovery prior to that point, the evidence relevant to the bond order and the Court’s interpreta-
20
tion of that evidence should have raised concern. At that point in time, WHGC’s obligation under Rule
21
11 increased as it thereafter became unreasonable to rely on the representations of Plaintiffs and lead
22
counsel when filing documents with the Court. Even when serving in the more limited role of local
23
counsel, WHGC had an obligation to review Judge Anello’s order and undertake a reasonable investiga-
24
tion into the merits of the case. WHGC did not do so, and instead continued to file documents with the
25
Court without performing the requisite reasonable inquiry under the circumstances. As such, every
26
27
28
19
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 20 of 21
1
WHGC filing after September 20, 2010, was in violation of Rule 11. Accordingly, the Court finds
2
sanctions to be warranted under Rule 11.9
3
With regard to the appropriate amount of Rule 11 sanctions, the sanction must be “appropriate”
4
and “limited to what suffices to deter repetition of the conduct or comparable conduct by others
5
similarly situated.” Fed. R. Civ. P. 11(c)(1), (2). Defendants ask that the Court hold WHGC jointly and
6
severally liable for the entire amount of Defendants’ requested attorneys’ fees. The Court finds this
7
suggestion excessive based on the nature of WHGC’s actions and their limited role as local counsel. It
8
appears that WHGC initially performed a reasonable inquiry under the circumstances, and the basis for
9
finding a Rule 11 violation arises only after the issuance of Judge Anello’s bond order. For this reason,
10
the Court finds that joint and several liability for the entirety of Defendants’ attorneys fees is dispropor-
11
tionate to WHGC’s violation. Alternatively, the Court considered limiting WHGC’s joint and several
12
liability to those fees billed by Defendants’ following the bond order, but concludes that this award
13
would also be excessive under the circumstances. At the hearing on the instant motion for fees,
14
Defendants’ counsel posed a more reasonable alternative. In their opposition to Defendants’ motion for
15
attorneys’ fees, WHGC compared the requested $13,465,331.01 in Defendants’ attorneys’ fees to the
16
$64,316.50 in fees billed by WHGC from the time it was retained in February 2009 through September
17
28, 2012. (Doc. No. 354 at 9.) Referencing WHGC’s billing total, Defendants suggest that the Court
18
impose sanctions in the amount of $64,316.50, which would, in essence, prevent WHGC from realizing
19
a profit for its services as local counsel in this case. The Court finds this to be a reasonable method of
20
determining the proper amount of sanctions to be imposed. This amount should deter local counsel from
21
filing documents without performing a reasonable inquiry under the circumstances, but it is also more
22
appropriate in this instance than basing the award upon Defendants’ attorneys’ fees incurred following
23
Judge Anello’s order. The lower award is appropriate considering the nature of the violation along with
24
WHGC’s limited role in the case. Accordingly, the Court imposes Rule 11 sanctions against WHGC in
25
the amount of $64,316.50. This amount shall be awarded to Defendants as attorneys’ fees.
26
27
28
9
Insomuch as the Court finds sanctions appropriate under Rule 11, the Court need not consider
the imposition of sanctions pursuant to its inherent authority. However, he Court notes that there
appears to be little evidence to support finding bad faith on the part of WHGC sufficient to warrant
sanctions under the Court’s inherent authority.
20
08cv1992
Case 3:08-cv-01992-AJB-MDD Document 371 Filed 02/01/13 Page 21 of 21
1
2
Conclusion
In sum, the Court awards Defendants’ $12,401,014.51 in attorneys’ fees as against Plaintiffs.
3
Accordingly, the $800,000 bond posted by Plaintiffs and any accrued interest will be distributed to
4
Defendants when this order becomes final. (Doc. No. 121). Plaintiffs will pay the remaining amount
5
of $11,601,014.51 to Defendants. Additionally, the Court awards $64,316.50 in attorneys’ fees as
6
against WHGC to be paid to Defendants.
7
For the reasons set forth above, it is ORDERED that Defendants’ Motion for Attorneys’ Fees as
8
to Plaintiffs and WHGC be, and it hereby is, GRANTED IN PART and DENIED IN PART. The Court
9
further ORDERS as follows:
10
1. The Court directs the Clerk of Court to pay the $800,000 in funds and any accrued interest
held in the Court’s registry to Defendants when this order becomes final;
11
2. Plaintiffs will pay $11,601,014.51 to Defendants within 60 days of this order; and
12
3. WHGC will pay $64,316.50 to Defendants within 60 days of this order.
13
IT IS SO ORDERED.
14
15
DATED: February 1, 2013
16
17
Hon. Anthony J. Battaglia
U.S. District Judge
18
19
20
21
22
23
24
25
26
27
28
21
I:\Chambers Battaglia\DJ CASES\2 Orders to be filed\08cv1992 Gabriel Final Order Granting Atty's Fees.wpd
08cv1992
UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF INDIANA
SOUTH BEND DIVISION
IN RE: BIOMET M2a MAGNUM HIP
IMPLANT PRODUCTS LIABILITY
LITIGATION (MDL 2391)
This Document Relates to All Cases
)
)
)
)
)
)
)
)
CAUSE NO. 3:12-MD-2391
ORDER REGARDING DISCOVERY OF ESI
Biomet has produced 2.5 million documents to plaintiffs in this docket’s
constituent cases, and the Plaintiffs’ Steering Committee believes production
should run to something closer to 10 million documents. The parties have set
forth their positions on the procedures or protocols that should be used to
facilitate identification, retrieval, and production of electronically stored
information in submissions filed on April 1 and 5. The parties seek my guidance
as to the direction discovery of ESI should take, and I believe the parties need a
prompt ruling more than they need extensive discussion of each point they raise.
Biomet began producing documents in cases eventually centralized here in
the summer of 2012. Some plaintiffs’ counsel, anticipating this docket’s formation,
told Biomet (occasionally in forceful terms) not to begin document production until
the Judicial Panel on Multidistrict Litigation decided whether to centralize.
Biomet, neither sold on centralization nor free of judicial exhortations in other
cases against it, started the process of identifying and producing documents.
Page 1 of 7
Biomet used a combination of electronic search functions to identify
relevant documents. Keyword culling was used first, reducing the universe of
documents and attachments from 19.5 million documents to 3.9 million
documents, comprising 1.5 terabytes of data. Removal of duplicates left 2.5
million documents and attachments. Statistical sampling tests of a random
sample projected, with a 99 percent confidence rate, that between .55 and 1.33
percent of the unselected documents would be responsive and (with the same
confidence level) that between 1.37 and 2.47 percent of the original 19.5 million
documents were responsive. In comparison, Biomet’s keyword/deduplication
approach had identified 16 percent of the original 19.5 million.
Biomet then employed technology-assisted review, or predictive coding, to
identify the relevant documents to be produced from the 2.5 million that emerged
from the keyword and deduplication processes. Predictive coding has found many
uses on the Internet. Under predictive coding, the software “learns” a user’s
preferences or goals; as it learns, the software identifies with greater accuracy just
which items the user wants, whether it be a song, a product, or a search topic.
Biomet used a predictive coding service called Axelerate and eight contract
attorneys to review a sampling of the 2.5 million documents. After one round of
“find more like this” interaction between the attorneys and the software, the
contract attorneys (together with other software recommended by Biomet’s ediscovery vendor) reviewed documents for relevancy, confidentiality, and privilege.
Page 2 of 7
To date, Biomet’s e-discovery costs are about $1.07 million and will total
between $2 million and $3.25 million.
Biomet invited the Plaintiffs’ Steering Committee to suggest additional
search terms and offered to produce the rest of the non-privileged documents from
the post-keyword 2.5 million so the Steering Committee can verify that Biomet is
producing the relevant documents. The Steering Committee has declined those
offers, believing they are too little to assure proper document production.
The Steering Committee contends Biomet’s initial use of the keyword
approach has tainted the process. They point to a recent article that mentioned
unidentified “literature stating that linear review would generate a responsive rate
of 60 percent and key word searches only 20 percent, and [the defendants in the
case being discussed] proposed that predictive coding at a 75 percent responsive
rate would be sufficient.” Barry Kazan and David Wilson, TECHNOLOGY-ASSISTED
REVIEW IS
A
PROMISING TOOL
FOR
DOCUMENT PRODUCTION, New York Law Journal
(Mar. 18, 2013). The Steering Committee sees Biomet’s approach as insufficient
because, although it employed predictive coding, Biomet began with the less
accurate keyword search. The Steering Committee sees Biomet’s offer to let the
Steering Committee propose search terms as unhelpful because the Steering
Committee’s unfamiliarity with Biomet terminology prevents them from making
suggestions advisedly.
The Steering Committee wants Biomet to go back to its 19.5 million
documents and employ predictive coding, with plaintiffs and defendants jointly
Page 3 of 7
entering the “find more like this” commands. Biomet objects on a variety of
grounds, including its estimate that virtually starting over would cost it millions
more than the millions it already has spent in document production. The Steering
Committee responds that Biomet gambled when it spent millions on document
production that several of plaintiffs’ counsel warned Biomet not to undertake until
the Panel had centralized the cases.
The issue before me today isn’t whether predictive coding is a better way of
doing things than keyword searching prior to predictive coding. I must decide
whether Biomet’s procedure satisfies its discovery obligations and, if so, whether
it must also do what the Steering Committee seeks. What Biomet has done
complies fully with the requirements of Federal Rules of Civil Procedure 26(b) and
34(b)(2). I don’t see anything inconsistent with the Seventh Circuit Principles
Relating to the Discovery of Electronically Stored Information. Principle 1.02
requires cooperation, but I don’t read it as requiring counsel from both sides to
sit in adjoining seats while rummaging through millions of files that haven’t been
reviewed for confidentiality or privilege. Both sides cite reports from the Sedona
Conference project, e.g., The Sedona Conference, The Sedona Conference
Commentary on Proportionality in Electronic Discovery (Jan. 2013); The Sedona
Conference, The Sedona Conference Best Practices Commentary on the Use of
Search and Information Retrieval Methods in E-Discovery, 8 Sedona Conf. J. 189
(2007); and The Sedona Conference, Conducting E-Discovery After Amendments:
Page 4 of 7
The Second Wave, 10 Sedona Conf. J. 215 (2009), and I don’t see Biomet’s
approach as running afoul of any of the principles set forth in those publications.
In contrast, the Steering Committee’s request that Biomet go back to Square
One (more accurately Square Two, since Biomet first collected the 19.5 million
documents) and institute predictive coding at that earlier stage sits uneasily with
the proportionality standard in Rule 26(b)(2)(C). Doing so would entail a cost in
the low seven-figures. The confidence tests Biomet ran as part of its process
suggest a comparatively modest number of documents would be found. The
Steering Committee challenges that conclusion by pointing to studies (one in
1985) indicating that, on average, Boolean searches identify less than a quarter
of the relevant documents in a set of documents. Boolean language provides the
basis for keyword searches, though I can’t find anything in this record that
equates today’s keyword searches to Boolean searches. In contrast, the Steering
Committee says predictive coding identified 75 to 95 percent of the relevant
documents — about four times more efficient than keyword searches. The 75
percent figure appears to come from the previously-cited recent New York Law
Journal article about technology-assisted review as part of document production.
The article itself doesn’t vouch for the accuracy of the 75 percent figure; the article
simply notes that in a Virginia state court case, Global Aerospace v. Landow
Aviation, No. CL 61040 (Va. Cir. Ct., Loudon County, Apr. 23, 2012), the
defendants “proposed that predictive coding at a 75 percent responsive rate would
be sufficient.” I can find no source for the 95 percent figure.
Page 5 of 7
It might well be that predictive coding, instead of a keyword search, at Stage
Two of the process would unearth additional relevant documents. But it would
cost Biomet a million, or millions, of dollars to test the Steering Committee’s
theory that predictive coding would produce a significantly greater number of
relevant documents. Even in light of the needs of the hundreds of plaintiffs in this
case, the very large amount in controversy, the parties’ resources, the importance
of the issues at stake, and the importance of this discovery in resolving the issues,
I can’t find that the likely benefits of the discovery proposed by the Steering
Committee equals or outweighs its additional burden on, and additional expense
to, Biomet. FED. R. CIV. P. 26(b)(2)(C).
The Steering Committee appears to argue that Biomet is estopped from
relying on proportionality arguments based on the incremental cost of what the
Steering Committee seeks because Biomet embarked on its document
identification in disregard of pre-centralization warnings and advice from some
counsel for plaintiffs in individual cases. It might be that the Steering Committee’s
argument could carry the day in some cases, but this one doesn’t seem to be such
a case. The Steering Committee hasn’t argued (and I assume it can’t argue) that
Biomet had no disclosure or document identification obligation in any of the cases
that were awaiting a ruling on (or even the filing of) the centralization petition.
Until the MDL Panel enters a centralization order under 28 U.S.C. § 1407 (or
transfers a tag along pursuant to an earlier centralization order), a transferee
court is free to act on pending matters. Indeed, through its conditional transfer
Page 6 of 7
orders, the Panel regularly encourages transferee courts to do so. To hold that a
party that behaves as the transferee court directs, or that follows the transferee
court’s standing procedures, does so only by forfeiture of the proportionality
provision of Rule 26(b)(2)(C), seems an uncongenial exercise of whatever discretion
I have. It also would seem inconsistent with the purposes of centralization under
§ 1407.
In making this ruling, I assume that Biomet will remain open to meeting
and conferring on additional reasonably-targeted search terms and to producing
the non-privileged documents included in the statistical sample. Beyond that, if
the Steering Committee wishes production of documents that can be identified
only through re-commenced processing, predictive coding, review, and production,
the Steering Committee will have to bear the expense.
ENTERED:
April 18, 2013
/s/ Robert L. Miller, Jr.
Judge, United States District Court
Northern District of Indiana
Page 7 of 7
UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF INDIANA
SOUTH BEND DIVISION
IN RE: BIOMET M2a MAGNUM HIP
IMPLANT PRODUCTS LIABILITY
LITIGATION (MDL 2391)
This Document Relates to All Cases
)
)
)
)
)
)
)
)
CAUSE NO. 3:12-MD-2391
MEMORANDUM AND ORDER
Biomet had begun searching its electronic files before the creation of this
multidistrict litigation docket through a keyword culling and de-duplication
process, winnowing the universe of potentially relevant documents (the universe
as Biomet saw it) from 19.5 million documents to 2.9 million. At that point,
Biomet turned to technology-assisted review, or “predictive coding,” to winnow the
potentially discoverable documents down further. On April 18, I resolved the
parties’ dispute over whether Biomet had to go back to square one in its document
production and use predictive coding. I ruled that Biomet didn’t have to do that,
though I encouraged Biomet to do what it said it would do and let the Plaintiffs’
Steering Committee suggest additional search terms.
Today’s dispute picks up where the April order left off. The Steering
Committee wants Biomet to produce the discoverable documents used in the
training of the “predictive coding” algorithm. Biomet reveals only that the
discoverable documents used in the seed set already have been disclosed to the
Steering Committee; Biomet won’t identify the seed set beyond that. Without
knowing the training documents, the Steering Committee says, it can’t
intelligently propose more search terms, since it doesn’t know what already has
been included in the search. Biomet says nothing in the law requires it to provide
what the Steering Committee seeks.
My understanding of the predictive coding process consists largely of what
was placed before me in resolution of the first discovery issue, although Biomet
added a little more information and terminology in the memoranda I requested
from both sides. As I understand it, a predictive coding algorithm offers up a
document, and the user tells the algorithm to find more like that document or that
the user doesn’t want more documents like what was offered up.
The Steering Committee wants the whole seed set Biomet used for the
algorithm’s initial training. That request reaches well beyond the scope of any
permissible discovery by seeking irrelevant or privileged documents used to tell
the algorithm what not to find. That the Steering Committee has no right to
discover irrelevant or privileged documents seems self-evident.
So I might have misunderstood the Steering Committee’s request. It might
be that the Steering Committee only wants to know which discoverable documents
were used in the seed set, placing the parties’ disagreement in a different light.
Biomet tells the court — and the Steering Committee is in no position to agree or
disagree — that it already has produced all of the discoverable documents that
were used in the seed set. Given that production, then, the Steering Committee
doesn’t seek the production of documents; it has all the documents it wants to
know about. The Steering Committee wants Biomet to tell how it went about
identifying and selecting the documents (not just the seed set) that it has
produced.
I’m puzzled as to the authority behind the Steering Committee’s request.
Federal Rule of Civil Procedure 26(b)(1) makes discoverable
2
any nonprivileged matter that is relevant to any party’s claim or
defense—including the existence, description, nature, custody,
condition, and location of any documents or other tangible things and
the identity and location of persons who know of any discoverable
matter. For good cause, the court may order discovery of any matter
relevant to the subject matter involved in the action. Relevant
information need not be admissible at the trial if the discovery
appears reasonably calculated to lead to the discovery of admissible
evidence. All discovery is subject to the limitations imposed by Rule
26(b)(2)(C).
The Steering Committee knows of the existence and location of each discoverable
document Biomet used in the seed set: those documents have been disclosed to
the Steering Committee. The Steering Committee wants to know, not whether a
document exists or where it is, but rather how Biomet used certain documents
before disclosing them. Rule 26(b)(1) doesn’t make such information disclosable.
The only authority the Steering Committee cites is a report of the Sedona
Conference that has had a significant, salutary, and persuasive impact on federal
discovery practice in the age of electronically stored information. Sedona
Conference Cooperation Proclamation, 10 Sedona Conf. J. 331 (Fall Supp. 2009).
Biomet, the Steering Committee says, isn’t proceeding in the cooperative spirit
endorsed by the Sedona Conference and the corresponding Seventh Circuit
project. But neither the Sedona Conference nor the Seventh Circuit project
expands a federal district court’s powers, so they can’t provide me with authority
to compel discovery of information not made discoverable by the Federal Rules.
Still, Biomet’s position is troubling. Biomet suggests no way in which telling
the Steering Committee which of the documents already produced were in the
seed set would harm it. Based on what I have been given in the parties’
memoranda, Biomet is right that it doesn’t have to identify the seed set, but the
Steering Committee is right that Biomet’s cooperation falls below what the Sedona
3
Conference endorses. An unexplained lack of cooperation in discovery can lead a
court to question why the uncooperative party is hiding something, and such
questions can affect the exercise of discretion.
But I don’t have any discretion in this dispute. I won’t order Biomet to reveal
which of the documents it has disclosed were used in the seed set, but I urge
Biomet to re-think its refusal.
ENTERED:
August 21, 2013
/s/ Robert L. Miller, Jr.
Robert L. Miller, Jr., Judge
United States District Court
4
Case 1:08-cv-00380-WMS-LGF Document 456 Filed 05/21/13 Page 1 of 7
UNITED STATES DISTRICT COURT
WESTERN DISTRICT OF NEW YORK
_____________________________________
GAIL HINTERBERGER, et al.,
Plaintiffs,
DECISION
and
ORDER
v.
08-CV-380S(F)
CATHOLIC HEALTH SYSTEM, INC., et al.,
Defendants.
_____________________________________
APPEARANCES:
THOMAS & SOLOMON, LLP
Attorneys for Plaintiffs
MICHAEL J. LINGLE,
SARAH E. CRESSMAN, of Counsel
693 East Avenue
Rochester, New York 14607
NIXON PEABODY, LLP
Attorneys for Defendants
MARK A. MOLLOY,
TODD R. SHINAMAN,
JOSEPH A. CARELLO, of Counsel
40 Fountain Plaza, Suite 500
Buffalo, New York 14202
JURISDICTION
By order of Hon. William M. Skretny, dated January 6, 2010 (Doc. No. 243), this
matter was referred to the undersigned for all non-dispositive pretrial matters pursuant
28 U.S.C. § 636(b)(1)(A). The matter is presently before the court on Plaintiffs’ motion,
filed October 5, 2012, to compel Defendants to meet and confer with respect to
establishing an agreed protocol for implementing the use of predictive coding software;
alternatively, Plaintiffs request the court to adopt and impose such protocol (Doc. No.
360).
Case 1:08-cv-00380-WMS-LGF Document 456 Filed 05/21/13 Page 2 of 7
BACKGROUND
Plaintiffs’ motion requests an order compelling Defendants “to engage in
meaningful meet and confer discussions regarding an ESI protocol with both parties’
respective ESI experts/consultants; and an order that if the parties are unable to agree
upon an ESI protocol by a deadline set by the court, that each side submit its own
proposed ESI protocol to the court for a ruling as to which protocol should be adopted in
this case.” (Doc. No. 360) (“Plaintiffs’ motion”). In support, Plaintiffs filed a
Memorandum Of Law In Support of Plaintiffs’ Motion To Compel (Doc. No. 360-1)
(“Plaintiffs’ Memorandum”) along with an Affirmation of Sarah E. Cressman (Doc. No.
361) (“Cressman Affirmation”) attaching Exhibits A - E (“Cressman Affirmation Exh(s).
___”). Defendants opposed Plaintiffs’ motion by filing on October 16, 2012 a
Memorandum Of Law In Opposition To Motion To Compel ESI Meet And Confer (Doc.
No. 369) (“Defendants’ Memorandum”) and an Attorney Declaration In Opposition To
Motion To Compel Meet And Confer (Doc. No. 369-1) (“Declaration of Todd R.
Shinaman” or “Shinaman Declaration”). On October 19, 2012, Plaintiffs filed a Reply
Memorandum Of Law In Further Support Of Plaintiffs’ Motion To Compel (Doc. No. 379)
(“Plaintiffs’ Reply Memorandum”). Oral argument was deemed unnecessary. Based on
the following, Plaintiff’s motion should be DISMISSED without prejudice.
FACTS1
For well-over a year, the parties have attempted, without success, to agree on
1
Taken from the pleadings and papers filed in connection with Plaintiffs’ m otion.
2
Case 1:08-cv-00380-WMS-LGF Document 456 Filed 05/21/13 Page 3 of 7
how to achieve a cost-effective review of Defendants’ voluminous e-mails using a keyword search methodology. At the last of a series of ESI discovery status conferences
with the court, conducted June 27, 2012 (Doc. No. 361) (“the June 27, 2012
conference”), the court expressed dissatisfaction with the parties’ lack of progress
toward resolving issues related to completion of reviewing and production of Defendants’
e-mails using the key-word search method, and pointed to the availability of predictive
coding, a computer assisted ESI review and production method, directing the parties’
attention to the recent decision of Magistrate Judge Andrew Peck in Moore v. Publicis
Groupe & MSL Group, 287 F.R.D. 182 (S.D.N.Y. 2012), approving use of predictive
coding in a case involving over 3 million e-mails. At the June 27, 2012 conference, the
parties were requested to submit a joint or individual protocols for a key-word search
methodology, which the parties had been attempting, albeit unsuccessfully, to implement
by August 14, 2012. Cressman Affirmation ¶ 3; Cressman Affirmation Exh. A at 3 (Email from Todd R. Shinaman to Sarah E. Cressman August 31, 2012) (“August 31, 2012
e-mail”). Thereafter, by Decision and Order, filed July 20, 2012 (Doc. No. 329) (“July 20,
2012 D&O), the court directed completion of ESI discovery by October 23, 2012, and
completion of non-ESI discovery by January 23, 2013. July 20, 2012 D&O at 5. On
August 31, 2012, Defendants informed Plaintiffs they would abandon use of the keyword search method, commence use of predictive coding, and requested Plaintiffs
confer with respect to “identification of [Defendants’] custodians for purposes of the
search.” August 31, 2012 e-mail. Approximately two hours prior to a scheduled
conference call among the parties on September 13, 2012, Defendants informed
Plaintiffs that Defendants objected to Plaintiffs’ use of Plaintiffs’ ESI consultants and
3
Case 1:08-cv-00380-WMS-LGF Document 456 Filed 05/21/13 Page 4 of 7
would refuse to confer on implementation of predictive coding with Plaintiffs’ ESI
consultants. Cressman Affirmation ¶ 4; Cressman Affirmation Exh. B at 1 (September
13, 2012 e-mail from Todd R. Shinaman to Sarah E. Cressman). Plaintiffs’ attorneys
complied with Defendants’ request, but the conference did not result in a protocol for
Defendants’ use of predictive coding. Cressman Affirmation ¶ ¶ 5-6. Following
Defendants’ motion to disqualify Plaintiffs’ ESI consultant, filed October 2, 2012 (Doc.
No. 351), Defendants provided Plaintiffs with a proposed ESI protocol for predictive
coding. Cressman Affirmation ¶ 8; Cressman Affirmation Exh. D. Plaintiffs objected to
Defendants’ proposed protocol by letter dated October 4, 2012 in which Plaintiffs noted
several technical issues which should be discussed with the assistance of Plaintiffs’ ESI
consultants and cooperatively resolved by the parties before any efforts by Defendants
to implement predictive coding of Defendants’ e-mails were initiated. Cressman
Affirmation ¶ ¶ 11-13; Cressman Affirmation Exh. E.
DISCUSSION
Plaintiffs contend that where a party intended to use predictive coding to assist in
the review and production of ESI,2 it is necessary that the parties negotiate a protocol to
2
E-Discovery is the “process of identifying, preserving, collecting, preparing, and producing
electronically stored inform ation (‘ESI’) in the context of the legal process.” The Sedona Conference
Glossary E-Discovery & Digital Inform ation Managem ent (Third Edition) (“Sedona Conference Glossary”)
at 18. See also Maura R. Grossm an and Gordon V. Corm ack, The Grossman-Cormack Glossary of
Technology-Assisted Review, Vol. 7 F ED ER AL C O U RTS L AW R EVIEW 1, 15 (2013) (E-Discovery is the
“process of identifying, preserving, collecting, processing, searching, reviewing, and producing
Electronically Stored Inform ation that m ay be relevant to a civil, crim inal, or regulatory m atter.”) (“The
Grossm an-Corm ack Glossary”). ESI includes electronic m ail, or e-m ail m essages, word processing files,
web pages, and databases created and stored on com puters, m agnetic disks (such as com puter hard
drives, optical disks (such as DVDs and CDs, and flash m em ory (such as “thum b” or “flash” drives) and
“cloud” based services hosted by third parties via the internet)). M AN AG ING D ISC O VER Y O F E LEC TR O N IC
IN FO R M ATION : A P O C KET G U ID E FO R J U D G ES , Second Edition, Federal Judicial Center (2012) at 2.
4
Case 1:08-cv-00380-WMS-LGF Document 456 Filed 05/21/13 Page 5 of 7
guide the use of predictive software for the case. Plaintiffs’ Memorandum at 1-2 (citing
Moore v. Publicis Groups & MSL Group, 287 F.R.D. 182, 185 (Peck, M.J.) (where use of
predictive coding is challenged court may require requesting party obtain documents —
seed set documents – that were used by the producing party to “train” the computerassisted coding system)). Plaintiffs maintain that Defendants’ protocol does not provide
Plaintiffs with any details regarding such documents as required by Moore. Plaintiffs’
Memorandum at 2 (citing Moore, 287 F.R.D. at 192 (approving agreement to provide
seed-set documents to plaintiff’s counsel)). In Moore, the court noted that “[e]lectronic
discovery requires cooperation between opposing counsel and transparency in all
aspects of preservation and production of ESI.” Moore, 287 F.R.D. at 191 (quoting
William A. Gross Constr. Assocs., Inc. v. Am. Mfrs. Mut. Ins. Co., 256 F.R.D. 134, 136
(S.D.N.Y. 2009) (Peck, M.J.)). Plaintiffs argue that Defendants’ refusal to discuss with
Plaintiffs Defendants’ protocol with the assistance of Plaintiffs’ ESI consultants has
impeded establishing an agreed protocol and completion of ESI discovery using
predictive coding in this case. Plaintiffs’ Memorandum at 2-3.
Defendants argue in opposition that as Defendants have not attempted to produce
the requested ESI at issue using predictive coding under Defendants’ protocol, Plaintiffs’
motion is premature. Defendants’ Memorandum at 3. Defendants further argue that
contrary to Plaintiffs’ contention, Moore does not require Plaintiffs be given access to
Defendants’ seed-set documents at this time. Id. at 5 (citing Moore, 287 F.R.D. at 192;
Romero v. Allstate Ins. Co., 271 F.R.D. 96, 110 (E.D. Pa. 2010) (parties expected to
voluntarily share information in order to effectively implement search methodology for
defendant’s ESI)). Defendants point to several considerations that warrant the exercise
5
Case 1:08-cv-00380-WMS-LGF Document 456 Filed 05/21/13 Page 6 of 7
of caution in directing production of ESI. Defendants’ Memorandum at 3 (quoting the
Sedona Conference Cooperation Proclamation: Resources for the Judiciary, 20-21
(August 2011) (https://thesedonaconference.org/download-pub/425).
Defendants are
willing to discuss ESI issues with Plaintiffs ESI consultants provided such consultants
were not disqualified from doing so based on prior services to Defendants.3 Defendants’
Memorandum at 6-7.
In response, Plaintiffs do not directly contest Defendants’ reading of the specific
holding in Moore regarding the absence of any requirement by that court that the parties
meet and confer regarding the producing party’s selection of a “seed set of documents.”
But see William A. Gross Const. Assoc., 256 F.R.D. at 136 (requiring counsel for parties
to cooperate in selecting appropriate key-words to facilitate computerized search for
relevant e-mails). Plaintiffs also point to Local Rule of Civil Procedure 26(f)(3) which, in
cases involving ESI discovery, requires the parties “discuss and attempt to reach
agreement as to the method of searching.” Plaintiffs’ Reply Memorandum at 2 (quoting
Local R.Civ.P. 26(f)(3)). Plaintiffs request that, in the event the court does not grant
Plaintiffs’ motion, Defendants be reminded of the possibility that upon Plaintiffs’ further
motion, the court may find Defendants’ ESI search methodology to be unreasonable and
thus non-compliant with Defendants’ production obligations in accordance with
Fed.R.Civ.P. 34. Plaintiffs’ Reply Memorandum at 4-5. See William A. Gross Const.
Assoc., 256 F.R.D. at 136 (“the proposed methodology must be quality control tested to
ensure accuracy”). Finally, Plaintiffs maintain that given the court-imposed ESI discovery
3
Defendants’ m otion to disqualify is directed to D4, LLC, a Rochester, New York com pany that
provides E-Discovery and ESI services.
6
Case 1:08-cv-00380-WMS-LGF Document 456 Filed 05/21/13 Page 7 of 7
deadline of October 23, 2012, Plaintiffs were required to preserve Plaintiffs’ objection to
Defendants’ expected ESI methodology by filing the instant motion. Id.
Here, Defendants indicate they are prepared to meet and confer with Plaintiffs and
Plaintiffs’ ESI consultants, who are not disqualified, regarding Defendants’ ESI production
using predictive coding. Shinaman Declaration ¶ 16; Defendants’ Memorandum at 6.
Based on Defendants’ expressed awareness of Defendants’ discovery obligations,
Shinaman Declaration ¶ 17 (“Defendants . . . will fulfill their obligations to comply with
[Plaintiffs’} document requests”) the court also need not, as Plaintiffs’ request, remind
Defendants of relevant considerations regarding Defendants’ use of predictive coding
regarding ESI document production obligations under Fed.R.Civ.P. 34(a). Accordingly, it
is not necessary for the court to further address the merits of Plaintiffs’ motion at this
time.
CONCLUSION
Based on the foregoing, Plaintiffs’ motion (Doc. No. 360) is DISMISSED without
prejudice.
SO ORDERED.
/s/ Leslie G. Foschio
________________________________
LESLIE G. FOSCHIO
UNITED STATES MAGISTRATE JUDGE
Dated: May 21, 2013
Buffalo, New York
7