Am Law 100 Firm Uses
Equivio>Relevance™ to Find More
Relevant Documents and to Find
Them Faster
an Epiq-Equivio Case Study
THE PROBLEM
In today’s litigation environment, completing discovery in a timely and cost-effective
manner can seem an impossible task. The volume of discovery documents exchanged in
routine litigation has risen exponentially over the past few years, and litigation budgets
simply can’t grow the same way. Moreover, quantity impacts quality, with the sheer
volume of documents and emails to be reviewed generating risks of error and omission
in the litigation review process. As discussed in the May 2009 Sedona Working Group
Series paper, “The Sedona Conference Commentary on Achieving Quality in the EDiscovery Process”, existing document review protocols are unable to effectively scale to
meet new demands, and litigation teams must consider new approaches to successfully
navigating this piece of the litigation lifecycle.
Regardless of their size, of course, document review objectives remain constant: (1)
identify relevant and key documents within the document population; (2) carry out the
task in a reasonable, defensible way that remains within the litigation budget.
New technology has led to tools that can better leverage a human’s subjective
understanding of a legal dispute to increase the productivity, accuracy, and consistency
of manual document review. However, a pervasive question when using such tools is
how well these tools actually work. These and other concerns have limited the degree to
which technology has been applied to enhance document review processes.
Recently, in a pilot conducted by Epiq Systems with an Am Law 100 New York-based
firm, Equivio>Relevance™ was tested under controlled conditions to compare its analysis
of an actual discovery document collection to the results of a purely human-based
review. The trial offers dramatic insight into the potential benefits of augmenting human
review with Equivio technology.
PURE HUMAN REVIEW VS. COMPUTER-ASSISTED REVIEW
The litigation team faced a discovery document population of approximately 100,000
documents. Culling exact duplicate documents and applying date range filters cut the
number of documents requiring substantive review by half, but with 47,000 documents
to review, the team had its work cut out for it. Ultimately, using raw manpower and
abundant overtime, the human review team completed its task in approximately 7
weeks. Now, it was time to see how Equivio>Relevance would perform. Key factors
used to measure performance included: (1) implementation time to prepare the
documents for human review; (2) accuracy of the Equivio>Relevance document scores;
and (3) impact on review speed and budget.
Deploying Equivio>Relevance was straightforward. Unlike some other analytical tools on
the market, Equivio>Relevance does not require keyword lists or other complex linguistic
2
analysis. Instead, the Equivio>Relevance process begins with a statistical sampling
algorithm to harvest batches of fifty (50) documents from the document collection. One
or more litigation team members, the designated “experts,” then determine which of
these documents are relevant and which are not. The software uses these relevance
designations to classify documents, and, in turn, to score documents in the collection by
their relevance. To improve the accuracy of the results, multiple sample sets are
analyzed by the experts, with each set of analysis further refining the Equivio>Relevance
document cores.
For this test, a single “expert” who was familiar with the case reviewed 25 sets of 50
documents for relevance. Although at first glance this seems like a large number of
documents, it represented just over 2% of the total document population and required
only about 18 person-hours of review time. Moreover, in a live case, expert analysis of
the sample documents would be incorporated into the remainder of the review, ensuring
that none of this work is duplicated.
Based on the expert analysis, Equivio>Relevance classified all documents in the review
population on a sliding scale of potential relevance. However, instead of taking seven
weeks to complete this task, the Equivio>Relevance scoring process took less than an
hour after the last of the expert relevance determinations had been fed back into the
system. However, speed isn’t everything. What was the quality of the
Equivio>Relevance analysis, compared to the human review that had taken place?
Several different approaches were used to test the value of the Equivio>Relevance
analysis. First, the relevance decisions made by the human review team and
Equivio>Relevance were compared to see how closely the two sets matched one
another. Document for document, the two analyses matched 91.4% of the time, even
though the Equivio>Relevance analysis took only a fraction of the time the human
review required. This is illustrated in Table 1. The human review team and Equivio
agreed on 43,543 documents (3048 responsive and 40,495 not responsive) out of a total
of 47,650 documents. Skeptics of computer-assisted review might argue that an even
higher overlap is required before computer-assisted analysis should be considered
reasonable and defensible. Such an argument, however, relies on the assumption that
the human review team decision was always the correct one—an assumption that was
also tested within this study.
3
Pointing to differences between two different analytical results is meaningless unless a
neutral benchmark can be used to determine whether one—or both—analyses is
incorrect. The standard approach is to use a human “Oracle” to make a final
determination of conflicting document classifications. Within this study, the Oracle
reviewed a statistically sampled selection of documents about which the human review
team and Equivio>Relevance had reached different conclusions regarding their
relevance. To avoid skewing the analysis, the Oracle was not informed which conclusion
each had reached, only that the two had differed for each document. The Oracle’s
decision would be the final word regarding a document’s relevance.
Out of the 4,107 documents in which the review analyses differed, the Oracle reviewed a
statistically drawn sample of 190 documents, or slightly less than 5% of the documents
in dispute. Of those 190 documents, the Oracle determined that Equivio>Relevance was
correct in 147 of the cases. The human review team, by comparison, prevailed in only
43 of the disputed sample documents. As will be discussed below, these findings harbor
considerable potential for enhancing the review team’s ability to effectively mine a
collection for responsive documents.
SIGNIFICANCE OF THE RESULTS
Computer-assisted review is not a replacement for the subjective analysis provided by
human reviewers, nor should it be seen as a replacement for document review projects.
However, computer-assisted review can dramatically increase the efficiency and
accuracy of a document review team’s work, and it makes possible new review
objectives and strategies that may not be possible using traditional document review
procedures.
4
EARLIER, BETTER CASE UNDERSTANDING
Organizing the review of documents by their potential relevance, rather than by more
traditional review criteria, offers a strikingly efficient way of finding key documents
within a review collection. Typically, the vast majority of documents within a discovery
document collection bear little if any relevance to the core legal disputes. Reviewing all
of these documents to find the small set of documents with actual substantive
importance consumes significant resources and wastes valuable time and expense.
In this trial of 47,650 documents, only about 4,624 documents, or about 10% of the
collection, were ultimately found to be responsive. Using traditional document review
procedures, these relevant documents were spread randomly throughout the collection
and were identified on a rolling basis during the seven-week human review. In contrast,
using Equivio>Relevance scores to prioritize the review would have exposed the legal
team to most of the key documents at the onset of the review process. More specifically,
in this trial, the first 20% of the document collection as organized by the
Equivio>Relevance relevance scores would have presented the reviewers with 75% of
the total number of responsive documents identified within the collection. This is shown
in Figure 1, which details the rate at which responsive documents were fed to the review
team in a the original review process, as against the rate that the review team would be
fed responsive documents following relevance prioritization by the Equivio product.
Clearly, earlier awareness of the responsive documents would materially influence future
litigation strategy and give the litigation team a significant competitive edge in terms of
case understanding.
5
GREATER CONSISTENCY WITHIN THE DOCUMENT REVIEW
Human review, especially team review, always contains differences of opinions. A
document that one reviewer deems non-responsive may be classified as highly relevant
by a different reviewer who has greater knowledge of the case. In the rush of
completing a large-scale document review, these differences of opinions are rarely
identified, much less resolved one way or the other.
One of the great advantages of computer-assisted review is that algorithms yield
consistent document classification across the collection. Based on the initial expert
review of a selection of documents, Equivio>Relevance applies the same review standard
to all documents within the collection, consistent with the original input from the senior
expert. Using Equivio>Relevance, it is possible to identify discrepancies – that is,
documents for which the computer-generated relevance scores differ from the human
review – earlier in the review process. Analysis of these discrepancies, in turn, allows
the review leaders to identify systematic errors being made by the review team and to
issue timely adjustments to the guidance provided to the reviewers.
6
QUALITY ASSURANCE TO CORRECT SUBJECTIVE CODING ERRORS
Because of the consistency of the Equivio>Relevance document analysis and its
application to the entire document population, the tool can also be used to enhance
review quality. Specifically, as a quality control procedure, second-level document
reviewers can focus on documents that differed in their Equivio and human relevance
categorization, quickly locating responsive documents incorrectly coded by the human
team as non-responsive, and non-responsive documents incorrectly coded as responsive.
Within the case study document collection, Equivio>Relevance ultimately identified more
than 1,000 additional responsive documents that had been overlooked or miscategorized by the human review team. Some of these omissions were simple mistakes
that could have been made when a reviewer was distracted or tired. Other omissions
reflected the increasing sophistication of the review team as they worked through the
collection and learned more about the facts of the case. Regardless of the cause,
Equivio>Relevance increased the reliability of the review without reducing the human
review team’s ability to further refine the Equivio>Relevance results.
FASTER COMPLETION OF THE REVIEW
A review team can further leverage Equivio>Relevance by using it in conjunction with
Equivio’s near-duplication tools to categorize entire sets of substantially similar
documents and to further condense the number of independent relevance determinations
that need to be made. Human reviewers would continue to hold responsibility for
making the subjective judgment calls about relevance and privilege, but the combination
of serving up relevant and related documents further multiplies reviewer productivity
while maintaining consistency in document classification decisions.
CONCLUSION
This test of Equivio>Relevance demonstrates the ability of the tool to improve
prioritization, monitoring, and correction of human document review projects.
Importantly, the client was able to integrate Equivio>Relevance into its ongoing litigation
process without discarding many of the document review protocols and procedures that
it had developed over time and with which it was already comfortable. Moreover, the
fact that Equivio>Relevance analysis is based on the knowledge of the firm’s case
experts, and Equivio>Relevance’s ability to provide measurable feedback on the quality
of the analytical results being achieved by the software provided an important level of
assurance for users. Ultimately, Equivio>Relevance scores are not intended to replace
human review, only to increase its efficiency. Human reviewers can and should still be
deployed to review the document collection. With Equivio>Relevance, however, they will
be exposed to high value documents much earlier in the review, and they will also have
the fallback of Equivio relevance scores to monitor and enhance review quality.
7
ABOUT EQUIVIO
Equivio develops text analysis software for e-discovery. Users include the DoJ,
the FTC, KPMG, Deloitte, plus hundreds of law firms and corporations. Equivio
offers Zoom, an integrated web platform for analytics and predictive coding.
Zoom organizes collections of documents in meaningful ways. So you can zoom
right in and find out what’s interesting, notable and unique. Request a demo at
[email protected] or visit us at www.equivio.com.
Zoom in. Find out.
Equivio™, Equivio Zoom™, Equivio>NearDuplicates™, Equivio>EmailThreads™, Equivio>Compare™, Equivio>Relevance™ are
trademarks of Equivio. Other product names mentioned in this document may be trademarks or registered trademarks of their
respective owners. All specifications in this document are subject to change without prior notice.
© Copyright 2012 Equivio
8