Utilizing Technology Assist Review to
Reduce an Organization's Redundant,
Obsolete and Trivial Data
Mid-Michigan ARMA Chapter Spring Seminar
April 18, 2016
©2016 Warner Norcross & Judd LLP. All rights reserved.
WNJ.com
Agenda
•
•
•
•
•
•
•
•
•
•
•
Why is IG Increasingly Important?
Developing a Business Case for IG Improvements
Contours of the Problem
Best Practices
How Technology-Assisted Review (TAR) and Predictive
Analytics Can Help
Background on TAR and Predictive Analytics
Proof of Concept – Library
POC for Clients
Available Cost Savings
Pros and Cons
Questions
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 2
Why is IG Increasingly Important?
Reason #1: Quantity of data doubling every two years
The average Fortune 1000 company manages approximately 1,200
terabytes of electronic information; that is almost 100 billion document
pages. By 2020 this amount is expected to increase by 50X!
1 Office Computer
40 Gb Workstation
(at 50%)
©2016 Warner Norcross & Judd LLP. All rights reserved.
1,500,000
pages
Page 3
>Eiffel Tower
571 feet
Why is IG Increasingly Important?
Reason #2: Organizational data is located in more places
• Computers
• Desktops
• Laptops
• Servers
• Portable Media
•
•
•
•
USB Devices
CD ROMs and DVDs
Tapes
Portable Hard Drives
• Small Scale Digital Devices
• Mobile Phones
• Tablets/eReaders
• GPS devices
• Internet
• Public/Extranet/Intranet
• Social Media
• Facebook/Twitter/LinkedIn
• Cloud Storage
• Public/Private
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 4
Why is IG Increasingly Important?
• Reason #3: Increased Data Security Risks
• December 2013: Target announces that stolen credit
and debit card data included personal information for 70
million customers
• May 2014: Target CEO resigns (after 35 years at
Target)
• August 2014: Target reveals data breach will cost
shareholders $148 million
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 5
Why is IG Increasingly Important?
Reason #4: Increased Employee Hours Spent
Searching
• Employees waste approximately 20% of work hours
searching through unmanaged records for information they
need.
Harvard University
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 6
Why is IG Increasingly Important?
• Reason #5: Increased Data Discovery Costs and
Sanctions
• In Re Fannie Mae Securities Litigation (2009)







Agreed to electronic discovery protocol
Over 400 search terms
660,000 potentially responsive documents
50 contract attorneys to review
$6 Million or 9% of the FNMA’s annual budget to complete
Civil contempt sanction for failure to meet stipulated deadline
Waiver of privilege for all un-reviewed documents as of
deadline
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 7
Why is IG Increasingly Important?
Reason #6: Increased Data Spoliation Sanctions
Organization
Data Preservation Risk
Ramification
Nartion Corporation
• Computer Records
Destroyed
• Claims Dismissed
Collectors Guild Ltd.
• Relevant Information
Destroyed
• Adverse Inference Jury
Instruction
Pribyl
• Overwrote Computer
Information
• Adverse Inference Jury
Instruction
Philip Morris, Inc.
• Failed to Suspend
Deletion of Emails
• $2.75 Million Sanction
• Witness Preclusion
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 8
Why is IG Increasingly Important?
• Reason #7: Increased Data Privacy and Retention
Obligations
• New York hospital entered into a $3.3 million settlement over
failure to adequately implement HIPAA requirements
• Certain statutes/regulations require quick production of
certain records. For example, if the FDA suspects tainted
food, records must be made available within 24 hours. 21
CFR §1.361
• FCPA & SOX: Willful/knowing violation - $5 million in
criminal fines and/or prison for up to 20 Years for individuals
and companies fined up to $20 million
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 9
Assessing Your IG: Asset or Liability?
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 10
Potential Cost Savings Associated
with IG Improvements
1. Reduction in Data Storage Costs
 Assume company has 25,000 Gb of active data
 Assume cost of $15 Gb annually to maintain and store
data
 Assume company can achieve a 35% reduction in data
via improved IG
 Annual estimated cost savings: $131,250
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 11
Potential Cost Savings Associated
with IG Improvements
2. Reduction in Data Breach Exposure
 Likelihood that company will be a victim of a data
breach next year: 11%
 Assume average data breach of 10,000 records
 Average cost per record in a data breach: $200
 Assume 50% reduction in data breach risk via
improved IG
 Annual estimated cost savings: $110,000
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 12
Potential Cost Savings Associated
with IG Improvements
3. Reduction in eDiscovery Costs
 Assume company has an average of three federal and
three state cases per year
 Assume that eDiscovery costs average $35,000 per
federal case and $17,500 per state case
 Assume company can achieve a 50% reduction in
eDiscovery costs via improved IG
 Annual estimated cost savings: $78,750
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 13
Potential Cost Savings Associated
with IG Improvements
4. Reduction in Employee Inefficiency
 Assume company has 100 employees
 Assume average employee spends 150 hours per year
(approximately 3 hours per week) searching through
unmanaged data for information they need
 Assume that it costs an average of $50/hour for each
hour worked by an employee
 Annual estimated cost savings: $75,000
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 14
Potential Cost Savings Associated
with IG Improvements
1.
2.
3.
4.
5.
Reduction in Data Storage Costs:
$131,250
Reduction in Data Breach Exposure: $110,000
Reduction in eDiscovery Costs:
$ 78,750
Reduction in Employee Inefficiency: $ 75,000
Miscellaneous
?
Total:
©2016 Warner Norcross & Judd LLP. All rights reserved.
$395,000
Annually
Page 15
Contours of the Problem
Data explosion
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 16
100,000,000 Pennies at Rockefeller
Center
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 17
What’s the Difference?
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 18
Best Practices
• Updated RIM Policy with appropriate definitions.
 Documents requiring retention = “RECORDS;” documents that may
be deleted at any time, unless a legal hold exists = “NONRECORDS.”
• Update retention schedule so it is current and defensible.
 Statutory/Regulatory
 Legal Considerations
 Business reasons
• Set auto-expiration for records passed their retention periods and nonrecords.
THE PROBLEM:
How to find the documents that must be retained?
How to find the documents that can be deleted?
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 19
Technology-Assisted Review and
Predictive Analytics: Basic Process
• Computer learns from human categorizations
• Computer categorizes non-human-categorized
documents
• Humans assess computer accuracy and act
accordingly
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 20
Decision Tree
Documents
Official Records - Subject to Records
Retention Periods on the Schedule?
Yes
Non-Records? Yes
Retention period
expired? Yes
Not accessed within
X years? Yes
No legal hold
applies? Yes
Defensible Disposition
©2015 Warner Norcross & Judd LLP. All rights reserved.
Page 21
Proof of Concept
Getting started and selecting a cross-functional team to select:
•
•
•
•
•
An appropriate data repository.
An appropriate department or function.
Available subject matter experts (“SMEs”).
A predictive coding vendor, such as Dagger.
A cost-effective document review team.
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 22
Proof of Concept
SAMPLE Librabry Services
Record Series
Retention Period
Creation
of record
+ 10 years
Permanent
Official Record Owner:
person responsible for
retaining and disposing of
record
Official Record
Format
Official Record Location(s)
Administrative
X
Library Services
Electronic
I drive
Billings and Accounting
X
Library Services
Electronic
I drive
Policies
X
Library Services
Both
Publishers
X
Library Services
Electronic
Hard copies, in staff desks;
electronic copies, on I drive
I drive
Vendor Contracts
X
Library Services
Both
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 23
Hard copies, in basement and
file room; electronic copies, on I
drive
Proof of Concept
•
•
•
•
WNJ library files
~500 records, ~1000 non-records
Exceedingly small number of training records
Uncommonly high prevalence of records
 Typical <=5%
 So also added 25K non-records.
Result:
90% accurate
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 24
Proof of Concept:
Differences from real world
Made POC easier
• No date-time restrictions
• No difficulty accessing
client data
• Human classifications
readily available
• One category of record
©2016 Warner Norcross & Judd LLP. All rights reserved.
Made POC harder
• Exceptionally few training
exemplars
Page 25
Pros and Cons: Pros
• Extraordinary savings and ROI
• In widespread use in legal proceedings and by
regulators (e.g., F.T.C., DoJ, S.E.C.)
• Uniformly approved by courts in litigation context
 U.S.: dozens of cases
 Recent Ireland, U.K. approval
• Throw bathwater out without tossing baby
• Find the “Lost Ark”: Identify records in locations
previously unknown
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 26
Pros and Cons: Cons
• Risks
 Erroneous deletion of records or otherwise desirable files
 Mitigation: earmark on retention schedule, place certain
custodians or repositories (e.g., legal) off limits, compare to
records accessibility sans TAR / predictive analytics
• Logistical difficulties granting access to all data repositories
• Unusual / exotic filetypes, database records difficult to
process
• PDFs / other images without extracted text (but can OCR)
• Think hard about accuracy
 Even impressive-sounding “99% accuracy” can be
misleading. You can easily have >99% accuracy and
delete everything you’re supposed to keep.
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 27
CONCLUSION
WITH THE RIGHT KNOWLEDGE
AND EFFORT CAN COME
GREAT REWARDS!
©2016 Warner Norcross & Judd LLP. All rights reserved.
Page 28
Questions?
Dawn Ward
Warner Norcross & Judd
[email protected]
616-396-3039
©2016 Warner Norcross & Judd LLP. All rights reserved.
B. Jay Yelton, III
Warner Norcross & Judd
[email protected]
269-276-8130
Page 29