UNCLASSIFIED
UNCLASSIFIED
Defense Forensic Science Center
Evaluating Probabilistic Genotyping
Results
Joel Sutton, Technical Leader
USACIL DNA Casework Branch
7th Annual Prescriptions for Criminal Justice Forensics , 3 June 2016
UNCLASSIFIED
UNCLASSIFIED
Disclaimer
The opinions or assertions contained herein are the
private views of the author and are not to be construed
as official or as reflecting the views of the Department of
the Army or the Department of Defense.
Names of commercial manufacturers or products
included are incidental only, and inclusion does not imply
endorsement by the authors, DFSC, OPMG, DA or DoD.
Unless otherwise noted, all figures, diagrams, media,
and other materials used in this presentation are created
by the respective author(s) and contributor(s) of the
presentation and research.
UNCLASSIFIED
2
UNCLASSIFIED
Outline
•
•
•
•
•
•
•
Background
Various methods for interpreting/reporting results
Testing different LR propositions
Testing multiple persons of interest
Verbal scale
Uninformative – the “new inconclusive”
Admissibility and testimony strategies
UNCLASSIFIED
3
UNCLASSIFIED
Background
• The Defense Forensic Science Center has been online
using probabilistic genotyping (PG) exclusively in
casework since Nov 2014
• Our primary casework is sexual assault evidence
• At the current time, we have testified in 18 courtmartials using PG results
• One thing we have learned is that the validation of
the software turned out to be the “easy” part
• Interpreting and reporting the results has actually
been the real challenge
• It’s not even the “PG” part that is the challenge, but
more with the likelihood ratio we are reporting
UNCLASSIFIED
4
UNCLASSIFIED
Practical Considerations
• As laboratories continue to move towards
interpreting and reporting DNA results with
probabilistic genotyping, there are considerations
the legal community needs to be aware of
 Training practitioners
 Training stakeholders
• The following slides give examples for some of
these considerations
UNCLASSIFIED
5
UNCLASSIFIED
Methods For Interpreting/Reporting Results
• DNA results traditionally have one of three conclusions:
 inclusion
 exclusion
 inconclusive
• PG is based on probabilities, most use a traditional LR framework
Pr( E | H 1, I )
LR 
Pr( E | H 2, I )
This is the basic LR
formula w/o adding in
the additional
probabilities for drop
• Some have advocated no longer using terms like “inclusion” or
“exclusion” for PG conclusions, but instead simply report out the LR
and let the number speak for itself
 An LR > 1 favors the prosecution hypothesis
 An LR < 1 favors the defense hypothesis
 An LR =1 is considered neutral
UNCLASSIFIED
6
UNCLASSIFIED
Methods For Interpreting/Reporting Results
• This has led to two general schools of thought
with interpreting PG results based on an LR
 Run all mixtures through the program
 Evaluate all the data first to determine suitability and
try to exclude the POI first
There are pros and cons to each approach
UNCLASSIFIED
7
UNCLASSIFIED
Methods For Interpreting/Reporting Results
Run everything through
• Pro: considered less
subjective
• Pro: with complex
mixtures, can you really
ever “include” or “exclude”
• Con: “junk in, junk out”
• Con: who’s the expert
Interpret first
• Pro: if one can interpret first,
you can “weed out” some of
the more difficult mixtures as
being uninterpretable
• Pro: The analyst still serves
as the expert for determining
inclusions/exclusions
• Con: There is some
subjectivity with this in terms
of interpreting complex
mixtures and making decisions
up front
UNCLASSIFIED
8
UNCLASSIFIED
Testing Different Hypotheses
• Hypotheses are also referred to as propositions
• Assumptions considered for proposition
 Number of contributors in the mixture
 Any known contributors
• Depending on these propositions, you can get very different
LRs
• When setting propositions, there are a few things to keep in
mind:
 We generally know the prosecution hypothesis
 We never know nor should we predict what the defense
hypothesis will be
• However, knowing the math behind the LR, we can predict:
 What is most reasonable
 What is most informative
UNCLASSIFIED
9
UNCLASSIFIED
LR Example
• Sexual Assault case – evidence is semen on a
vaginal swab from Victim (V)
• DNA results - mixture of two individuals, with all
the alleles consistent with the V and Suspect (S)
• Prosecution hypothesis – V + S
• Defense hypothesis options – unknown individual
(U) contributed the semen
 V+U
 U+U
UNCLASSIFIED
10
UNCLASSIFIED
LR Example
• (V+S/U+U) – this LR would be large
 The more U on bottom compared to top LR goes
“waaay” up
 Has to do with the math
• (V+S/V+U) is smaller
 The ratio of U in the LR has been minimized
 Reasonable to assume the V on her own swab
 Genotype possibilities have been constrained to the
foreign DNA contributor which makes the LR lower
but also more informative (less genotype possibilities
promote better exclusionary power)
UNCLASSIFIED
11
UNCLASSIFIED
Which Proposition To Use
•It is conventional wisdom to try and minimize the
ratio of unknowns with the LR where possible
• This reduces the LR (gives a more conservative
estimate)
• Otherwise the LR can blow up based on the math
which does not benefit the accused
• Making reasonable assumptions for known
contributors and keeping the number of contributors
the same between propositions is generally preferred
• If unsure, test and include multiple propositions to
assess at trial
UNCLASSIFIED
12
UNCLASSIFIED
Multiple Persons Of Interest
•Two choices to make with your proposition:
 Evaluate each individual separately in the mixture
 Evaluate them together in the mixture
• If your software can, better to evaluate together
• Case circumstances and pre-trial conferences will
dictate which proposition will be reasonable for the
defense
UNCLASSIFIED
13
UNCLASSIFIED
Verbal Scale
• Often these numbers associated with the LRs
being reported will be drastically different
• What does “50 times more likely” mean
compared to “50 septillion times more likely”
• The differences in the numbers speak to the
weight
• A verbal scale is mostly arbitrary, but may help in
verbalizing what these differences mean
UNCLASSIFIED
14
UNCLASSIFIED
USACIL DNA Verbal Scale Using STRmix™
LR =
Verbalized as…
10 billion or greater
“Very strong support”
10 million to 10 billion
“Strong support”
10,000 to 10 million
“Moderate support”
1000 to 10,000
“Weak support”
>10 to 1000
“Very weak support”
1 to 10
“Uninformative”
0 to 1
“Supports an exclusion”
This scale is data dependent, based on the
deconvolution, and used when requested to put the
numbers in some sort of general context
UNCLASSIFIED
15
UNCLASSIFIED
What Does “Uninformative” Mean?
This paper demonstrates that LR distributions for competing propositions are
very well separated with good quality data, but converge to 1 with low quality
data. At that point, the LR becomes uninformative for inclusions/exclusions ,
essentially inconclusive for either proposition
UNCLASSIFIED
16
UNCLASSIFIED
Admissibility And Courtroom Testimony
This paper drives home some important points on this subject
and introduces the concept of a software-expert pair (SEP)
UNCLASSIFIED
17
UNCLASSIFIED
Admissibility
• From the authors’ point of view PG is reliable if
properly validated
 ISFG recommendations
 SWGDAM validation guidelines
 OSAC is writing validation standards
• Peer-reviewed papers from the developers is crucial –
the software may be new, but not the concepts
• The concepts of an LR, MCMC, modeling stutter and
peak height variability, and calculating match
probabilities are not novel and well-published
UNCLASSIFIED
18
UNCLASSIFIED
Courtroom Testimony
• The courts expect the results and any software
not only be properly validated, but also that it is
appropriately applied
• “Appropriately applied” relates to the expert
using it and testifying to it. The software and
expert work together (SEP)
•“It is imperative to note that to have a successful
SEP in use for forensic casework requires not only
a validated software tool, but a fully trained and
competent expert that uses the chosen software”
UNCLASSIFIED
19
UNCLASSIFIED
Admissibility And Courtroom Testimony
• FRE 702 speaks to experts and the right for the
defense to confront these witnesses
• It’s the expert after all, not the software that will
have to testify
• “It is accepted that no analyst is required to
understand the mathematics and computer
program to the extent that they could recreate the
system, except the developers themselves.
However it is an expectation that analysts at least
understand the workings of any system they use
to be able to understand and explain the results.”
UNCLASSIFIED
20
UNCLASSIFIED
Summary
• Lots of PG systems out there, all slightly different
• Individual laboratory protocols may use the same
system differently
• Read the literature to better understand the
fundamental concepts – training staff is key
• Understand the LR framework
• These are not expert systems – do not treat it as
a “black box”
• Work with your customers and stakeholders –
they need to be trained as well
UNCLASSIFIED
21
UNCLASSIFIED
Questions?
Thank you for your time!
Joel Sutton
[email protected]
404-469-4469
USACIL DNA Technical
Leader
UNCLASSIFIED
22