Changing the Game of
Antibody Validation
David McAdams
Professor of Economics, Duke
Antibody Validation Conference, Asilomar
September 2016
1
A bit about me …
1. Research: Economic theorist
– economic theory of games with private information, esp.
equilibrium foundations and market design
– now: economic theory of medicine, esp. antibiotic resistance
– NOT an expert in biochemistry
2. Teaching: Game theory
3. Practice: Game-Changer Files
2
A bit about me …
1. Research: Economic theorist
2. Teaching: Game theory
– teach “art of transforming strategic situations” to MBAs
– focus on creative thinking / brainstorming
– examples of great student projects:
• “A Scourge of Piracy in Somali Seas”
• “Drug-Seeking Patients in the ED”
• “The Spanx Dilemma”
– inspired me to engage in real-world
challenges, starting in 2012
3. Practice: Game-Changer Files
researched in 2012
A bit about me …
1. Research: Economic theorist
2. Teaching: Game theory
3. Practice: Game-Changer Files
– Oct 2013: “Changing the Debt-Ceiling Game” [NYT op-ed]
– Aug 2014: “Prescription Drug Overdose in NC” [NPR interview]
– Feb 2015: “Alternatives to a Military Response to Islamic State”
[BBC Radio interview]
– May 2015: “Game Theory of Antibody Validation” [unsolicited
email to David Rimm after reading about his validation work]
– June 2016: joined Impact Assessment Team for Chemical
Probes Portal, a Structural Genomics Consortium spinoff
focused on problem of invalid chemical-probe use
4
My Goals for this Talk
Game theory of standards
– conceptual framework [to facilitate discussion]
– examples outside of science [to help spark ideas]
Application to antibody validation
– promote discussion [I hope!]
– highlight some non-obvious potential risks
5
Game = Any situation with multiple
decision-makers (or “players”) whose
choices impact one another.
.”
HOSPITAL
OFFICE
SCHOOL
Games
of Life
LAB
HOME
Research-Antibody Use
[Strategic Ecosystem]
SCIENTISTS
PRODUCERS
JOURNALS
MANY OTHER PLAYERS
8
Many Decisions Impact Others [Games!]
SCIENTISTS
–
–
–
–
antibody choice
self-validation
methods reporting
sharing information
about performance
–…
9
Many Decisions Impact Others [Games!]
PRODUCERS
– validation investment
– reagent identification
– sharing information
about products
–…
SCIENTISTS
–
–
–
–
antibody choice
self-validation
methods reporting
sharing information
about performance
–…
10
Many Decisions Impact Others [Games!]
PRODUCERS
– validation investment
– reagent identification
– sharing information
about products
–…
SCIENTISTS
JOURNALS
–
–
–
–
antibody choice
– paper acceptance
self-validation
– requirements
methods reporting – …
sharing information
about performance
–…
11
Many Decisions Impact Others [Games!]
PRODUCERS
– validation investment
– reagent identification
– sharing information
about products
–…
SCIENTISTS
JOURNALS
–
–
–
–
antibody choice
– paper acceptance
self-validation
– requirements
methods reporting – …
sharing information
about performance
–…
MANY OTHER PLAYERS
12
Three Reasons for Standards
1. HELP PEOPLE MAKE BETTER CHOICES
2. ACHIEVE COORDINATION
3. INCENTIVIZE PROSOCIAL BEHAVIOR
13
1. Making Better Choices
KEY FEATURE: You “internalize” consequences
of your own decisions
– e.g. designing an experiment using antibodies
Good
Choice
“YOU”
Bad
Choice
14
1. Making Better Choices
By offering recommended
choice, standard can help
people avoid mistakes
Good
Choice
“YOU”
Bad
Choice
15
2. Achieving Coordination
KEY FEATURES: Social benefits from coordination
+ Incentive to behave like others
– e.g. standard validation methods
Choice
#1
Choice
#2
16
2. Achieving Coordination
By providing focal choice,
standard can facilitate /
accelerate coordination
Choice
#1
Choice
#2
17
3. Incentivizing Prosocial Behavior
KEY FEATURES: Lack of individual incentive for
prosocial behavior
– e.g. sharing results of self-validation studies
Benefits
Others
“YOU”
Benefits
Self
18
3. Incentivizing Prosocial Behavior
By designating required /
customary choice, standard
can induce prosocial behavior
Benefits
Others
“YOU”
Benefits
Self
19
Key to Successful Standards
Purpose of
Standard
Complementary policies & institutions to:
Promote
awareness
Induce / compel Induce / compel
adoption
maintainence
Better choices
Coordination
[brand-new standard]
Coordination
[switching standards]
Prosocial behavior
20
Roadmap
Game theory of standards
– basic purposes of standards
• empower, coordinate, and/or incentivize
– basic types of standards
• unconditional vs conditional
• fixed vs accumulative
– illustrative analogies outside of science
Application to antibody validation
21
Unconditional vs Conditional
What hand to shake with?
UNCONDITIONAL STANDARD
Unconditional standards
specify the same behavior
regardless of circumstance.
When to cross intersection?
CONDITIONAL STANDARD
Conditional standards
specify different behavior
in different circumstances.
22
Two Types of Dynamic Standards
FIXED STANDARD
Fixed standards cannot be
easily changed or replaced
[“Standards Trap”]
ACCUMULATIVE STANDARD
Accumulative standards
build on previous standards
[escape Standards Trap!]
23
Roadmap
Game theory of standards
– basic purposes of standards
– basic types of standards
– illustrative analogies outside of science
• professional certifications
• food-quality designations
• social-network interoperability
Application to antibody validation
24
Microsoft Certified Professionals
“Absolutely the certification changed
my life: The moment I got my first MCP
it was that much easier to get the
interviews and ultimately secure a job.”
-- Sajid Ali, Microsoft Certified Professional
25
Impacts of Certification
Qualified
Unqualified
“YOU’RE HIRED!!”
hiring firm
Unemployed Workers
1. identify qualified workers
26
Impacts of Certification
Qualified
Unqualified
“YOU’RE HIRED!!”
hiring firm
Unemployed Workers
1. identify qualified workers
2. incentivize workers to
become qualified
27
Organic-Food Certification
• Certifying “organic” food ingredients
– application / “Organic System Plan” [each farm must
meet standards but can do so in different ways]
– on-site inspection [to verify application]
• Multiple grades for “organic food”
– “100% organic”
– “organic” if >95% organic ingredients
– “made with organic ingredients” if >70% organic
28
Impacts of Certification
• empower: reduces consumer confusion
• coordinate: more organic farms leads to …
– more organic-certified food processors
– better/more organic seed varieties
– dedicated space on store shelves for organic
• incentivize: premium “organic” price encourages
farmers to go organic
29
Roadmap
Game theory of standards
– basic purposes of standards
– basic types of standards
– illustrative analogies outside of science
• professional certifications
• food-quality designations
• social-network interoperability
Application to antibody validation
30
Facebook’s Strategic Ecosystem
Reader
Sharer
Advertiser
31
Facebook-Free Sharing
32
Facebook-Free Reading
“Geek alert! I have just discovered how to
add an RSS feed from a Facebook Page into
my Hootsuite stream!”
- “How to add a Facebook Page RSS feed to your reader”
by Julia Doherty, green-umbrella.biz, November 2013
Facebook’s FUTURE Ecosystem
Reader
Sharer
Advertiser
34
Impacts of inter-operability
among databases [more broadly]
• enables multi-homing / data aggregation
– users can take advantage of multiple databases
• enables complementary services
• encourages entry of complementary databases
• limits dominant database’s ability to exercise
market power or block new competitors
35
Roadmap
Game theory of standards
Applications to antibody validation
– validation-method standards (Mon AM-PM)
– recombinant antibodies (Mon PM)
– antibody databases (Mon PM)
– producers and certification (Tu AM)
– training standards (Tu AM)
– journal standards (Tu PM)
36
Validation-Method Standards
Validation is valuable. We all know that.
But do we need a standard validation method?
Yale, Dept of Pathology
Structural Genomics, U Toronto
Bordeaux et al
Marcon et al
Biotechniques 2010 (for IHC/QIF)
Nature Methods 2015 (for IP)
37
Validation-Method Standards
Validation is valuable. We all know that.
But do we need a standard validation method?
“Five Pillars”
Uhlen et al
Nature Methods (in press)
38
Validation-Method Standards
Validation is valuable. We all know that.
But do we need a standard validation method?
1. can best-practice be pre-specified?
YES [electrician]
Licensed +
Checklist Inspection
NO [furniture maker]
“Master” Designation
+ Expert Evaluation
39
Validation-Method Standards
Validation is valuable. We all know that.
But do we need a standard validation method?
1. can best-practice be pre-specified?
2. are there perverse incentives that need to be
corrected? My own take: “Not really.”
3. are there coordination benefits to adopting a
standard method? My own take: “Yes.”
40
Coordination Benefits
1. MORE IMPACT
– easier for others to understand / reproduce
2. EASIER TO MONITOR AND ENFORCE (if necessary)
3. MORE INFORMATION SHARING
– more journal space devoted to validation findings
4. MORE SUPPORTIVE INFRASTRUCTURE
– “economies of scale” reduce cost of validation
– better customer service from producers
41
Potential Coordination Harms
METHODOLOGICAL STRAIGHTJACKET
– substantially better (non-standard) approaches may
be available in some applications
– BUT if funders/journals require standards be
followed, scientists may feel compelled to do so
DAMPER ON INNOVATION
– if new-and-improved methods are not adopted,
scientists have little reason to innovate
42
Benefits without the Harms
STRIKE A CONDITIONAL BALANCE
– different applications call for different methods
 design a conditional standard
– that said, there is a tradeoff as less-conditional
standards induce stronger economies of scale
BUILD AN ACCUMULATIVE STANDARD
– updating the standard as we learn more
– expanding the standard to cover more reagents and
more application areas
43
Meaningful Comparisons
Means to compare affinity reagents are needed
– help scientists choose among reagents
– allow best-practice standards to be updated
Type of
Reagent
Application
WB
IHC
IP
…
Polyclonal
Monoclonal
Recombinant
…
44
Meaningful Comparisons
What can we learn from comparison sites?
Summary
statistic …
Details allowing for
customer customization
Example: Scoring Reproducibility
polyclonal
monoclonal
monoclonal w/ best-practice certification
recombinant
recombinant w/ public sequence
Roadmap
Game theory of standards
Applications to antibody validation
– validation-method standards (Mon AM-PM)
– recombinant antibodies (Mon PM)
– antibody databases (Mon PM)
– producers and certification (Tu AM)
– training standards (Tu AM)
– journal standards (Tu PM)
47
Antibody Databases
Validation standards may increase
coordination benefits of antibody databases
– user-generated self-validation data could
drive scientists to “home” at one database
Yet as antibody databases become more
valuable, they may also become more
powerful …
48
Market-Power Concern?
EXAMPLE #1 [Naked abuse of power]
A dominant distributor rewards users for
sharing self-validation findings, which are
only shared with “subscribers.”
The distributor then leverages its hold on
data to squeeze both producers and users.
49
Market-Power Concern?
EXAMPLE #2 [Somnolent market leader]
A dominant antibody database has become
so useful that everyone now relies on it.
New databases with innovative approaches
may be unable to break into the market.
The dominant database is then under little
pressure to improve.
50
Antibody-Database Interoperability
Interoperability minimizes market-power
concerns, while also increasing benefits
that antibody databases can deliver:
– scientists can develop and publish
programmable tools that draw on databases
– incumbent databases need to continually
deliver valuable services to keep users
51
Roadmap
Game theory of standards
Applications to antibody validation
– validation-method standards (Mon AM-PM)
– recombinant antibodies (Mon PM)
– antibody databases (Mon PM)
– producers and certification (Tu AM)
– training standards (Tu AM)
– journal standards (Tu PM)
52
Impacts of Validation Certification
Validated Unvalidated
scientist
Antibodies for Target X
NO VALIDATION STANDARDS
+ NO CERTIFICATION
“I’d like to use a valid antibody.”
1. identify valid antibodies
“But it’s hard to tell which is valid.”
“I guess I’ll just use the one that
other people have published with.”
53
Impacts of Standard + Certification
Validated Unvalidated
scientist
Antibodies for Target X
VALIDATION STANDARDS
+ CERTIFICATION
“I’d like to use a valid antibody.”
1. identify valid antibodies
“Let me limit my search to those
that are certified validated.”
54
Impacts of Standard + Certification
Validated Unvalidated
scientist
Antibodies for Target X
VALIDATION STANDARDS
+ CERTIFICATION
“I’d like to use a valid antibody.”
1. identify valid antibodies
2. incentivize producers to
invest in validation
“Let me limit my search to those
that are certified validated.”
55
Should Certification be Required?
If journals or funders were to require that only
certified-validated antibodies be used,
producers would be effectively compelled to
“adopt” the certification process.
Is compelling certification in the public interest?
– is there an important coordination benefit?
– is certification necessary to induce some important
prosocial behavior?
My own take is “No”
56
Case Against Mandatory Certification
The purpose of certification is to allow firms to
prove something about themselves to customers
Trusted producers do not need to prove anything
– if they say that an antibody is validated
according to standards, then it is
Certification is valuable mainly for those whose
customers need reassurance, e.g., a firm that
previously had a reputation for low quality
57
Roadmap
Game theory of standards
Applications to antibody validation
– validation-method standards (Mon AM-PM)
– recombinant antibodies (Mon PM)
– antibody databases (Mon PM)
– producers and certification (Tu AM)
– training standards (Tu AM)
– journal standards (Tu PM)
58
Incentives for Training
lab leader
publish more
or train more?
PhDs
• Lab leaders want what
is best for their PhDs
and post-docs.
• In the scientific job
market, publications
are observable while
training is not.
• Lab members’
incentives therefore
tilt toward less focus
on training.
59
Impact of Training Standards
lab leader
publish more
or train more?
TRAINING STANDARDS
• Well-designed curriculum
lowers cost of training
 PhDs will choose to
train more
– empowering PhDs
PhDs
60
Impact of Training Standards
lab leader
TRAINING STANDARDS
• Well-designed curriculum
lowers cost of training
TRAINING CERTIFICATION
• Certification makes
training observable in the
job market  even more
incentive to train
publish more
or train more?
PhDs
– empowering employers
– incentivizing PhDs
61
Roadmap
Game theory of standards
Applications to antibody validation
– validation-method standards (Mon AM-PM)
– recombinant antibodies (Mon PM)
– antibody databases (Mon PM)
– producers and certification (Tu AM)
– training standards (Tu AM)
– journal standards (Tu PM)
62
Journal Standards as “Coordination Game”
Suppose that best-practice standards are established.
Will journals require authors to adopt best practices?
Representative
Journal
Other Journals
Require
Not Require
Require
Not Require
If other journals do not require, each journal has an
incentive to follow suit, to avoid losing submissions
63
Journal Standards as “Coordination Game”
Suppose that best-practice standards are established.
Will journals require authors to adopt best practices?
Representative
Journal
Other Journals
Require
Not Require
Require
Not Require
If other journals require, each journal has an incentive
to follow suit, now to avoid low-quality “lemons”
64
Journal Standards as “Coordination Game”
Suppose that best-practice standards are established.
Will journals require authors to adopt best practices?
Representative
Journal
Other Journals
Require
Not Require
Require
Not Require
“Everyone requires” and “No one requires” are both
stable equilibria  Coordinated action is necessary to
65
switch to “Everyone requires”
Key to Successful Standards
Purpose of
Standard
Complementary policies & institutions to:
Promote
awareness
Induce / compel Induce / compel
adoption
maintainence
Better choices
Coordination
[brand-new standard]
Coordination
[switching standards]
Prosocial
behavior
66
Key to Successful Standards
Purpose of
Standard
Complementary policies & institutions to:
Promote
awareness
Induce / compel Induce / compel
adoption
maintainence
Better choices
Coordination
[brand-new standard]
Coordination
[switching standards]
Prosocial
behavior
Journals / Funders
Can Play Key Role
67
Key to Successful Standards
Purpose of
Standard
Better choices
Complementary policies & institutions to:
Promote
awareness
Induce / compel Induce / compel
adoption
maintainence
Avoid Imposing
Stifling Standards
Coordination
[brand-new standard]
Coordination
[switching standards]
Prosocial
behavior
Journals / Funders
Can Play Key Role
68
Key to Successful Standards
Purpose of
Standard
Complementary policies & institutions to:
Promote
awareness
Induce / compel Induce / compel
adoption
maintainence
Avoid Imposing
Stifling Standards
Better choices
Coordination
[brand-new standard]
Coordination
[switching standards]
Prosocial
behavior
Accumulative
Standards Avoid
Difficult Switching
Journals / Funders
Can Play Key Role
69
Key to Successful Standards
Purpose of
Standard
Better choices
Complementary policies & institutions to:
Promote
awareness
Induce / compel Induce / compel
adoption
maintainence
Ensure “Decision
Aids” are Open +
Interoperable
Avoid Imposing
Stifling Standards
Coordination
[brand-new standard]
Coordination
[switching standards]
Prosocial
behavior
Accumulative
Standards Avoid
Difficult Switching
Journals / Funders
Can Play Key Role
70