A Popperian Platform for
Programming and Teaching the
Global Brain
Karl Lieberherr
Ahmed Abdelmeged
Northeastern University, CCIS, PRL,
Boston
7/31/2017
1
A claim is …
• information about one’s performance when
interacting with another clever being in a
specific domain.
• information about the performance of one’s
program.
4/24/2011
Crowdsourcing
2
Outline
Introduction
Theory
Methods
Results
Conclusion
7/31/2017
Introduction
Theoretical Background
Methods of Exploration
Results
Conclusions and Future Work
3
Introduction
Theory
Methods
Results
Conclusion
Introduction
SCG = Scientific Community Game = Specker Challenge Game
• Explanation: SCG as a general pattern behind many
different competitions: topcoder.com, kaggle.com,
tunedit.org, Renaissance, …
• Make SCG a part of cyber-infrastructure (e-science) to
support teaching and innovation in constructive
domains.
• SCG usage for teaching
– Innovation Success with Undergraduates using SCG on
piazza.com: Qualitative Data Sources & Analysis
– Avatar competitions are not for teaching (but for
competitive innovation)
• Theoretical Properties of SCG
7/31/2017
4
Introduction
Theory
Methods
Results
Conclusion
Specker
• Claims:
– Specker(set X, set Y(X), function f(X,Y)->[0,1], constant
c): ForAll x in X Exists y in Y(X): f(x,y)≥c
• Example 1
–
–
–
–
X = Conjunctive Normal Forms with various restrictions
Y(X) = Assignments to CNFs
f(x,y) = fraction of satisfied clauses in x under y
c in [0,1], e.g., c= 0.61
• Example 2
–
–
–
–
7/31/2017
X = [0,1]
Y(X) = [0,1]
f(x,y)=x*y+(1-x)(1-y^2))
c in [0,1], e.g., c=0.61
5
Introduction
Theory
Methods
Results
Conclusion
Popper
• One of the philosophers of science who has
had the biggest impact.
• Popper’s solution: Falsificationism: A claim is
falsifiable if you can imagine an observation
that would cause you to reject the theory.
• That a claim is "falsifiable" does not mean it is
false; rather, that if it is false, then some
observation or experiment will produce a
reproducible result that is in conflict with it.
7/31/2017
6
Introduction
Theory
Methods
Results
Conclusion
What SCG helps with
• How to identify experts?
• How to decide if an answer is worthwhile?
– Use scholars to choose the winners
• How to organize egoistic scholars to
produce social welfare: knowledge base
and know-how how to defend it.
– The scholars try to reverse engineer the
solutions of winning scholars.
7/31/2017
7
Claim involving Experiment
Claim ExperimentalTechnique(X,Y,q,r)
I claim, given raw materials x in X,
I can produce product y in Y of quality q
and using resources at most r.
4/24/2011
Crowdsourcing
8
Who are the scholars?
• Scientists
• Students in a class room
– High school
– University
• Members of the Gig Economy
– Between 1995 and 2005, the number of selfemployed independent workers grew by 27 percent.
• Potential employees (Facebook on kaggle.com)
• Anyone with web access; Intelligent crowd.
7/31/2017
9
Kaggle.com Competitions
2012
• Facebook recruiting competitions
– Task: Data scientist
– Reward: Job
– Teams: 197
• Heritage Health Prize
– Task: Hospital admissions
– Reward: $ 3 million
– Teams: 1118
• Chess ratings – Elo versus the Rest of the World
– Task: Predict outcome of chess games
– Reward: $ 617
– Teams: 257
7/31/2017
10
Kaggle.com Competitions
2012
• Eye Movements Verification and Identification
– Task: Identify people
– Reward: Kudos
– Teams: 51
• EMC Data Science Global Hackathon
– Task: Air Quality Prediction
– Reward $ 7030
– Teams: 114
7/31/2017
11
Introduction
Theory
Methods
Results
Conclusion
What Scholars think about!
• If I propose claim C, what is the probability that
– C is successfully refuted
– C is successfully strengthened
• If I try to refute claim C, what is the probability
that I will fail.
• If I try to strengthen claim C, what is the
probability that I will fail?
• Scholars are free to invent; game rules don’t limit
creativity!
7/31/2017
12
Introduction
Theory
Methods
Conclusion
Results
Degree of automation with SCG(X)
avatar Bob
scholar Alice
degree of automation used by scholar
1
0
no automation
human plays
some automation
human plays
more applications:
test constructive knowledge
7/31/2017
full automation
avatar plays
transfer to reliable,
efficient software
13
happy = can contribute, not ignored, can be creative, can thrive, have opportunity to learn
Organizational Problem Solved
• How to design a happy scientific community
that encourages its members to really
contribute.
• Control of scientific community
– tunable SCG rules
– Specific domain, claim definition to narrow scope.
7/31/2017
14
Introduction
Theory
Methods
Results
Conclusion
What is a loose collaboration?
• Scholars can work independently on an aspect of
the same problem.
• Problem = decide which claims in playground to
oppose or agree with.
• How is know-how combined? Using a protocol.
– Alice claimed that for the input that Alice provides,
Bob cannot find an output of quality q. But Bob finds
such an output. Alice corrects.
– Bug reports that need to be addressed and
corrections.
Playground = Instantiation of Platform
7/31/2017
15
Introduction
Theory
Methods
Results
Conclusion
Theory
• Extensive Form Representation of Game
• Community Property: All faulty actions can be
exposed.
• SCG Equilibrium
• Convergence to optimum claim
7/31/2017
16
Introduction
Theory
Methods
Results
Conclusion
Extensive-form representation
1. the players of a game: 1 and 2
2. for every player every opportunity they have
to move
3. what each player can do at each of their
moves
4. what each player knows for every move
5. the payoffs received by every player for every
possible combination of moves
7/31/2017
17
Large Action Spaces
• Thick arrows mean: select from a usually large
number of choices
1
2
7/31/2017
18
Introduction
Theory
Methods
Conclusion
Results
Refutation Protocol
• Collects data given to predicate p. Alternates.
refute(C,proposer,other)
other tries to make
p false while proposer
tries to make p true.
p(C, …)?(1,-1):(-1,1)
claim
payoff for proposer
if p true (defense)
payoff for other
if p true (defense)
payoff for proposer payoff for other
if p false (refutation) if p false (refutation)
7/31/2017
p false means successful
refutation.
p true means successful
defense.
19
Introduction
Theory
1 scholar
2 scholar
refute(C, proposer,other)
p(…)?(proposer,other):
(proposer,other)
refute attempt C
refute(C,1,2)
p(C, …)?(1,-1):(-1,1)
Methods
Conclusion
Results
1
propose claim C from Claims
agree attempt C
2
strengthen attempt C’ => C
refute(C’,2,1)
refute(C,2,1)
p(C’, …)?(1,-1):(-1,1)
p(C’, …)?(-1,1):(1,-1)
7/31/2017
p(C, …)?(1,-1):(-1,1)
p(C, …)?(0,0):(1,-1)
20
Reinterpret Refutation
• Refutation leads to successful strengthening
or successful agreement.
7/31/2017
21
Introduction
Theory
Methods
Results
Conclusion
Game Rules for Playground
• All objects exchanged during protocol must be
legal and valid.
• Each move must be within time-limit.
• Scholar who first violates a playground rule,
loses.
7/31/2017
22
Example: Independent Set
• Alice = proposer, Bob = other.
• Protocol / claim: AtLeastAsGood. Alice claims
to be at least as good as Bob at IS.
– Bob provides undirected graph G.
– Bob computes independent set sB for G (secret).
– Alice computes independent set sA for G.
– Alice wins, if size(sA) >= size(sB) (= p(sA,sB)).
7/31/2017
23
Introduction
Theory
Methods
Results
Conclusion
Logic with Soundness
claims
sentences
good
bad
not just true/false claims, but optimum/non-optimum claims:
good: true/optimum
bad: false/non-optimum
4/24/2011
Crowdsourcing
24
Introduction
Theory
Methods
Results
Conclusion
Scientific Community Game Logic
with Community Principle
claims
sentences
good
bad
disagreed by two scholars
agreed by two scholars
there exists two-party certificate
to expose misclassification
4/24/2011
Crowdsourcing
25
Comparison Logic and SCG
Logic
• sentences
Scientific Community Game
• sentences = claims
– true
– false
• proof for being true
– good
– bad
• evidence for goodness
– proof system, checkable
– guaranteed defense
• proof for being false
– defense, checkable
– uncertainty of defense
• evidence for badness
– proof system, checkable
– guaranteed refutation
• Universal sentences
4/24/2011
– refutation, checkable
– uncertainty of refutation
• Personified sentences
Crowdsourcing
26
Introduction
Theory
Methods
Results
Conclusion
Community Property
• For every faulty decision action there exists an
exposing reaction that blames the bad
decision.
– Reasons:
• We want the system to be egalitarian.
– It is important that clever crowd members can shine and
expose others who don’t promote the social welfare of the
community.
• Faulty decisions must be exposable. It may take effort.
7/31/2017
27
Community Property
Alternative formulation
• If all decisions by Alice are not faulty, there is
no chance of Alice losing against Bob.
– if Alice is perfect, there is no chance of losing.
• If there exists a faulty decision by Alice, there
is a chance of Alice losing against Bob.
– egalitarian game
7/31/2017
28
Summary: faulty decisions
1.
2.
3.
4.
5.
6.
7.
propose(Alice,C),C=false
propose(Alice,C),C=not optimum, C=true
refute(Alice,Bob,C),C=true
strengthen(Alice,Bob,c,cs),c=optimum
strengthen(Alice,Bob,c,cs),c=false
agree(Alice,Bob,c),c=false
agree(Alice,Bob,c),c=not optimum, c=true
7/31/2017
29
Introduction
Theory
Methods
Results
Conclusion
SCG Equilibrium
• Points (reputations) of scholars are stable.
• The science does not progress; bugs are not
fixed, no new ideas are introduced.
• Extreme, desirable situation: All scholars are
perfect: they propose optimal claims that can
neither be strengthened nor refuted.
7/31/2017
30
Claims: convergence to optimum
over strengthening
1
false claims
(refutable)
correct valuation
quality
strengthening
true claims
(defendable)
0
7/31/2017
31
31
Introduction
Theory
Methods
Results
Conclusion
Convergence
• Given a family of claims totally ordered with
respect to strengthening and containing an
optimum claim. If every faulty action is
exposed, convergence to the optimum claim is
guaranteed by using a search strategy such as
binary search.
7/31/2017
32
Introduction
Theory
Methods
Results
Conclusion
Methods of Exploration
• Developed Platform SCG Court = Generator of
teaching/innovation playgrounds
– http://sourceforge.net/p/generic-scg/code-0/11
0/tree/GenericSCG/
– Developed numerous playgrounds for avatars.
• Developed Algorithms Course using Piazza based
on SCG Court experience
– role of scholar played by humans
– piazza.com: encourages students to answer each
other’s questions.
7/31/2017
33
Introduction
Theory
Methods
Results
Conclusion
Avatar Interface
• AvatarI
– public List<Claim> propose(List<Claim> forbiddenClaims);
– public List<OpposeAction> oppose(List<Claim>
claimsToBeOpposed);
– public InstanceI provide(Claim claimToBeProvided);
– public SolutionI solve(SolveRequest solveRequest);
• from http://sourceforge.net/p/generic-scg/code0/110/tree/GenericSCG/src/scg/scg.beh
7/31/2017
34
Introduction
Theory
Methods
Results
Conclusion
Instance Interface
• InstanceI
– boolean valid(SolutionI solution, Config config);
– double quality(SolutionI solution);
7/31/2017
35
InstanceSet Interface
• InstanceSetI
– Option<String> belongsTo(InstanceI instance);
– Option<String> valid(Config config); }}
7/31/2017
36
Protocol Interface
• ProtocolI
– double getResult(Claim claim, SolutionI[]
solutions, InstanceI[] instances);
– ProtocolSpec getProtocolSpec();
– boolean strengthenP(Claim oldClaim, Claim
strengthenedClaim);
7/31/2017
37
Claim Class, for all playgrounds
• Claim
– public Claim(InstanceSetI instanceSet, ProtocolI
protocol, double quality, double confidence)
7/31/2017
38
Introduction
Theory
Methods
Results
Conclusion
Protocol Library
•
•
•
•
•
ExistsForAll.java
ForAllExists.java
Renaissance.java
AsGoodAsYou.java
Survivor.java
7/31/2017
39
Introduction
Theory
Methods
Results
Conclusion
Claim Kinds in SCG Court
• Claim C(instance, q)
• Claim C(InstanceSet, q)
• Claim
MaxResource(Algorithm,InstanceSet,n,ResExp)
• Claim
MinResource(Algorithm,InstanceSet,n,ResExp)
• Claim IAmTheBest(), AtLeastAsGoodAsYou()
• Claims from predicate logic and some of its
generalizations (IF Logic)
7/31/2017
40
Piazza: Gale-Shapley
• Gale-Shapley (GS) algorithm is fixed.
• Alice provides secret instance of size n, solves
it with GS (counts number of iterations) and
publishes it: Ac(n).
• Bob provides secret instance of size n, solves it
with GS and publishes it Bc(n).
• Alice wins if Ac(n) > Bc(n).
7/31/2017
41
Reverse Engineering
Playground Design Issues
• input, output, algorithm unknown
– for some problems input-output pairs help to
invent the algorithm
• input unknown, output, algorithm
– e.g., output for worst-case running-time, find
input which is worst-case
– for some problems finding
7/31/2017
42
Introduction
Theory
Methods
Results
Conclusion
Playground Design
• Make sure that the scholars don’t have to
reveal their secrets. If they do.
• Remember who was first?
• Provenance of winning idea.
7/31/2017
43
Introduction
Theory
Methods
Results
Conclusion
SCG with piazza.com
• use piazza.com for posting
– playground definition by playground designer
– for scholars
• claims
• executions of refutation protocols
• Community policing
• Used JSON to exchange claims, instances and
solutions.
• Used links when they did not fit in piazza window.
7/31/2017
44
Introduction
Theory
Methods
Results
Conclusion
JSON example
• We define the JSON notation for defining a
preference p as follows:
– {"n":3,
– "manPref" : [[2,1,0],[1,0,2],[0,1,2]],
– "womanPref : [[2,1,0],[1,3,2],[3,1,2]] }
• Easy to parse in a variety of programming
languages and also readable by humans.
7/31/2017
45
Piazza Experience
Gale-Shapley
• We propose that, for all integers n > 0, the
maximum iterations the Gale-Shapely
algorithm with n men and n women can
produce is n(n-1)+1.
Note: Thus far, the algorithm used for all other
claims arrives at only (n(n+1))/2.
7/31/2017
46
Piazza Experience
• Leaf Covering: Improved running time from
quadratic to constant time.
7/31/2017
47
Introduction
Theory
Methods
Results
Conclusion
Results
SCG = Scientific Community Game = Specker Challenge Game
• Explanation: SCG as a general pattern behind
many different competitions: topcoder.com,
kaggle.com, Operations Research Competitions,
tunedit.org, http://eterna.cmu.edu/ …
• SCG usage for teaching using forum
– Innovation Success with Undergraduates using SCG on
piazza.com: Qualitative Data Sources & Analysis
• Avatar competitions are not for teaching (but
good for competitive innovation)
• Theoretical Properties of SCG
7/31/2017
48
Competition tuning: minimum
• For each scholar
– count claims that were successfully opposed
(refuted or strengthened)
• encourages strong claims
• gather information from competitors for free
– count claims that were not successfully agreed
• Good for teaching
– students want minimum competition
– good students want to build social capital and
help weaker students
7/31/2017
49
Introduction
Theory
1 scholar
2 scholar
refute(C, proposer,other)
p(…)?(proposer,other):
(proposer,other)
refute attempt C
refute(C,1,2)
p(C, …)?(1,-1):(-1,1)
Methods
1
Conclusion
Results
High competition
propose claim C from Claims
agree attempt C
2
strengthen attempt C’ => C
refute(C’,2,1)
refute(C,2,1)
p(C’, …)?(-1,1):(1,-1)
7/31/2017
p(C, …)?(0,0):(1,-1)
50
Introduction
Theory
1 scholar
2 scholar
refute(C, proposer,other)
p(…)?(proposer,other):
(proposer,other)
s: successful
refute attempt C
u: unsuccessful
refute(C,1,2)
p(C, …)? (0,0) :(0,1)
Methods
1
Conclusion
Results
Low competition
propose claim C from Claims
agree attempt C
2
strengthen attempt C’ => C
refute(C’,2,1)
refute(C,2,1)
p(C’, …)?(0,1): (0,0)
7/31/2017
p(C, …)?(0,0): (1,0)
51
Introduction
Theory
Methods
Results
Conclusion
Piazza Results
• Lower competition knob for teaching.
• For optimization claims got significant
scientific discourse.
• Playgrounds cannot have too many scholars,
otherwise they are overwhelmed.
– about 5 is a good size
– use hierarchical playgrounds: winning teams
compete again
7/31/2017
52
Introduction
Theory
Methods
Results
Conclusion
Piazza Results
• Do not give hints at solutions. This
significantly decreased the amount of
discourse taking place.
7/31/2017
53
Conclusions and Future Work
• We propose a systematic gamification of
teaching STEM domains:
– Design an SCG playground where the winning
students demonstrate superior domain
knowledge.
STEM = Science, Technology, Engineering, and Mathematics
7/31/2017
54
Introduction
Theory
Methods
Results
Conclusion
Gamification of Software Development
for Computational Problems
• Want reliable software to solve a
computational problem? Design an SCG
playground where the winning team will
create the software you want.
• playground design = requirements
– Programming the Global Brain
• socio-technical system (playground) will
produce solution to requirements.
4/24/2011
Crowdsourcing
55
Clear Feedback
Sense of Progress
What we get
Authenticity
• Engaged software developers
– let them produce software that models an
organism that fends for itself in a real virtual world
while producing the software we want. Have fun.
Focus them.
– let them propose claims about the software they
produce. Reward them when they
• defend their claims successfully or
• oppose the claims of others successfully.
Possibility of Success
4/24/2011
Crowdsourcing
56
Introduction
Theory
Methods
Results
Conclusion
Conclusions
• Flexible use of SCG using a forum environment
with threads and replies using optimization
playgrounds is productive:
– teams took turns leapfrogging each other
– reached state-of-the-art and even improved it
• SCG has desirable theoretical properties.
– faulty decision –> exposing reaction
– equilibria
– convergence to optimum claim
7/31/2017
57
Introduction
Theory
Methods
Results
Conclusion
Future Work
• Make SCG part of cyber-infrastructure (e-science)
both for avatars and human scholars.
• Polish SCG Court
– The administrator software needs to be very reliable (to
avoid cheating by avatars).
– Playground development and testing needs tool support.
• Further develop SCG with forum software
– Playground design defines requirements for know-how.
– Hierarchical playgrounds: partitioning into balanced
groups.
– Restart playground after publishing all current ideas in
playground (if optimum is not yet reached).
7/31/2017
58
The End
• More Questions?
7/31/2017
59
Links
• SCG Home
– http://www.ccs.neu.edu/home/lieber/evergreen/specker/scghome.html
• Piazza page for Algorithms
– http://piazza.com/class#winter2012/cs4800/0
• Algorithms Home
– http://www.ccs.neu.edu/home/lieber/courses/algorithms/cs4800/sp1
2/course-description.html
• Algorithms Feedback
– http://www.ccs.neu.edu/home/lieber/courses/algorithms/cs4800/sp1
2/feedback/
• SCG Court Source
– http://sourceforge.net/p/generic-scg/code-0/110/tree/GenericSCG/
7/31/2017
60
Extra slides
7/31/2017
61
Essence of Game Rules
without Payoff
blamed decisions:
propose(1,C)
refute(1,2,C)
strengthen(1,2,C,C’)
agree(1,2,c)
• scholars: 1, 2
• LifeOfClaim(C) = propose(1,C) followed by
(oppose(1,2,C)|agree(1,2,C)).
• oppose(1,2,C) =
(refute(1,2,C)|strengthen(1,2,C,C’)), where
stronger(C,C’).
• strengthen(1,2,C,C’) = !refute(2,1,C’).
• agree(1,2,C) = !refute(2,1,C)
7/31/2017
62
Introduction
Theory
1 scholar
2 scholar
refute(C, proposer,other)
p(…)?(proposer,other):
(proposer,other)
s: successful
refute attempt C
u: unsuccessful
refute(C,1,2)
p(C, …)?(1,-1):(-1,1)
u:1 2
s:1 2
p(C’, …)?(1,-1):(-1,1)
p(C’, …)?(-1,1):(1,-1)
s:1 2
7/31/2017
u:1 2
Methods
Conclusion
Results
1
propose claim C from Claims
agree attempt C
2
strengthen attempt C’ => C
refute(C’,2,1)
refute(C,2,1)
p(C, …)?(1,-1):(-1,1)
p(C, …)?(0,0):(1,-1)
s:1 2 u:1 2
63
Introduction
Theory
1 scholar
2 scholar
refute(C, proposer,other)
p(…)?(proposer,other):
(proposer,other)
s: successful
refute attempt C
u: unsuccessful
refute(C,1,2)
p(C, …)? (0,0) :(0,1)
u:1 2
s:1 2
p(C’, …)?(0,1): (0,0)
s:1 2
7/31/2017
u:1 2
Methods
1
Conclusion
Results
Low competition
propose claim C from Claims
agree attempt C
2
strengthen attempt C’ => C
refute(C’,2,1)
refute(C,2,1)
p(C, …)?(0,0): (1,0)
s:1 2 u:1 2
64
Introduction
Theory
Methods
Results
Conclusion
Conclusions for Teaching
• Transition
– refute: (1,-1):(-1,1) -> (0,0) :(0,1)
– strengthen: (-1,1):(1,-1) -> (0,1): (0,0)
– agree: (0,0):(1,-1) -> (0,0): (1,0)
• creates better playgrounds for learning by
lowering competition and increasing teaching
between scholars.
7/31/2017
65
Claims
• Protocol. Defines scientific discourse.
• Scholars make a prediction about their
performance in protocol.
• Predicate that decides whether refutation is
successful. Refutation protocol collects data
for predicate.
• As a starter: Think of a claim as a
mathematical statement: EA or AE.
– all planar graphs have a 4 coloring.
7/31/2017
66
Introduction
Theory
Methods
Results
Conclusion
More examples of Protocols
• Let f(x,y)=x*y+(1-x)(1-y^2)). Alice claims
Math(0.61): Bob constructs an x in [0,1] and Alice
construct a y in [0,1], and Alice guarantees that
f(x,y)> 0.61. True claim but can be strengthened
to 0.618.
• Alice claims Solar(RawMaterials,m,0.61). Bob
constructs raw materials r in RawMaterials and
Alice constructs a solar cell s in Solution from r
using money m and so that efficiency(s)> 0.61.
7/31/2017
67