Ch11
Distributed Agreement
Outline
Distributed Agreement
Adversaries
Byzantine Agreement
Impossibility of Consensus
Randomized Distributed Agreement
Exponential Time Shared Memory Consensus
Outline
Distributed Agreement
Adversaries
Byzantine Agreement
Impossibility of Consensus
Randomized Distributed Agreement
Exponential Time Shared Memory Consensus
Distributed Agreement
Adversaries
When proving correctness or analyzing an algorithm,
it is convenient to assume that:
the inputs,
the failure times,
failure behaviors
any system variables
are under the control of an adversary who, intuitively,
makes as much difficulty for the algorithm as possible
Worst case analysis: worst case choices of the adversary
Outline
Distributed Agreement
Adversaries
Byzantine Agreement
Impossibility of Consensus
Randomized Distributed Agreement
Exponential Time Shared Memory Consensus
Distributed Agreement
The Agreement or the Consensus Problem:
Assume
P = {p1,…,pM} is the set of all the processors in the system
Some processors in P are faulty
Let F be the set of all faulty processors in P
Every processor p in P has a value p.Val
The requirement: devise a distributed algorithm that lets each
processor p computes a value p.A such that when the execution of
this distributed algorithm terminates, the following two conditions
hold:
1. (agreement value) For every pair of processors p and q, q.A = p.A
2. The agreement value is a function of the initial values {p.Val} of
non-faulty processors
Outline
Why Distributed Agreement is an interesting problem?
Processor p is the leader
Processor p has the right to enter the critical section
Processor p has failed
Distributed Agreement
Byzantine Agreement:
Assumptions
A failed processor can send arbitrary messages
A non-failed processor always responds to a message within T
seconds
When a processor receives a message, it can reliably determine
the sender of that message
Distributed Agreement
Byzantine Agreement:
The Byzantine Generals Problem:
Byz_A2
Byz_A4
Some byzantine generals are “corrupted”
G2
G4
“Non-corrupted” generals knew that
they will be victorious only if they
attack simultaneously
“ENEMY”
the Sultan’s army
G3
Byz_A3
G1
Byz_A1
Loyal generals must find a consensus
to attack or to retreat
Distributed Agreement
Byzantine Agreement:
The Byzantine Generals Problem (basic idea)
Each general has to make a decision based on the opinions
it gets from the other generals
All loyal generals must make the same decision
If all loyal generals get the same set of opinions
for making the decision, then all loyal generals can achieve a
consensus using the same procedure to decide
How can we ensure that all loyal generals get the same
set of opinions ?
Distributed Agreement
Byzantine Agreement:
The Byzantine Generals Problem (basic idea)
To ensure that each loyal general gets the same set of values, it is sufficient that
each loyal general uses the same value Vj for every other general Gj in order to
decide
The Byzantine Generals Problem is then reduced to agreement
by generals on the value sent by a particular general: a commanding
general
Formally, we must have:
1. If the sender ps is loyal and sends the value Vs, the loyal generals will decide
that the value sent is Vs
2. If the sender ps is treacherous, the loyal general will agree on the
same value
This problem is known as the interactive consistency problem
Distributed Agreement
Byzantine Agreement:
The Byzantine Generals Problem (continued)
Assuming that each general can reliably broadcast its opinion,
the loyal generals can reach an agreement!
How and under which conditions?
Distributed Agreement
The Byzantine Generals Problem (continued)
Question 1
assuming that there is a reliable protocol for broadcast,
is it possible to reach an agreement with three generals
with one disloyal ?
The answer is NO!
Why?
disloyal
C
C
retreat
attack
retreat
L1
retreat
L2
attack
attack
attack
L1
attack
L2
disloyal
Distributed Agreement
The Byzantine Generals Problem (continued)
Question 2
assuming that there is a reliable protocol for broadcast,
is it possible to reach an agreement with four generals
with one disloyal ?
The answer is Yes!
Why?
Two cases are in order:
The commanding general is disloyal
The commanding general is loyal
Distributed Agreement
The Byzantine Generals Problem (continued)
Justification of the answer to Question 2
Disloyal
commanding
General:C
C
L1
L2
L3
Round 1
Round 2
By the end of the second round,
attack
retreat
L1 has 2 attack and 1 retreat
L2 has 2 attack and 1 retreat
L3 has 2 attack and 1 retreat
Each Lieutenant obeys
the majority
Distributed Agreement
The Byzantine Generals Problem (continued)
Justification of the answer to Question 2
C
L1
L2
Round 1
L3
Disloyal
Lieutenant: L3
Round 2
By the end of the second round,
attack
retreat
L1 has 2 attack and 1 retreat
L2 has 2 attack and 1 retreat
L3 has 3 attack and 1 retreat
Each Lieutenant obeys
the majority:
Each Loyal General decides
“attack”
Distributed Agreement
The Byzantine Generals Problem (continued)
Theorem:
Assuming a synchronous system with M processors ,of which up
to t can be faulty, the loyal generals can reach a consensus only if
M  3t+1
The algorithm to solve the Byzantine Generals Problem is
parameterized by k the maximum number of disloyal generals
This algorithm is BG(k)
Distributed Agreement
The Byzantine Generals Problem (continued)
The BG(k) idea:
The algorithm works in rounds of messages exchange.
Round 1
L(C)=the set of Lieutenants for C;
size of L(C) = M-1
C
L1
L2
If no message is sent to a Lieutenant
that Lieutenant takes “retreat” as
the default value
LM-1
L(C)
Distributed Agreement
The Byzantine Generals Problem (continued)
The BG(k) idea:
Every processor p1 in L(C) acts as the commanding General
Round 2
L(p1:C)=the set of Lieutenants for p1
with respect to C’s opinion;
size of L(p1:C) = M-2
p1
L1
L2
LM-2
L(p1)
p1 sends M-2 messages
p1 receives M-2 messages
p1.v(2) := majority(V) where
V = {p1.v(1)}  {p1.Rq(2) : q in L(p1:C)}
Distributed Agreement
The Byzantine Generals Problem (continued)
The BG(k) idea:
Every processor p2 in L(p1), for each p1 in L(C), acts as the
commanding General
Round 3
L(p2:p1)=the set of Lieutenants for p2
with respect to p1’s opinion;
size of L(p2:p1) = M-3
p2
p2 sends M-3 messages
p2 receives M-3 messages
L1
L2
LM-3
L(p2)
p2.vr(3,p1) := majority(V) where
V = {p2.vr(2,p1)}  {p2.Rq(3,p1) : q in L(p2:p1)}
Distributed Agreement
The Byzantine Generals Problem (continued)
The BG(k) idea:
Every processor p3 in L(p2), for each p2 in L(p1), p1 in L(C),
acts as the commanding General
Round 4
L(p3:p2)=the set of Lieutenants for p3
with respect to p2’s opinion on ...;
size of L(p3) = M-4
p3
L1
L2
LM-3
L(p3)
p3 sends M-4 messages
p3 receives M-4 messages
p3.vr(4,p2) := majority(V) where
V = {p3.vr(3,p2)}  {p2.Rq(4,p2) : q in L(p3:p2)}
Distributed Agreement
The Byzantine Generals Problem (continued)
The BG(k) idea:
Every processor pi-1 in L(pi-2), for each pi-2 in L(pi-3), …,
p2 in L(p1), p1 in L(C), acts as the commanding General
Round i
L(pi-1:pi-2)=the set of Lieutenants for pi-1
with respect to pi-2’s opinion on ...;
size of L(pi-1:pi-2) = M-i
pi-1
L1
L2
LM-i
L(pi-1)
pi-1 sends M-i messages
pi-1 receives M-i messages
pi-1.vr(i,pi-2) := majority(V) where
V = {pi-1.vr(i-1,pi-2)}  {pi-1.Rq(i,pi-2) : q in L(pi-1:pi-2)}
Distributed Agreement
The Byzantine Generals Problem (continued)
The BG(k) idea:
Round k+1, BG(0)
Every processor pk in L(pk-1), for each pk-1 in L(pk-2), …,
p2 in L(p1), p1 in L(C), acts as the commanding General
L(pk:pk-1)=the set of Lieutenants for pk
with respect to pk-1’s opinion on ...;
size of L(pk) = M-k-1
Pk sends M-k-1 messages
pk receives M-k-1 messages
pk
L1
L2
LM-k-1
L(pk)
pk.vr(k+1,pk-1) := majority(V) where
V = {pk.vr(k,pk-1)}  {pk.Rq(k+1,pk-1) : q in L(pk)}
For each pk-1,
pk decides pk.vr(k+1,pk-1)
Distributed Agreement
The Byzantine Generals Problem (continued)
The BG(k) idea:
Round k+1
Processor pk decides using (1+M-k-1)
opinions
So, if M=3k+1, then we have that
pk decides using 2k+1 opinions
pk
L1
L2
LM-k-1
L(pk)
Since at most k processors can be
faulty, it follows that all non faulty
processors make the same decision
pk decides pk.vr(k+1,pk-1)
Distributed Agreement
The Byzantine Generals Problem (continued)
The BG(k):
BG(0):
1. The commanding sends its value to all the other n-1 processors
2. Each processor uses the value it receives from the commanding
or uses the default value
Distributed Agreement
The Byzantine Generals Problem (continued)
The BG(k):
BG(k), k>0:
1. The commanding sends its value to all the other n-1 processors
2. Let vp be the value the processor p receives from the commanding
general, or the default value if no value is received.
Processor p acts as the commanding in BG(k-1) to send the value
vp to each of the other M-2 processors.
3. For each processor p, let vq be the value received from processor q
(q p). Processor p uses the majority({v}  {vq: q in L(C)})
where v is the value processor p received from
the commanding general
Distributed Agreement
The Byzantine Generals Problem (continued)
The BG(k): number of messages sent
Following the presentation gave above, one can see that
the number of messages sent is proportional to
(M-1)(M-2)(M-3)…(M-k-1)
Since k can be (M-1)/3, it follows that
the number of messages is O(Mk)
Outline
Distributed Agreement
Adversaries
Byzantine Agreement
Impossibility of Consensus
Randomized Distributed Agreement
Exponential Time Shared Memory Consensus
Distributed Agreement
The Byzantine Generals Problem (continued)
Impossibility result
If the system is asynchronous (no bound on the relative speeds
of processors or the communication delays), then it can be
shown (Fisher, Lynch, Paterson 1985) that agreement is
impossible if even one processor can fail, and even if the failure is
a crash failure
Outline
Distributed Agreement
Adversaries
Byzantine Agreement
Impossibility of Consensus
Randomized Distributed Agreement
Exponential Time Shared Memory Consensus
Distributed Agreement
The Randomized Distributed Agreement
Randomization:
processors can flip coin
Assumptions:
The system consists of N processors, of which up to t can be faulty
Processors communicate by using shared registers
The shared registers are non-faulty
The accesses to the shared register are sequentially consistent
Atomic reads and writes of the contents of the registers
The system is asynchronous
Outline
Distributed Agreement
Adversaries
Byzantine Agreement
Impossibility of Consensus
Randomized Distributed Agreement
Exponential Time Shared Memory Consensus
Distributed Agreement
The Randomized Distributed Agreement
The Exponential Time Shared Memory Consensus:
The Naïve algorithm:
Assume :
the system is synchronous
each processor p has a initial value Vp to prefer
The idea (algorithm for processor p):
while I have not yet decided do
1. Read the set {Vi} of values of every other processors
2. If for all i, Vi = Vp then
decide Vp
else
Vp := coin_flip()
end
Distributed Agreement
The Randomized Distributed Agreement
The Exponential Time Shared Memory Consensus:
Transformation of the Naïve synchronous algorithm into an
asynchronous algorithm
Idea 1:
“simulate” the synchronous algorithm:
add a round variable at each processor
The naïve algorithm becomes
Distributed Agreement
The Randomized Distributed Agreement
The Exponential Time Shared Memory Consensus:
Idea 1 continue (algorithm for processor p):
while I have not yet decided do
1. Read the set {Vi} of values of every other processors
2. If for all i, Vi = Vp and p.round = I.round then
decide Vp
else
Vp := coin_flip();
p.round := pround+1
end
Problem :
some processors can fail; a fail processor may not increment
its round variable when it executed
Distributed Agreement
The Randomized Distributed Agreement
The Exponential Time Shared Memory Consensus:
Idea 2:
At any moment, the set of processors can be regarded as
consisting of
FP: the set of the largest round value;
LP: the other processors
If ( p,q in FP: Vp = Vq ) and
eventually ( s in LP: Vs = Vq , q in FP )
then
one can decides on Vq, q in FP
How can we achieve the eventually part ?
Distributed Agreement
The Randomized Distributed Agreement
The Exponential Time Shared Memory Consensus:
Idea 2(continued):
How can we achieve the eventually part ?
Intuitively, the idea is to make the slower processors prefer to
accept the value of faster processors.
Distributed Agreement
The Randomized Distributed Agreement
The Exponential Time Shared Memory Consensus:
The algorithm :Variable used
V[1..M] shared array of records, one per each processor
V[i].value : the preferred decision of processor i;
V[i].round : execution round of processor i;
Local_V[1..M] local copy of V[1..M]
Leaders : the processors that have the largest round values
in round Local_V
Distributed Agreement
The Randomized Distributed Agreement
The Exponential Time Shared Memory Consensus:
The algorithm :Functions used
leader_set(Local_V) : returns the set of leaders
Flip() : randomly returns either 0 or 1.
Distributed Agreement
The Randomized Distributed Agreement
The Exponential Time Shared Memory Consensus:
The algorithm :Initially
V[i].value := NIL;
V[i].round :=0;
/* not necessary */
Local_V[i].value := NIL;
Local_V[i].round := 0;
leaders := empty
Distributed Agreement
The Randomized Distributed Agreement
The Exponential Time Shared Memory Consensus:
The algorithm :
SM_Consensus(self, preference)
(V[self].value, V[self].round) := (preference,1)
while I have not made a decision do
read V into Local_V;
leaders := leader_set(Local_V);
if (self  leaders) and
( i : Local_V[i].value  Local_V[self] : Local_V[i].round < Local_V[self].round -1)
then
decide(V[self].value)
elseif (i,j in leaders: Local_V[i].value = Local_V[j].value) then
(V[self].value, V[self].round) := (V[i].value, V[i].round) for a i in leaders
elseif V[self].value  NIL then
(V[self].value, V[self].round):=(NIL, V[self].round)
else
(V[self].value, V[self].round) := (Flip(), round+1)
Distributed Agreement
The Randomized Distributed Agreement
The Exponential Time Shared Memory Consensus:
The probability that all leaders choose the same value:
O(2-N)
The expected number of rounds:
O(2N)