CS4231
Parallel and Distributed Algorithms
AY 2006/2007 Semester 2
Lecture 6
Instructor: Haifeng YU
Review of Last Lecture
 Chapter 9 “Global Snapshot”
 What is our goal?
 Formalizing consistent global snapshot
 Protocol for capturing a consistent global snapshot
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
2
Today’s Roadmap
 Chapter 12 “Message Ordering”
 FIFO ordering
 Already discussed last lecture
 Causal ordering and its application
 Total ordering and its application
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
3
A Internet Chat Room Example
Alice (process1)
Introducing
Bob to Cary
Hi Bob, I just
introduced you
to Cary
Bob (process2)
Hi Cary,
this is Bob
Cary (process3)
???
The notion of “causal order”.
Sorry, I don’t
know any Bob.
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
4
Formalizing Causal Order
 If a “send” event s1 caused a “send” event s2
 Causal order requires the corresponding receive event r1 be before
r2 (r1 and r2 on the same process)
 But how do we know whether s1 caused s2?
 Based on “Happened-Before” relation
 This is the only ordering observable to external users
 If a “send” event s1 happens before a “send” event s2
 Then s1 may have caused s2
 Let’s be pessimistic and be safe -- Assume s1 indeed caused s2
 Causal order: If s1 happens before s2, and r1 and r2 are on the
same process, then r1 must be before r2
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
5
How to Ensure Causal Ordering – Intuition
Message1
A to C
Message1
A to B
Alice (process1)
PS:
Message1
A to C
Bob (process2)
PS:
Message1
B to C
Message1
A to C
Cary (process3)
Message1
A to B
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
6
How to Ensure Causal Ordering – Protocol
 Each process maintains n*n matrix M
 M[i,j] denotes that number of messages sent from i to j, as known
by the local process

If process i send a message to process j
 On process i: M[i,j]++
 Piggyback M on the message
 Upon process j receiving a message from process i with matrix
M’ piggybacked
 If M[k,j]  M’[k,j] for all k except k = i, AND M[i,j] == M’[i,j] -1,
then deliver message and M = pairwise-max(M,M’)
 Else delay message
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
7
Example Run for the Protocol
(0,0,1) (0,1,1)
(0,0,0) (0,0,0)
(0,0,0) (0,0,0)
Alice (process1)
can deliver
(0,1,1)
(0,0,1)
(0,0,0)
Bob (process2)
Message1
B to C
(0,0,0)
(0,1,1)
(0,0,0)  (0,0,0)
(0,0,0)
(0,0,0)
Cary (process3)
can’t deliver
(0,0,0)
(0,0,0)
(0,0,0)
(0,1,1)
(0,0,1)
(0,0,0)
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
8
Correctness Proof of the Protocol
 (The textbook does not have an explicit correctness
proof.)
 s1, s2, r1, r2, where s1 happened before s2
 We want to prove that r1 is never delivered after r2
 Consider the case where s1 and s2 are on different
processes
 W.l.o.g., assume s1 on process 1, s2 on process 2, r1
and r2 on process 3
 We focus on the first 3 elements of the 3rd column of
the matrices
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
9
Correctness Proof of the Protocol
Alice (process1)
x1
y1
0
x2
y2
0
x2  x1
y2  y1
x3
y3
0
x4
y4
0
x4  x3  x2
y4 > y3  y2
Bob (process2)
x1
y1
0
x4
y4
0
Cary (process3)
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
10
Correctness Proof of the Protocol (continued)
 Prove by contradiction: Assume Red delivered before
Blue
 After delivering Red: 3rd column of the matrix on process 3
will be  x4
 y4
0
 The values in the matrix never decreases: Blue can never be
delivered
Cary (process3)
x1
y1
0
x4
y4
0
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
x4  x1
y4 > y1
11
Need More Correctness Proof
 What we proved so far:
 If s1 happened before s2, then r1 is never delivered after r2
is delivered
 We don’t know if r1 and r2 will be delivered at all!
 We will now prove that
 At any given time, there must be one message that can be
delivered to the process – induction will take care of the
other messages
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
12
More Correctness Proof
 Consider a process and its corresponding column in
the matrix
x_1
x_2
…
x_n
 If there is any message (from process i) undelivered
to the process, then there must be messages whose
matrix has the column
?
call such message as
successor messages
?
x_i + 1
?
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
13
More Correctness Proof
 Consider the non-empty set of all successor
messages and their corresponding “send” events
 There must be a send event s that does not have any other
send events that happened before s
 Call the corresponding message first successor message
 Claim: Any first successor message can be delivered
 W.l.o.g, assume the message is from process 1
x_1
x_1 + 1
x_2
y_2
…
…
x_n
y_n
The matrix column
on the receiver
The matrix column
in the message
need to show
x_i  y_i for i
between 2 and n
W.l.o.g, assume
i=2
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
14
More Correctness Proof
 Prove by contradiction: Suppose y_2 > x_2
process1
this is not a first
successor message
then!
process2
The last delivered  x_1
message from
x_2
process 2
…
x_1 + 1
must be another
un-delivered
message
y_2
…
Receiver process
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
15
Summary of Correctness Proof
 All messages will eventually be delivered
 The delivery order satisfies causal order
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
16
Causal Ordering of Broadcast Messages
 Broadcast: Every message is sent to all people
(including the sender itself)
 Modeled as n point-to-point messages
 Application: Internet chat room
 For same reason as before, we may need to ensure
causal order among messages
 Each process uses the previous protocol
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
17
A Simpler Protocol for Causal Ordering
Broadcast
 Each process maintains a message log for all
messages delivered
 When sending a message, append the message to log and
send the whole log
 Upon receiving a log, scan the log sequentially, for any
message not seen before: Append the message to local log
and deliver it
 Can be used for point-to-point as well
 But inefficient..
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
18
Total Ordering of Broadcast Messages
 All messages delivered to all processes in exactly the
same order
 Also called atomic broadcast
 Total ordering does not imply causal ordering
 Causal ordering does not imply total ordering
 Thus…there is something called “causal total order”
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
19
Application: Internet Chat Room
 We want to assign numbers to people have said
 The number have to be consistent across all users
[0] Alice: Welcome!
[1] Bob: Hello, Alice
[2] Alice: Let’s try to prove P=NP
[3] Bob: OK, how do we start?
….
….
….
[1712] Bob: I am confused, what were we trying to prove?
[1713] Alice: Please refer to message [2]
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
20
Application: Replicated State Machine
 State machine can be an abstract of many things
 E.g., A database
 Typically deterministic
 Replicated state machine
 Multiple copies of the same state machine for performance /
availability
 Each “instruction” is broadcast to all state machines
 The state machines must execution the “instruction” in
exactly the same order!
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
21
Using a Coordinator for Total Order Broadcast
 A special process is assigned as the coordinator
 To broadcast a message
 Send a message to the coordinator
 Coordinator assigns a sequence number to the message
 Coordinator forward the message to all processes with the
sequence number
 Messages delivered according to sequence number order
 Problem:
 Coordinator is performance bottleneck
 What if coordinator fails?
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
22
Skeen’s Algorithm for Total Order Broadcast
pick the max clock value
received as message number
process 1
broadcast
message
acknowledge
process 2
notify message number
process 3
put msg in
buffer
 Each process maintains
reply with
current logical
clock value
 Logical clock and a message buffer for undelivered messages
 A message in the buffer is delivered / removed if
 All messages in the buffer have been assigned numbers
 This message has the smallest number
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
23
Correctness Proof for Skeen’s Algorithm
pick the max clock value
received as message number
process 1
broadcast
message
acknowledge
process 2
notify message number
process 3
put msg in
buffer
reply with
current logical
clock value
 Claim: All messages will be assigned message numbers
 Claim: All messages will be delivered
 Claim: If message A has a number smaller than B, then B is
delivered after A – Prove by contradiction (continue on next slide)
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
24
Correctness Proof for Skeen’s Algorithm -Continued
process 3
B placed in
buffer
B’s number A must be
received and placed in
B delivered buffer after
B delivered
reply for A
key: Process 3’s logical clock now must be larger than B’s number
 Suppose A is delivered on process 3 after B.
 A must be placed in buffer after B is delivered
 A must have a number larger than B – Contradiction.
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
25
Summary
 Chapter 12 “Message Ordering”
 FIFO ordering for point-to-point messages
 Already discussed last lecture
 Causal ordering for point-to-point messages
 Applications
 Protocol to ensure causal ordering
 Causal ordering for broadcast messages
 Protocol
 Total ordering for broadcast messages
 Application
 Skeen’s algorithm
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
26
Homework Assignment
 Page 206, Problem 12.3. For each pair (x, y) of the three pairs (i.e.,
C1 & C2, C1 & C3, C2 & C3)
 Prove x implies y and y implies x, OR
 Prove x implies y, and a counter example that satisfies y but not x, OR
 Prove y implies x, and a counter example that satisfies x but not y, OR
 A counter example that satisfies y but not x, and a second counter
example that satisfies x but not y
 Think about why the protocol on slide 18 will deliver all messages in
causal order
 For the protocol on slide 22, think about whether the resulting total
order satisfy causal order? Why?
 Homework due a week from today
 Read Chapter 13
CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2
27