Replication (1)
Topics
 Why Replication?
 System Model
 Consistency Models – How do we reason about the
consistency of the “global state”?


Data-centric consistency
Client-centric consistency
 We will examine consistency protocols which
describe an implementation of a specific
consistency model.
 Other Implementation Issues
 Examples
Readings
 Van Steen and Tanenbaum: 6.1, 6.2 and 6.3,
6.4
 Coulouris: 11,14
Why Replicate?
 Replication refers to the maintenance of
copies at multiple site
 Reliability
If one replica is unavailable or crashes, use
another
 Avoid single points of failure

 Performance
 Placing copies of data close to the processes
using them can improve performance through
reduction of access time.
 If there is only one copy, then the server could
become overloaded.
Common Replication Examples
 DNA naming service
 Web browsers often locally store a copy of
a previously fetched web page.
 This is referred to as caching a web
page.
 Replication of a database
 Replication of game state
System Model
 A collection of replica managers (RMs) provide a
service to clients

One replica manager per replica
 The front-end for a client allows a client to see a
service that gives it access to logical objects,
which are in fact replicated at the RMs
 Clients request operations: some are read-only
operations and some are update operations
•
System Model
Requests and
replies
C
RM
RM
FE
Clients
Front ends
C
Service
FE
RM
Replica
managers
Clients’ request are handled by front ends. A
front end makes replication transparent.
•
Phases in Executing Request
 Issue request
 the FE either
• sends the request to a single RM that passes it on to the
others
• or multicasts the request to all of the RMs
 Coordination
 The RMs decide whether to apply the request; and
decide on its ordering relative to other requests
(according to FIFO or total ordering)
 Execution
 The RMs execute the request
 Agreement
 RMs agree on the effect of the request, e.g., perform
'lazily' or immediately
 Response
 One or more RMs reply to FE.
Group Communication
 Replication systems need to be able to add/remove
RMs
 A group membership service provides:

Interface for adding/removing members
• Create, destroy process groups, add/remove members
• A process can generally belong to several groups

Implements a failure detector
• This monitors members for failures (crashes/communication),
and excludes them when unreachable


Notifies members of changes in membership
Expands group addresses
• Multicasts addressed to group identifiers
• Group expanded to a list of delivery addresses
• Coordinates delivery when membership is changing
Services provided for process
groups
A process outside
the group sends to
the group without
knowing the
membership
Membership service
provides leave and
join operations
Group
address
expansion
Group
send
Multicast
communication
The group address is
expanded
Leave
Fail
Group membership
management
Join
Process group
Failure detector notes failures and
evicts failed processes from the
group
Members are
informed when
processes join/leave
•
Group Communication
 Replication systems need to be able to add/remove
RMs
 A group membership service provides:

Interface for adding/removing members
• Create, destroy process groups, add/remove members
• A process can generally belong to several groups

Implements a failure detector
• This monitors members for failures (crashes/communication),
and excludes them when unreachable


Notifies members of changes in membership
Expands group addresses
• Multicasts addressed to group identifiers
• Group expanded to a list of delivery addresses
• Coordinates delivery when membership is changing
Developing Replication Systems
 Consistency
 Faults
 Changes in Group Membership
A Replication Problem
 Multiple copies may lead to consistency
problems.
 Whenever a copy is modified, that copy
becomes different from the rest.
 Modifications have to be carried out on all
copies to ensure consistency.
 The type of application has an impact on
the consistency requirements needed and
thus on the implementation.
Consistency Model
 A consistency model describes the rules to
be used in updating replicated data
 The rules define the order of operations.
 Rules used depend on the application
 Consistency defined within the context of
read and write operations on shared data is
data-centric
Strict
 Sequential
 Causal
 FIFO

Strict Consistency
 Strict consistency is defined as follows:
 Read is expected to return the value resulting
from the most recent write operation
 Assumes absolute global time
 All writes are instantaneously visible to all
 Suppose that process pi updates the value
of x to 5 from 4 at time t1 and multicasts
this value to all replicas
Process pj reads the value of x at t2 (t2 > t1).
 Process pj should read x as 5 regardless of the
size of the (t2-t1) interval.

Strict Consistency
 What if t2-t1 = 1 nsec and the optical fiber
between the host machines with the two
processes is 3 meters.
The update message would have to travel at 10
times the speed of light
 Not allowed by Einsten’s special theory of
relativity.

 Can’t have strict consistency
Sequential Consistency
 Sequential Consistency:
 The result of any execution is the same as if the (read
and write) operations by all processes on the data were
executed in some sequential order
 Operations of each individual process appears in this
sequence in the order specified by is programs.
 We have seen this in the banking example
 One implementation used Lamport’s clocks.
Causal Consistency
 Causal Consistency: That if one update, U1, causes
another update, U2, to occur then U1 should be
executed before U2 at each copy.
 Application: Bulletin board

Possible Implementation: Using vector clocks
FIFO Consistency
 Writes done by a single process are seen
by all other processes in the order in which
they were issued
 … but writes from different processes may
be seen in a different order by different
processes.
 i.e., there are no guarantees about the
order in which different processes see
writes, except that two or more writes
from a single source must arrive in order.
FIFO Consistency
 Caches in web browsers
 All updates are updated by page owner.
 No conflict between two writes
 Note: If a web page is updated twice in a very
short period of time then it is possible that the
browser doesn’t see the first update.
 Implementation:
 Each process adds the following to an update
message: (process id, sequence number)
 Each other process applies the update
messages in the order received from a single
process.
Implementation Options:
Sequential Consistency
 We saw how to use Lamport’s logical clocks
for sequential consistency.
 Another option is to have a centralized
processor that is a sequencer.
Implementation Options:
Sequential Consistency
 We saw how to use Lamport’s logical clocks
for sequential consistency.
 Another option is to have a centralized
processor that is a sequencer.
 Each update request it sent to the
sequencer which
Assigns the request a unique sequence number
 Update request is forwarded to each replica
 Operations are carried out in the order of their
sequence number

Implementation Options:
Sequential Consistency
 The use of a sequencer also does not solve the scalability
problem.


It may become a performance bottleneck.
What if it goes down?
 A combination of Lamport timestamps and sequencers may
be necessary.
 The approach is summarized as follows:
 Each process has a unique identifier, pi, and keeps a sent
message counter ci. The process identifier and message
counter uniquely identify a message.
 Active processes (or a sequencer) keep an extra counter:
ti. This is called the ticket number. A ticket is a triplet
(pi, ti, (pj, cj)).
 All other processes are passive
Implementation Options:
Sequential Consistency
 Approach Summary (cont)



Passive processes (non-sequencer) send their messages
to their sequencer.
Lamport’s totally ordered multicast algorithm is used
among the sequencers to determine the order of update
operations.
When an operation is allowed, each sequencer sends the
ticket to its associated passive processes. It is assumed
that the passive process receives these tickets in the
order sent.
Implementation Options:
Sequential Consistency
 Approach Summary (cont)
If a sequencer terminates abnormally, then one
of the passive processes associated with it can
become the new sequencer.
 An election algorithm may be used to choose
the new sequencer.

Implementation Options:
Sequential Consistency
 Let’s say that we have 6 processes: p1,p2,p3,p4,p5,p6
 Assume that p1,p2 are sequencers; p3,p4 are
associated with p1 and p5,p6 are associated with p2
 Let’s say that p3 sends a message which is
Ticket number
identified by (p3 , 1).
 p1 generates a ticket as follows: (p1, 1, (p3 , 1))
 The ticket number is generated using the Lamport
clock algorithm.
Implementation Options:
Sequential Consistency
 Let’s say that p5 sends a message which is
identified by (p5 , 1).
 p2 generates a ticket as follows: (p2, 1, (p5 ,
1))
 Which update gets done first? Basically,
p1,p2 will apply Lamport’s algorithm for
totally ordered multicast.
 When an update operation is allowed to
proceed, the sequencers send messages to
their associated processes.
Data-Centric Consistency Models
 The consistency models just discussed are
called data-centric consistency models.
 Assumptions:
Concurrently processes may be simultaneously
updating
 Updates need to be propagated quickly.
