Consistency and Replication
Chapter 6
Topics
• Reasons for Replication
• Models of Consistency
– Data-centric consistency models
– Client-centric consistency models
• Protocols for Achieving Consistency
Replication
• Reasons:
– Reliability: increase availability when servers crash
– Performance: load balancing; scale with size of
geographical region
– Availability: local server likely to be available
• When one copy is modified, all replicas have
to be updated
• Problem: how to keep the replicas consistent
Object Replication
•
Approach 1: application is responsible for replication
–
•
Application needs to handle consistency issues
Approach 2: system (middleware) handles replication
–
Consistency handled by the middleware: Simplifies application
development but makes object-specific solutions harder
Replication and Scaling
• Replication and caching used for system scalability
• Multiple copies:
– Improves performance by reducing access latency
– But higher network overheads of maintaining consistency
– Example: object is replicated N times
• Read frequency R, write frequency W
• If R <= W, high consistency overhead and wasted messages
• Consistency maintenance is itself an issue
– What semantics to provide?
– Tight consistency requires globally synchronized clocks!
• Solution: loosen consistency requirements
– Variety of consistency semantics possible
Data-Centric Consistency Models
Consistency Model: contract between processes
and the data store. If processes follow contract,
the data store works correctly.
Strict Consistency
Def.: Any read on a data item x returns a value
corresponding to the result of the most recent write
on x (regardless of which copy was written to).
Behavior of two processes, operating on the same data item.
(a) A strictly consistent store.
(b) A store that is not strictly consistent.
Sequential Consistency (1)
Def.: The result of any excution is the same as if the operations
by all processes on the data store were executed in some sequential
order and the operations of each individual process appear in this
sequence in the order specified by its program.
• Sequential consistency is weaker than strict consistency
• All processes see the same interleaving of operations
a)
b)
A sequentially consistent data store.
A data store that is not sequentially consistent.
Sequential Consistency (2)
Sequential consistency comparable to serializability of transactions
Process P1
Process P2
Process P3
x = 1;
print (y, z);
y = 1;
print (x, z);
z = 1;
print (x, y);
•
•
•
•
Any valid interleaving is allowed
All agree on the same interleaving
Each process preserves its program order
Nothing is said about “most recent write”
Sequential Consistency (3)
x = 1;
print (y, z);
y = 1;
print (x, z);
z = 1;
print (x, y);
x = 1;
y = 1;
print (x,z);
print(y, z);
z = 1;
print (x, y);
y = 1;
z = 1;
print (x, y);
print (x, z);
x = 1;
print (y, z);
y = 1;
x = 1;
z = 1;
print (x, z);
print (y, z);
print (x, y);
Prints: 001011
Prints: 101011
Prints: 010111
Prints: 111111
Signature:
001011
(a)
Signature:
101011
(b)
Signature:
010111
(c)
Signature:
111111
(d)
Four valid execution sequences for the processes of
the previous slide. The vertical axis is time.
Linearizability
Assumption: Operations are timestamped (e.g., Lamport TS)
Def.: The result of any execution is the same as if the operations
by all processes on the data store were executed in some
sequential order and the operations of each individual
process appear in this sequence in the order specified by
its program. In addition, if tsOP1(x)<tsOP2(y), then OP1(x)
should precede OP2(y) in this sequence.
• Linearizable data store is also sequentially consistent
• Lineralizability is weaker than strict consistency, but stronger
than sequential consistency - adds global TS requirements to
sequential consistency.
Causal Consistency (1)
• Writes that are potentially causally related must be
seen by all processes in the same order.
• Concurrent writes may be seen in a different order
on different machines.
• Causal consistency is weaker than sequential
consistency
Causal Consistency (2)
This sequence is allowed with a causally-consistent store,
but not with sequentially or strictly consistent store.
• W2(x)b may depend on R2(x)a and therefore depends
on W1(x)a  a must be seen before b at other processes
• W2(x)b and W1(x)c are concurrent
Causal Consistency (3)
a)
b)
A violation of a causally-consistent store: W2(x)b depends
on W1(x)a.
A correct sequence of events in a causally-consistent store.
FIFO Consistency (1)
• Writes done by a single process are seen by all
other processes in the order in which they were
issued.
• Writes from different processes may be seen in a
different order by different processes.
• FIFO consistency is weaker than causal
consistency.
• Simple implementation: tag each write by
(Proc ID, seq #)
FIFO Consistency (2)
A valid sequence of events of FIFO consistency
FIFO Consistency (3)
Process P1
Process P2
Process P3
x = 1;
print (y, z);
y = 1;
print (x, z);
z = 1;
print (x, y);
Process P1’s view
x = 1;
print (y, z);
y = 1;
z = 1;
Process P2’s view
x = 1;
y = 1;
print (x, z);
z = 1;
Process P3’s view
y = 1;
z = 1;
print (x, y);
x = 1;
Prints: 00
Prints: 10
Prints: 01
Signature: 001001
Statement execution as seen by the three processes. The statements
in bold are the write-updates originating from other the processes.
Signature 001001 not possible with sequential consistency. In
sequential consistency, all processes have the same view.
FIFO Consistency (4)
Process P1
Process P2
x = 1;
if (y == 0) kill (P2);
y = 1;
if (x == 0) kill (P1);
• Sequential consistency: 6 possible statement orderings;
none of them kills both processes
• FIFO consistency: both processes can get killed
Models Based on a Sync Operation
• No consistency is enforced until a synchronization
operation is performed. This operation can be
done after local reads and writes to propagate the
changes throughout the system.
• Weak Consistency
• Release Consistency
• Entry Consistency
Weak Consistency (1)
• Often not necessary to see all writes done by a
process
• Weak consistency enforces consistency on a group
of operations; not individual read/write statements
• Synchronization point:
– Propagate changes made to local data store to remote
data stores
– Changes made by remote data stores are imported
• Weak consistency is weaker than FIFO consistency
Weak Consistency (2)
Properties:
• Accesses to synchronization variables associated with
a data store are sequentially consistent (i.e., all
processes see all operations on synchronization
variables in the same order)
• No operation on a synchronization variable is allowed
to be performed until all previous writes have been
completed everywhere (i.e., guarantees all writes have
propagated)
• No read or write operation on data items are allowed
to be performed until all previous operations to
synchronization variables have been performed (i.e.,
when accessing data items, all previous
synchronizations have completed)
Weak Consistency (3)
a) A valid sequence of events for weak consistency.
b) An invalid sequence for weak consistency.
Release Consistency (1)
• More efficient implementation than weak
consistency by identifying critical regions
– Acquire: ensure that all local copies of the data are
brought up to date to be consistent with (released)
remote ones
– Release: data that has been changed is propagated
out to remote data stores
• Acquire does not guarantee that locally made
changes will be sent to other copies immediately
• Release does not necessarily import changes
from other copies
Release Consistency (2)
A valid event sequence for release consistency.
Release Consistency (3)
Rules:
• Before a read or write operation on shared data is
performed, all previous acquires done by the process
must have completed successfully.
• Before a release is allowed to be performed, all
previous reads and writes by the process must have
completed
• Accesses to synchronization variables are FIFO
consistent (sequential consistency is not required).
Release Consistency (4)
Different implementations:
– Eager release consistency: process doing the
release pushes out all the modified data to all
other processes.
– Lazy release consistency: no update messages
are sent at time of release. When another process
does an acquire, it has to obtain the most recent
version.
Entry Consistency (1)
• Every data item is associated with a
synchronization variable.
• Each data items can be acquired and released
as in release consistency.
• Acquire (entry) gets most recent value.
• Advantage: increased parallelism
• Disadvantage: increased overhead
Entry Consistency (2)
A valid event sequence for entry consistency.
Summary of Consistency Models
Consistency
Description
Strict
Absolute time ordering of all shared accesses matters.
Linearizability
All processes must see all shared accesses in the same order. Accesses are
furthermore ordered according to a (nonunique) global timestamp
Sequential
All processes see all shared accesses in the same order. Accesses are not ordered in
time
Causal
All processes see causally-related shared accesses in the same order.
FIFO
All processes see writes from each other in the order they were used. Writes from
different processes may not always be seen in that order
(a)
Consistency
Description
Weak
Shared data can be counted on to be consistent only after a synchronization is done
Release
Shared data are made consistent when a critical region is exited
Entry
Shared data pertaining to a critical region are made consistent when a critical region is
entered.
(b)
a)
b)
Consistency models not using synchronization operations.
Models with synchronization operations.
Client Centric Consistency (1)
• Strong consistency for data store often not necessary
• Consistency guarantees from a clients perspective
• Clients often tolerate inconsistencies (e.g., out of date
web-pages)
• Assumptions:
– Client may move to a different replica during a
single session
– Eventual consistent data store: total propagation
and consistent ordering
• Trade-off: consistency vs. availability
Client Centric Consistency (2)
• The principle of a mobile user
accessing different replicas of a
distributed database.
Data Storage Model (1)
• Client uses Server that operates on a Database
• Database holds complete copy of replicated
data store
• Server executes Read and Write operations
• Every Write operation has a globally unique
write-ID (WID)
• A Session is the context in which reads and
writes occur
Data Storage Model (2)
Client1
Client2
Client3
Server1
Server2
Server3
DB
DB
DB
Eventually consistent replicated data store
Writes are propagated in a lazy fashion among servers. To ensure that the session
guarantees are met, the servers at which an operation can be performed must be
restricted to a subset of available servers that are sufficiently up-to-date.
Data Storage Model (3)
• DB(S,t) ::= ordered sequence of Writes that
have been received by server S up to time t
• WriteOrder(W1,W2) ::= Write W1 should be
executed before Write W2
• Write set WS: set of WIDs
• Write set WS is complete for Read R and
DB(S,t), iff WS  DB(S,t) and for all WS2
with WS  WS2  DB(S,t): result of R
applied to WS2 is the same as the result of R
applied to DB(S,t)
Data Storage Model (4)
• RelevantWrites(S,t,R) ::= function that
returns the smallest set of Writes that is
complete for Read R and DB(S,t)
• Note: such a set exists, since DB(S,t) is itself
complete for any Read
• Intuitively, RelevantWrites is the latest write
for each data item whereas DB(S,t) has the
entire history of writes.
Monotonic Reads (1)
• Definition: If Read R1 occurs before R2 in a
session and R1 accesses server S1 at time t1
and R2 accesses server S2 at time t2, then
RelevantWrites(S1,t1,R1)  DB(S2,t2)
• That is R2 sees the same as R1 or a more
recent value.
• Example: Calendar updates
Monotonic Reads (2)
S1
W1 R1
S2
S1
S2
W1 R2
W0 W1 R1
W0
R2
Valid
Invalid: R2
doesn‘t see
W1
Assumption: W1RelevantWrites(S1,t,R1)
Monotonic Writes (1)
• Definition: If Write W1 precedes Write W2
in a session, then, for any server S2, if
W2  DB(S2,t) then W1  DB(S2,t) and
WriteOrder(W1,W2)
• Like monotonic reads except the writes force
consistency.
• Example: Software Update
Monotonic Writes (2)
S1
W1
S2
S1
S2
W1 W2
W0 W1
W0
W2
Valid
Invalid
Read Your Writes (1)
• Definition: If Read R follows Write W in a
session and R is performed at server S at
time t, then W  DB(S,t)
• Example: Password update propagation
Read Your Writes (2)
S1
W1 W2
S2
S1
S2
W1 W2 R
Valid
W1 W2
W1 R
W2
Invalid
Write Follows Read (1)
• Definition: If Read R1 precedes Write W2 in a
session and R1 is performed at server S1 at
time t, then, for any server S2, if W2 
DB(S2,t) then any W1 
RelevantWrites(S1,t,R1) implies W1 
DB(S2,t) and WriteOrder(W1,W2)
• Example: Newsgroup - my message W2 is a
response to my reading (R1) message W1, so
W1 should proceed W2 in all servers.
Write Follows Read (2)
S0
S1
S2
W1
S0
S1
S2
W1
W1 R
W2
Valid
W1 W2
W1 R
W2
Invalid
W2 W1
Assumption: W1RelevantWrites(S1,t,R)
Implementation Summary
Guarantee
Session state
updated on
Session state
checked on
Read Your Writes
Write
Read
Monotonic Read
Read
Read
Write Follows Read Read
Write
Monotonic Writes
Write
Write
Replica Placement
The logical organization of different kinds of
copies of a data store into three concentric rings.
State vs. Operations
•
Design choices of update propagation:
1. Propagate only a notification of an update (e.g.,
invalidation protocols)
2. Transfer data from one copy to another
3. Propagate the update operation to other copies
(a.k.a. active replication)
Pull vs. Push Protocols
Issue
Push-based
Pull-based
State of server
List of client replicas and caches
None
Messages sent
Update (and possibly fetch update later)
Poll and update
Response time at
client
Immediate (or fetch-update time)
Fetch-update time
• A comparison between push-based and pull-based
protocols in the case of multiple client, single server
systems.
Epidemic Protocols
•
•
•
Useful for eventual consistency
Propagating updates to all replicas in as few
messages as possible
Update propagation model:
–
–
–
•
•
Infective: node holds update and is willing to spread
Susceptible: node willing to accept update
Removed: updated node not willing to spread
Anti-entropy: pick nodes at random
Exchanging updates between nodes P and Q:
1. P only pushes its own updates to Q
2. P only pulls in new updates from Q
3. P and Q send updates to each other
Consistency Protocol Implementations
• Primary based (each data item has an
associated primary):
– Remote write
– Local write
• Replicated write (write operations carried
out at multiple replicas, update anywhere):
– Active replication
– Quorum based protocols
Remote-Write Protocols (1)
Primary-based remote-write protocol with a fixed server to
which all read and write operations are forwarded.
Remote-Write Protocols (2)
• The principle of primarybackup protocol. Compare
eager and lazy update.
Local-Write Protocols (1)
• Primary-based local-write protocol in which a
single copy is migrated between processes.
Local-Write Protocols (2)
• Primary-backup protocol in which the primary migrates to the
process wanting to perform an update.
Alternative to ROWA
• Write quorum: all write quorums must have a
non-empty intersection.
• Read quorum: any read quorum must have a
non-empty intersection with all write quorums.
• Write operation: write new value to all copies
in the quorum and update version number.
• Read operation: read all copies in read quorum
and pick the most recent (copies must have
write timestamp or version number).
Quorum-Based Protocols
Constraints on read quorum NR and write quorum NW:
1. NR + NW > N
2. NW > N/2
Three examples of the voting algorithm:
a)
A correct choice of read and write set
b)
A choice that may lead to write-write conflicts
c)
A correct choice, known as ROWA (read one, write all)
Example
• Causally consistent lazy replication
• (Wuu1984) Efficient Solutions to the
Replicated Log and Dictionary Problem, by
Wuu G. T. and Bernstein A. J.,
• NEXT: Fault Tolerance