Distributed Transactions
Chapter 14
•Atomic Commit
•Logging and Recovery
1
Distributed Atomic Commit
• The distributed commit problem is how get all
members of a group to perform an operation
(transaction commit) or none at all.
• It is needed when a distributed transaction
commits.
• It is necessary to find out if each site was
successful in performing his part of the transaction
before allowing any site to make that transaction's
changes permanent.
2
Two Phase Commit (Gray 1987)
• Assumptions: Reliable communications and a
designated coordinator. Simple case: no failures.
• First phase is the voting phase
– Coordinator sends all participants a VOTE REQUEST
– All participants respond COMMIT or ABORT
• Second phase is decision phase. Coordinator decides
commit or abort: if any participant voted ABORT,
then decision must be abort. Otherwise, commit.
– Coordinator sends all participants decision
– Participants (who have been waiting for decision) commit
or abort as instructed and ack.
3
2PC: Participant Failures
• If a participant, P, fails before voting, coordinator, C,
can timeout and decide abort. (cannot decide commit
because p1 may vote abort)
• If p1 fails after voting, if he voted commit, he must be
prepared to commit the transaction when he recovers.
He must ask the coordinator or other participants what
the decision was before doing so. If he voted abort, he
can abort the transaction when he recovers.
• So BEFORE the participant votes, he must log on
stable storage his position.
4
2PC: Coordinator Failures
• If coordinator fails before requesting votes,
participants timeout and abort.
• If coordinator fails after requesting votes (or after
requesting some votes),
• Ps who did not get the vote request timeout, abort
• Ps who voted abort timeout and abort
• Ps who voted commit cannot unilaterally abort as all
Ps may have voted commit, and C decided commit and
some P may have received the decision. Must either
wait until the coordinator recovers or contact ALL the
other Ps. If one is unreachable, he cannot decide.
5
Two-Phase Commit (1)
a)
b)
The finite state machine for the coordinator in 2PC using
the notation (msg rcvd/msg sent).
The finite state machine for a participant.
6
Two-Phase Commit: Recovery
State of Q
Action by P
COMMIT
Make transition to COMMIT
ABORT
Make transition to ABORT
INIT
Make transition to ABORT
READY
Contact another participant
Actions taken by a participant P when residing in state
READY and having contacted another participant Q.
7
Two-Phase Commit
actions by coordinator:
write START _2PC to local log;
multicast VOTE_REQUEST to all participants;
while not all votes have been collected {
wait for any incoming vote;
if timeout {
write GLOBAL_ABORT to local log;
multicast GLOBAL_ABORT to all participants;
exit;
}
record vote;
}
if all participants sent VOTE_COMMIT and coordinator votes COMMIT{
write GLOBAL_COMMIT to local log;
multicast GLOBAL_COMMIT to all participants;
} else {
write GLOBAL_ABORT to local log;
multicast GLOBAL_ABORT to all participants;
}
Outline of the steps taken by the coordinator
in a two phase commit protocol
8
Two-Phase Commit
actions by participant:
Steps taken by
participant
process in
2PC.
write INIT to local log;
wait for VOTE_REQUEST from coordinator;
if timeout {
write VOTE_ABORT to local log;
exit;
}
if participant votes COMMIT {
write VOTE_COMMIT to local log;
send VOTE_COMMIT to coordinator;
wait for DECISION from coordinator;
if timeout {
multicast DECISION_REQUEST to other participants;
wait until DECISION is received; /* remain blocked */
write DECISION to local log;
}
if DECISION == GLOBAL_COMMIT
write GLOBAL_COMMIT to local log;
else if DECISION == GLOBAL_ABORT
write GLOBAL_ABORT to local log;
} else {
write VOTE_ABORT to local log;
send VOTE ABORT to coordinator;
}
9
Two-Phase Commit
actions for handling decision requests: /* executed by separate thread */
while true {
wait until any incoming DECISION_REQUEST is received; /* remain blocked */
read most recently recorded STATE from the local log;
if STATE == GLOBAL_COMMIT
send GLOBAL_COMMIT to requesting participant;
else if STATE == INIT or STATE == GLOBAL_ABORT
send GLOBAL_ABORT to requesting participant;
else
skip; /* participant remains blocked */
Steps taken for handling incoming decision requests.
10
2PC: Blocking Problem
• If coordinator fails after deciding but before sending
all decision messages, problem for P’s
• If the P got the decision message, he carries out the
decision (and tells others if asked)
• If the P voted abort, he just aborts.
• If the P voted commit, he must wait until the
coordinator recovers or he gets a response from ALL
participants (P must know who the other Ps are). The
crash or unavailability of coordinator and one P
results in a BLOCK.
11
How to Solve Blocking?
• All participants could be in same state or in states
adjacent to that one state, but no further away since their
moves are "coordinated". They move out of current
state when directed to by coordinator.
• Problem: when coordinator plus one or more Ps fails, the
other Ps must be able to determine the outcome based on
the states of the living participants.
12
Three Phase Commit
Solve the blocking problem of Two Phase
Commit by adding another “state”.
13
Participant State Diagrams
P’s can move out of current state only when directed to by C.
C gives the command only when he receives confirmation that
all Ps have made the transition. Assume this is SD of P
If one P is in state 1, what are the possible states of the others?
1
5
2
3
6
Coordinator has given the command to leave state 5.
C would have given the command to leave state 1
only if he knew all P’s were in state 1.
4
7
Possible states are {5,1} or {1,2,3}
14
Participant State Diagrams
If one P is in state 3, what are the possible states of the others?
1
5
2
3
6
Case 1: C has given the command to leave state 1
but has not yet received confirmation that all Ps are
in state 3. (C may have died after sending command
to some.) Possible states are {1,2,3}
4
7
Case 2: C gave command to leave state 3, but this P
has not received it yet. Possible states are {3,4,6}
State sets that are not possible: {3,5} and {1,4} and {2,4}
15
How to Solve Blocking
So the conditions for a non-blocking atomic
commit protocol are • There is no state from which it is possible to make a
transition directly to both commit and abort.
– That is, no {commit,abort}
• Any state from which a transition to commit is
possible has the property that it is possible for
participants to decide without waiting for the
recovery of the coordinator or dead participants.
– P can decide based on the sets of possible states.
16
Three Phase Commit
• Vote phase, Decision phase, (if commit) Commit phase.
• Vote phase
– Coordinator sends vote request
– All respond commit or abort (and change state)
• Decision phase
– After getting all votes, coordinator decides commit if all voted
commit and abort otherwise.
– Coordinator sends Abort or Prepare-to-commit
– All respond ack
• Commit phase
– After getting all acks, coordinator sends commit command.
– All ack.
17
Three Phase Commit
a)
b)
Finite state machine for the coordinator in 3PC
Finite state machine for a participant
18
Three Phase Commit
Failure cases - when P recovers:
• If P dies before voting, P and C can assume abort.
• If P dies after voting abort, he unilaterally aborts.
• If C dies and P has voted abort, then abort.
Two interesting cases to look at:
• Coordinator dies after requesting votes and some
parts may be dead also.
• And coordinator times-out because some part died.
19
Three PC: Case 1
• C dies after requesting votes, some Ps dead. Living Ps
must decide what to do based on their states.
• If some in init state and some ready: abort as dead ones
may be in abort state.
init
• Any in abort state: abort since some
ready
may be in ready state, but none
have committed
abort
precommit
commit
20
3PC: Case 1 continued
• If some in ready state, some in precommit: commit
since all must have voted commit
• If all in ready state: abort since some dead parts may
have voted abort or C may not have
init
received all votes before crashing.
If all voted commit then coordinator
ready
crashed, well won't hurt to abort
if all agree.
abort
precommit
commit
21
3PC: Case 2
• Coordinator times-out because some P died
• C in wait state: some P died before/while voting: decide abort
• C in pre-commit state: some part has already voted commit but
failed to ack the prepare command: commit transaction: when
part recovers and inquires it will commit the transaction.
22
Checkpointing, Logging, Recovery
• Recovery is what happens after the crash. Logs and
checkpoints make it possible.
• A checkpoint is a durable record of a consistent
state that existed on this node at some time in the
past.
• A log is a durable record or history of the significant
events (such as writes but not reads) that have
occurred at this site (either since the start or since
the last checkpoint).
23
Recovery Strategies
• Backward recovery: Bring the system back to a
previous correct and consistent state. Checkpoint is
made periodically during normal operations by
recording (on stable storage) the current state.
(Problems with on-going transactions). After a
failure, the state can be restored from the checkpoint
info. Issue: taking a checkpoint is a lot of overhead.
• Forward recovery: Bring the system to a new
correct state after a crash. May involve asking
another site what the current state is if data is
replicated.
24
Recovery Strategies
• Combination: Use both checkpoint and recovery
log.
– Take checkpoint and delete old log and start new
log.
– Log all significant messages, transactions, etc, up to
the next checkpoint.
– When recovering from failure, restore checkpoint
state then replay log and re-do these operations.
– Often used in databases
25
Checkpoints in a Distributed System
• Taking a checkpoint is a single site operation,
whereas in a DS, there is a global state that must
remain consistent.
• If one process, P2, fails and rolls back to a
previous checkpoint, that point may be
inconsistent with the rest of the sites in the DS.
• This means that the other sites may have to roll
back also. This is a problem if the taking of
checkpoints is not coordinated -- need a global
snapshot.
26
Checkpointing
P2 fails and rolls back to previous checkpoint. It is now
inconsistent with P1, so P1 must roll back, which causes P2 to
roll back one more checkpoint to final recovery line.
27
Independent Checkpointing
The problem with independent checkpoints is the
domino effect or cascading rollbacks to find a
consistent state: need distributed snapshot!
28
Logging
• All events which affect the system state are
logged sequentially starting with the last
checkpoint.
– Messages received
– Updates from clients
• After failure, the system state is restored from
the checkpoint, then the log is replayed to
bring the system to a consistent state.
29
Questions?
30