50.530: Software Engineering
Sun Jun
SUTD
Week 6: Race Detection
Agenda
• A remainder of concurrent bugs.
• Common ways of coordinating
threads/processes
• How to detect potential concurrent bugs?
Concurrent Bugs
Thread1 Thread2
0
0
count++
1
count++
00 count = 0

count = 1 01
10 count = 1
11
1
count = 2
count should be 2?
Concurrent Bugs
Thread1 Thread2
0
0
r1
1
r1
02
1
2
i2
i2
12
03
2
w1
00
01
r2
i1
3
r2
w2
3
Is it OK if it could be
1 or 2?
13
23
11
i1
w1
10
20
22
w2
33
30
21
31
32
count=1
r2
00
01
r1
02
i2
12
03
13
w1
23
10
11
i1
20
33
30
21
22
w2
What do we do to make sure the red
transitions won’t happen?
31
32
Coordinate the threads based on timing
“You wait for 5 minutes, I will start
right away”
Using Thread.sleep()
• Example: a program with two threads, one
printing 1,2,3,5 and the other printing 4
repeatedly such that the print out is
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5
…
Using Thread.sleep()
Thread1 Thread2
0
0
print 1,2,3
wait for 5 seconds
1
1
print 4
wait for 10 seconds
2
print 5
3
2
Expected Behavior
LeftThread
RightThread
print 1,2,3
sleeping
5
sleeping
print 4
10
print 5,1,2,3
sleeping
15
sleeping
print 4
20
print 5,1,2,3
sleeping
25
sleeping
print 4
30
print 5,1,2,3
sleeping
time
This would work except it doesn’t
SleepExample.java
r2
00
01
r1
02
i2
12
03
13
w1
23
10
11
i1
20
33
30
21
22
w2
What do we do to make sure the red
transitions won’t happen?
31
32
Coordinate the threads based on locking
If an object is to be shared by multiple
threads, make sure the object is locked
and there is only one key to unlock it.
Using Locks
Thread1 Thread2
acquire
lock
0
0
1
1
r1
r2
2
i1
2
i2
3
w1
release
lock
acquire
lock
3
w2
4
4
5
5
release
lock
ThirdBlood.java
Rules for Locks
• Update related state variables in a single atomic
operation
• For each mutable variable that may be accessed by
more than one thread, all assesses to that variable
must be performed with the same lock held.
• Every shared, mutable variable should be guarded by
exactly one lock. Make it clear to maintainers which
lock that is.
• For every invariant that involves more than one
variable, all the variables involved in that invariant
must be guarded by the same lock.
r2
00
01
r1
02
i2
12
03
13
w1
23
10
11
i1
20
33
30
21
22
w2
What do we do to make sure the red
transitions won’t happen?
31
32
Coordinate the threads by sending
messages
“I am working on this. I will send you a
message when I am done.”
wait() and nofity()
Producer/Consumer Pattern
BoundedBuffer
Producer Thread 1
Producer Thread 2
…
Consumer Thread 1
Consumer Thread 2
…
removeItem
addItem
BufferFixed.java
Race Conditions
A race condition occurs when two or more
threads can access shared data at the same time
and at least one of the threads is writing.
Thread 1
Thread 2
count++
count++
Deadlock
public class LeftRightDeadlock {
private final Object left = new Object ();
private final Object right = new Object ();
public void leftRight () {
synchronized (left) {
synchronized (right) {
doSomething();
}
}
}
public void rightLeft () {
synchronized (right) {
synchronized (left) {
doSomethingElse();
}
}
}
}
Thread A
lock left
try to lock right
wait forever
Thread B
lock right
wait for lock left
wait forever
Example
public void transferMoney (Account from, Account to, int amount) {
synchronized (from) {
synchronized (to) {
if (from.getBalance() < amount) {
//raiseException
}
else {
from.debit(amount);
to.credit(amount)
}
}
}
}
Is it deadlocking?
Example
public void transferMoney (Account from, Account to, int amount) {
synchronized (from) {
synchronized (to) {
if (from.getBalance() < amount) {
//raiseException
}
else {
from.debit(amount);
to.credit(amount)
}
}
}
}
How can transferMoney deadlock?
Thread A: transferMoney(myAccount, yourAccount, 1)
Thread B: transferMoney(yourAccount, myAccount, 1)
Check out: DemonstrateDeadlock.java
Research Discussion
Would Delta Debugging work for concurrent programs?
Or the bug localization methods?
Savage et al. SOSP’97
ERASER: A DYNAMIC DATA RACE
DETECTOR FOR MULTI-THREADED
PROGRAMS
Static vs. Dynamic Analysis
Static (program) analysis: is the analysis of
computer programs that is performed without
actually executing the programs.
– Often would result in false alarms
Dynamic (program) analysis: is the analysis of
computer programs that is performed by executing
programs on a real or virtual processor.
– Often not exhaustive
Dynamic Analysis
Instrument the program
Obtain execution traces
e.g., adding print statement
everywhere
e.g., which are sequences of
statements
Analyze the trace and wonder:
Is there a potential problem like race condition?
Example
Trace
1. Thread 1: count++
2. Thread 2: count++
Can we spot any problem from this trace?
Answer: Yes, it seemed that nothing would
prevent us from reordering statement 1 and 2.
Since they modify the same variable, a race
condition is found!
Example
Trace
1. Thread 1: lock(mu)
2. Thread 1: count++
Happens-before
3. Thread 1: unlock(mu)
4. Thread 2: lock(mu)
Happens-before
5. Thread 2: count++
Happens-before
6. Thread 2: unlock(mu)
Can we spot any problem from this trace?
Answer: No, because of the happens-before
relationship.
Example
Trace
1. Thread 1: count++
2. Thread 1: send message to thread 2
3. Thread 1: v = v + x;
4. Thread 2: y = z + x;
5. Thread 2: get message from thread 1
5. Thread 2: count++
Can we spot any problem from this trace?
Answer: No, because of the happens-before
relationship.
Happens-before
Happens-Before Approach
Input: a sequence of statements <s1, s2, …>
Output: true if there is a race condition
For any pair of statements s and t (assuming s is “earlier”)
if (s and t are from two different threads and
s and t access some shared memory and
either s modifies the memory or t does) {
if (there is no happens-before relation from s to t) {
report potential race condition;
}
}
}
Example
Trace
Thread 1: obj1.methodA()
Thread 2: obj2.methodB()
Can we spot any problem from this trace?
It depends on whether obj1 and obj2 are disjoint.
Example
Trace
A
Thread 1: obj1.methodA()
Thread 2: obj2.methodB()
B
The question is: whether A and B are disjoint?
Define Happens-Before
Given a sequence of statements
<a, b, c, …>
• If statement f and u (u is after f in the sequence) are
from the same thread, f happens-before u.
• If f sends a message and u receives the message, f
happens-before u.
• If f unlocks an object and u locks the same object, f
happens-before u.
• …
• If f happens before u and u happens-before y, f
happens-before y.
Is this complete?
Subtle Happens-Before
Trace
Thread 1: count++
Thread 2: while (x <= 0) {
Thread 2: Thread.yield()
Thread 2: endwhile
Thread 1: x++
Thread 2: while (x <= 0) {
Thread 2: endwhile
Thread 2: count++
Do these form a race condition?
Subtle Happens-Before
Trace
Thread 1: count++
Thread 2: while (x <= 0) {
Thread 2: Thread.yield()
Thread 2: endwhile
Thread 1: x++
Thread 2: while (x <= 0) {
Thread 2: endwhile
Thread 2: count++
Happens-Before Algorithm
• In order to get precise happens-before
relation, we often need to analyze the whole
program.
• Race condition detection based on happensbefore relation is very costly – no known
efficient implementation so far.
• Race condition detection based on happensbefore relation is not exhaustive even with a
perfect happens-before relation.
Exercise 1
Trace
Thread 1: count++
Thread 1: lock(mu)
Thread 1: x++
Thread 1: unlock(mu)
Thread 2: lock(mu)
Thread 2: x++
Thread 2: unlock(mu)
Thread 2: count++
Is there a race condition according to the happens-before approach?
Is there a race condition?
Lockset Algorithm
Recall (rules for locks):
• For each mutable variable that may be
accessed by more than one thread, all
assesses to that variable must be performed
with the same lock held.
The lockset algorithm is designed to check if the
rules is properly implemented. If not, it reports
potential race condition.
Lockset Algorithm
For each shared variable x
set lockset(x) = *;
Endfor
For each access to x by thread t
lockset(x) = {locks held by t at the time} intersect lockset(x)
if (lockset(x) is an empty set) {
report race condition;
}
}
* is a special flag denoting the set of all possible locks.
Example
Trace
Locks held by thread 1 and 2
Lockset(count)
Thread 1: lock(mu1)
{} for thread 1; {} for 2
{mu1, mu2}
Thread 1: count++
{mu1} for thread 1; {} for 2
{mu1}
Thread 1: unlock(mu1)
{mu1} for thread 1; {} for 2
{mu1}
Thread 2: lock(mu2)
{} for thread 1; {} for 2
{mu1}
Thread 2: count++
{} for thread 1; {mu2} for 2
{}
Thread 2: unlock(mu2)
Exercise 2
Trace
Thread 1: count++
Thread 1: lock(mu)
Thread 1: x++
Thread 1: unlock(mu)
Thread 2: lock(mu)
Thread 2: x++
Thread 2: unlock(mu)
Thread 2: count++
How does Lockset find the race in this example?
Patching Lockset
Locking may not be necessary in some scenarios
• Initialization: shared variables are frequently
initialized without holding a lock
• Read-share data: Some shared variables are
written during initialization only and are readonly thereafter. These can be safely accessed
without locks.
• Read-write locks: Read-write locks allow multiple
readers to access a shared variable, but allow
only a single writer to do so.
Improving Lockset
For each shared variable,
we only monitor its status
according to the state
machine and check for
race condition only at
state “Shared-Modified”
Improving Lockset
When a variable is in Shared-Modified state, we
do the checking slightly differently.
For each read-access of x by thread t {
lockset(x) = {locks held by t at the time} intersect lockset(x)
if (lockset(x) is an empty set) {
report race condition;
}
}
For each write-access of x by thread t {
lockset(x) = {locks held by t at the time in write mode} intersect lockset(x)
if (lockset(x) is an empty set) {
report race condition;
}
}
Example: On Variable count
Trace
Locks held
Write Locks held Status
Lockset(count)
T1: lock(mu, R)
{} for T1 and T2
{} for T1 and T2
Virgin
*
T1: x = count+1
{mu} for T1;
{} for T2
{} for T1;
{} for T2
Virgin
*
T1: unlock(mu)
{mu} for T1;
{} for T2
{} for T1;
{} for T2
Virgin
*
T2: lock(mu, W)
{} for T1;
{} for T2
{} for T1;
{} for T2
Virgin
*
T2: count++
{} for T1;
{mu} for T2
{} for T1;
{mu} for T2
Exclusive
*
T2: unlock(mu)
{} for T1;
{mu} for T2
{} for T1;
{mu} for T2
Exclusive
*
Exercise 3
Trace
Thread 1: count++
Thread 1: lock(mu)
Thread 1: x++
Thread 1: unlock(mu)
Thread 2: lock(mu)
Thread 2: x++
Thread 2: unlock(mu)
Thread 2: count++
Draw the table for this example.
Efficiency of Lockset
• Implemented at the binary level
• Maintain a status and lockset for each
memory location
• Performance penalty: applications with
lockset implementation slows down 10 to 30
times.
Effectiveness
False Alarms are produced in some scenarios.
• Memory reuse: the same memory is reused
for different purposes later (with different
locking policy).
• Private locks: user defines their locks without
using the ones from the library
• Benign races: Race condition is found but
deemed to be benign.
False Alarm: Example
Which is worse for users: false alarms or missing some true races?
O’Callahan et al. PPoPP’03
HYBRID DYNAMIC DATA RACE
DETECTION
Example
class Main {
int globalFlag;
ChildThread childThread;
void execute () {
globalFlag = 1;
childThread = new ChildThread(this);
childThread.start();
…
synchronized (this) {
childThread.interrupt();
}
}
}
class ChildThread extends Thread {
Main main;
ChildThread (Main main) {
this.main = main;
}
void run() {
if (main.globalFlag == 1) …;
…
main.childThread = null;
}
}
Is there a race condition?
Example: Lockset Algorithm
class Main {
int globalFlag;
ChildThread childThread;
void execute () {
globalFlag = 1;
childThread = new ChildThread(this);
childThread.start();
…
synchronized (this) {
childThread.interrupt();
}
}
}
class ChildThread extends Thread {
Main main;
ChildThread (Main main) {
this.main = main;
}
void run() {
if (main.globalFlag == 1) …;
…
main.childThread = null;
}
}
What would the lockset algorithm report?
Example: Lockset Algorithm
class Main {
int globalFlag;
ChildThread childThread;
void execute () {
globalFlag = 1;
childThread = new ChildThread(this);
childThread.start();
…
synchronized (this) {
childThread.interrupt();
}
}
}
class ChildThread extends Thread {
Main main;
ChildThread (Main main) {
this.main = main;
}
void run() {
if (main.globalFlag == 1) …;
…
main.childThread = null;
}
}
What would the (not improved) lockset algorithm report?
Two potential races: one globalFlag and the other on childThread.
The former is false alarm, why?
The Idea
Race condition detection
with Lockset
Potential race conditions
Define a limited easy-to-check
happens-before relation
Filter false alarms with the
happens-before relation
Filtered race conditions
Exercise 4
class Main {
int globalFlag;
ChildThread childThread;
void execute () {
globalFlag = 1;
childThread = new ChildThread(this);
childThread.start();
…
synchronized (this) {
childThread.interrupt();
}
}
}
class ChildThread extends Thread {
Main main;
ChildThread (Main main) {
this.main = main;
}
void run() {
if (main.globalFlag == 1) …;
synchronized (main) {
main.childThread = null;
}
}
}
What would the hybrid approach report?
Koushik Sen. PLDI’08
RACE DIRECTED RANDOM TESTING
OF CONCURRENT PROGRAMS
Motivation
Hybrid dynamic race detection:
• 39 out of 51 reported race conditions for
tomcat are false alarms
Static race detection (from Stanford, 2006)
• 13 out of 19 reported race conditions for hedc
(a software) are false alarms
Would you be happy to use these tools?
Motivation
Thread1 Thread2
0
0
r1
1
01
r1
r2
02
1
i1
2
i2
i2
12
03
2
w1
3
r2
00
w2
3
Even if we are sure there is a race
condition, users might not be
convinced if you can’t show it!
13
23
11
i1
w1
10
20
22
w2
33
30
21
31
32
count=1
Question
Thread1 Thread 2
Thread k
…
11
21
k1
12
22
k2
…
…
1n
How about we show the problematic
trace by testing the system many
times?
…
2n
kn
How many possible traces are there?
Scheduling
threads
Thread1
Thread2
Thread3
Thread4
Scheduler
How can we control the scheduler so that it would show the error, if there is?
Proposal: PaceFuzzer
1. Use hybrid dynamic race detection to find
potential race conditions, in the form of pairs of
statements (s, t).
2. PaceFuzzer executes the program with a random
scheduler
3. Control the scheduler such that it keeps delaying
s and t
4. Report race condition if s and t can be executed
by different thread next to each other
Example
What are the race conditions according to the hybrid algorithm?
Assume there are two test cases, one in which thread 1 runs to
finish first and the other in which thread 2 runs to finish first.
Example
• Test 1: thread 1 runs first and then thread 2
– Lockset based algorithm:
• race condition between statement 1 and 10
• race condition between statement 5 and 7
– happen-before filtering
• The race condition between 1 and 10 is removed as
statement 4 happens-before 8
Exercise 5
• Test 2: thread 2 runs first and then thread 1
– Lockset based algorithm:
– happen-before filtering:
What are the race conditions reported in this case?
Example: Case (1, 10)
Statement 1 is enabled, so delay it and let thread 2 go.
Example: Case (1, 10)
Statement 1 is enabled, so delay it and let thread 2 go.
Example: Case (1, 10)
Statement 1 is enabled, so delay it and let thread 2 go.
Example: Case (1, 10)
Statement 1 is enabled, so delay it and let thread 2 go.
Example: Case (1, 10)
Thread 2 has finished; thread 1 proceeds to execute statement 1 and more
Example: Case (1, 10)
Report: No real race condition found! False alarm!
Example: Case (5, 7)
Statement 7 is enabled, so delay it and let thread 1 go.
Example: Case (5, 7)
Report: Real race condition identified!
The Algorithm
For each pair of statements (s, t) {
postponed = {}
while (some thread is enabled) {
randomly pick one enabled thread which is not in postponed;
if execute the picked thread would execute s or t {
if a thread in postponed is about to execute the other statement {
report race condition and terminate;
}
else {
add the thread to postponed;
}
}
else {
execute the picked thread for one step;
}
if (enabled == postponed) { remove one from postponed };
}
report deadlock if some thread is not finished;
}
Exercise 6
lock(L);
x = 1;
Apply the algorithm with this modified example for (2, 10).
Empirical Study
Conclusion
• There are proposals for detecting race
conditions.
• The algorithms/tools must work with large
programs.
• The algorithms/tools are often neither sound
nor complete
– Sound: a race condition found is always a realone.
– Complete: all race conditions are found.
Example
Initially: x = y = 0
thread 1 {
x = random.nextInt(1,100000);
count++;
}
thread 2 {
if (x==3771) {
count++;
}
}
Would any dynamic analysis approach be able to find the race condition?
And be sure it is not a false alarm?
Can We Do Better?
• Recall the rules
– Update related state variables in a single atomic operation
– For each mutable variable that may be accessed by more
than one thread, all assesses to that variable must be
performed with the same lock held.
– Every shared, mutable variable should be guarded by
exactly one lock. Make it clear to maintainers which lock
that is.
– For every invariant that involves more than one variable,
all the variables involved in that invariant must be guarded
by the same lock.
–…
Can we do better using other rules?
Example
Thread 1
Thread 2
lock(mu)
x++;
y++;
unlock(mu)
lock(mu)
x++;
unlock(mu)
lock(mu)
y++;
unlock(mu)
Assume that we have an invariant x==y.