Distributed
Programming
Lesson 06
COMP5126
Overview
We’ve been covering shared-memory techniques:
– busy waiting
– semaphores
– monitors
Most OSs for uniprocessors and shared-memory
multiprocessors use shared variables for process interaction
– However, these OS also provide message passing primitives so that
processes can communicate with other machines on a network
Distributed systems involve:
– Interconnection network can be a variety of possibilities:
– “tightly-coupled” — within the same cabinet, or adjacent cabinets
– “loosely-coupled”, “cluster of workstations” — within the same
room, floor, or building
– Internet — “grid computing”, “client-server”, etc.
The word distributed means spread out across space.
– Thus, distributed computing is an activity performed on a spatially
distributed system.
2
1
Overview II
Semaphores provide a synchronisation
technique
Messages extend semaphores by adding data
Tools
– shared — MPD, Pthreads, Java
– distributed — MPD, MPI, Java
Message-passing communication does not
need to use critical section or mutual
exclusion to guard/protect shared variables.
They instead guard/protect shared
communication links (channels with or
without buffers).
Bottom line: both shared memory and
message passing must share something to be
able to interact:
– they share either variables or communication
links to which access can be synchronised
and protected by mutual exclusion
3
Message Passing
Generalize:
– semaphores: provide synchronisation among processes
– messages: synchronisation plus they add the ability to
transfer data among processes
Channel structure
– generic message structure
– queue of messages
chan name (type1, type2, ...)
– each type represents data fields in messages transferred
over the channel
– can be zero or more types.
– zero types represents a message that is solely providing
synchronisation
4
2
Message Passing Primitives
send name( expr1, expr2, ...)
sends data to the channel
receive name( var1, var2, ...)
receives data from the channel
both send and receive are atomic and they synchronize with
each other
For synchronous system:
– send (i.e. call) and receive block until both have been reached by their
respective processes
– Known as rendezvous technique
– implies use of a bounded queue
For asynchronous system:
– receive blocks
– send does not block
– implies use of an unbounded queue
5
Asynchronous Message Passing
Example 1:
chan toB(i)
Process A:
send toB(1)
send toB(2)
Process B:
receive toB(x)
receive toB(y)
– Can conclude: {x == 1 ∧ y == 2}
– Channel is ordered: send’s from one source are in order in
the channel and received in order
6
3
Example 2:
chan toB(int)
chan toD(int)
Process A:
send toB(1)
send toD(2)
Process B:
receive toB(x)
receive toB(y)
Process C:
send toB(3)
send toD(4)
Process D:
receive toD(i)
receive toD(j)
== 1 ∧ y == 3) ∨ (x == 3 ∧ y == 1)) ∧
((i == 2 ∧ j == 4) ∨ (i == 4 ∧ j == 2))
Can conclude: ((x
send’s from the same source produce ordered results
send’s from different sources produce results in any order!
7
General Process Interaction Patterns
Overview
– filters (pipes) — 1 way data flow
– client / server — 2 way data flow, asymmetric
– peers — 2 way data flow, symmetric
Filters (pipes) — producer/consumer pattern
–
–
–
–
–
–
a|b|c
shared memory — Unix
buffers between processes
write on one side of pipe, read on the other side
non-blocking on both sides (usually)
but, can fill up the buffer!
On distributed memory:
– buffers become channels
– send to the channel to deposit
– receive from channel to fetch
8
4
1. Filters (pipes) Pattern (Cont.)
Example, Figure 7.1, page 297.
•Implements a pipe that takes in characters and outputs lines:
chan input(char), output(char [MAXLINE]);
process Char_to_Line {
char line[MAXLINE]; int i = 0;
while (true) {
receive input(line[i]);
while (line[i] != CR and i < MAXLINE) {
# line[0:i-1] contains the last i input characters
i = i + 1;
receive input(line[i]);
}
line[i] = EOL;
send output(line);
i = 0;
}
}
9
2. Client/Server Pattern
May be many requests
Server may be on the same, or different, machine from
the Client
– Example: a Server that satisfies one type of request, such as a
read() operation on a database:
chan read(int sender_id, ...), reply[1:n](...)
process Client[my_id = 1 to n] {
send read(my_id, ...)
... can do stuff here ...
receive reply[my_id](...)
}
process Server {
local vars
while (true) {
receive read(id, ...)
...process request...
send reply[id](...)
}
}
Array of reply channels, indexed by sender id. Allows Server to
respond to correct sender.
Looks a lot like a procedure, where the call and return are separated
into two calls at the client.
10
5
Multiple operations:
– supports both a read() and a write() operation on a database.
With a Server process: might have just 1 Server
type
type
type
chan
chan
op_kind = enum(READ, WRITE);
arg_type = union(arg1, ..., argn);
result_type = union(res1, ..., resn);
request(int sender_id, op_kind kind, arg_type args);
reply[n](res_type);
process Client[my_id = 1 to n] {
send request(my_id, READ/WRITE,
myargs)
receive reply[my_id](myresults)
}
process Server {
while (true) {
receive request(id, kind, args)
if (kind == READ) {
perform read & prepare reply
}
else if (kind == WRITE) {
perform write & prepare reply
}
send reply[id](...)
}
}
11
Request/release using client/server
Sometimes the Server may have to “hold onto” the request
and reply “later”
– Use a queue for pending requests that cannot be answered “just yet”
Example code see Fig. 7.7 (page 307)
Its similarity with monitor:
– The messages that the Server receives and sends correspond to the
encapsulated procedures of a monitor
– The queue for delayed processes corresponds to a condition variable
Usually, server is implemented as a multithreaded program
keeping a single input channel to record requests, but multiple
input and output channels for interactions
– This allows to serve multiple client requests in parallel
12
6
3. Interacting Peers Pattern
– – 13
2 way data flow, symmetric
Two-process exchange:
– chan p1_to_p2(int), p2_to_p1(int);
int my_value = 14;
send p1_to_p2(my_value);
receive p2_to_p1(&my_value);
int my_value = 42;
send p2_to_p1(my_value);
receive p1_to_p2(&my_value);
Problem:
Consider a collection of processes that each has a value.
We want to arrange the values in order from largest to
smallest, and let each process know the resulting array of
values.
Useful interactive patterns are:
– Centralised
– Star
– Ring
14
7
Centralised solution (Star topology)
One worker, P1, collects the values from all of the
other processes, sorts them, and distributes the
resulting array
chan number (int i), results(int x[1:n])
int my_x[1:n];
for [i = 2 to n]
receive number(my_x[i]);
...sort my_x...
for [i = 2 to n]
send results(my_x);
P7
P1
int my_x[0:n-1];
send number(my_x[my_id]);
receive results(my_x);
How many messages?
– n sends to P1 + n sends from P1 = 2 * n
How much time for the sort?
15
– n2 or nlogn, depending on the algorithm used
Symmetric solution (Mesh topology)
each worker sends its value to all of the other workers
each worker then determines where in the result array its own value goes
each worker then sends the location information to all of the other
workers
channel model does not work easily
chan number[1:n](int x), results[1:n](int pos, int arr[1:n])
Pi:
int my_x[1:n], another, new_pos = 1;
for [j = 1 to n st j != i]
send number[i](my_x[my_id]);
for [j = 1 to n st j != i]
receive number[i](another);
if (another < my_x[i])
new_pos++;
my_x[new_pos] = my_x[i];
for [j = 1 to n st j != i]
send results[j](new_pos,my_x[new_pos]);
for [j = 1 to n st j != i]
receive results[i](a_pos,my_x[a_pos]);
P6
How many messages?
– n2 + n2 = 2 * n2
How much time for sort?
– linear for each process
16
8
Ring solution (Ring topology)
Each process sends to its neighbor around the ring
For the sort problem:
P6
P1: starts by sending its value to P2
P2..Pn: Inserts its value into the array and forwards
the array
completed array has to be forwarded around the
ring
How many messages?
– n + (n - 1) = 2n - 1
How much time for sort?
– each process does O(n) work to insert its value
= O(n2) for n processes
Note: see errata for correction to Figure 7.13, page
317.
17
Synchronous Message Passing
sender and receiver perform send and receive
simultaneously
A sender is blocked until the message is
received, and the receiver is blocked until the
message is sent
Used in some massively parallel architectures
(where processors are not spatially distributed)
In many multicomputer parallel architectures or
computers on LANs/WANs to implement this is
too expensive
– So, asynchronous message passing is used
18
9
MPD Channel Implementation
declare message channel using op:
op values(int, int)
define the implementation in the body:
as a receive
process zap {
... some code here...
receive values(my_x, my_y)
... some more code here...
}
19
Example: exchange
values between n
processes
# Figure 7.12, page 315.
resource exchange()
int numProcs = 1; getarg(1, numProcs);
int numRounds = 1; getarg(2, numRounds);
op values[numProcs](int); # msg channel
process Worker[i = 1 to numProcs] {
int myvalue, maxvalue, newvalue;
for [j = 1 to numRounds] {
# generate a new random value
myvalue = int(ceil(random(1000)));
# send myvalue to the other processes
for [k = 1 to numProcs st k != i]{
send values[k](myvalue);
}
# receive n-1 other values & find max
maxvalue = myvalue;
for [k = 1 to numProcs-1] {
receive values[i](newvalue);
maxvalue = max(maxvalue, newvalue);
}
# Worker 1 prints result each round
if (i == 1) {
write("round", j,"maximum value
is", maxvalue);
}
}
20
} /* Worker */
end
10
CSP
Communicating Sequential Processes.
– first described by Tony Hoare (1978)
– guarded synchronous communication
– CSP has influenced occam, Ada, MPD
Execution of matching input and output
statements in concurrent processes can be viewed as
a distributed assignment
Process A { … B!e; … }
Process B { … A?x; … }
B!e # output value of expression e to process B
A?x # input value from process A into variable x
21
CSP - continued
Direct naming of source and destination processes
in CSP input and output statements
CSP derived language occam uses named channels
for communication – this makes programs more
flexible and easier to develop and change
Example of CSP copy process
Process Copy {
char c;
do true ->
West?c;
# input a character from process West
East!c;
# output a character to process East
od
}
22
11
CSP –guarded commands
Guarded commands
B -> S
Where B is a Boolean expression (the guard) and S is a
statement list
Guarded communication supports non-deterministic
communication patterns
B; C -> S ;
Where B is a Boolean expression, C is a communication
operation, and S is a statement list. B and C together are the
guard
if B1 ; C1 -> S1;
[] B2; C2 -> S2;
fi
[] separates the arms of non-detministic guarded choice
23
CSP – buffered copy
Copy with Guarded commands
process Copy {
char c;
do West?c -> East!c; od
}
Copy process with buffer of two characters
process Copy {
char c1, c2;
West? C1;
do West?c2 -> East!c1; c1=c2;
[] East!c1;
}
24
12
occam language
Invented by David May in mid-1980’s
Very small language syntax
occam2 developed in 1987
occam3 under development in 1990’s
Developed in association with the transputer, a
microprocessor designed for parallel systems, and
was essentially the machine language of the
transputer
Basic units of the language are:
– Process
– Assignment
– Input and output
25
occam language - primitives
Primitive processes are combined into convential
processes using “constructors”
–
–
–
–
–
–
SEQ
PAR
ALT
IF
CASE
WHILE
-- a sequences of processes
-- a concurrent exection
-- guarded input selection
-- selection
-- selection
-- iteration
Program contains a static number of processes and
static communication paths
Procedures and functions are used to modularise a
program
26
13
occam language - examples
Examples:
-- a sequence of assignment statements
INT x,y:
SEQ
X := x + 1
Y := y + 1
-- a loop that inputs and outputs characters
WHILE TRUE
BYTE ch :
SEQ
keyboard ? ch
Screen ! ch
27
occam language - examples
-- concurrent reader and writer processes
CHAN OF BYTE comm :
PAR
-- keyboard reader process
WHILE TRUE
BYTE ch :
SEQ
keyboard ? ch
comm ! ch
-- screen writer process
WHILE TRUE
BYTE ch :
SEQ
comm ? ch
screen ! ch
28
14
occam language - examples
-- buffered copy process, but no output guards
PROC Copy(CHAN OF BYTE West, East, Ask)
BYTE c1, c2 dummy :
SEQ
West ? c1
WHILE TRUE
ALT
West ? C2
–– West has a char ready to send
SEQ
East ! C1
c1 := c2
Ask ? dummy
-- East wants a char
SEQ
East ! C1
West ? c1
29
Java Sockets — Sec. 7.9
Sockets are an API originally written for BSD
Unix. They have since been ported to (most) other
platforms.
– An OS provided service
– Interface is “file-like” — provides the ability to read and
write data into socket
Applications using sockets identify themselves by
the hostname + port #
– port is a 16-bit integer value
– there are “well-known” port numbers reserved for
specific protocols (see /etc/services) (0-1024)
– other applications are assigned port numbers as needed
– only one application (one socket) can use a specific port
number on a machine at a time
30
15
Java Sockets
There are two classes, both within java.net
package:
– ServerSocket
• creates a “passive listener” socket that can accept connections;
that is, the Server side of a client-server pair
– Socket
• creates an “active” socket that can initiate connections; that is,
the Client side of a client-server pair
31
Example: Java Client/Server
FileReader w/ threads.
– Each request to the Server causes the Server to spawn a server thread
and to open a file and provide a sequence of lines from the file to the
client.
The Server closes the file when it has been read, and then
closes the socket connection.
FileReaderServer.java
constructs a “passive” socket, using
java.net.ServerSocket
listens for connection requests on this
socket
accepts requests
creates a thread to handle each request
Client.java
constructs an “active” socket, using
java.net.Socket, to contact the
Server
sends filename
receives lines from the Server
FileReaderThread.java
receives the socket from the Server
opens the requested file
reads and sends lines from the file
closes the file and the socket
32
16
FileReaderServer.java code:
import java.io.*;
import java.net.*;
public class FileReaderServer {
public static void main(String args[]) {
ServerSocket listen;
InetAddress myAddr;
Socket socket;
FileReaderThread new_thread;
try {
// create server socket to listen for
// connection on port 9999, backlog of 5
listen = new ServerSocket(9999, 5);
myAddr = listen.getInetAddress();
System.out.println(
"server: on host "+ myAddr.toString());
while (true) {
System.out.println();
System.out.println(
"server: waiting for connection");
// wait for a client
socket = listen.accept(); # call blocks
System.out.println(
"server: creating thread for socket");
new_thread = new FileReaderThread(socket);
new_thread.start();
}
}
catch (Exception e) { // report any exceptions
System.err.println(e);
}
} // main
} // FileReaderServer
constructors for creating a ServerSocket
variations support the different ways of identifying a host (hostname or IP
address)
specify a port number: 9999 = port number, 5 = backlog (how many
connections can be pending)
33
ServerSocket listen = new ServerSocket(9999, 5);
FileReaderThread.java code
// get filename from client & check if it exists
filename = from_client.readLine();
inputFile = new File(filename);
if (!inputFile.exists())
to_client.println("cannot open " + filename);
else {
// read lines from filename & send to client
System.out.println("reading from file "
+ filename);
input = new BufferedReader(
new FileReader(inputFile));
while ((line = input.readLine()) != null) {
to_client.println(line); sleep(100);
}
}
to_client.close(); from_client.close();
mySocket.close();
} catch (Exception e) {
System.out.println("caught an exception!");
}
} // run
InputStreamReader(mySocket.getInputStream()));} // FileReaderThread class
to_client = new
PrintWriter(mySocket.getOutputStream());
34
// Thread to handle 1 server connection
import java.io.*;
import java.net.*;
public class FileReaderThread extends Thread {
private Socket mySocket;
public FileReaderThread(Socket socket) {
mySocket = socket;
} // FileReaderThread constructor
public void run() {
InetAddress inetAddr;
String filename, line;
BufferedReader from_client, input;
PrintWriter to_client;
File inputFile;
try {
inetAddr = mySocket.getInetAddress();
System.out.println("thread: connection from "
+ inetAddr.toString());
// create client input & output streams
from_client = new BufferedReader(new
17
Client.java code
// Get file from RemoteFileServer & print it on stdout // open socket, then input & output
// usage: java Client "hostname" "filename" ["tabs"]
streams to it
import java.io.*;
socket = new Socket(host, 9999);
import java.net.*;
from_server = new BufferedReader(new
public class Client {
public static void main(String[] args) {
InputStreamReader(socket.getInputStream()))
String line, host, filename;
;
int tabs;
to_server =
Socket socket;
new
BufferedReader from_server;
PrintWriter(socket.getOutputStream());
PrintWriter to_server;
// send filename to server, then read &
try {
print
// read command-line arguments
// lines until the server closes the
if (args.length < 2) {
connection
System.out.println("need at least 2 args");
to_server.println(filename);
System.exit(1);
to_server.flush();
}
while ((line=from_server.readLine()) !=
host = args[0];
null) {
filename = args[1];
for (int i = 0; i < tabs; i++)
tabs = Integer.parseInt(args[2]);
System.out.print("\t");
System.out.println(line);
}
constructs an “active” socket
}
35
can specify host with domain name catch
or IP (Exception
address e) { // report any
exceptions
specify port number — which application to talk to on the host
System.err.println(e);
Summary
Message passing uses channel structure
Three types of interaction patterns:
– Filter
– Client/server
– Interacting peers
Next: RPC and rendezvous
36
18