InterProcess Communications
1
Outline
 IPC fundamentals
 UNIX sockets
 Remote procedural call
 Shared memory and large memory spaces
 IPC in Windows NT
 Shared memory IPC
 Shared memory and large address spaces
 Windows NT IPC Mechanisms
2
IPC Fundamentals
 What is IPC?
 Mechanisms to transfer data between
processes
 Why is it needed?
 Not all important procedures can be
easily built in a single process
3
Why do processes communicate?
 To share resources
 Client/server paradigms
 Inherently distributed applications
 Reusable software components
 Other good software engineering
reasons
4
The Basic Concept of IPC
 A process needs to send data to a
receiving process
 Sender wants to avoid details of
receiver’s condition
 Receiver wants to get the data in
an organized way
5
IPC from the OS Point of View
Private
address
space
Private
address
space
OS address space
Process A
Process B
6
Fundamental IPC Problem for the OS
 Each process has a private address
space
 Normally, no process can write to
another process’s space
 How to get important data from process
A to process B?
7
OS Solutions to IPC Problem
 Fundamentally, two options
1. Support some form of shared address space

Shared memory
2. Use OS mechanisms to transport data from
one address space to another

Files, messages, pipes, RPC
8
Fundamental Differences in OS
Treatment of IPC Solutions
 Shared memory



OS has job of setting it up
And perhaps synchronizing
But not transporting data
 Messages, etc


OS involved in every IPC
OS transports data
9
Desirable IPC Characteristics
 Fast
 Easy to use
 Well defined synchronization model
 Versatile
 Easy to implement
 Works Remotely
10
IPC and Synchronization
 Synchronization is a major concern for IPC



Allowing sender to indicate when data is
transmitted
Allowing receiver to know when data is ready
Allowing both to know when more IPC is
possible
11
IPC and Connections
 IPC mechanisms can be connection less or
require connection


Connectionless IPC mechanisms require no
preliminary setup
Connection IPC mechanisms require
negotiation and setup before data flows
12
Connectionless IPC
 Data simply flows
 Typically, no permanent data structures
shared in OS by sender and receiver
+ Good for quick, short communication
+ Less long-term OS overhead
- Less efficient for large, frequent
communications
- Each communication takes more OS
resources per byte
13
Connection-Oriented IPC
 Sender and receiver pre-arrange IPC delivery
details
 OS typically saves IPC-related information for
them
 Advantages/disadvantages pretty much the
opposites of connectionless IPC
14
Basic IPC Mechanisms
 File system IPC
 Message-based IPC
 Procedure call IPC
 Shared memory IPC
15
IPC Through the File System
 Sender writes to a file
 Receiver reads from it
 But when does the receiver do the read?

Often synchronized with file locking or lock
files
 Special types of files can make file-based IPC
easier
16
File IPC Diagram
Process A
Data
Process B
17
Message-Based IPC
 Sender formats data into a formal message

With some form of address for receiver
 OS delivers message to receiver’s message
input queue (might signal too)
 Receiver (when ready) reads a message from
the queue
 Sender might or might not block
18
Message-Based IPC Diagram
OS
Process A
Data sent
from A to B
B’s message
queue
Process B
19
File IPC Diagram
main () {
.
call();
.
.
.
Data as parameters
Data as return values
.
.
.
server();
.
.
.
}
Process A
Process B
20
Shared Memory IPC
 Different processes share a common piece of
memory

Either physically or virtually
 Communications via normal reads/writes
 May need semaphores or locks

In or associated with the shared memory
21
Shared Memory IPC Diagram
main () {
.
x = 10
.
.
.
write variable x
x: 10
read variable x
.
.
.
print(x);
.
.
.
}
Process A
Process B
22
Synchronizing in IPC
 How do sending and receiving process
synchronize their communications?
 Many possibilities

Based on which process block when
 Examples that follow in message context, but
more generally applicable
23
Blocking Send, Blocking Receive
 Both sender and receiver block



Sender blocks till receiver receives
Receiver blocks until sender sends
Often called message rendezvous
24
Non-Blocking Send, Blocking Receive
 Sender issues send, can proceed without
waiting to discover fate of message
 Receiver waits for message arrival before
proceeding

Essentially, receiver is message-driven
25
Non-Blocking Send, Non-Blocking
Receive
 Neither party blocks
 Sender proceeds after sending message
 Receiver works until message arrives


Either receiver periodically checks in nonblocking fashion
Or some form of interrupt delivered
26
Addressing in IPC
 How does the sender specify where the data
goes?
 In some cases, the mechanism makes it
explicit (e.g., shared memory and RPC)
 In others, there are options
27
Direct Addressing
 Sender specifies name of the receiving
process
 Using some form of unique process name
 Receiver can either specify name of expected
sender

Or take stuff from anyone
28
Indirect Addressing
 Data is sent to queues, mailboxes, or some
other form of shared data structure
 Receiver performs some form of read
operations on that structure
 Much more flexible than direct addressing
29
Duality in IPC Mechanisms
 Many aspects of IPC mechanisms are duals of
each other
 Which implies that these mechanisms have the
same power
 First recognized in context of messages vs.
procedure calls
 At least, IPC mechanisms can be simulated by
30
each other
So which IPC mechanism to
build/choose/use?
 Depends on model of computation
 And on philosophy of user
 In particular cases, hardware or existing
software may make one perform better
31
Typical UNIX IPC Mechanisms
 Different versions of UNIX introduced
different IPC mechanisms






Pipes
Message queues
Semaphores
Shared memory
Sockets
RPC
32
Pipes
 Only IPC mechanism in early UNIX systems
(other than files)




Uni-directional
Unformatted
Un interpreted
Inter process byte streams
 Accessed in file-like way
33
Pipe Details
 One process feeds bytes into pipe
 A second process reads the bytes from it
 Potentially blocking communication mechanism
 Requires close cooperation between processes
to set up

Named pipes allow more flexibility
34
Pipes and Blocking
 Writing more bytes than pipe capacity blocks
the sender

Until the receiver reads some of them
 Reading bytes when none are available
blocks the receiver

Until the sender writes some
 Single pipe can’t cause deadlock
35
UNIX Message Queues
 Introduced in System V Release 3 UNIX
 Like pipes, but data organized into messages
 Message component include:



Type identifier
Length
Data
36
Semaphores
 Also introduced in System V Release 3 UNIX
 Mostly for synchronization only

Since they only communicate one bit of
information
 Often used in conjunction with shared memory
37
UNIX Shared Memory
 Also introduced in System V Release 3
 Allows two or more processes to share some
memory segments
 With some control over read/write permissions
 Often used to implement threads packages for
UNIX
38
Sockets
 Introduced in 4.3 BSD
 A socket is an IPC channel with generated
endpoints
 Great flexibility in it characteristics

Intended as building block for communication
 Endpoints established by the source and
destination processes
39
UNIX Remote Procedure Calls
 Procedure calls from one address space to
another

On the same or different machines
 Requires cooperation from both processes
 In UNIX, often built on sockets
 Often used in client/server computing
40
More on Sockets
 Created using the socket() system call
 Specifying domain, type, and protocol
 Sockets can be connected or connectionless

Each side responsible for proper setup/access
41
Socket Domains
 the socket domain describes a protocol family
used by the socket

Generally related to the address family
 Domains can be:



Internal protocols
Internet protocols
IMP link layer protocols
42
Socket Types
 The socket type describes what the socket does
 Several types are defined





SOCK_STREAM
SOCK_DGRAM
SOCK_SEQPACKET
SOCK_RAW
SOCK_RDM
43
Socket Protocols
 This parameter specifies a particular protocol
to be used by the socket
 Must match other parameters

Not all protocols usable with all domains and
types
 Generally, only one protocol per socket type
available
44
Some Examples of Sockets
 Socket streams
 Socket sequential packets
 Socket datagrams
45
Socket Streams
 Of type SOCK_STREAM
 Full-duplex reliable byte streams
 Like 2-way pipes
 Requires other side to connect
46
Socket Sequential Packets
 Similar to streams

But for fixed-sized packets
 So reads always return a fixed number of bytes

Allow easy use of buffers
47
Socket Datagrams
 Like sequential packets

But non-reliable delivery

Which implies certain simplifications

And lower overhead
 send(), rather than write(), used to send data
48
Socket Options
 Connection or connectionless
 Blocking or non-blocking
 Out-of-band information
 Broadcast
 Buffer sizes (input and output)
 Routing options
 And others
49
Binding Sockets
 Binding is the process of preparing a socket for
use by a process
 Sockets are typically bound to local names

For IPC on a single machine
 Often accessed using descriptors
 Requires clean-up when done
 Binding can be to IP addresses, as well
50
Connecting to Sockets
 Method for setting up the receiving end of a
socket
 In local domain, similar to opening file
 Multiple clients can connect to a socket

Program establishing socket can limit connections
51
Remote Procedural Call
 Method of calling procedures in other address
spaces
 Either on the same machine

Or other machines
 Attempts to provide interface just like local
procedure call
 Request/reply communications model
52
Semantics of RPC
 Similar to regular procedure call
1. Calling procedure blocks
2. Set of parameters transferred to called
procedure
3. Called procedure computes till it returns
4. Return value delivered to calling procedure
5. Calling procedure continues
53
High-Level RPC Mechanics
 Hide details from applications
 Clients pass requests to stub programs
 Client-end stub sends request to server stub
 Server-end stub calls user-level server
 Results travel in reverse direction
 Network transport or OS actually moves data
54
Diagram of RPC in Action
Client (caller)
call
Server (callee)
reply
reply
Client stub
call
Server stub
OS or Network
55
What do the stubs do?
 Stubs handle complex details like:

Marshaling arguments

Message construction

Data format translation

Finding the server process
56
Setting Up RPC
 Caller must have a way to find the called
procedure
 But it can’t be found at link time

Unlike local procedure
 Potential servers must make their presence
known
57
Registering Servers
 Either register the server at a “well-known” port
 Or register it with a name server

Which in turn is at a “well-known” port
 Calling procedure “addresses” RPC to that port
58
Binding to a Service
 A client process binds to the service it wants
 Two major sub-problem:


Naming
Location
59
Binding: The Naming Problem
 How does a caller name the server program?
 Depends on the RPC mechanism

And perhaps the protocol or type within it
 Do you name the server explicitly?
 Or do you name the service and let some
other authority choose which server?
60
Binding: The Location Problem
 Where is the remote server?
 Some naming schemes make it explicit

Some don’t
 If it’s not explicit, system must convert
symbolic names to physical locations
61
Binding in Cedar RPC
 Client applications bind to a symbolic name

Composed of:


Type
Instance
 Type is which kind of service
 Instance is which particular implementer
62
Locating Cedar RPC Service
 Names do not contain physical location
 So Cedar consults a database of services
 Services register with database
 Binding call automatically looks up location in
database
63
Binding in UNIX RPC
 bind() system call used by servers

Allows servers to bind naming/location
information to particular socket
 connect() system call used by clients

Using information similar to that bound by the
server
 Automatic code generation hides details
64
UNIX Binding Information
 Like most socket operations, it’s flexible
 Fill in all or part of a socket address data
structure
 Create an appropriate socket
 Then call bind to link socket to address
information
 connect works similarly
65
UNIX Binding Example
 On server side,
struct sockaddr_un sin;
int sd;
strcpy(sin.sun_path,”./socket”);
sd = socket(AF_UNIX, SOCK_STREAM, 0);
bind(sd, &sin, sizeof(sin));
66
UNIX Binding Example, Con’t
 For client side,
struct sockaddr_un sin;
int sd;
strcpy(sin.sun_path, “./socket”);
sd = socket(AF_UNIX, SOCK_STREAM, 0);
connect(sd, &sin, sizeof(sin));
67
Locating Remove Services in UNIX
RPC
 Similar to Cedar methods
 Register services with the portmapper
 The portmapper is typically called through
automatically generated code
 the portmapper runs on each machine
 Another service (e.g., NIS) deals with
intermachine requests
68
Using RPC
 Once it’s bound, how does the client use
RPC?
 Just call the routines

As if they were local
 And the results come back
69
What’s happening under the covers?
 When a client calls a remote routine, he really
calls a local stub program
 Stub program packages up request to remote
server
 And sends it to local transport code
 When reply arrives, local transport code
returns results to stub
 Which returns to client program
70
What happens at the server side?
 A request comes in to the RPC transport
code
 It routes it to the appropriate server stub
 Which converts it into a local procedure call
 Which is made within the context of the
server
71
Conceptual Diagram of RPC
call_procedure();
client program continues
Message from client
stub to server stub
Client stub converts
message to return value
Network transport
Network transport
Message received by
server stub
Return value formatted as
message to client stub
call_procedure();
return(…);
72
Transport for RPC
 In Cedar, special-purpose RPC transport
protocol
 In UNIX RPC, can use either UDP or TCP for
transport

Typically protocol is chosen automatically by
stub generator program
73
Other RPC Issues
 Mostly related to intermachine RPC




Data format conversions
Security and authentication
Locating remote services
Multipacket RPC
74
Shared Memory
 A simple and powerful form of IPC
 Most multiprogramming OS’s use some form
of shared memory

E.g., sharing executables
 Not all OS’s make shared memory available
to applications
75
Shared Memory Diagram
a: __
z: __
b: __
x: 10
y: 20
Process A
Process B
76
Problems with Shared Memory
 Synchronization
 Protection
 Pointers
77
Synchronization
 Shared memory itself does not provide
synchronization of communications
 Except at the single-word level
 Typically, some other synchronization
mechanism is used


E.g., semaphore in UNIX
Events, semaphores, or hardware locks in
Windows NT
78
Protection
 Who can access a segment? And in what
ways?
 UNIX allows some read/write controls
 Windows NT has general security monitoring
based on the object-status of shared memory
79
Pointers in Shared Memory
 Pointers in a shared memory segment can be
troublesome
 For that matter, pointers in any IPC can be
troublesome
80
Shared Memory Containing Pointers
a: __
z: __
b: __
y: 20
x: 10
w: 5
Process A
Process B
81
A Troublesome Pointer
a: __
z: __
b: __
y: 20
x: 10
w: 5
Process A
Process B
82
So, how do you share pointers?
 Several methods are in use



Copy-time translation
Reference-time translation
Pointer swizzling
 All involve somehow translating pointers at
some point before they are used
83
Copy-Time Pointer Translation
 When a process sends data containing
pointers to another process
 Locate each pointer within old version of the
data
 Then translate pointers are required
 Requires both sides to traverse entire
structure
 Not really feasible for shared memory
84
Reference-Time Translation
 Encode pointers in shared memory segment
as pointer surrogates
 Typically as offsets into some other segment
in separate contexts
 So each sharer can have its own copy of
what is pointed to
 Slow, pointers in two formats
85
Pointer Swizzling
 Like reference-time, but cache results in the
memory location
 Only first reference is expensive
 But each sharer must have his own copy
 Must “unswizzle” pointers to transfer data
outside of local context
 Stale swizzled pointers can cause problems
86
Shared Memory in a Wide Virtual
Address Space
 When virtual memory was created, 16 or 32
bit addresses were available
 Reasonable size for one process


But maybe not for all processes on a machine
And certainly not for all processes ever on a
machine
87
Wide Address Space Architectures
 Computer architects can now give us 64-bit
virtual addresses
 A 64-bit address space, consumed at 100
MB/sec, lasts 5000 years


Orders of magnitude beyond any process’s
needs
40 bits can address a TB
88
Do we care?
 Should OS designers care about wide
address space?
 Well, what can we do with them?
 One possible answer:


Put all processes in the same address space
Maybe all processes for all time?
89
Implications of Single Shared
Address Space
 IPC is trivial

Shared memory, RPC
 Separation of concepts of address space and
protection domain
 Uniform address space
90
Address Space and Protection
Domain
 A process has a protection domain


The data that cannot be touched by other
processes
And an address space

The addresses it can generate and access
 In standard systems, these concepts are
merged
91
Separating the Concepts
 These concepts are potentially orthogonal
 Just because you can issue an address
doesn’t mean you can access it
 (Though clearly to access an address you
must be able to issue it)
 Existing hardware can support this separation
92
Context-Independent Addressing
 Addresses mean the same thing in any
execution context
 So, a given address always refers to the
same piece of data
 Key concept of uniform-address systems
 Allows many OS optimizations/improvements
93
Uniform-Addressing Allows Easy
Sharing
 Any process can issue any address

So any data can be shared
 All that’s required is changing protection to
permit desired sharing
 Suggests programming methods that make
wider use of sharing
94
To Opal System
 New OS using uniform-addressing
 Developed at University of Washington
 Not intended as slight alteration to existing
UNIX system
 Most of the rest of material specific to Opal
95
Protection Mechanisms for UniformAddressing
 Protection domains are assigned portions of
the address space
 They can allow other protection domains to
access them



Read-only
Transferable access permissions
System-enforced page-level locking
96
Program Execution in UniformAccess Memory
 Executing a program creates a new
protection domain
 The new domain is assigned an unused
portion of the address space
 But it may also get access to used portions
 E.g., a segment containing the required
executable image
97
Virtual Segments
 Global address space is divided into
segments

Each composed of variable number of
contiguous virtual pages
 Domains can only access segments they
attach to
 Attempting to access unattached segment
causes a segment fault
98
Persistent Memory in Opal
 Persistent segments exist even when
attached to no current domain
 Recoverable segments are permanently
stored

And can thus survive crashes
 All Opal segments can be persistent and
recoverable
 Pointers can thus live forever on disk
99
Code Modules in Opal
 Executable code stored in modules

Independent of protection domains
 Pure modules can be easily shared

Because they are essentially static
 Can get benefit of dynamic loading without
run-time linking
100
Address Space Reclamation
 Trivial in non-uniform-address systems
 Tricky in uniform-address systems
 Problem akin to reclaiming i_nodes in the
presence of hard links
 But even if segments improperly reclaimed,
only trusting domains can be hurt
101
Windows NT IPC
 Inter-thread communications

Within a single process
 Local procedure calls

Between processes on same machine
 Shared memory
102
Windows NT and Threads
 Windows NT support multiple threads of
control in a single process address space
 Threads share address space

So communication among them is through
memory
103
Windows NT and Client/Server
Computing
 Windows NT strongly supports the
client/server model of computing
 Various OS services are built as servers,
rather than part of the kernel
 Windows NT needs facilities to support
client/server operations
 Which guide users to building client/server
solution
104
Client/Server Computing and RPC
 In client/server computing, clients request
services from servers
 Service can be requested in many ways

But RPC is a typical way
 Windows NT uses a specialized service for
single machine RPC
105
Local Procedure Call (LPC)
 Similar in many ways to RPC
 But optimized to only work on a single
machine
 Primarily used to communicate with protected
subsystems
 Windows NT also provides a true RPC facility
for genuinely distributed computing
106
Basic Flow of Control in Windows
NT LPC
 Application calls routine in an application
programming interface
 Which is usually in a dynamically linked
library
 Which sends a message to the server
through a messaging mechanism
107
Windows NT LPC Messaging
Mechanisms
 Messages between port objects
 Message pointers into shared memory
 Using dedicated shared memory segments
108
Port Objects
 Windows NT is generally object-oriented
 Port objects support communications
 Two types:


Connection ports
Communication ports
109
Connection Ports
 Used to establish connections between
clients and servers
 Named, so they can be located
 Only used to set up communication ports
110
Communication Ports
 Used to actually pass data
 Created in pairs, between given client and
given server
 Private to those two processes
 Destroyed when communications end
111
Windows NT Port Example
Connection port
Client process
Server process
112
Windows NT Port Example
Connection port
Client process
Server process
113
Windows NT Port Example
Connection port
Communication ports
Client process
Server process
114
Windows NT Port Example
Connection port
Send request
Communication ports
Client process
Server process
115
Message Passing through Port
Object Message Queues
 One of three methods in Windows NT to pass
messages
1. Client submits message to OS
2. OS copies to receiver’s queue
3. Receiver copies from queue to its own
address space
116
Characteristics of Message Passing
via Queues
 Two message copies required
 Fixed-sized, fairly short message

~256 bytes
 Port objects stored in system memory

So always accessible to OS
 Fixed number of entries in message queue
117
Message Passing Through Shared
Memory
 Used for messages larger than 256 bytes
 Client must create section object


Shared memory segment
Of arbitrary size
 Message goes into the section
 Pointer to message sent to receiver’s queue
118
Setting up Section Objects
 Pre-arranged through OS calls
 Using virtual memory to map segment into
both sender and receiver’s address space
 If replies are large, need another segment for
the receiver to store responses
 OS doesn’t format section objects
119
Characteristics of Message Passing
via Shared Memory
 Capable of handling arbitrarily large transfers
 Sender and receiver can share a single copy
of data

i.e., data copied only once
 Requires pre-arrangement for section object
120
Server Handling of Requests
 Windows NT servers expect requests from
multiple clients
 Typically, they have multiple threads to
handle requests
 Must be sufficiently general to handle many
different ports and section objects
121
Message Passing Through Quick
LPC
 Third way to pass messages in Windows NT
 Used exclusively with Win32 subsystem
 Like shared memory, but with a key
difference

Dedicated resources
122
Use of Dedicated Resources in
Quick LPC
 To avoid overhead of copying


Notification messages to port queue
And thread switching overhead
 Client sets up dedicated server thread only
for its use
 Also dedicated 64KB section object
 And event pair object for synchronization
123
Characteristics of Message Passing
via Quick LPC
 Transfers of limited size
 Very quick
 Minimal copying of anything
 Wasteful of OS resources
124
Shared Memory in Windows NT
 Similar in most ways to other shared memory
services
 Windows NT runs on multiprocessors, which
complicates things
 Done through virtual memory
125
Shared Memory Sections
 Block of memory shared by two or more
processes

Created with unique name
 Can be very, very large
126
Section Views
 Given process might not want to waste lots of
its address space on big sections
 So a process can have a view into a shared
memory section
 Different processes can have different views
of same section
 Or multiple views for single process
127
Shared Memory View Diagram
view 1
view 2
Section
view 3
Process B
Process A
Physical memory
128