An Exchange Object
for
Parallel Multimodeling
PILOT Presentation
Michael Harmon
November 14, 2005
Multimodeling
• Simulations that involve interactions
between different types of computational
models
• Models may operate on different spatial
and temporal resolutions
• Models may utilize different mathematical
approaches to simulation
Parallel Multimodeling
• Allow simulations to run simultaneously on
separate processors
• Simulations exchange their data with each
other
How to Communicate?
• Must allow the models to run with different
resolutions (spatial, temporal, etc.)
• Exchange heterogeneous data
WBFish Implementation
• Parallel fish simulation
• Sequential wading bird simulation
• Exchange for message/data passing
Model for Original WBFish
Exchange
• Resident on separate processor
• Follows set exchange cycle
 Fish communicates first
 Bird communicates second
 Cycle repeats
Limitations of original Exchange
• Connected models are forced to synchronize
data in each exchange cycle before advancing
computation
 Leads to substantial time spent blocking on
communication requests
• Lacks scalability/flexibility for adding more
models to the simulation
• No multi-way communication
• Lacks support for heterogeneous data types
between models
An Alternate Approach
• Exchange runs as a central repository for
data being passed amongst the model
• Models can interact with the Exchange at
any point in execution
• Buffering to assist averaging speed
differences between models
• Each model uses an Exchange Object to
communicate
Objectives of New Exchange
• Performance enhancement
• Support multi-way communication
between models
• Scalability
• Flexibility
 Multiple data types
 Dynamic or configurable exchanges
Language
• Exchange: a repository for transferring
data from one model to
another.
• Exchange Object: The unit that contains
the data and
communicates with
the Exchange
Model for New Exchange
• Centralized depot for data and messaging
running on a separate processor
• Each model has a single Exchange Object
that registers with the Exchange
• Exchange Objects make requests to and
from the Exchange as necessary.
• Internal buffering of deposited data
Buffering
Execution Modes
1. Registration - accepts
requests to add
Exchange Objects to
the registry
2. Running - accepts
requests to add, probe,
remove data, etc from
the Exchange
3. Suspended - No op
state. Ignores
communication
requests
4. Shutdown - Terminal
state.
Exchange Object Methods
• registerObject: add an Exchange Object to the
Exchange’s registry.
• put: deposit data into the Exchange
• probe: poll the Exchange for the existence of
data
• get: retrieve data from the Exchange
• halt: move Exchange into the suspended state
• resume: return Exchange to the running state
• shutdown: request to remove Exchange Object
from the Exchange’s registry
Exchange Object Methods
• setValues: set the data in the
Exchange Object
• getValues: extract the data from the
Exchange Object
• setListeners: set which other
Exchange Objects that
are interested in the
data
Code Example
MPI_Comm_rank(MPI_COMM_MYEXCH, &myrank);
MPI_Comm_size(MPI_COMM_MYEXCH, &size);
// exchange processor
if (myrank == DEF_MODEL_EXCHANGE) {
Exchange theExchange(myrank, MPI_COMM_MYEXCH,size-1);
theExchange.start();
}
// computation processors
else {
ExchangeObject myExchObj(myrank, MPI_COMM_MYEXCH);
myExchObj.registerObject(DEF_MODEL_EXCHANGE);
// set listeners
if (myrank == DEF_MODEL_EXCHANGE + 1) listeners = size-1;
else listeners = myrank - 1;
myExchObj.setListeners(listeners, 1);
// launch simulations
launchSimulation(myrank, myExchObj);
}
Code Example
// generic launchSimulation
// computation
etc.
etc.
// communication sequence
myExchObj->setValues(dataSize, dataBuffer);
// write data to exchange
myExchObj->put(DEF_MODEL_EXCHANGE, sendKey);
// grab data
myExchObj->get(DEF_MODEL_EXCHANGE, fromWho, getKey);
myExchObj->getValues(dataBuffer);
// after final sequence
myExchObj->shutdown(DEF_MODEL_EXCHANGE);
Exchange Protocol
Performance Enhancement
Achieve additional speedup by parallelizing
internal data flow by dividing the old
communication exchange cycle into
independent communication requests to
deposit, probe or remove data.
Exchange Performance
Exchange Performance
Old
# Processors Start Date
End Date
Start Time
End Time
Run Time
6
1/1/65
12/31/70
22:35:50
0:03:08 01h27m18s
7
1/1/65
12/31/70
21:20:40
22:31:34 01h10m54s
8
1/1/65
12/31/70
17:51:36
18:53:17 01h01m41s
9
1/1/65
12/31/70
20:03:44
20:59:54 56m10s
10
1/1/65
12/31/70
16:44:38
17:37:46 53m08s
11
1/1/65
12/31/70
19:09:19
20:00:41 51m22s
12
1/1/65
12/31/70
12:48:16
13:37:48 49m32s
13
1/1/65
12/31/70
18:07:52
18:56:33 48m41s
14
1/1/65
12/31/70
11:52:02
12:39:40 47m38s
14
1/1/65
12/31/95
22:10:01
2:19:50 04h09m49s
New
# Processors Start Date
End Date
Start Time
End Time
Run Time
6
1/1/65
12/31/70
15:14:04
16:41:52 01h27m48s
7
1/1/65
12/31/70
13:54:08
15:05:05 01h10m57s
8
1/1/65
12/31/70
10:28:03
11:28:37 01h00m34s
9
1/1/65
12/31/70
10:22:32
11:17:41 55m09s
10
1/1/65
12/31/70
14:33:08
15:22:04 48m56s
11
1/1/65
12/31/70
9:09:11
9:54:26 45m15s
12
1/1/65
12/31/70
13:43:49
14:26:19 42m30s
13
1/1/65
12/31/70
6:21:54
7:02:01 40m5s
14
1/1/65
12/31/70
11:07:32
11:45:46 38m14s
14
1/1/65
12/31/95
23:00:55
2:22:40 03h21m45s
Performance Enhancement
• Original Exchange follows a set cycle of fish
deposit followed by bird extraction.
• New Exchange allows for deposits/extractions to
occur in an arbitrary order.
• Allows fish code to advance and make deposits
without having to wait for the bird model’s
synchronization request
• Performance gain is dependent on how fast the
fish code completes
Exchange Performance
Exchange Performance
Where to Now?
• Deploy multimodeling at a larger scale
• Threaded implementation
• Data interfacing
 Visualization
 Real-time analysis
• Data tracing/archiving