Quality-based Adaptive Resource
Management Architecture (QARMA):
A CORBA Resource Management Service
Presenter: David Fleeman { [email protected] }
D. Fleeman, M. Gillen, A. Lenharth, M. Delaney,
L. Welch, D. Juedes and C. Liu
Center for Intelligent, Distributed & Dependable Systems
Ohio University
Athens, OH
WPDRTS 2004
April 26, 2004
Presentation Overview
•
QARMA
–
–
–
–
–
–
•
Overview of Architecture
GME/VEST-based Modeling Tool
System Repository Service
Resource Management Service
Enactor Service
Experimental Results
Integration Efforts
– RT CORBA Dynamic Scheduling Service (URI)
– Utah CPU Broker (Utah)
•
Technology Transition
– DARPA PCES BBN OEP (UAV)
– DARPA ARMS MLRM component
– Place services into TAO
•
Conclusions and Future Work
2
Generic Resource Management Architecture
Configuration
Files
Detectors
Decision-Maker
Monitors
Information
Repository
Enactors
Environment
Resource Instrumentation
Software Instrumentation
Resource and Software
Management Commands
Computing Environment &
User Applications
3
CORBA Resource Management Architecture
Resource Management Architecture
QARMA Components
Specification
Tool
Configuration
Files
System
Repository
Service
Resource
Management
Service
Enactor
Service
Replaceable Components
Hosts/Networks
Host and Network
Host and Network
Monitor
Host and
/ Detector
Network
Monitor / Detector
Monitor / Detector
Resource Management Service Goal
•
Each chain of applications achieves a
“benefit” based on it’s service attribute
settings (e.g., frame rate) and extrinsic
attribute settings (e.g., importance).
•
The Resource Management Service
changes service attributes in order to
optimize total benefit, subject to the
constraint that all tasks meet their realtime requirements.
Software
Software
Performance
Software
Performance
Monitor
Performance
/ Detector
Monitor / Detector
Monitor / Detector
UAV Video
Enactor
Enactor
(QualityEnactor
Connector)
(Quality Connector)
(Quality Connector)
ATR
Tracking
Application Layer (independent of QoS mechanism) 4
QARMA System Repository Service
• Purpose
– Central repository for communicating all specification and state data to
management components
• Benefits
– Reduced complexity of other components
– Other components become stateless
– Need for replication and fault tolerance of other components is
decreased
– Flexibility in replacing storage mechanism behind System Repository
(e.g., memory, XML files, database)
– Integration with components developed by other organizations hinges
only on agreement of the data stored in the System Repository
• Drawbacks
– Becomes a single point of failure
– Not scalable for arbitrarily large systems
– These drawbacks can be handled with standard fault-tolerance and
replication mechanisms.
5
QARMA Resource Management Service
• Purpose
– Translate specification into algorithmic model.
– Make intelligent allocation decisions.
• Benefits
– Ability to perform global feasibility analysis for tasks
that require multiple resources.
– Ability to perform global optimization for service
quality settings.
• Drawbacks
– Not scalable for arbitrarily large systems.
• Hierarchical decision-makers, such as the one presented this
morning may be used in place of this component.
6
QARMA Resource Management Service
Code Snippets
IDL Interface
Invocation by Host Detector
module QARMA {
…
interface RM_Service {
typedef sequence<string> Hint;
typedef sequence<Hint> Hint_seq;
// Create a hint for the RM Service
QARMA::RM_Service::Hint_seq hints;
hints.length(1);
hints[0].length(2);
hints[0][0] =
CORBA::string_dup(“Overload”);
hints[0][1] =
CORBA::string_dup(“host1”);
void reactive_rm(in Hint_seq hints);
};
…
};
// Invoke the RM Service
rm_service->reactive_rm(hints);
7
QARMA Resource Management Service
The Key Problems
• Information Acquisition
• Information Modeling
• Resource Allocation Algorithms
–
–
–
–
Feasibility Testing
Optimization Objective
Control Mechanisms
Pluggable
8
Information Acquisition
• Obtained from System Repository Service
– Resource Specification:
• Describes hosts, network devices, and network connectivity.
– Software Specification:
• Describes the extrinsic attributes that affect performance of
applications and the service attributes that can be used to
control application resource usage and quality.
• Describes the application connectivity and dependencies,
performance requirements, and performance characteristics.
– Monitoring Data:
• Host and network resource usage state and statistics
• Software performance state and statistics
9
Information Modeling
• Specification data is transformed into
mathematical models.
– Based on the models presented in chapters 3 and 9
of Jane Liu’s “Real-Time Systems” book.
– Service and extrinsic attributes have been added to
extend the model presented in the book.
• Four models are created:
–
–
–
–
Resource Model
Task Model
Profile Model
Benefit Model
10
Example: Specification => Data Structure => Model
Host ou4.cidds {
Memory 256 MB;
…
Network {
Interface eth0 {
Name “10.0.0.11”;
…
}}}
Host ou3.cidds {
…
Network {
Interface eth0 {
…
}}}
Switch lan1 {
Network {
Interface eth0;
Interface eth1;
}}
Network Topology {
Link { Interface lan1 eth0;
Interface ou4.cidds eth0;
Speed 100 MB/sec;
Latency 0.0 sec;
}…}
…
struct NetworkInterface { string name; … }
struct Host {
string name; long memory;
sequence<NetworkInterface> interfaces;
}
typedef Host NetworkSwitch;
struct NetworkLink {
sequence<StringPair> source_interfaces;
sequence<StringPair> sink_interfaces;
double link_speed;
doulbe link_latency;
}
I
H
I
L
L
S
L
ou4.cidds
with
loopback
interface
fullduplex
link
H
L
switch
lan1
ou3.cidds
fullwith
duplex loopback
link
interface
11
QARMA Control Mechanisms
• Long term goal is to have all the following:
–
–
–
–
–
–
–
Service Level Adaptation
Process Allocation (Startup capabilities)
Process Migration (Ability to stop and restart processes)
Process Replication
Dynamic Network Routing
Network Reservation and Prioritization (IntServ / DiffServ)
CPU Reservation and Prioritization (Utah CPU Broker)
• Currently implemented in QARMA:
– Service Level Adaptation
– Process Allocation (simplest)
– Process Migration (simplest)
12
Example Resource Allocation Algorithm:
Greedy Service Level Optimization
• INPUT:
– Resource Model, Task Model, Profile Model, Benefit Model
• OUTPUT:
– Setting of Service Levels such that the resource utilization on any
resource is less than 69% and utility is maximized.
• ALGORITHM
– Choose lowest quality settings for all tasks
– Rank/order systems by importance
– For each system in order of importance:
• Set service level to the highest
• While not feasible, decrease service level
– Return service level settings
13
QARMA Enactor Service
• Purpose
– Enact changes to the allocation of resources as directed by the
Resource Management Service.
• Benefits
– Delegates change directives to lower-level enactors.
– Specification for enactors allows enactment mechanisms to be
swapped out.
• Drawbacks
– ???
14
Integration with the BBN OEP
Resource Management Architecture
Specification
Tool
Configuration
Files
System
Repository
Service
Resource
Management
Service
Enactor
Service
Replaceable Components
Hosts/Networks
Host and Network
Host and Network
Monitor
Host and
/ Detector
Network
Monitor / Detector
Monitor / Detector
Quality
Connector
Instance
Software
Performance
Monitor / Detector
(Quo Enactor)
Application Layer
QuO Middleware
** System Condition objects were added.
** Contracts were modified to allow RM
Service to force a chosen region.
QuO
QuO
Contracts
Contracts
Event
Channel
read
sys
sys
sys
cond
sys
cond
cond
cond
set
QuO Kernel
Application Layer
BBN OEP UAV System
Video
Source
Process
Video
Distributor
QuO
Video Stream
Video
Display
Proxy
QuO
Video
Display
QuO
Video File
MOBILE VIDEO
SOURCE HOST
VIDEO
DISTRIBUTION
HOST
VIDEO DISPLAY HOST
15
Global Resource Mgmt. Demo Configuration
CORBA
Services
CORBA RM
Services
OU1
Receiver1
Viewer1
Receiver2
Viewer2
Host Mon
LOAD
Distrib1
UAV1
Distrib2
UAV2
Host Mon
Host Mon
OU3
OU2
•
There are two Sender-DistributorReceiver streams.
•
Receiver1 is set to the highest priority
(because it is viewing a target).
•
Use Hourglass to apply CPU load on
the Receiver node.
•
OU4
Load increases every 90
seconds as follows:
–
–
–
–
–
40% at time 40
60% at time 130
90% at time 220
98% at time 310
100% at time 400
16
Global RM vs. Local RM - CPU Load Experiment
Frame Rate
Local RM
Global RM
QARMA Managed CPU Load - Frame Rate - Viewer1
QuO Managed CPU Load - Frame Rate - Viewer1
40
35
35
30
30
25
25
Frame Rate
Frame Rate
40
20
20
15
15
10
10
5
5
0
0
0
25
50
75
100
125
150
175
200
225
250
276
301
327
352
378
405
431
458
483
0
25
50
75
100
125
150
175
200
Seconds
257
282
310
335
360
385
410
435
460
485
356
381
406
431
456
481
Seconds
QuO Managed CPU Load - Frame Rate - Viewer2
40
40
35
35
30
30
25
25
Frame Rate
Frame Rate
232
QARMA Managed CPU Load - Frame Rate - Viewer2
20
20
15
15
10
10
5
5
0
0
0
25
50
75
100
125
150
175
200
225
250
Seconds
276
301
327
351
378
404
431
458
483
0
25
50
75
100
125
150
175
200
231
256
Seconds
281
306
331
Global RM vs. Local RM - CPU Load Experiment
Frame Rate Results
35
30
25
20
Quo Viewer 1
Quo Viewer 2
QARMA Viewer 1
QARMA Viewer 2
15
10
5
0
1
2
3
4
Frame Rate Average (frames per second)
Baseline Viewer1
5.48
Baseline Viewer2
6.01
QARMA Viewer1
8.73
QARMA Viewer2
2.67
Global RM vs. Local RM - CPU Load Experiment
Latency
Local RM
Global RM
QARMA Managed CPU Load - Latency - Viewer1
QuO Managed CPU Load - Latency - Viewer1
6000
5000
5000
4000
4000
Microseconds
Microseconds
6000
3000
3000
2000
2000
1000
1000
0
0
0
25
50
75
100
125
150
175
200
225
250
276
301
327
352
378
405
431
458
483
0
25
50
75
100
125
150
175
200
Seconds
257
282
310
335
360
385
410
435
460
485
Seconds
QARMA Managed CPU Load - Latency - Viewer2
CuO Managed CPU Load - Latency - Viewer2
6000
6000
5000
5000
4000
4000
Microseconds
Microseconds
232
3000
3000
2000
2000
1000
1000
0
0
0
25
50
75
100
125
150
175
200
225
250
Seconds
276
301
327
351
378
404
431
458
483
0
25
50
75
100
125
150
175
200
231
256
Seconds
281
306
331
356
381
406
431
456
481
Global RM vs. Local RM - CPU Load Experiment
Latency Results
250
Average Latency
Standard Deviation
700
600
200
500
Microseconds
150
400
Quo Viewer 1
Quo Viewer 2
100
QARMA Viewer 1
QARMA Viewer 2
300
50
200
0
Baseline QARMA
100
0
Baseline Baseline
V1
Latency Average
Latency Max
Standard Deviation
Baseline Viewer1
236 ms
5770 ms
624 ms
Baseline Viewer2
229 ms
5751 ms
621 ms
QARMA Viewer1
106 ms
3270 ms
321 ms
QARMA Viewer2
89 ms
3479 ms
292 ms
V2
QARMA QARMA
V1
V2
Global RM vs. Local RM - CPU Load Experiment
Conclusions
• Applications controlled by the global RM
• demonstrate greater stability
• exhibit greater determinism
• have lower average latency
• delivered
• higher frame rate for more important streams
• higher average frame rate
Future Work
• Integrate other resource management mechanisms
• Investigate and develop more advanced allocation
algorithms
• Provide fault tolerance and stabilization of the Resource
Management Service as well as reflexive management.
• Continue integration, transition and validation efforts
– Validation with RMBench
– Validation with PCES OEP (Boeing and BBN)
– Integration with PCES technology
• Scheduling Service, Kokyu, CPU Broker
– Transition into DARPA ARMS MLRM infrastructure
– Transition into TAO as CORBA Services
22
Envisioned PCES Component Integration
Resource Management Architecture
Specification
Tool (OU/URI)
System
Repository
Service (OU)
Configuration
Files
Resource
Management
Service (OU)
Enactor
Service (OU)
Resource
Resource
Resource
Enactors
(KU)
Enactors
Enactor (KU)
(KU)
CPU
Broker
CPU
Broker
CPU
Broker
(Utah)
(Utah)
(Utah)
Other PCES Components
Host and Network
Host and Network
Resource
Monitor
/ Detector
Monitor / Detector
Scheduling
Service (URI)
Monitoring (KU)
Hosts/Networks
RT CORBA
Pluggable
Scheduler
Kokyu (WUSTL)
RT CORBA
scheduling
point
UAV Video
(OU)
...
...
ATR
Enactor
Enactor
Quality
Enactor
(Quality
Connector)
(Quality Connector)
Tracking
Adaptation Parameter
Adjustments
(i.e., frame rate)
Application Layer
23