Design Principles and Architectures for
End-to-End Quality of Service
Chris Gill
[email protected]
Center for Distributed Object Computing
Department of Computer Science
Washington University, St. Louis, MO
http://www.cs.wustl.edu/~cdgill/End_to_End.ppt
Theme: Layering in Distributed Systems
• End-to-end layers
– Application components, middleware services, and transport
protocols distributed over multiple end-systems
• Hop-to-hop layers
– Network: internetworking e.g., IP routing between LANs
– Data Link: e.g., frame management in LANs, bridges
– Physical: EE aspects for a medium: voltages, timing
• Different time scales
– End-to-end vs. hop-to-hop, in-band vs. out-of-band mechanisms
• Design Trade-offs
– How to decide what functionality belongs where
The End-to-End Argument
• J.H. Saltzer, D.P. Reed and D.D. Clark, “End-To-End
Arguments in System Design”, ACM Transactions on
Computer Systems, 4(4):277-288, November 1984.
• Design Principle: “A lower level subsystem that supports a
distributed application may be wasting its effort providing
a function that must by nature be implemented at the
application level anyway” - depends on other design forces
• Examples
– FTP correctness, message delivery, data encryption, duplicate
message suppression, FIFO message ordering, transactions
The End-to-End Argument: FTP
Correctness
• Question: where to perform bit error detection and/or
correction for file transfer in a distributed system?
• End-to-end path: I/O device to application to OS to
protocol stack to LAN to router to protocol stack to router
to LAN to protocol stack to OS to application to I/O device
• Hop-to-hop error detection/retransmit, or error correction
is overly expensive unless link is highly unreliable, and
will not solve the entire problem, as sender crashes, I/O bit
errors, and other threats still require end-to-end solution
• Application-specific solution is an end-to-end checksum
every so many bytes and at EOF
The End-to-End Argument: Message
Delivery
• A message is much shorter than a byte stream (e.g., FTP)
and is at less risk for corruption
• However, there are still many threats along the path
– E.g., packet dropping due to router congestion
• Hop-to-hop ACK/retransmit may be useful for congestion
control depending on link reliability and router
capabilities, but still cannot ensure “perfect” reliability
– E.g., receiver crashes right after receipt, unbounded priority
inversion on receiver, etc.
• Need an application-specific end-to-end solution such as a
transactional commit protocol
The End-to-End Argument: Secure
Transmission
• If hop-to-hop encryption is performed, network nodes will
need to securely manage encryption keys
• Unless the application also performs decryption, the
message will be vulnerable on the destination end-system
• The application will also need to perform authentication
of the messages it receives
• Therefore, intermediate encryption is redundant here
• However, intermediate encryption still might be useful to
ensure no unauthorized exposure of information occurs,
say outside a departmental firewall: additional requirement
The End-to-End Argument: Duplicate
Messages
• For efficiency, and possibly correctness, it is useful to
discard duplicate messages
• Even if the network discards all duplicate messages, it has
no way to detect duplication at the application level
– E.g., a repeated user login request due to slow system response
• Since the application needs a way to suppress duplicates
anyway, that function can also be used to suppress
duplicates originating in lower layers
• Therefore eliminating duplicates in lower layers is
wasteful, unless motivated by another design force
The End-to-End Argument: FIFO Ordering
• Independent paths are useful for robustness in the face of
link failures (e.g., spanning tree re-convergence for
routing), for priority-preserving virtual circuits, etc.
• Latency differences (e.g., due to congestion,
retransmission) between consecutive messages routed
along independent paths may result in reordering
• However, distributed applications over multiple endsystems are asynchronous, so for many application if the
order of messages is important, then a higher level
message synchronization protocol is needed anyway
The End-to-End Argument: Applied
• Distributed reads and writes in the SWALLOW distributed
data storage system - anticipates DOC applications (1984)
• Object id plus message version is sufficient to detect
duplicate writes at application layer of server, plus
duplicate replies for reads are easily discarded at client
• Writes on server are ACKed back to client after data is
stored safely, and the read value response is the ACK
• Performing duplicate elimination and acknowledgement at
the application level simplifies the lower layers, and
improves performance by reducing both in-band overhead
and message traffic from lower level ACKs
The End-to-End Argument: Endpoints
• The message delivery and secure transmission examples
highlight a more fundamental issue: where a path ends
• Different applications may apply requirements to different
path segments, and may have different overall paths
– Client-server application: two-way reply appears on the screen
– Manufacturing application: remote robot arm rotates 15 clockwise
• Different inherent semantics for various approaches
– CORBA 2-way calls vs. transactional MOM
• Different applications may produce different decisions
WRT the end-to-end argument
QoS Architectures
• Aurrecoechea, C., Campbell, A.T. and L. Hauw, "A Survey
of QoS Architectures," ACM/Springer Verlag Multimedia
Systems Journal , Special Issue on QoS Architecture, Vol.
6 No. 3, pg. 138-151, May 1998
• Presents a set of QoS design principles
• Presents an architecture (they use the term framework) for
QoS specification and enforcement
• Surveys other QoS management architectures, circa 1998
• Terrific bibliography (modulo a few erroneous dates) will highlight a few papers that proposed design principles
QoS Design Principles
• Integration principle - QoS must be configurable,
predictable, and maintainable over all layers
– Kurose, J.F. “Open Issues and Challenges in Providing Quality of
Service Guarantees in High Speed Networks”, ACM Computer
Communications Review, vol. 23, no. 1, pp. 6-15, Jan 1993
• End-to-end QoS depends on all modules traversed in the
end-to-end path, each of which must offer:
– QoS configurability based on some specification
– Resource guarantees and enforcement mechanisms
– Maintenance of ongoing flows
QoS Design Principles, Continued
• Separation principle - media transfer, control, and
management are three functionally distinct activities
– Lazar, A.A., “A Real-time Control, Management, and Information
Transport Architecture for Broadband Networks”, Proc.
International Zurich Seminar on Digital Communications, pp. 281295, 1992
• Three activities at three scales of complexity
– Media transfer - simple transmission: hop-to-hop forwarding
– Control - more complex but related to media transfer: routing,
queue scheduling, fine-grain monitoring etc.
– Management - distributed, possibly end-to-end: reservations,
reconfiguration, large-scale monitoring, etc.
QoS Design Principles, Continued
• Transparency principle - applications should be shielded
from the complexity of the underlying QoS policies and
management mechanisms
• Reduces accidental complexity by decoupling QoS aspects
from application functionality
• Delegates inherent complexity to a layer below the
application
– Combined with the end-to-end argument, this makes a good case
for QoS management in middleware
QoS Design Principles, Continued
• Multiple time scales principle - guides division of QoS
control and management functionality between modules
– (also from) Lazar, A.A., “A Real-time Control, Management, and
Information Transport Architecture for Broadband Networks”,
Proc. International Zurich Seminar on Digital Communications,
pp. 281-295, 1992
• Three activities at three scales of time
– Media transfer - rapid transmission
– Control - additional overhead but still close to media time scale
– Management - distributed, possibly end-to-end time scales
• Balancing design force to the end-to-end argument
QoS Design Principles, Continued
• Performance principle - a collection of widely agreed
design idioms pertaining to overall system performance
– The end-to-end argument
– Avoiding multiplexing
• Tennenhouse, D.L., “Layered Multiplexing Considered
Harmful”, Protocols for High-Speed Networks, Elsevier
Science Publishers (North-Holland), 1990
– Structuring communications protocols
• Clark, D. and Tennenhouse, D.L., “Architectural Consideration
for a New Generation of Protocols”, ACM SIGCOMM 1990
– Pushing protocol processing down into hardware
QoS Architecture: Specification
• Each category captures a class of application level QoS
requirements as declarative (what rather than how) policies
that are enforced by layer-specific mechanisms
– Flow synchronization - degree of frame tightness between flows
• interestingly, applies to frame-based event-driven applications
– Flow performance - throughput, delay, jitter, loss
– Level of service - deterministic, predictive, best effort gurantees
– QoS management policy - adaptation and scaling capabilities
– Cost of service - competitive, collaborative, individual/social good
QoS Architecture: Mechanisms
• Provisioning
– Initial flow establishment and end-to-end re-negotiation
– Setting up configurations of resources and functionality in support
of media transfer and processing
– End-to-end time scale
• Control
– In-band handling of flows on time-scales close to media transfer
• Management
– Out-of-band reconfiguration of flows to ensure contracts are met
– Time scales between media transfer and end-to-end
QoS Mechanisms: Provisioning
• QoS mapping
– Transforms application QoS requirements into policies for
configuring and operating mechanisms in systems layers
• Admission control
– Determines whether new requirements can be met given current
reservations and resources
• Resource reservation
– Once admission control has been performed, allocates (binds)
resources to the new flow(s)
QoS Mechanisms: Control
• Flow shaping
– In-band traffic pacing, smoothing, pruning to improve overall good
• Flow scheduling
– Forwarding/processing based on per-flow or aggregate policy
• Flow policing
– Detects misbehaving flows
• Flow control
– Sender pacing or receiver feedback adaptation
• Flow synchronization
– Control of flows based on aggregate policies
QoS Mechanisms: Management
• QoS monitoring
– Each layer may track QoS achieved by other modules (feedback)
• QoS maintenance
– Tunes operations based on perceived/expected QoS
• QoS degradation
– QoS cannot be corrected by lower layers, must handle or pass up
• QoS availability
– Performance monitoring/assessment/signaling intervals
• QoS scalability
– Filtering and adaptation over heterogeneous resources/end-systems
Survey of Related QoS Architectures
•
•
•
•
•
•
•
•
•
•
IBM European Networking Center (Heidelberg) - HeiRAT
Columbia U - eXtended integrated Resource Model (XRM)
U Pennsylvania - OMEGA
IETF - IntServ
Lancaster U, Alcatel - QoS-A
OSI - QoS Framework
UC Berkeley - Tenet Architecture
TINA-C - QoS Framework
U Pierre et Marie Curie - MASI End-to-End model
Washington U (Gopal) - End System QoS Framework
Conclusions
• Application-specific QoS requirements must be considered
– Timeliness, correctness, liveness, security
• Application-specific design forces combine with the
inherent complexities of distribution
– Asynchrony, failures, latency, etc.
• System design principles articulate key idioms/forces
• Design patterns and pattern languages appear helpful
– Identifying and codifying trade-offs driven by design forces
• End-to-end argument supports the case for middleware
– Need flexibility to plug in application-specific high-level policies
– Encapsulation of common mechanisms in middleware reduces
accidental complexity for the application