CARNEGIE
Depar[men[ of Elec[rical
MELLON
and Cornpu[e~ Engi~~eering
Real-Time
IEEE 802.5 Token Ring
Implementation
Studies
Thomas E. Marchok
1989
Real.Time
IEEE802.5 Token Ring
Implementation
Studies
Thomas Edward Marchok
Carnegie Mellon University
May8, 1989
This report is submitted in partial fulfillment of the requirementsfor a Masters Degreefrom the Departmentof Electrical &
ComputerEngineering.
Table of Contents
Abstract
1. Introduction
1
2
2. Token-Ring
AdapterChipsetArchitectural Considerationsfor Real-TimeSystems
4
2.1 Background
2.2 Implementation Constraints
2.2.1 Impact on Periodic Task Scheduling
2.2.2 Impact on Asynchronous Message Response Time
2.2.2.1 Alert Class Messages
2.2.2.2 MediumUrgency Messages
2.2.2.3 LowUrgency Messages
2.3 The Impact of Priority Inheritance
on the The TMS380LANAdapter Chipset
2.4 Schedulability Impact of Priority Inheritance
2.4.1 Bounded Blocking Time Theorem
2.4.2 An Algorithm for Determining Worst Case Blocking Time
2.4.3 Periodic Messages
2.4.4 Aperiodic Messages
2.5 Limitations of Priority Inheritance
2.6 An Idealized IEEE 802.5 Implementation
2.6.1 Concurrency Control and Buffer Management
2.6.2 An Ideal Implementation
2.7 Summary
3o End to End Time
3.1 Components of End to End Time
3.1.1 Transmit Time
3.1.2 propagation Time
3.1.3 Receive Time
3.2 Factor Which Influence End to End Time
3.2.0.1 Physical Constraints
3.2.0.2 Host Adapter DMA
Interface
3.2.0.3 Internal Adapter Buffer Size
3.2.0.4 Numberof Allocated Internal Adapter Buffers
3.3 Time Measurements
4. Conclusions
References
4
4
6
8
9
10
10
11
13
13
14
15
15
16
18
19
19
21
22
22
22
23
23
23
23
23
24
24
24
25
26
ii
List of Figures
Figure 1-1:
Figure 2-1:
Figure 2-2:
Figure 2-3:
Figure 2.4:
Figure 2-5:
Figure 2-6:
Figure 2-7:
Figure 2-8:
Figure 2-9:
Figure 2-10:
Figure 2-11:
Figure 3-1:
ARTS Testbed
LANInterface Configuration
Deadline Missed Due to Priority Inversion
Alert Class Message Response Time No Longer Guaranteed
LowUrgency Message Causes Periodics to Miss Deadline.
Original state of transmit queues.
Transmit Queues After Update Priority(4) commandissued at Node
Transmit Queues after messages at Node A have been transmitted.
In SomeInstances Priority Inheritance Guarantees Timing Correctness
Priority Inheritance Salvages Alert Class MessageResponsiveness
Packetization Requires Packet Pacing Considerations
Ideal Implementation Uses Priority Queues
MACLayer to MACLayer Delivery Time
2
5
7
9
10
12
12
13
16
17
18
20
22
Abstract
Previous work [15] developed the algorithms, models and methodologiesto realize highly responsive, deterministic communications services utilizing the IEEE802.5 Token Ring Standard. Simulation studies were used to validate the timing
determinism property, to demonstrate greatly enhanced asynchronous response limes, and to introduce guaranteed
asynchronousalert class messageservice. This work reports the results of implementationstudies utilizing commercially
available IEEE802.5 chip sets and boards. Serious implementationflaws were discovered that prevent the full potential of
the IEEE802.5 TokenRing Standard from being realized for real-time applications. Specifically, unboundeddelays can be
introduced which not only destroy the desired response time determinismoffered by algorithmic scheduling, but also negate
asynchronousalert class guarantees and unnecessarily increase other asynchronousresponse times. This work analyzes the
scheduling properties of current implementations and proposes modifications to those implementations to allow them to
realize the full potential offered by the IEEE802.5 Standard for real-time applications. This work also documentsend to
end message delivery time measurementsfrom the ARTShardware testbed.
1. Introduction
This work involves real-time communicationimplementation studies using the IEEE 802.5 Token Ring Standard [5]. The
introduction of timing correcmess distinguishes real-time communicationsfrom the more general communicationsproblem.
In a nonreal-time setting, it is sufficient to verify the logical correctness of a communicationssolution; however,in a
real-lime setting it is also necessaryto verify the timing correctness. Historically, real-time communications
were provided
via point-to-point connections. Networkingsuch as the IEEE802.5 token ring introduced media contention which must be
carefully managed
in order to maintainstable, predictable liming behavior.
/n the past, timing verification has been performed using Time DomainMultiplexing (TDM)Techniques. In TDM
techniques all access to shared mediais statically boundand predetermined, thus the most limiting factor associated with TDM
techniques is the lack of run time flexibility. The TDM
methodlimits the system’s ability to respond to changing needs.
Run-timeoperation mayonly be altered by re-programmingthe system.
Experience with real-time processor scheduling has shownthat as the number of tasks and task periods increase, TDM
schedules tend to becomead hoc in nature, painfid to generate, and difficult to modify[3]. The real-time communications
scheduling problem for distributed systems is even more complexas the communicationsmediumis generally required to
handle a larger numberof tasks and is typically subject to a more highly dynamicenvironmentthan the processor environment. To avoid propagating the problems of TDM-basedtime managementto the real time communicationsenvironment,
Strosnider [16] developedthe algorithms, models, and methodologiesnecessary to realize highly responsive, deterministic
communicationsservices. These algorithms dynamically bind resource allocation, unlike TDM
techniques which statically
bind resource allocation to fixed time slots. The algorithmic scheduling models for the IEEE 802.5 Token Ring had
previously been verified via simulation only [15]. Here we examine a real-time communicationprotocol on the ARTS
hardwaretestbed [21], [22] to realize the desired real-time environment.
Disk
Disk
T
T
Disk
Disk
T
Figure 1-1: ARTSTestbed
The work initially involved writing a device driver for a commerciallyavailable netwonkingcard (the Proteon proNET
Modelp1542). This established a communicationlink between three Sun 3 wodcstalions via the IEEE 802.5 Token Ring
Standard. After the desired hardware environment was realized, the real-time protocol was studied at the media access
(MAC)level in order to accurately predict the performance of the software interface 1. This knowledgeallowed the
characterization of end-to-end communication.Efficient implementation of the real-time protocol was considered with
regard to the data structures used for implementation.
1Readersinterested in real-time comrnunicationas applied to the higher layers of the ISOReferenceModel[19] are referred to [23] and [24].
The ARTStestbed, pictured in Figure 1-1 includes an IEEE 802.5 Token Ring Networkwith three Sun 3/140 Workstations
as well as an IBMPC/ATused as a network monitor. The Sun workstations use Proteon p1542 NetworkingCards, while th~
IBMPC/ATcontains a Proteon p1344 network monitor card. Both the p1542 and p1344 networking cards are built around
the Texas Instruments TMS380Token Ring Adapter Chipset. The ComputerScience Ethernet backbone connects to the
target machineson the token ring to permit downloadingof the experimental ARTSkernel and debugging.
The original intent of this research was to duplicate the simulations developed in [15], in order to demonstrate the advantages of algorithmic scheduling in the communicationsdomainover conventional TDM
techniques° Unfortunately there
were a numberof obstacles which prevented this, including:
¯ Load Generation: the simulations ran at a ring utilization near 70%.The hardware testbed was not able to
provide this level of ring utilization with only three nodes transmitting messagesonto the ring. Eachstation was
only able to utilize approximately5%of available ring bandwidth.Financial constraints prevented the procurement of additional Sun Workstations and Proteon p1542networking cards.
¯ Measurementot’ Time: whereas the simulations monitored messagedelivery time to the nearest 0.1 msec, the
internal clock on the Sun 3 workstation has a 10 msec granularity. The size of the packet transmitted was
increased in an attempt to lengthen the delivery time of the messages. Howeverdue to memorylimitations of
the Proteon p1542 NetworkingCard and the TMS380
Adapter Chipset, it was not possible to make the packet
large enoughsuch that the timing granularity wassufficient.
Although it was not possible to duplicate the simulations from [15], muchwas learned from the development of the
hardware testbed. Beyondthe aforementioned physical constraints, commercial implementations of the IEEE 802.5 Token
Ring Standard have features which compromise
the abovelisted requirements and artificially limit the perfomaancepotential
of the IEEE802.5 Standard for real-time systems. Chapter 2 examines the flaws found in commercialimplementations of
the IEEE 802.5 TokenRing Standard, demonstrates howthese flaws reduce ring schedulability, and makes general recommendationsregarding token ring adapter chipset architectural considerations to support real-time communication.Chapter 3
documentsmessagetransmission times as observed on the ARTStestbed.
4
2. Token-RingAdapter Chipset Architectural Considerations for Real-Time Systems
2.1 Background
The introduction of individual message response time requirements distinguishes real-time communicationsfrom more
general pulpose communication where notions of aggregate throughput are sufficient. The real-time communication
problem inherits all of the problemsof the general purpose domainplus the added difficulty of guaranteeing individual
messagerequirements. Several research efforts have addressed this area. IBM[2] extended the Multi-Level Multi-Access
(MLMA)
Protocol originally developed by Rothauser and Wild [12] to support priority based media arbitration on buses.
Zhao, Stankovic and Ramaritham [25] developed a sliding windowprotocol which provides an alternative non-TDM
strategy for scheduling time constrained messages in networks. LeLann[6] has developed a variant of the CSMA/CD
protocol which uses an adaptive tree walk strategy for back-offs that provides another non-TDM
alternative for real-time
communications. The SAE-gBHigh Speed Ring Bus [4] committee has recognized the limitations of TDMscheduling for
real-time applications and has adopted a priority-based media arbitration approachwhichsupports algorithmic scheduling.
Previous work by Strosnider, et. al. [15], [17] develops an algorithmic scheduling modelfor the IEEE802.5 TokenRing
Standard. This approach provides the disciplined timing behavior that allows the prediction of system timing behavior via
algorithmic media access scheduling as opposed to conventional Time DomainMultiplexing (TDM)based scheduling. The
work demonstratedthat the a priori timing predictions can be met providing a guaranteed alert class service and greatly
enhanced asynchronousresponsiveness while still maintaining synchronous class guarantees. Fulther, asynchronous message response time improvements of nearly two orders of magnitude were demonstrated over the conventional TDM
approach. This research demonstratedthat the IEEE802.5 Standard can be used to form the backboneof predictable, stable,
extensible real-time systems.
This work reports implementation flaws which were discovered during an implementation study on the ARTS
Tested [21], [22]. These flaws prevent the IEEE802.5 TokenRing Standard from realizing its full potential for real-time
applications. Section 2.2 covers the major implementation flaws in commercially available implementations of the IEEE
802.5 Standard. Section 2.3 introduces a possible solution to the flaws. Section 2.4 illustrates howthis solution impactsring
schedulabifity, and Section 2.5 covers the limitations of this solution. Finally, Section 2.6 summarizesimplementation
requirements necessary for the IEEE802.5 Standard to support real-time communication.
2.2 Implementation Constraints
The IEEE 802.5 Token Ring Standard includes attributes which allow the standard to support algorithmic media access
scheduling. Algorithmic LANscheduling has been shown to have advantages over TDM-basedLANscheduling for
real-time applications [15]. Theattributes to support algorithmic scheduling include:
¯ three priority bits (8 levels of priority) in each token to provide priority granularity for both messagesand free
tokens.
¯ an algorithmic mediaaccess technique based upontoken priority [5].
Priorities are used to determine in which order messages gain access to the network. Algorithmic scheduling assigns
priorities to messagesin order to guarantee messagedelivery time. The Texas Instruments TMS380
Adapter Chipset ([20])
and IBMToken Ring Chipset are commercially available implementations of the IEEE 802.5 Token Ring Standard.
Unfortunately these implementationshave features whichcompromisethe abovelisted requirements and artificially limit the
performance potential of the IEEE 802.5 Standard for real-time systems. This work focuses on the Texas Instruments
2.
TMS38.0Adapter Chipset
WPheIBMToken Ring Chipset exhibits manyof the same problems as the TMS380
Chipset. The authors constructed and experimented with a testbed
which uses the Texas Instruments TMS380
Adapter Chipset, and therefore focus on the TMS380
implementation.
The first constraint which the TMS380
Chipset imposesis that is provides only half of the required priority levels to the
application (4 levels as opposedto 8). This lack of priority granularity significantly restricts the applications that can
supported to those with a small numberof unique priority levels. Applications with a rich set of timing requirements are
either not supported, or supported at arbitrarily low schedulable utilization levels since insufficient priority granularity
reduces schedulableutilization [7].
HO~T
CPU Main
Memory
Adapter Memory
TokenI
RingI Transmit
Receive
Ada’ptel&~LBuffer Buffer
System Bus
to TokenRing
l~igure 2-1: LANInterface Configuration
The TMS380Adapter Chipset constitutes the LANinterface to the token ring network. Whena message is queued for
transmission, it is transferred from host memoryto that node’s Transmit Buffer, which resides in Adapter memory(as
illustrated in Figure 2-1). Unfortunately the TransmitBuffer is maintainedas a FIFOqueue3. The priority of the messageat
the front of the FIFOqueueis used to contend for access to the token ring. Priority inversion occurs whena higher priority
messageis blocked behind a lower priority messagealready in the Transmit Buffer. Priority inversion is the phenomenon
where a higher priority job is forced to wait for a lower priority job to complete its execution [9]. Priority inversion
introduces unbounded delays into message response times. These unbounded delays destroy the predictability and
guaranteed schedulability developedin theoretical studies and validated via simulation. (The simulation modelof the IEEE
802.5 Token Ring Adapter is based upon the assumption that at each node the highest priority pending messages are
transmitted before lower priority messages4.) In result, priority inversion causes a degradation in both response time
performanceand the percentage utilization of mediawhich maybe used such that all messagedeadlines are still guaranteed
(ring schedulability). As an illustration, consider the followingexample.
Example0: Suppose that there are 3 nodes X, Y and Z on a token ring network. Multiple messages may be queued for
transmission at a single node simultaneously, howeverat each node ring access contention is performedusing the priority of
the pendingmessagewhicharrived at the queue at the earliest point in time. Let there be a low priority messageon node X,
and two mediumpriority messages on nodes Y and Z when a free token is generated on the network. Before any data
transmission occurs, a high priority messagebecomesready for transmission on node X. However,the network interface at
node X cannot transmit the high priority messageuntil after the low priority messagethere has been transmitted. This
duration of waiting represents priority inversion for the high priority messageat node X. Unfortunately, the duration for
whichthe high priority messageat node X is blocked due to priority inversion is unbounded.This is because all present and
future mediumpriority messageson nodes Y and Z would be transmitted before the low priority messageon node X can be
transmitted, makingwayfor the high priority message.
Priority inversion is also a problem when receiving frames. Frames are inserted into that node’s Receive Buffer when
3Ideally the Transmit Buffer is maintained as a Priority Queue. Thoughperformance concerns accompanythe use of Priority
[10] demonstratedthat Priority Queuesneed not translate into performancelosses.
4Messagesof the same priority are transmitted in the order in which they were queued.
Queues, Ralya
received. Framesare then transferred from the Receive Buffer in adapter memory
to host memoryin the order in which they
were received. Messagepriorities are used to determine in which order messagesshould be Iransferred to host memoryfor
processing at the upper layers of the end-to-end protocol [14], [19]. Priority inversion occurs whena higher priority
messageis received while a lower priority messageis already in the queue. The complicationsat the receive end are distinct
from those at the transmit end. Wewill concentrate on the implementationproblemsas they relate to the transmit process,
and later relate those problemsto the receive process.
Strosnider [16] developed the Deferrable Server Algorithm, an advanced scheduling algorithm which provides dramatic
speedups in aperiodic message response times while still guaranteeing the periodic messageresponse times [15]. The
Deferrable Server algorithm relegates aperiodic messagesinto three separate classes: a high urgency(alert) class, a medium
urgency class, and a low urgency (background) class. The priority inversion problem inherent to the TMS380
TokenRing
Adapter Chipset degrades the improvementsdemonstratedin the simulation studies [15] in a numberof ways:
¯ The magnitudesof the guaranteedresponse times for alert class messagesare seriously degraded.
¯ The meanand standard deviation for mediumurgency messageresponse times are both greatly increased.
¯ The presence of backgroundmessagesserves to funlaer magnify the effects of the above complications as they
block higher priority asynchronousaccesses.
Unfortunately, the only remedyfor the unboundedpriority inversion problem in the TMS380
LANAdapter Chipset is an
expensive one. The only possible way to preempt the message pending transmission is to issue the Transmit Halt command, which purges all frames queued for transmission. The cost of the Transmit Halt commandis the time required to
re-DMAeach of the frames purged (not just the frame at the beginning of the Transmit Queue) from Host to Adapter
memory.
Suppose that a high priority messagebecomesready for transmission at a node whena low priority messageis already
pending within the LANadapter chipset at that samenode. To avoid priority inversion, the host operating system needs to
purge the low priority frame to allow transmission of the higher priority frame. The low priority frame wouldbe re-queued
for transmission later. 5 Clearly, the data corresponding to all purged frames must be re-copied into the adapter chipset
buffers at somefuture point. Thus, the overhead due to the Transmit Halt operation along with the retransmission costs can
be rather expensive. In a prioritized real-time system, there could be several priority levels in use and the cumulative
overheadof the abort operations can be prohibitive.
The remainder of this section studies the impact of unboundedpriority inversion in the transmit process in the TMS380
TokenRing Adapter Chipset. Wefirst illustrate howpriority inversion destroys the guaranteed response time for periodic
messages offered by algorithmic scheduling. Wethen examinethe impact of priority inversion on asynchronous message
response lime performance.
2.2.1 Impact on Periodic Task Scheduling
The unboundeddelays caused by priority inversion maycause periodic messagesto miss their deadlines.
followingset of rate orderedperiodic messages:a, b ..... g.
Consider the
Example1: Figure 2-2 pictures a token ring LANwith 4 stations, A, B, C, and D. The ring has a task set consisting of
periodic messagesa through g. Each messagerequires 5 time units of ring time for transmission and delivery. The Rate
Monotonicscheduling algorithm [8] assigns the highest priorities to messageswith the shortest periods. Consistent with the
rate monotonicscheduling algorithm, periodic messageswith a period of 50 (a, j, k) are assigned priority 3. Periodic
5There might be a small time windowwhenan ongoing frame transmission is completed just when the Transmit Halt commandis issued. Hence, it is
possible that a purgedframe is retransmitted twice. End-to-endtransfer protocols can deal with this situation, say, identical to the loss of an acknowledge
frame.
Access
Priority
2
I ! I k I h ( e I b ~I i~ 2
0
25
2
22 (f~[ c ( 2~I dI a 12
2
50
Figure 2-2: Deadline MissedDue to Priority Inversion
messageswith a period of 100 (b, c, d, e, f, g, h, i) are assigned priority 2. The notation used to represent queuedmessages
in Transmit Buffers will be uniform in all Examplesthroughout this work. Referring to Figure 2-2, the letter in the lower
right hand comer of a Transmit Buffer entry denotes the name of the message, while the numberin the upper right hand
comerrepresents the priority of that message.
Wequote Theorem1 which was proved by Liu and Layland [8] under the assumption of independent tasks, i.e. whentasks
do not block one another due to resource sharing.
Theorem1: A set of n periodic tasks scheduled by the rate-monotonic algorithm can always meet their deadlines
if
Cn
T’--~ + "’" +-zn <n(21/n-1)
whereCi and Ti are the executiontime and period of task zi respectively.
Wenowapply this Theoremto the communicationdomain, where we consider messages to be the tasks 6. For Example1,
the periodic messageset of each station is revealed in Figure 2-2. ApplyingTheoremI to the messageset of Example1:
C
a
Cg
-~--.+
... +~-< ll(21/H--1)
a
g
6NowCi represents the time for transmission and delivery of message i, and Ti the period of message i
which becomes
(3 × ~) + (8 ×~--z-x,) < 0.7154
5-~
~u~5
The set of periodic messages in Example 1 meets the requirements of Theorem1, and thus the message set should be
schedulable on the ring. Therefore all periodic messageswill meet their deadlines provided that tasks do not block one
another whencontendingfor the token ring, and provided that messagesare assigned the proper priority according to the rate
monotonicscheduling algorithm.
Supposethat the messagesfrom Example1 arrive at their nodes’ Transmit Buffer FIFOQueueat nearly the same instant and
in the order indicated in Figure 2-2. Althoughpessimistic, this is the worst case ordering for the messageset at NodeA, and
thus provides the most rigorous scheduling test. Messagea is blocked behind lower priority messagesb, c, and d at node
A. The timeline at the bottomof Figure 2-2 indicates that messagea misses its deadline because it was blocked behind lower
priority messagesb, c, and d7. Even though messagea was queuedand ready for transmission, NodeA contended for ring
access with priority 2, the priority of the messageat the front of the queue. AlthoughNodeA had a pending messageof
higher priority than nodes B and C, each node had equal access to the ring while messagesb, c, and d were at the head of the
Transmit Buffer at Node A.
Our evaluation of Theorem1 guarantees that the task set from Example1 wouldbe schedulable in the absence of blocking.
Unfortunately priority inversion introduces delays whichnullify ring schedulability. Whenunboundedpriority inversion is
present in a system, response times cannot be guaranteed.
2.2.2 Impact on Asynchronous Message Response Time
Guaranteeingperiodic task sets, though important, is only one of the objectives in real-time communicationscheduling.
Concernswith regard to aperiodic messageperformance also exist. Simulation studies offered proof that the Deferrable
Server scheduling algorithm dramatically enhances aperiodic messageswhile still maintaining guaranteed service for periodic messages.
Noting that there is no value to the system for the periodic messagescompleting early, the Deferrable Server algorithm
assigns higher priority to aperiodics up until the point wherethe periodics wouldbe late. Recall that the Deferrable Server
algorithm relegates aperiodic messagesinto three separate classes, including a high urgency (alert) class, a mediumurgency
class, and a low urgency (background)class. Alert and mediumurgency class messagesare assigned priorities higher than
periodic messages, while backgroundmessagesare assigned the lowest possible priority.
Whenboth periodic and aperiodic messagesare present in a system, it is common
for messageswith different priorities to be
transmitted from the same node. Because the order of queueing cannot be predetermined, transmitting messages with
different priorities fromthe samenode can lead to priority inversions. Theremainderof this section will demonstratethat if
priority inversion exists, alert and mediumclass messages mayno longer receive good response times. Furthelmore, the
existence of backgroundaperiodic messagescan result in a dramatic increase in blocking times for periodic messages,, which
further reduces ring schedulability.
7Thetime line in Figure 2-2 arbitrarily assumes that free tokens perambulatethe ring in a clockwisemanner.This assumptionholds for all examplesin
this thesis.
9
2.2.2.1 Alert Class Messages
The simulation studies guaranteed a highly responsive alert class message, and demonstrateda dramatic decrease in medium
urgency response time (nearly two orders of magnitude) over traditional scheduling algorithms. Howeverpriority inversion
severely degrades the high degree of responsiveness which advancedscheduling algorithms offer. Consider the following
example.
Example2: Here we use the samebasic messageset as in Example1, with the addition of an alert class messagen at Node
C. Once’againsupposeall messagesarrive at nearly the sameinstant (t = 0). The period for all priority 2 periodics is 100,
and priority 3 periodics have a period of 50. Each messagerequires 5 units of ring time for transmission. The ordering of
the Transmit Buffer at NodeC in Figure 2-3 corresponds to the worst case response time for messagen.
Media
Access
Priority
2
el
h
6 I
Figure 2-3: Alert Class MessageResponse
Time No Longer Guaranteed
Messagen is the highest priority messagein the system, and should be transmitted with the next free token after it is queued
for transmission. Howeveraccording to the arrival sequenceof the messagesin Figure 2-3, priority inversion exists at node
C wherepriority 2 periodic messagese, f, and g block alert class messagen. NodeC competesfor ring access with priority
2 until all periodics have beentransmitted. NodeC finally transmits messagen at t = 50, thus yielding a response time of 55.
If higher.priority messageswere transmitted before lower priority messages, the worst case response time for messagen
wouldbe 10 time units.
-10
2,2,2,2 MediumUrgency Messages
Priority inversion degrades the response time of mediumurgency messagesin the samemannerthat it degrades the response
time of alert class messages. For example, suppose messagen from Figure 2-3 is a mediumurgency messagewith priority
5. Onceagain, as in Figure 2-3, all lower priority messagesare transmitted before messagen. The response time of medium
urgency messagen is degraded from what it wouldbe if priority inversion were not present. In general, priority inversion
increases both the meanand standard deviation of the response times for mediumurgencyas well as alert class messages.
2.2.2.3 LowUrgency Messages
Backgroundmessagesare often transmitted from the same node as higher priority messages. While this is often necessary,
background messages block periodic messages as well as alert and mediumurgency aperiodic messages. Whena backgroundmessageis at the head of the TransmitBuffer, all messageswith priority greater than 0 (backgroundpriority) queued
at all other nodes are transmitted before that backgroundmessage.The result is an increase in the worst case blocking time
for each higher priority messageat that node.
Example3: Consider the same basic task set as in Example1, with the addition of a low urgency message at Node B
(message m). The Transmit Buffer at Node B in Figure 2-4 corresponds to the worst case arrival sequence there. Once
again, supposeall messagesarrive at nearly the sameinstant (t = 0). Theperiod for all priority 2 periodics is 100, priority
periodics havea period of 50, and each messagerequires 5 time units of ring time for transmission.
Media
Access
Priority
h c
0
f
i
25
d a
m
k
50
Figure 2-4: Low Urgency Message Causes
Periodics to Miss Deadline.
With priority 0 message m in the front of the Transmit FIFOBuffer the worst case blocking time for messages j and k
consists of the time in whichall messageswith priority less than 3 are signalled while the priority 0 messagemis at the head
11
of NodeB’s Transmit Buffers . This time includes the ring time required to transmit all priority 2 messagesat NodesA, C,
and D. Since there are a total of eight priority 2 periodic messagesat those nodes, the blockingtime of messagesj and k is 40
time units 9. The timeline at the bottomof Figure 2-4 showsthat periodic messagesj and k miss their deadline at time t = 50
because they are blocked by the backgroundmessagem.
2.3 The Impact of Priority
Inheritance
on the The TMS380 LAN Adapter Chipset
There is a knownsolution to the unboundedpriority inversion in the processor scheduling domainproblemcalled priority
inheritance [9]. The basic idea of priority inheritance is that whena high priority messageis waiting behind a low priority
messagein a Transmit FIFOBuffer, the low priority messageinherits (uses) the priority of the high priority messageto
contend for network access 1°. The otherwise unboundedpriority inversion time is then bounded according to network
queueing models. For a simple example, reconsider Example0. With a priority inheritance mechanism,the low priority
messageon node X wouldbe transmitted at high priority on the network, and therefore allow the high priority messageat
node X to be transmitted before any mediumpriority messageswere transmitted from nodes Y or Z.
The Rate Monotonicscheduling algorithm assigns priorities in a mannerwhich insures that periodic messages will meet
their timing constraints. In reducing the effects of priority inversion, priority inheritance not only boundsthe duration that
messageshave to wait before transmission, but also allows for high schedulable utilization of the LAN.As a result, message
timing constraints can be guaranteedto be met at high levels of LAN
traffic. In contrast, if the priority inversion problemis
allowed to cause unboundeddelays, the timing constraints of messagesmaynot be met even at low traffic levels.
Recall that before a messageis signalled onto the ring, that the frame11 (along with its priority) is copied by the adapter into
its internal Transmit Buffers. The MediaAccessPriority, whichis the priority that the node uses contend for access to the
ring, is set to equal the priority of the flame at the head of the Transmit FIFO Buffer. Whensubsequent frames are
transferred to the internal adapter buffer, there exists no mechanismfor modifying the Media Access Priority. Hence
priority inheritance cannot be implemented.If a high priority frame becomesready for transmission whena lower priority
messageis already in the transmit queue, the scenario of Examples0 and 1 occurs.
Anattractive solution to the unboundedpriority inversion problem in the TMS380
Adapter Chipset is the creation of an
Update Priority(Pri) commandwhich may be issued to the adapter chipset. Whenthis command
is issued to the adapter
chipset, the priority used at that node to contend for ring access is adjusted to Pri. Note that the node’s MediaAccess
Priority is adjusted, and that framepriorities are not changed.This is an importantdistinction, as the priority of frameswill
retain their original value during processing in the receive queues at the destination node. The Media Access Priority
remainselevated until the node transmits a frameat a priority equal to priority Pri, at whichpoint it is set to the priority of
the frame currently at the beginningof the transmit queue.
Each time a frame Fhost is readied for transmission a check is done. If frame Fhost has a priority higher than all frames
currently in the Transmit Queue, then the host issues an UpdatePriority(Pri) command
with Pri = priority of Fhnsr If the
MediaAccessPriority was elevated to Pri, the MediaAccessPriority of that node will then remain at Pri until the Adapter
signals a frame with priority Pri. Immediatelyafter doingso, the MediaAccessPriority of that node is adjusted to equal the
highest priority of any frame in the TransmitQueue.
Sin section 2.4.2 we develop a WorstCase BlockingTimeAlgorithmin order to quantify this calculation.
9Notice that the blocking time does not include the ring time for messagea, since messagea does not have a lower priority than messagesj and k.
1°Notethat the actual priority of the messagedoes not change. Onlythe priority that the node uses to contendfor ring access is affected. Theoriginal
priority of the messageis preservedsince this priority is used to determinethe order of processingat the receiving node.
11in this explanation the terms messageand frame are used interchangeably, in general, large messages maybe divided and transmitted in multiple
frames.
12
In summary,the priority inheritance operation consists of the followingsteps:
* There are two Scheduling instances: the completion of a frame transmission and the enqueueingof a frame by
the host.
- At each scheduling instance, the host sets the MediaAccess Priority to the maximum
of the priority of the host
frameFlaost and the priority of all framescurrently in the transmit queue, F] .... Fn, where n is the number of
buffer frames in the transmit queue. That is, MediaAccessPriority = max(Fhost,
NodeA
NodeB
NodeC
Transmit
Queues
Media
Access
Priority
802.5 TokenRinq
Figure 2-5: Original state of transmit queues.
NodeA
NodeB
NodeC
Transmit
Queues
,, [-;-]
Media
Access
Priority
802.5 TokenRing
Figure 2-6: Transmit QueuesAfter UpdatePriority(4)
commandissued at Node A.
Figure 2-5 illustrates a situation in which unboundedpriority inversion exists at NodeA. There a priority 0 messageis
blocking a messageof priority 4. If the MediaAccess Priority at NodeA remains 0 all of the lower priority messagesat
NodesB and C will be transmitted before the priority 4 messageat NodeA, since these messagesare all at a priority higher
than the MediaAccess Priority of NodeA. The situation is said to be unboundedsince the blocking time is dependent upon
the numberof messages queuedat Nodes B and C at that instant. Figure 2-6 shows the adjusted MediaAccess Priorities
after an Update Priority(4) command
is issued at NodeA. Note that the MediaAccess Priority at NodeA is 4 even though
the priority of the messageat the head of the queueis 0. NodeA will have access to the ring as long as the MediaAccess
Priority at NodeA is higher than at nodesB and C. Figure 2-7 illustrates the state of the TransmitBuffers at each node after
NodeA has transmitted the two packets at the head of its Transmit Buffer, and the Media Access Priority returns to the
priority of the frameat the headof the transmit queue.
The UpdatePriority command
is entirely transparent to applications software. Instead the operating system software that
communicateswith the adapter chipset ]2 issues the UpdatePriority command
whennecessary. The device driver software
retains a FIFOqueue of the priority of all frames in the Transmit Buffer. Each time a frame is DMA’ed
to the Transmit
12Commonlyreferred to as the device driver.
13
Node A
Node B
Node C
Transmit
Queues
Media
Access
Priority
802.5 Token Ring
Figure 2-7: Transmit Queuesafter messages
at NodeA have been transmitted.
Buffers the driver software checks to see if the queuecontains any framesat a lowerpriority than that frame. If so, then the
driver issues an UpdatePriority(Pri) command
with the Pri parameterset to the priority of that frame. As a result, both the
unboundedpriority inversion problem and the cosily overhead of aborting and retransmitting frames are avoided. The
priority inheritance mechanismalso bounds the duration that a high priority frame is blocked by a lower priority frame.
Mostimportantly, this command
makesit possible to guarantee that the timing constraints of real-time frames are met.
The UpdatePriority command
is conceptually simple and its implementation of the idea maynot require silicon changes.
Instead the change might be facilitated through changing the microcode which runs the TMS380Adapter Chipset. If
implemented, the Update Priority command
would result in a simple and cost-effective solution to support prioritized
scheduling on the token-ring. Howeverthe schedulability recovered via a priority inheritance mechanismis bounded. Not
all of the schedulability lost to priority inversion maybe recovered with a priority inheritance scheme. The next section
discusses the limitations of a priority inheritance scheme.
2.4 Schedulability
Impact of Priority Inheritance
The Priority Update commandprovides a cost effective mechanismby which to bound the time that a message may be
blocked due to priority inversion. This section incorporates the ability to guarantee a worst case boundfor blocking time
with an algorithm which maybe used to a priori guarantee ring schedulability. An algorithm is then presented for
determining the worst case blocking time for a given message. Finally, Example5 is extended to illustrate that in some
cases timing correctness maybe guaranteed in the presence of priority inversion through the use of a priority inheritance
mechanism.
2.4.1 Bounded Blocking Time Theorem
Previous work by Sha, Rajkumar, and Lehoczky[9] includes the worst case blocking time in the scheduling equation in
order to compensatefor priority inversion. The worst case blocking time, Bi, is the maximum
amountof time that a task
must wait on lower priority tasks to completetheir execution. This alters Theorem1 and producesthe following.
Theorem
2: A set of n periodic tasks using the priority ceiling protocol can be scheduled by the rate monotonic
algorithmif the followingconditions are satisfied:
~¢i, l <_i<_n,
C1
C
2
C
B
¯ .. + ~..i +i -- < i(21/~1).
Remark:The first i terms in the above inequality constitute the effect of preemptions from all higher priority
messagesand messagei’s ownexecution time, while Bi of the last term represents the worst case blocking time
due to all lowerpriority tasks.
14
Theorem2 indicates that ring scheduling maybe guaranteed if the worst case blocking time is knownand all possible
inequalities from Theorem2 are satisfied. Next we present an algorithm for determining the worst case blocking time for
any message.
2.4.2 An Algorithm for Determining Worst Case Blocking Time
The BoundedBlocking TimeTheoremrequires that the worst case blocking time Bi be knownfor every message. The worst
case blocking time consists of the blocking time due to both messages within and outside of the node from which that
messageis transmitted. The following algorithm maybe used to determine the worst case blocking time for any message
with priority I’RI whenpacketization13 is not present in the system.
The worst case blocking time of a messageis comprised of two components:
¯ the worst blocking time due to lower priority messagesat that node.
¯ the worst blocking time due to lower priority messagesat other nodes.
The worst blocking time due to lower priority messagesat that messagesownnode is calculated in the following manner:
1. Order the messagesin the TransmitBuffer to force the worst case blocking time. Place the lowest priority messagesat
the front of the node, and place the next highest priority messagesdirectly after them. In this mannerthe highest
priority messageswill be at the rear of the TransmitBuffer.
2. The worst blocking time for any messageis the amountof time required to signal onto the ring all messagesin front of
that messagein the TransmitBuffer.
The worst blocking time due to lower priority messagesat other nodes is calculated by considering every other station in the
network:
1. Onceagain order the messagesin each node’s Transmit Buffer so as to force the worst case blocking time for each
message.
2. Identify all other stations which have at least one messagewith priority greater than PRI. For those stations the
contribution to the worst case Mockingtime is the sumof the ring time of all messageswith priority less than PRI.
3. Identify all other stations whosehighest priority messageis equal to PRI. The worst case blocking due to that station is
the ling time of all messageswith priority less than PRI.
Whenlarge messagesare present in a system, they are often divided into several smaller messagesin order to provide a
higher degree of preemptabflity of the network. This process is called packetization. Without packefizafion, large lowpriority messagescan cause long blocking times. Howeverwhenpackefization is present in the system, care must be taken
when evaluating the worst case blocking time for use in Theorem 2. Packetization requires that packet pacing
[11] considerations be taken into account. Section 2.5 discusses the impact of packet pacing in detail.
Thus far the concept of priority inheritance and the Priority Update command
have been introduced. A scheduling model
has been introduced which compensatesfor blocking time due to priority inversion, and an algorithm for determining worst
case blocking time has been developed. In order to demonstrate the scheduling advantage of using a priority inheritance
mechanismwe now incorporate these ideas and extend previous examples such that each station has the capability to
perform the Priority Update command.
13packetization is the process by whichlarge messagesare divided into several smaller messagesand transmitted across the network. This is done to
provide a higher degree of preemptionin the system. Later we will examinethe effects of packetization.
15
2.4.3 Periodic Messages
There are instances where a priority inheritance mechanismguarantees the schedulability of periodic task sets which may
otherwise not be scheduledwhenpriority inversion is present. Consider the following example:
Example4: Figure 2-8 shows the task set from Example1 with the MediaContention Priority at each node updated via a
priority inheritance mechanism.Messagea is the only message which exists at the same node with messages of a lower
priority, and thus only messagea maybe blocked due to priority inversion. The Transmit Buffer at NodeA in Figure 2-8
corresponds to the worst case blocking time for messagea. This worst case blocking time consists of the transmission time
for the lowerpriority messagesb, c, and d (for a total of 15 time units). Notice that messagesj and k mayalso be transmitted
before messagea. Howeverthis time is not considered part of the worst case blocking time for messagea, since messagesj
and k also havea priority of 3.
Nowthat the worst case blocking time for all messageshas been established, we proceed to insert the appropriate values into
Theorem2 to determine if timing correctness can be guaranteed. Since message a is the only message which can be
blocked, we will insert the correspondingworst case blocking time into the most stringent inequality possible~4 accordingto
Theorem2.
C
Cj
C
B
ma+ _ + __k + a__ < 3(21/3-1).
-a
rarjrkT
where
Ba
Cb Cc
Cd
-_-. ~ar__.t.~.._
"~aaZaZaa
Inserting the value of 5 for all C and 50 for Ta, Tj, and Tic satisfies the aboveinequality. Therefore timing correctness is
guaranteed and all periodic messagedeadlines will be met. Suppose all periodic messagesfrom Example1 arrive nearly
instantaneously at time t = 0 in the worst case arrival sequence indicated in Figure 2-8. With a priority inheritance
mechanismpresent via the Priority Updatecommand,NodeA contends for token ring access with priority 3 even though the
messageat the front of the Transmit Buffer has priority 2. This allows messagea to meet its deadline according to the
timeline at the bottomof Figure 2-8. Therefore if the requirements of Theorem2 axe satisfied, it is possible that timing
correcmessfor a set of periodic messagesmaybe guaranteed whena priority inheritance mechanismis present to counteract
the effects of blockingdue to priority inversion.
2.4.4 Aperiodic Messages
Example2 illustrates that priority inversion is capable of destroying the high degree of responsiveness whichalgorithmic
scheduling provides for alert class messages. Using a priority inheritance mechanism,someof the high degree of responsiveness may be salvaged. Wenowaugment Example2 such that each node mayperform a Priority Update command.
Example5: Recall that priority 3 messages have a period of 50 and priority 2 messages a period of 100. Once again
supposethat all messagesarrive at nearly the sameinstant at time t=0. The messageordering in the TransmitBuffer at node
C in Figure 2-9 corresponds to the worst case response time for alert class messagen. Nowthat a priority inheritance
mechanism
exists, node C contends for access to the ring with priority 6 because the alert class messagen is blocked by the
14Note that n inequalities are possible according to Theorem2. Thoughnot shownhere, it can be proven that messageordering does not influence
whetheror not a single inequality maybe found which is not satisfied. Wetherefore proceed by defining the most stringent inequality as the one which
includes the greatest quantity on the left side of the inequality. This includes the utilizations for all priority 3 messages(a, j, and k), and the blockingtime
for messagea.
16
Priority
/
0
25
eadline met
50
Figure 2-8: In SomeInstances Priority Inheritance
Guarantees Timing Correctness
lower priority periodic messagese, f, and g. NodeC wins access to the ring and the messagesat node C are transmitted.
Priority inheritance is able to prevent priority inversion from seriously degradingthe high degree of responsivenessof alert
class aperiodic messages.Priority inheritance salvages someof the responsivenessand transmits messagen in 20 time units,
whereasmessagen required 55 time units for transmission whena priority inheritance mechanismwas not present.
Section 2.2 introduced the concept of priority inheritance and developed the Priority Update command
to counteract the
effects of priority inversion on ring schedulability. Section 2.3 then gave examplesof howthe priority update command
maybe used to both schedule a task set which is not schedulable whenpriority inversion is not present, and to salvage the
response time of alert class and mediumurgency messageswhenpriority inversion is present. The next section examines
the limitations of priority inheritance.
2.5 Limitations of Priority Inheritance
Somemessagesets are schedulable whenpriority inversion exists. Howeverin general one cannot rely on priority inheritance to counteract priority inversion. There are numerouscases where, with priority inversion, priority inheritance
cannot guarantee the schedulability of a set of messageswhichsatisfy Theorem1. This section explores the limitations of
the priority inheritance technique.
Remember
that only a fraction of all possible periodic messagesets maybe scheduledif priority inversion exists. Toolarge
a worst case blocking time violates the constraints of Theorem2 and renders the messageset unschedulable.
There are instances whereadditional constraints must be taken into consideration in order to insure the schedulability of
messagesets in whichpriority inheritance is used to counteract the effects of priority inversion. This extra consideration is
17
Figure 2-9: Priority Inheritance Salvages Alert
Class MessageResponsiveness
termed packet pacing [11], and must be taken into consideration when packefization is used anywherein the system.
Packefization is whenlarge messagesare divided into multiple smaller messages(called packets) and transmitted accordingly. Packet pacing is concerned with at which point in time individual packets from multi-packet messagesshould be
submitted for transmission.
Example6: The task set in this example consists of four messages from two nodes. At node A messages a and b each
require 1 messageto be transmitted every 4 time units, and messagec requires that a messagewhich requires 10 units for
transmission and delivery be signalled every 100 time units. At node B, messaged requires 1 messagebe transmitted every
10 time units. Messagesa, b, and d require 1 time unit for transmission and delivery. Messagec is too large to be
transmitted in a single messageand is therefore divided into 10 individual packets, each of whichrequires 1 time unit for
transmission and delivery. These packets are then submitted for transmission in succession. For messagec to meet its
deadline, 10 packets must be delivered every 100 time units. Only one messagec packet maybe queuedfor transmission at
any instant.
Accordingto the rate monotonicalgorithm, messagesa and b are assigned the highest priority, messagec the next highest,
and messaged the lowest. Supposethat messagesa, b, c, and d are assigned priorities 3, 3, I, and 2, respectively. Messages
a, b and d are capable of being blocked by the lower priority message c. Since only one packet from messagec maybe
queuedat any instant, one might be tempted to assumethat the maximum
blocking time for messagesa, b or d is 1 time unit.
If we insert the value 1 for the worst case blocking time Bi for messagesa, b, and d, then the messageset satisfies Theorem
2.
18
)1 1 1 1_< 3(21/3_1
C
Cb Cd +B
T +’~b+’-T-~d
dmTd< - 3(21/3-1).
__aa
Howeverthe critical section analysis at the bottomof Figure 2-10 showsthat there is an instance wheremessaged misses its
deadline. In this scenario messagesa, c, and d arrive for transmission at time t = 0 in the order indicated in the Transmit
FIFOs. Since messagea is blocked behind the lower priority messagec, node A contends for the token ring with priority 3.
Node A wins contention and transmits a packet from messagec and then message a. While messagea is being transmitted
another packet from message c is queued for transmission. Before message c is transmitted, message b is queued for
transmission, and therefore node A contends for the ring using the priority of messageb. This interleaving continues such
that every time a packet from messagec is at the front of the queue, it is blockingone of the higher priority messages(a or
b). NodeA contends for the ring at priority 3 while node B contends for ring access at priority 2. Therefore node A utilizes
100%of ring capacity until all packets from messageb have been transmitted, causing messaged to miss its deadline.
Messaged could have easily met its deadline had the packets from message c been spaced in order to allow message d
access to the ring. If only 1 packet from messagec were submitted for transmission every 10 time units, all higher priority
messageswould meet their deadline. In summary,the blocking time for messaged is actually 10 time units (the total
transmission time for messagec) whenpacket pacing is not used. The messageset does not satisfy Theorem2 if we use the
value Bi = 10. Howeverusing packet pacing, the maximum
blocking time Bi is 1 time unit, and Theorem2 is satisfied. In
summary,packet pacing must be taken into consideration when messages are divided into smaller packets. This adds
software scheduling overhead complexity to the token ring adapter operating system interface. An ideal IEEE 802.5
Standard implementationwouldeliminate this consideration.
Media
Access
Priority
o
,¢,alOlblCla,
tC
IlblC[a[I IlblCllallCllbll
I I
0
5
10
15
IlalClbl I
20
Figure 2-10: Packetization Requires Packet
Pacing Considerations
2.6 An Idealized
IEEE 802.5 Implementation
Whenimplementing a Token Ring Chipset or a Network Adapter Card, system designers must incorporate a number of
requirements in order to realize the performancepotential of the IEEE802.5 TokenRing Standard. These requirements fall
under the area of concurrency control and buffer management.In general, whenthere is concurrency, preemptiveprioritybased arbitration should be used. Wherethere is queueing, priority queues, as opposed to FIFOqueues, should be used.
Failure to adhere to these basic guidelines can result in the loss of predictability and serious degradationin the responsetime
19
performance. In the following paragraphs, we provide general implementation guidelines for IEEE802.5 implementations
along with specific recommendationsfor implementations using Texas Instruments TMS380
Adapter Chipset.
2.~i.1 Concurrency Control and Buffer Management
In real-time systems, it is important that consistent priority-based concurrencymanagement
be used betweenand within each
layer of the protocol stack. Ideally the priority granularity, resource sharing, and the degree of preemptability should be
uniformup and downthe protocol stack. In reality, there are significant variations up and downthe protocol stack with tt-~e
higher layers in the protocol stack typically havinga greater degree of priority discriminationthan the lower levels. Further,
the degree ofpreemptability can vary radically. As an implementor,it is important to minimizethese differences so as not to
introduce undesirable artifacts into the communications
subsystem.
Whena process accesses a queue of pending messages, the pending message with the highest priority should be serviced
first. Otherwise system schedulability and responsiveness are compromisedby priority inversions. Priority inversion can
occur at any level within the protocol stack. Queuesin the higher layers of the protocol stack are typically software queues,
which are flexible and easy to implement.Thus existing FIFOqueues at these layers are easily modified to tree Priority
Queues15. Queuesat lower layers require higher operating speeds. Thus lower layers, including the ISOReference Model’s
Data Link Layer16, often use less-flexible but faster hardwarequeues. Here we are concernedwith priority inversion caused
by the hardware implementations using FIFO queueing. If FIFO queues must be used, a priority inheritance mechanism
should be used to minimizeperformancedegradations. Priority inheritance methodsreduce the blocking time and expedite
the service of high priority tasks.
Concurrencymanagementguidelines are summarizedas follows:
* Whenconcurrency is supported within a protocol layer, priority-based arbitration should be used to insure
threads are serviced in priority order. At least three bits of priority are required, and eight bits of priority should
be provided wherever possible. Ties between equal priorities can be broken arbitrarily. However,a userprogrammable
tie-breaking facility is desirable.
o Priority granularity should be consistent from the bottom to the top of the protocol stack. Whennot possible,
resources should be provided to support mappingpriorities between protocol layers. These mappingfunctions
should be programmableby the network managementfunction.
o At all times, peer-to-peer priority consistency and visibility should be maintained. For example,the priority at
the transmit side shouldbe visible at the receive side.
Whenthere are queues within and between the protocol layers, they should be priority queues wherever
possible. Aneight deep queueis sufficient for all but the most rigorous applications. If a FIFOqueue must be
used, it should not be morethan 2 deep and should support priority inheritance.
To bound blocking durations and maintain responsiveness, the maximum
length of block transfers and frames
should be programmableunder the control of higher level network management
functions.
2.6.2 An Ideal Implementation
The Priority Update commandprovides an acceptable start, however priority inheritance results with a loss in
schedulability and performanceas well as complicating the ring schedulability analysis. In order to avoid these losses and
realize the full potential of the real-time communication
scheduling algorithms as demonstratedin the simulation studies, a
better implementationis needed. Priority Queues, whichallocate a separate FIFOQueueto each priority level, provide a
better implementation. Consider the following example.
Example 7: As indicated in Figure 2-11, Exanaple 7 has the same task set as Example 1. As before suppose that all
15priority Queuesoffer the task with the highest priority for execution, regardless of the arrival time of other pendingtasks.
16TheData Link Layer is usually implemented
on the networkadaptercard.
20
messagesarrive at nearly the same instant at time t--0. Howeverat each node the Token Ring Adapters utilize Priority
Queues for the Transmit Buffers: Priority Queues are composed of multiple FIFO queues, where one FIFO queue is
allocated to each priority level for both the transmission and reception of frames. Whenqueuedfor transmission, a message
is inserted corresponding to its priority. Messageswhich have the same priority and which are queuedfrom the samenode
are transmitted in the order of their arrival. Thefimeline at the bottomof Figure 2-11 indicates that with priority queuesall
periodic messagesmeet their deadline regardless of the order in whichthey were queued.
Receive
1
2 1 0
Receive
Transmit
2
0
:5
2
0
Transmit
5
:5
Transmit
2 t 0
1 0
Transmit
Receive
5 2
0
:5
2 1 0
Receive
EP,~es Priority Queues
o Figure 2-11: 2~deal Implementauon
Furthermore, with priority queues the guaranteed response times of high urgency aperiodic messagesare preserved, as are
the improved response times of mediumurgency messages. Priority queues prevent blocking, and therefore aperiodic
messagesresponsetime is not artificially increased.
At the receive end, a major concernis to transfer messagesfrom adapter to host memoryin the proper order. Higherpriority
messagesshould be transferred before lower priority pendingmessages,regardless of the arrival order of those messages.If
the device is slow, then multiple received frames mayreside in the Receive Buffer simultaneously. Whenmultiple messages
reside in the Receive Buffer the highest priority message should be transferred to host memoryfirst. In the current
implementation, the Receive Buffer is implementedas a FIFOQueuesuch that messagesare transferred to host memoryin
the order of receipt, regardless their priority. Since messagesare competingfor transfer from a single FIFOQueue, the
priority inversion problemat the receive end is distinctly different from that at the transmit end17. As a result, priority
inheritance mechanismsmayhave no effect on the priority inversion problem at the receive end. Using a priority inheritance mechanismat the receive end would have no impact at all on the order in which messagesare transferred from
adapter to host memoryuponreceipt. Thus the only way to circumventthe priority inversion problemat the receive end is
to use Priority Queuesto implementthe Receive Buffers. Ralya [10] demonstrated that the overhead associated with the
management
of priority queuesdoes not cause noticeable performancelosses if architectural support is provided.
17At the transmit
end, messages from multiple
F~O queues contend for transmission
onto a commonmedia.
21
2.7 Summary
The Texas Instruments TMS380Adapter Chipset is a commercially available implementation of the IEEE 802.5 Standard.
Unfortunately the TMS380
Chipset has a numberof features which artificially limit the performance potential of this
realization of the IEEE802.5 Standard, including artificially limiting priority resolution and introducing priority inversion.
For the former, the chipset provides only 4 priority levels in lieu of the 8 levels specified in the IEEE802.5 Standard.
This work focusses primarily upon the latter problem of priority inversions introduced due to improper concurrency and
buffer managementpolicies. The problem arises because the Transmit Buffers in the Adapter Chipset RAM
are managedas
a FIFOQueue. Frames are DMA’ed
into the Transmit Buffers before being signalled onto the ring. Nodescontend for ring
access using the priority of the frame at the head of their Transmit FIFOQueue. Whena higher priority frame enters the
TransmitFIFOand finds itself behind a lower priority frame, it is desirable to be able to suspendtransmission of the lower
priority frame(s) in favor of the higher priority frames. Unfortunately the TMS380Chipset does not feature a Frame
Suspend capability.
Instead a more expensive Transmit Halt commandis provided. Wheninvoked, this command
destroys the contents of Transmit FIFOQueueand terminates the Transmit Process. That station must re-issue the transmit
command
before it can transmit subsequent frames. Alternatively, the signalling station could use the priority of the higher
priority frame when contending for media access for the lower priority frames18. This wouldbound the blocking time of
high priority messages. Unfortunately the TMS380
Chipset does not support a Priority Inheritance mechanismeither. The
absence of these capabilities degrades both performanceand analyzability.
To provide full support for real-time communication,priority queuesshould be used to implementboth the TransmitBuffers
and ReceiveBuffers. Priority queues prevent priority inversion, whichguarantees the schedulability of periodic task sets
(according to Theorem1), the guaranteed response times of alert class messages, and the high responsiveness of medium
class aperiodic messages.
lSHere the frame of lower priority inherits the higher priority for media contention purposes only. The receiving node uses the original frame priority
whenprocessing the frame.
22
3. End to End Time
3.1 Components of End to End Time
In this section we characterize the componentsof Media Access (MAC)Layer to MAC
Layer transmission time19. Wewill
refer to this time to as end to end transmission time. First we investigate the componentswhichcomprise messagedelivery
time, and someof the factors whichaffect this time. Next we report the results of messagetransmission time as observedon
the ARTStestbed.
Endto end defivery time maybe divided into 3 separate times:
¯ TransmitTime:The time to required for the transmitting node to sigual the node onto the ring.
¯ propagation Time: The time required for the frame to propagate from the source node to destination node.
¯ ReceiveTime:The time required for the receiving node to receive the frame.
Recall the token ring adapter configuration illustrated in Figure 2-1. Messagesoriginate in host memory,and are then
transferred via DMA
to the Transmit Buffer residing in Token Ring Adapter memory.The Adapter Chipset then contends
for mediaaccess, and signals the messageonto the ring uponcapturing a free token. The frame propagates around the ring
and is copied into the Receive Buffer of the receiving node. There the flame is subsequently transferred to host memory
and
processed at the higher layers of the ISO Reference Model. The following paragraphs more distinctly characterize these
actions.
device driver overhead.
DMAtime
TMS380software
overhead
signal propogation time
TMS380software
overhead
DMAtime
device driver
overhead.
Figure 3-1: MACLayer to MACLayer Delivery Time
3.1.1 Transmit Time
In Figure 3-1, TransmitTimeis comprisedof the time necessary to fulfill all necessary actions to signal the messageonto
the ring. Beginningat the Data Link Layer, there are 3 basic steps whichmust be taken to activate messagetransmission.
First the applications software must makea call to the operating system software which communicateswith the TokenRing
Adapter Chipset. This software is commonlyreferred to as a device driver. The device driver readies the message for
transmission and notifies the Adapter Chipset that a frame is pendingtransmission. If the TransmitBuffer is not full, the
Adapter Chipset supervises the DMA
transfer of that message from host memoryto adapter memory.The Adapter Chipset
then contends for access to the ring, and signals the messageonto the ring uponcapturing a flee token.
19In terms of the ISO Reference Model [ 14], this is the time to for a message to travel from the Data Link Layer of the transmitting
Layer of the receiving node.
node to the Data Link
23
3.1.2 propagation Time
The propagationtime is the time the electric signal takes to propagate completelyaroundthe ring. Eachmessageis signalled
onto the ring by the transmitting node, copied into the ReceiveBuffers at the destination node, and stripped from the ring by
the transmitting node.
3.1.3 Receive Time
After a messageis copied into the ReceiveBuffer at the destination node, the TMS380
Chipset supervises the transfer of the
message (via DMA)
from adapter memoryto host memory.Next the adapter chipset notifies the host that a message has
been received. The device driver software then passes this information along for processing at the upper protocol layers.
3.2 Factor Which Influence
End to End Time
The componentswhich determine the amountof time that each of the above actions requires may be determined according
to the physical attributes of the ring. Theproblemis that manyof these actions overlapin time, and it is therefore difficult to
accurately predict end to end delivery time. [1]. Even moreimportant than the amountof time required for messagedelivery
are the variables which affect that time. There are a numberof parameters which are established by the system software
whichaffect messagedelivery time. This section examinesthose factors.
3.2.0.1 Physical Constraints
The electrical signal perambulatesthe ring at a finite speed. As a general rule, the signal propagation speed for wire is
approximatelyone-half the speed of light. Therefore the propagationtime is a function of the distance the signal must travel
(i.e. the length of the ring). Furthermoreeach node through whichthe signal travels causes a 4-bit delay, and therefore
propagationtime is also dependentupon the numberof stations on the ring.
3.2.0.2 Host Adapter DMAInterface
The system bus is a shared medium,just as the token ring. Each messagemust traverse three shared media en route from
source node to destination node. First the messagemust gain access to the host system bus and be transferred from host to
adapter memory.After transmission over the token ring, the message must be transferred across the destination nodes’
system bus en route to host memory.The work presented here examinesthe scheduling of only the token ring. Howeverin
order to guarantee system-wide schedulability, the DMA
scheduling domainmust be examined.
The TMS380Chipset a/lows two modes of host DMA
operation: burst modeand cycle-steal mode. During a burst mode
block DMA
transfer, the adapter arbilrates for control of the host system’s memorybus and then retains control until an
entire block transfer is completed. In cycle-steal mode,the adapter arbitrates for the system bus on each individual DMA
transfer, releasing the bus whenthe transfer is complete. The arbitration sequenceis the samefor both types of transfers,
howeverthe burst modetransfer movesa larger amountof information per arbitration sequence. The Proteon p1542is not
capable of attaining burst modeDMA
transfer on the VMEBus [18], and DMA
transfers are performed in cycle-steal
2°.
mode
The disadvantageto burst modetransfer is that it increases the latency of other devices whichare trying to obtain the system
bus. This can cause both tasks in the processor domainand messagesin the communications
domainto miss their deadlines.
If burst modeis used to DMA
transfer a frame betweenhost and adapter memory,a higher priority task running on the CPU
might be blocked from accessing memory.Consider the following example. Suppose that the token ring adapter gains
control of the system bus, and initiates a burst modeDMA
transfer of a 1 Kbyte low priority messagefrom host to adapter
memory.Immediatelyafter beginning this burst modetransfer, the CPUrequests use of the bus so that it mayread one byte
of memoryto allow it to continue processing a high priority task in the processor domain. Howeverthe CPUcannot gain
access to the system bus until the token ring adapter has relinquished control, and therefore the high priority task in the
processor domainmisses its deadline. Alternatively, suppose a 64 byte high priority periodic messagewas received by the
2°The response time of the TMS380
Chipset is too slow to attain burst modeDMA
transfers.
adapter immediatelyafter the low priority messagehad entered into burst modeDMA
transfer. The higher priority message
misses its deadline because it is delayedfrom accessing the ring by the lowerpriority message.
In order to guarantee timing correctness in both the processor and communications
domain,access to the system bus (for use
in DMA
transfers) must be granted in a deterministic mariner. Sprunt, Kirk, and Rajkumar[13] have proposed a prioritybased D/VIAarchitecture which reflect the boundsthe time that a device must wait to gain access to the system bus. Until
the issue of systembus schedulability is resolved, guaranteedend to end response times cannot ge guaranteed.
3.2.0.3 Internal AdapterBuffer Size
Adapter memoryis partitioned into multiple internal adapter buffers. The size of each buffer is configured by the device
driver. If burst modeDMA
transfer is used, the internal adapter buffer size places an upper boundon the amountof data that
can be transferred between host memoryand adapter memoryin one DMA
burst modetransfer. In this respect, performance
is typically enhancedwhenfewer buffers are required to hold a frame. Furthermore, the TMS380
Adapter Chipset requires
additional overheadfor each internal adapter buffer used to transmit or receive a message.The moreinternal adapter buffers
used, the longer the execution time required by the Adapter Chipset to process the frame. The relationship betweeninternal
adapter buffer size and ring throughputis well characterized and documented
in [1].
3.2.0.4 Numberof Allocated Internal Adapter Buffers
The internal adapter buffers are divided betweenthe transmit and receive processes. A certain numberare allocated for use
in frame transmission, and the remainder are used to receive frames. The TMS380
Adapter Chipset is capable of pipelined
operation. If a frame is contained in multiple buffers, then the data from messagei+1 maybe transferred from host to
adapter while messagei is being transmitted onto the ring. Howeverfor this pipelining to occur, enoughinternal buffers
must be allocated to the transmit process to hold two messages. This relationship between the numberof internal adapter
buffers and ring throughputis also well documented
in [1].
3.3 Time Measurements
Measurementshave been made on the ARTStestbed. These measurements, listed in Appendix A, accurately reflect the
influences mentioned above.
25
4. Conclusions
The major contribution of this work has been to discover and documentthe flaws in current generation implementationsof
the IEEE 802.5 Token Ring Standard which prevent those implementations from successfully supporting real-time communication. This work identified and demonstrated howthese flaws degrade the potential of the IEEE 802.5 Standard for
real-6me communication,and determined the impact of these flaws on ring schedulability. Documentationfrom this work
has been presented to engineers whoare responsible for their companiesnext generation implementationsof the IEEE802.5
TokenRing implementation, including Jim Carlo and HowardRubin of Texas Instruments, and Ken Christiansen of IBM.
Finally this work has identified the scheduling of the system bus as a system-wide aspect which must be resolved before
messagedelivery times can be guaranteed, allowing for a real-time communication
protocol.
26
References
1. Alexander, C. PerformanceCharacteristics of the TMS380
TokenRing LANAdapter. Texas Instnunents, Inc. Internal
Document.
2. Dailey, G.E. Distributed Systems Data Bus, External Audit - PDR,03J/04273/03. IBMFSD,Manassas, Virginia, May,
1983.
3. Hood, P. and Grover, V. Designing Real TimeSystems in Ada. Tech. Rept. 1123-1, SofTech, Inc. 460 Totten Pond
Road, Waltham,MA,8 January, 1986.
4. SAE-9B.SAEAE9-BHigh Speed Data Bus Standard. Tech. Rept. Issue 1, Draft 2, Society of AutomotiveEngineers,
Subcommittee9-B, January 21, 1986.
5. IEEE Standards for Local Area Networks. Token Ring Access Method, ANSI/IEEEStandard 1S0 Draft Proposal. The
Institute of Electrical and Electronics Engineers, Inc., 345 East 47th Street, NewYork, NY10017, 1986.
6. Le Lann, G. Real-Time Communications Systems. Real-Time Communications Workshop, NASA-JSC,Houston Texas,
January 21-22, 1988.
7. Lehoczlcy, J.P. and Sha, L. "Performance of Real-TimeBus Scheduling Algorithms". ACMPerformanceEvaluation
Review, Special Issue 14, 1 (May1986).
8. Liu, C.L. and Layland, J.W. "Scheduling Algorithms for Multiprogrammingin a Hard Real-TimeEnvironment". JACM
20 (1) (1973), 46 - 61.
9. Sha, L., Rajkumar,R., and Lehoczky,J.P. "Priority Inheritance Protocols, An Approachto Real-TimeSynchronization".
To Appear in IEEE Transactions on Computers(1989-90).
10. Ralya, T. Real-TimeOperating SystemArchitecture .. Workstepsand Related Subjects. Proc. 5th Workshopon
Real-TimeSoftware and Operating Systems, May,1988.
11. Ralya, Tom. Internal Technical Memo,Software Engineering Institute,
Pittsburgh, PA. December,1988.
12. Rothauser, E.H. and Wild, D. "MLMA-A
Collision-Free Multi-Access Method". Proc.lFIP Congress77 (1977).
13. Sprnnt, H.M.B., Kirk, D. and Rajkumar, R. A DMA
Architecture for Real-TimeSystems. Departmentof Electrical &
ComputerEngineering, Pittsburgh, PA.
14. Stallings,
William. Data and ComputerCommunications.MacmillanPub. Co., 1988.
15. Strosnider, J.K., Marchok,T.E. and Lehoczky, J.P. "AdvancedReal-TimeScheduling Using the IEEE 802.5 Token
Ring". Proceedings of lEEE Real-Time Systems Symposium(1988).
16. Strosnider, Jay K. Highly Responsive Real-Time Token Rings. Ph.D. Th., Carnegie MellonUniversity, August1988.
17. Strosnider, J.K. and MarchokT.E. "Responsive, Deterministic IEEE802.5 Token Ring Scheduling"° Real-Time
Systems Journal (September 1989).
18. Ludemarm,J. User’s Guide to the SUN~3VMEBus.1985. Sun Microsystems, Inc. Mountain View, CA.
19. Tannenbaum,AndrewS. ComputerNetworks. Prentice-Hall,
1981.
20.. TMS380Adapter
Chipset User’s Guide. Texas Instruments, Inc, 1986.
21. Tokuda, H., Kotera, M. A Real-TimeTool Set for the ARTSKernel. Proceedings of 9th IEEEReal-Time Systems
Symposium.
22. Tokuda, H. and Mercer, C. W. ARTS:A Distributed Real-Time Kernel. To appear in ACMOperating Systems
Review.
23. Tokuda, H., Mercer, C.W. and MarchokT.E. TowardsPredictable Real-Time Communication.Proc. 6th IEEE
Workshopon Real-Time Software and Operating Systems, May, 1989.
24. Tokuda, H., Mercer, C.W., Ishikawa, Y. and MarchokT.E. Priority Inversions in Real-Time Commtmication.ACM
SIGCOMM,1989.
27
25. Zhao W., RamamithamK. "A Vi~tual Time CSMA
Protocol for Hard Real Time Communications". IEEE Real-Time
Systems Symposium(1986).