1072
IEEE TRANSACTIONS ON C O M P U T E R S , V O L . C-34,
NO.
12,
DECEMBER
1985
A Distributed VLSI Architecture for Efficient Signal
and Data Processing
JEAN-LUC GAUDIOT, MEMBER, IEEE, REX W. VEDDER, GEORGE K. TUCKER,
DENNIS FINN, AND MICHAEL L. CAMPBELL
Abstract — Future signal and data processing applications will
require billions of operations per second, and yet low hardware
and software development costs. Architectural improvements in
the form of multiprocessors must be used in order to reach these
high performance levels. Von Neumann models cannot easily
implement concurrent operations and data-flow principles are
one alternative for sequencing instructions in a parallel environ­
ment. The machine described here, the Hughes Data-Flow
Multiprocessor (HDFM), is a high-performance, scalable,
fault-tolerant, highly programmable multicomputer designed
for embedded signal and data processing applications. The ar­
chitecture of the machine is described in detail, and the influence
on the final design of various requirements such as weight, size,
power consumption, performance level, and reliability are
shown. The processing elements have been designed so as to re­
duce the number of VLSI component types required and for
modularity of the physical system. The modular nature of the
architecture allows a range of throughput and reliability require­
ments to be met. The model of execution derived from original
data-flow principles is presented as well as the different soft­
ware tools which give the system its high-level language programmability (compiler, allocator, etc.). Complex constructs
(such as large structure handling) are demonstrated. Finally,
the results of a deterministic simulation of the machine show
that a 64 processing element machine may provide real through­
put of 64 million instructions per second (MIPS).
Index Terms—Allocation, asynchronous execution, data-flow
multiprocessor, distributed computing, multiprocessor architec­
ture, signal and data processor.
I.
INTRODUCTION
T
HE Hughes Data-Flow Multiprocessor (HDFM) project
was prompted by the need for high-performance, reli­
able, and easily programmable processors for embedded sys­
tems. V L S I now allows a manyfold increase in the amount of
circuitry that can fit within the small volume and power
limitations of embedded processors. This has led to the in­
creased use of parallel processors to achieve high perfor­
mance. However, synchronization of independent processors
must usually be explicitly specified by the programmer. The
programmer is faced with the highly complex task of par­
titioning and scheduling the various processes among the
Manuscript received May 1, 1985; revised August 20, 1985. This work was
supported in part by the Hughes Aircraft Company under the Data-Flow Pro­
cessor Task of the Advanced Digital VLSI Architectures Internal Research and
Development Project C274H2A2 and in part by the National Science Founda­
tion under Grant ECS-8404345.
J. -L. Gaudiot is with the Department of Electrical Engineering-Systems,
University of Southern California, Los Angeles, CA 90089-0781.
R.W. Vedder, G. K. Tucker, D. Finn, and Μ. L. Campbell are with the
VLSI Systems Department, Electro-Optical and Data Systems Group, Hughes
Aircraft Company, El Segundo, CA 90245.
processors. The goal of this architecture has been to develop
a high-performance multiprocessor system which is pro­
grammed in a high-level language as would be a single com­
puter, while the multiprocessor organization is transparent to
the user. In addition to enabling fast mapping of the applica­
tion programs for various configurations, this approach al­
lows easy fault recovery and is useful for meeting changing
real-time performance requirements.
The architecture described in this paper answers both the
performance requirements and the hardware and software
cost issues. The programming environment allows applica­
tion coding in a functional high-level language called the
Hughes Data-Flow Language (HDFL). The HDFL is com­
piled into a data-flow graph, which is then automatically
partitioned and distributed to the various processing elements
(PE's). The PE's are interconnected by a three-dimensional
grid of buses and are organized in a cube topology. All com­
munications are based on the message passing model. Design
of the PE's has been made for easy implementation in VLSI.
Additional PE's can easily be added due to the modular nature
of the communication network. Due to the underlying data­
flow principles of execution, a distributed processing en­
vironment has been created with no central controller or
dispatcher. Simulation results presented in this paper have
demonstrated high-performance operation with high-level
language programmability.
The HDFM architecture was designed using the following
guidelines.
1) U s e c u s t o m H u g h e s V H S I C (Very H i g h S p e e d
Integrated Circuitry) 1.25 μπι CMOS/SOS chips for high
throughput, low power consumption, and reduced size and
weight.
2) Statically reconfigure around failed PE's for high
reliability.
3) Use hundreds of PE's for high throughput.
4) Use a functional high-level language for high pro­
grammability and compile-time parallelism detection.
5) Design for implementation in 1985-1987 timeframe.
6) Use homogeneous PE's for low cost.
7) Use static program allocation for lower run-time sched­
uling overhead.
8) Use a micro data-flow architecture to take advantage of
low-level inherent parallelism for high throughput.
9) Use a new cubed-bus packet-switching network com­
munication scheme optimized for small packet size (high
throughput and high reliability).
Section II (Model of Execution) describes the data-flow
principles of execution and the programming environment.
0018-9340/85/1200-1072$01.00
© 1985 IEEE
1073
GAUDIOT et al.: A DISTRIBUTED VLSI ARCHITECTURE
Section III (Architecture) presents the details of the hardware
design. Advanced constructs such as data structures are ex­
plained in Section IV (Array Handling). The methods for
partitioning the program among the processors of the ma­
chine are presented in Section V (Allocation). Simulation
results are shown and interpreted in Section VI (Simulation).
Finally, concluding remarks are drawn in Section VII (Con­
clusions and Future Research).
II.
MODEL OF EXECUTION
A. Elementary Data-Flow
Principles
As formally laid out by Dennis [8], the data-flow princi­
ples of execution do away with the two crucial notions of
von Neumann languages: the program counter and the central
memory system. This is accomplished by sequencing the
instructions upon the availability of the operands themselves
(no program counter to decide on the readiness of an opera­
tion for execution). Also, memory cells do not exist at the
lowest level because results must be passed directly between
operations. In summary, a data-flow program can be repre­
sented as a graph where instructions are nodes which commu­
nicate results to other instructions over arcs. Executability of an
operation is decided by the following rule.
"A data-flow actor can fire when it has data tokens on
all its input arcs. It then produces result token(s) on its
own output arc(s)."
Note that the execution is purely functional
in that the
result is exclusively a function of the input operands. No
notion of the state of the machine is added. This implies that
control can be fully distributed through the various PE's of
the machine.
Β. Acknowledgment
Scheme—Instruction
Enabling
The principles described above ensure that sequencing of
the instructions will be made on a data-driven mode. This
form of control can obviously be efficiently distributed over
many processors. However, the underlying idea behind these
principles is the concept of single assignment (this means
that identifiers can only be assigned once throughout the life
of the program). In other words, there cannot be any vari­
ables but only values which can only exist or not exist. Only
one value may transit on the arcs between instructions of a
loop. This implies that additional precautions must be taken
if the tokens are to be taken in the right order. This is accom­
plished by enforcing the following additional rule.
"An instruction is declared executable when it has
tokens on all its input arcs and there are no tokens on the
output arcs."
This extra provision ensures that there can be no more than
one token on any arc at any one moment. This can be imple­
mented by imposing any actor to acknowledge
the con­
sumption of the input tokens to the parent actors. After the
execution of the instruction, result tokens are produced on the
desired arcs of the actor, and acknowledge tokens are sent
back to the originating actors. For each data arc there is a dual
acknowledge arc in the opposite direction. As remarked by
Gajski et al. [11], this shows that communication through the
machine is doubled and that the bandwidth of the commu­
nication network must be able to support at least twice the
rate of communication of "true" d a t a — 1 0 0 percent over­
head. Another implication is that the "parent" template must
be known either by fixing connections at program load time
or by further increased communication overhead.
1) Acknowledgment
Arc Removal: The acknowledgment
scheme restricts the number of tokens on any arc to only one.
This is a direct consequence of the underlying hardware
scheme, which does not allow more than one token packet in
each instruction template. The system of acknowledgment
arcs ensures that no pileup can occur. However, this is a
redundant process: in some cases, the program constructs are
such that, if the presence of only one token on one arc of the
graph is guaranteed, no more than one token will ever be
present on one specific other arc of the graph. This would be
the case (again, depending on the program construct), regard­
less of whether the second is "protected" by an acknowl­
edgment arc or not.
Indeed, Brock et al. [4] have demonstrated several meth­
ods for suppressing acknowledgment arcs when their removal
does not introduce unsafe constructs (i.e., graphs which may
create accumulation of tokens on one arc). The compiler
creates a fully acknowledged data-flow graph from highlevel language input. The reduction algorithm is then applied
to the resulting graph. Acknowledgment arcs are taken away
following some simple rules.
2) Software Acknowledgments:
In the conventional ac­
knowledgment interpreter, once an actor has fired, special
Boolean tokens are sent to the parent actors. These Boolean
tokens are treated just like regular input operands by the
parent actors, but the values are ignored. Indeed their pres­
ence only serves to decrement the token count to the actor
until the counter reaches zero, thereby indicating the actor is
ready to fire again. For a dyadic operation (2 input operands),
which dispatches its result(s) to 3 destinations, the initial
token count is thus set to 5.
We perceive for our project three fundamental problems in
implementing the above approach. First, some actors pro­
duce a variable number of output tokens, depending on input
data. Second, hardware considerations restrict the maximum
token count to 2. Finally, some constructs such as function
invocation utilize templates capable of receiving input from
various sources, with no way of selecting the proper parent
actor to acknowledge. The solution is software acknowl­
edgment arcs, which will now be described.
Acknowledgment
blocks are inserted into the data-flow
graph, each including several templates. On the input arcs to
each block, ENABLE actors are inserted (Fig. 1). The other
input of each ENABLE actor is generated when all the de­
pendent ENABLE actors guarding the next block have fired.
A binary tree structure as shown, a linear chain, or the
COUNT actor described later can be used to determine when
all have fired. As the ENABLES fire they allow a new "wave"
of data into the block and propagate the acknowledgment
signal backward to the previous block. The compiler initially
generates small blocks, just large enough to enclose code
1074
IEEE TRANSACTIONS O N C O M P U T E R S , V O L . C-34,
RESULTS
DECEMBER
1985
arcs may even belong to different categories. For instance, in
a SWITCH actor, the result will be sent to only one of two
output arcs, depending on the value of the Boolean operand.
This example would correspond to two lists of result point­
ers. Note again that there is no implementation-defined limit
on the number of pointers in any list.
INCOMING VALUES
Fig. 1.
N O . 12,
READY
An acknowledge block.
which does not follow normal rules. Optimization will use
defaults and programmer directives specifying performance
requirements to merge small blocks into larger blocks when­
ever feasible. In the Dennis scheme, succeeding iterations in
a loop are allowed to be pipelined (program level pipelining).
Instruction / belonging to iteration J may be executed simul­
taneously with instruction / + 2 of iteration J - 1. This is
allowed because instruction / + 1 of iteration J — 1 has
already sent its acknowledgment to instruction / , thereby
rendering it usable for the next "wave." This is an important
aspect of the parallelism displayed in such a data-flow ma­
chine and must not be discarded.
Conceptually, the acknowledgment scheme is respected
with the acknowledgment blocks; only the size of a pipeline
stage has been modified. When excessive acknowledgment
overhead prevents pipelining, we can duplicate code to pro­
duce identical blocks capable of running in parallel on differ­
ent sets of data using different sets of processors.
C. Instruction Set
The elementary actor set of the Hughes Data-Flow Multi­
processor is geared toward numerical processing. In addi­
tion, noncomputational actors must be included for program
control purposes. The most essential features of the basic
actors are now introduced before a detailed description.
The basic instruction set of the Hughes data-flow machine
can be divided into three categories: categories I and II in­
clude actors which are purely functional (i.e., actors which
compute results regardless of their own history). Category III
comprises actors with some state information.
Actors of category I and II are "conventional" and imple­
ment the various basic functions of arithmetic and logical
operations. These include addition, subtraction, multi­
plication, division, etc. The actors of category III implement
a deviation from the pure functional model of computation.
Some amount of "state" information is associated with each
actor. This renders them history sensitive, as the results pro­
duced are not only a function of the input arguments but also
of the previous arguments that have traversed the actor.
By initial hardware considerations, the actors are confined
to a maximum fan-in (number of input arguments) of 2. This
precludes the use of more complex functions such as triadic
operators, etc. Fan-out is, however, unlimited, and output
1) Simple Actors: Category I actors include arithmetic
and logical operations which respect the functional principles
of execution. Since the machine is designed to operate either
in single-precision mode (16 bits) or double-precision mode
(32 bits), each of these actors exists in two incarnations
(1 opcode for each type: single or double precision). ADD,
ENABLE, SWITCH, FORWARD, and LE5, all shown in
Fig. 2, exhibit the range of complexity attainable in type I
actors.
Category II actors include some amount of state informa­
tion, which is readable only by the actor itself. This indicates
that, from the outside, their mode of operation is purely
functional since the "state," together with the opcode, can be
considered the characteristics of the actor to which the input
operands are applied. No modification internal to the actor
can occur as a result of its firing. At the hardware level,
different processing is required: when a category I actor is
declared fireable by application of data availability, it is in­
deed readily executable. On the other hand, a category II
instruction packet first needs to access some memory where
it finds its "state information" before the operation can be
executed. The details of these mechanisms will be explained
in more detail in the next section. An example of a type II
actor is NOTIFY, identical to FORWARD except the state
word is added to the incoming address to form the destination
address.
2) History Sensitive Actors — State Information:
In their
basic inception, category III actors are data-flow operators
which obey the same sequencing rules as any other actor
(execution upon availability of the input tokens). However,
their action is not limited to the production of output tokens,
which depend upon the input set. Instead, a category III actor
is associated with one or several state words. Computation of
the result tokens is made on a combination of the current state
and of the input arguments. Also, new state information is
then calculated and stored for the next incarnation of the
actor.
While this mode of operation may appear to fundamentally
violate the data-flow assumptions, it can be easily justified:
conceptually, the altered "state" words are equivalent to to­
kens, which are emitted by one invocation of the template and
immediately loop back to distinct operands of the same tem­
plate. In this view, functionality is preserved. At the hard­
ware level our alternate implementation is required because
the number of actual operands is limited to two, while the
additional "state" words allow "back door" inputs if needed.
An example type III actor is the C O U N T actor. It main­
tains a state word which is the current count of how many
tokens have arrived on its input arc. It also has a state word
which indicates its terminal count. When a token arrives,
these two state words are fetched, the current count is in­
cremented, and the updated count is compared to the terminal
1075
GAUDIOT et al.: A DISTRIBUTED VLSI ARCHITECTURE
ENABLE
ADD
(hIGhTeVEL
L A N G U A G E ^
IF I = 2 T H E N
Χ + 5
ELSE Χ + 10
VI
V2
W
COMPILER
τ
ENABLE
^^DATA
R««—
FLOW G R A P H ^ )
D2
D1
D 1 * - V 1
V1 + V2
D A T A FLOW
PROCESSOR
DESCRIPTION
FILE
D2«*-V2
SWITCH
F O R W A R D
t
t
ALLOCATOR
L
V
•^ALLOCATED DATA^N
V
F L O W G R A P K I ^ /
(^^^ITCH^^
T /
F O R W A R D
D A T A FLOW
PROCESSOR
* F
IF Ρ T H E N T ^ — V
V I SS E N T T O A D D R E S S
E L S E F « « — V
f
Ί
ACK
A
ACK««—V
LE5
V1
IF V 1 < = V 2
THEN
V2
The software environment.
B«*-TRUE,
TV*-V1,
T2«*-V2
ELSE
Fig. 3.
B ^ - F A L S E ,
F2«*-V2
VTV-.
Fig. 2.
Variations on type I actors.
F1*~V1,
T1
ci
T2
count. Upon a match, a control token is generated by the
count actor. The count actor can be used for loop control.
D. HLL Programming
Environment
The HDFM is programmed in HDFL, which is a functional
high-level language based on VAL (Ackerman [1] and
McGraw [19]). A compiler which translates from HDFL to a
parallel data-flow graph form has been developed. This graph
is then distributed to the PE's of the multiprocessor by a
software tool called the allocator. The allocator employs
static graph analysis to produce a compile-time assignment of
the program graph to hardware that attempts to maximize the
number of operations which can proceed in parallel while
minimizing the inter-PE communication. The software envi­
ronment for the HDFM is shown in Fig. 3 . Parallel devel­
opment of high-level language, compiler, and architecture
has allowed many hardware/software tradeoffs to be studied,
resulting in a design which can be efficiently implemented.
1) The Hughes Data-Flow Language: One of the primary
goals of our project was to provide a high-level language
capability for a multiprocessor system in order to reduce
software cost. This entailed choosing a high-level language
which could easily express the parallelism inherent in many
problems. Current sequential languages like Fortran and Pas­
cal were eliminated because of their inherent sequentiality.
Ada and other multitasking languages were rejected for they
require explicit programmer concern with the creation and
synchronization of multiple tasks. This would add complex­
ity and cost to software development. Within a specific pro­
cess, these languages are also subject to the same lack of
parallelism as the Fortran-class languages. It was determined
that an applicative data-flow language such as VAL (McGraw
[19]) or Id (Arvind [2]) was needed in order to allow an
effective extraction of parallelism and efficient mapping onto
multiprocessor hardware. This led to the development of the
HDFL.
HDFL is a general-purpose high-level programming lan­
guage for data-flow computers which is designed to allow
full detection of parallelism. It is an applicative language
(Backus [3]), but includes the usage of familiar algebraic
notation and programming language conventions. HDFL
borrowed many of its features from VAL, and now differs
from the newest version of VAL in details.
The language is value oriented, allowing only singleassignment variables. Its features include strong typing,
data structures including records and arrays, conditionals
(IF T H E N E L S E ) , iterations ( F O R ) , parallel iterations
(FORALL), and iteration over elements of streams and/or
arrays (FOREACH). An HDFL program consists of a pro­
gram definition and zero or more function definitions.
There are no global variables or side-effects; values are ex­
changed via parameter passing.
Fig. 4 shows a simple example of HDFL which illustrates
the flavor of the language. The example consists of a function
"foo," which takes four parameters (one record and three
integers) and returns one record and one integer. "Result" is
a keyword beginning the body of a function, and "endfun"
terminates it. The function body consists of a list of arbi-
1076
IEEE TRANSACTIONS O N C O M P U T E R S , V O L . C-34,
type
xy
= record [ x,
function foo(
returns
constant
constant
y:
integer];
Υ1
N O . 12,
DECEMBER
1985
Y2
xyvar: xy; x0,yl,y2:
integer
x y , i n t e g e r ) %Two r e s u l t s
offset = 1;
scale = 2;
result
if yl > y2
%two r e s u l t s
produced
then xyvar, y2*scale + o f f s e t
e l s e x y ( x O , y l ) , y l + xO
endif
endfun
Fig. 4.
An HDFL program.
trarily complex expressions separated by commas, with one
expression per return value. In this example, the first expres­
sion in the function body is a record type constructor,"
which assigns values to the fields of the record result. The
conditional below it evaluates to an integer value. Constants
and types may be declared before the function header or
before the body. Functions may be nested.
2) HLL Compiler: The compiler translates HDFL into a
data-flow graph intermediate form composed of primitive
data-flow actors. Operation proceeds in three phases — syn­
tax checking and parse tree construction, semantics checking
and augmentation, and finally, code generation. Each phase
is table driven, using the Hughes Translation Table Generator
(Tucker [24]). Following table-driven code generation is an
optimization/postprocessing stage. The latter consists of
many operations common to traditional code optimizers,
generating executable code for array operations, and mod­
ification of acknowledgment blocks. The graph intermediate
form generated by the compiler includes syntactic informa­
tion and other information which is used by the allocator.
4
For each construct in the high-level language the compiler
has a corresponding data-flow graph composed from the
primitive actors that implements that function. For example,
the data-flow graph generated from the HDFL conditional
expression, "if y l < = y2 then y2 * 2 + 1 else y l + xO
endif" is shown in Fig. 5. The then and else branches of the
conditional are merged together by sending tokens on these
arcs to the same location; this is indicated by merging the
output arcs together. Note also that the LE5 actor has some
stub output arcs which are not used. The ENABLE actor is
present so that when the result of the expression is generated
this guarantees that all actors in the graph have fired and the
graph is available for further use if desired.
3) Allocatof: The allocator is a software tool which takes
the data-flow graph intermediate form from the compiler and
a description of the number and configuration of PE's and
assigns each actor to an actual PE. Note that each PE may
have many actors assigned to it. The allocator attempts to
satisfy three conflicting goals: 1) maximize the parallelism
of execution in the data-flow graph by assigning actors that
can fire in parallel to different PE's; 2) minimize the commu­
nication traffic between PE's by assigning actors that are
connected by arcs to the same PE; and 3) balance the process­
ing load.
The allocator algorithm works in a divide-and-conquer
fashion by partitioning the input graph into smaller modules,
each of which is assigned to a subset of the PE's. The modules
can then be further divided and assigned to smaller sets of
Fig. 5.
A simple data-flow graph.
PE's until the trivial assignment of an actor or group of actors
to one PE is made. At each step, the assignment of modules
to PE's is guided by heuristic functions, which estimate the
communications overhead and parallelism of different distri­
butions of actors so that the best solutions may be selected.
The partitioning algorithm splits up the data-flow graph
along boundaries implied by the syntax of the high-level
language source code. To guide this process, the data-flow
intermediate graph form includes the syntactic parse tree
from the compiler. For example, the first step of the allocator
algorithm is to partition the parse tree into subtrees, which
correspond to high-level language functions. Each of these
subtrees corresponds to a module, which can then be further
partitioned into still smaller modules corresponding to
smaller language constructs such as loop bodies or expres­
sions. Since each construct in our language is side-effect
free, we expect this partitioning to achieve good locality of
reference and therefore help minimize interprocessor com­
munication. The more detailed allocation mechanisms are
described in Section V of this paper.
III.
ARCHITECTURE
The HDFM consists of one to hundreds of identical PE's
connected by a global packet-switching network. This net­
work is a three-dimensional bused-cube network as shown in
Fig. 6. Packet transmission proceeds via a store-and-forward
protocol, which allows any PE to transfer data to any other
PE. Each PE can execute the instruction set and perform the
data-flow sequencing and addressing. Each PE has its own
local memory for both program and data storage; there is no
global memory. The program, which consists of data-flow
actors (templates), is allocated to the local memories of the
PE's at compile time. The PE's are targeted for VLSI imple­
mentation, resulting in a simplified design to reduce I/O pins
and minimize the number of different types of VLSI chips.
The communication network is integrated with each PE to
allow growth of both processing power and extension of the
communication network with the addition of a single modular
unit. Each PE consists of a communication chip (COM), a
processing chip (PROC), and memory as shown in Fig. 7.
1077
GAUDIOT et al.'. A DISTRIBUTED VLSI ARCHITECTURE
ROWS
WORD 1
PLANE
2
9
PE ADDRESS
PLANE ί COL i R O W
TYPE
COLUMNS
6
9
TYPE
PE ADDRESS
PLANE j COL j R O W
TEMPLATE ADDRESS
WORD 3
DATA
TEMPLATE ADDRESS
17
1
PLANE
2
/
I/O CONTROLLERS
Fig. 6.
Fig. 8.
2
17
17
WORD 2
PLANE
I/O
PLANE
32 BIT DATA PACKET
16 BIT DATA PACKET
6
17
DATA (LS)
Two types of data packets.
The cubic bus interconnection network.
COLUMN BUS
PLANE
COLUMN
ROW
t
ι
t
ROW B U S
Γ
I
PLIQ
PLOQ
COIQ
COOQ
ROIQ
ROOQ
I
C O M M U N I C A T I O N S CHIP
• PACKET ROUTING
A N D FAULT T O L E R A N C E
C O M (1 V L S I )
PRIQ
|_COM
DESTINATION
MEMORY
(DM)
MULTIPLE
RAM CHIPS
P R O C E S S I N G CHIP
• O P E R A N D FETCH
• DATA FLOW SEQUENCING
• INSTRUCTION EXECUTION
• SEND OUT RESULTS
P R O C (1 V L S I )
Fig. 7.
A. Communication
TEMPLATE
MEMORY
(TM)
MULTIPLE
RAM CHIPS
A processing element.
Network
Instruction level data flow requires a flexible highbandwidth communication network which can operate effi­
ciently for small packets (tokens) consisting of only a single
data item. In addition, the high-reliability requirements for
the intended class of applications require that the commu­
nication network be highly fault tolerant. Thus, the HDFM
interconnection network is optimized for small packet trans­
fer and allows reconfiguration around faulty PE's.
As previously shown, the PE's are arranged in a threedimensional cube. Each PE has a unique 9 bit address, corre­
sponding to its position in the cube, which consists of a 3 bit
plane field, a 3 bit column field, and a 3 bit row field. This
allows up to 8 PE's per bus for a maximum configuration of
512 (8X8X8) PE's.
Within each PE are multiple templates (actors), which are
accessed by a template address. To implement the data-flow
model the results of one template (actor) are sent to another
in the form of packets. Each packet consists of a type field,
an address, and data. The type field indicates the type of
packet, for example, a token packet or a program load packet.
The address indicates where the packet should be sent and
consists of a PE address and a template address. The data
field contains one or more words of 17 bit data. The actual
data paths of the interconnection network are 17 bits wide so
that packets are broken into multiple words as shown in
Fig. 8. Variable length packets are supported by the network
protocol with the last word of a packet transmission indicated
by an end-of-packet bit.
PLIQ
PLOQ
ROIQ
ROOQ
COIQ
COOQ
PLANE INPUT QUEUE
PLANE OUTPUT QUEUE
R O W INPUT QUEUE
R O W OUTPUT QUEUE
COLLUMN INPUT QUEUE
C O L U M N OUTPUT QUEUE
Fig. 9.
Τ
PROQ
J
TO PROC CHIP
PRIQ
P R O C E S S O R INPUT QUEUE
PROQ
P R O C E S S O R OUTPUT QUEUE
ERROR M E M O R Y
The COM chip.
In the bused-cube interconnection network not all PE's are
directly connected, so a store-and-forward packet-routing
technique is used. This algorithm is implemented in the com­
munications chip (COM) (Fig. 9). The COM connects a pro­
cessing chip (PROC) to the row, column, and plane bus. The
chip acts like a crossbar in that it takes a packet from any of
its four ports and routes it in the proper direction, providing
buffering as needed, with a number of FIFO queues. Packets
traveling from one PE to another always take the same path.
This principle, called single-path routing, is required to pre­
serve the order of tokens in time-ordered data groups such as
streams.
Each PE continually monitors its plane, column, and row
buses to look for packets it should accept. PE's accept pack­
ets addressed directly to themselves and also accept packets
which need to be rerouted to other P E ' s . For example, if a
packet is put on a plane bus, then all PE's on that bus examine
the packet address, and the PE whose plane address matches
the packet's plane address accepts the packet.
The communication network is designed to be reliable,
with automatic retry on garbled messages, distributed bus
arbitration, alternate path packet routing, and failed PE trans­
lation tables to allow rapid switch-in and use of spare PE's.
Static fault tolerance is fully supported. Upon determination
of a PE failure, a spare PE can be loaded with the failed PE's
templates, and the operation can be continued. This creates
two problems. One, the spare PE has a different address than
the PE it replaced, and two, messages which were to be
routed through the failed PE must be routed around the failed
PE.
The first problem is solved by two methods. In the long
1078
IEEE TRANSACTIONS ON C O M P U T E R S , V O L . C-34,
MtCROMACHINES
C O M : C O M M U N I C A T I O N S / F A U L T TOLERANCE
T M C : TEMPLATE M E M O R Y CONTROLLED
ALU. ALU/MICROPROCESSOR A N D MICROMEMORY
DT: DESTINATION TAGGER
D M C : DESTINATION M E M O R Y CONTROLLER
MEMORIES
D M : DESTINATION M E M O R Y
TM: TEMPLATE M E M O R Y
F D M : FIRE D E T E C T M E M O R Y
N O . 12,
DECEMBER
1985
QUEUES
FCQ: FROM C O M M U N I C A T I O N QUEUE
FRQ: FEEDBACK QUEUE
FU FIRING QUEUE
RQ: RESULT QUEUE
DQ: D t S T I N A T I O N QUEUE
AIQ: A S S O C I A T E D INFORMATION QUEUE
I C Q TO C O M M U N I C A T I O N Q U E U E
Fig. 10. The PROC chip.
term (days to months), the application program can be real­
located using the allocator software tool during a scheduled
maintenance period. In this case the application program is
remapped to the new set of PE's excluding the failed PE(s).
For immediate recovery (several seconds), a small number of
failed PE address translation registers, called the error
memory, are provided in each PE. Upon failure, the address
of the failed PE is entered in the error memory of each PE
followed by the address of its replacement. Each generated
packet is checked against the error memory, and if a match is
made, then the replacement address is substituted for the
failed address — this allows operation without reallocation,
although it may slightly degrade performance because the
template allocation has been altered.
The second problem is that packets must be routed around
failed PE's. This is accomplished by having each PE keep
track of which directly connected PE's are operating and
which have failed. The routing table in each PE is updated
whenever the processor is reconfigured. In the case of failed
PE's, the sending PE routes packets to alternate buses.
Dynamic fault tolerance can be provided by running two or
more copies of critical code sections in parallel in different
PE's and voting on the results. In the HDFM it is extremely
simple to implement a voter; it is simply one or more equal
( = ) actors. Unlike difficulties encountered in other parallel
processors, the data-flow concept avoids synchronization
problems by its construction, and interprocess commu­
nications overhead is minimized because it is supported in
hardware. This software approach to dynamic fault tolerance
minimizes the extra hardware required for this feature.
B. Processing
Elements
Each PE consists of two parts — the above-mentioned
communications functionality (COM) and a processing en­
gine (PROC), which performs the primitive actor operations
and data-flow sequencing control. The COM receives pack­
ets from the routing network and either forwards them on to
other PE's or sends them to its attached PROC. When the
PROC receives a packet, it checks if this packet (token) has
enabled a template to fire. If so, the operands and opcode for
this template are sent to the ALU. The ALU will perform the
indicated operation and send the result to be matched with its
destination address and sent either back to this same PE or out
into the routing network.
The PROC is a three-stage pipelined processor with the
following three o p e r a t i o n s o v e r l a p p e d : 1) instruction/
operand fetch and data-flow firing rule check; 2) instruction
execution; and 3) matching results with destination ad­
dresses to form packets. Because there is some variance in
the service times of each of these units for different instruc­
tions, queueing is provided between the units, as is shown in
Fig. 10.
The instruction fetch and data-flow firing rule check is
performed by two parallel units, the template memory con­
troller (TMC) and the destination memory controller (DMC).
The templates are spread across three independent memories:
the template memory (TM), the destination memory (DM),
and the fire detect memory (part of the T M C ) , as is shown in
Fig. 11, to allow high-speed access of templates and data.
When a packet arrives at the PROC, the status of the template
to which the packet is addressed is accessed from the fire
detect memory and a decision on whether the template is
ready to fire is made. The status bits are stored in the on-chip
fire detect memory to allow fast access and update of tem­
plate status. If the template is not ready to fire, then the
arriving token is stored in the T M .
If the template is ready to fire, then the TMC fetches the
template opcode and operands and sends this information to
the firing queue (FQ), from which the ALU will execute the
operation. Simultaneously, the DMC begins fetching the re­
sult destinations to which the template result should be sent
and stores these in the destination queue. Since a template
result may need to be sent to multiple destinations, there is
additional destination overflow storage in the DM to accom­
modate lists of destinations for a single actor.
The results of the actor execution performed in the ALU
are put into the result queue (RQ). The results in the RQ and
the destinations in the destination queue are combined to-
1079
GAUDIOT et al.: A DISTRIBUTED VLSI ARCHITECTURE
TEMPLATE M E M O R Y
OPCODE
6
FIRE
DETECT
MEMORY
DESTINATION
MEMORY
|
OPERAND
STATUS
RESULT DESTINATION
j
17
4
32
I
1
1
TEMPLATE
STORE
1
!
DISTRIBUTED
ARRAYS
TEMPLATE
D E S T I N A T I O N LIST
OVERFLOW
RESULT QUEUE OVERFLOW
the structure could signify the recopying of the entire array.
This clearly imposes an inordinate overhead. Instead, several
methods of representing data structures have been proposed
(Dennis [8], Arvind and Thomas [27], Gaudiot et al. [12],
McGraw and Skedzielewski [20], Catto et al. [7], Caluwaerts et al. [5], and Gaudiot [13], [14]). These methods are
overgeneral and assume the presence of large amounts of
additional hardware. The constraints in an on-board pro­
cessor do not allow the inclusion of specialized arrayhandling resources. This is why a special kind of arrays was
chosen for this development.
LOCAL
ARRAYS
A. Array-Handling
Fig. 11.
Mapping templates and arrays in memory.
gether into packets by the destination tagger and sent back to
the TMC or to other P E ' s .
The HDFM is designed to handle both confined (to one PE)
and distributed (across the machine) arrays. Local arrays are
stored entirely in one PE, can be accessed only from that PE,
and are stored in the D M . Local arrays can be accessed
directly by the ALU providing fast array access. The pro­
grammer can specify, or the allocator will designate an array
as either confined or distributed. Each type of array uses a
distinct set of array access actors.
Distributed arrays are arrays which are located in one or
more PE's and accessible from any PE. These arrays are
stored in the TM and are accessed by using special array
packet types — fetch and update packets. For example, to
fetch an array item, an array fetch packet is sent to the array
location (PE and specific address of array item). The array
fetch packet contains the return destination address (RDA) to
which the array value is sent once it is fetched from the TM.
The PE which receives the fetch packet forms a packet with
the RDA as its destination address and the array value as its
data, and sends this packet to the communication network.
A stand-alone PE is capable of 2 - 4 MIPS processing per­
formance, depending on the instruction mix used. In this
case, a MIPS is defined as a primitive actor instruction; these
vary in complexity from a simple 16 bit add which is com­
pleted in one cycle (microinstruction) to some array address­
ing instructions which take over 10 cycles or a 16 bit divide
which requires approximately 25 cycles. The design goal of
minimizing chip types and power consumption resulted in a
simple ALU design, and multiplication is performed by a
modified Booth's algorithm technique. Each of the two VLSI
chips has less than 150 pins, consists of approximately 15K
gates, and operates at a 20 MHz clock rate. Preliminary floor
plans and cell layouts for important macro cells have been
completed for both chips using the Hughes VHSIC 1.25 μιτι
CMOS/SOS technology.
IV.
ARRAY HANDLING
It is widely known that handling arrays is the cause of
many problems in functional computing environments. Re­
call that functional principles imply that no modification of
data can occur; only new values can be created. In turn, this
means that when data need to be structured, as they would in
an array or a matrix, modification of the whole cannot be
envisioned economically, for the updating of a single item of
Scheme
In his doctoral dissertation, Mowbray [22] has described a
scheme for handling arrays in a VAL high-level environment.
It implicitly remarks that data-flow arrays as currently con­
ceived by researchers in the field are overly "asynchronous,"
i.e., do not take sufficient advantage of the data dependence
information carried by the program graph; while / structures
and M.I.T. trees correspond to one end of the spectrum, von
Neumann representation of arrays would represent the op­
posite in the lack of data dependence implied by their imple­
mentation. In his proposal, arrays would be allocated in a
sequential fashion, as they are in a conventional von Neu­
mann environment. Note, however, that some additional
information must be associated with the array in order to
indicate its dimensionality and its distribution across the
processors of the machine. SELECT actors can readily be
implemented as they consist of a simple READ operation.
Safety and correct execution of WRITE operations are a
compile-time task. Upon encountering an APPEND opera­
tion, the compiler may create one of these constructs:
1) array update if the original array is not used later;
2) runtime test if the new array is synthesized only for a
test; and
3) copy operation if both the old and new values are
required.
The advantages of the scheme are severalfold.
1) Reduction of the number of memory accesses compared
to the heaps as proposed by Dennis [8]. In fact, since the
system of pointers has now been replaced in favor of a se­
quential organization, any read or write access does not have
any inherent overhead.
2) Better possibility of distribution of an array structure
across the various units of the machine. This is also an imme­
diate consequence of the suppression of the pointer mecha­
nism as the distribution information may be made more
implicit (i.e., the index number of an element will uniquely
identify its allocated unit).
These undeniable improvements come at the price of the
following.
1) Possible induction of spurious data dependences, or
loss of runtime parallelism. This is due to the fact that the
compiler is not necessarily aware of the possibility of paral­
lelism that can be detected only at runtime. For instance, the
compiler may impose dependences on two arrays A and Β
that are related by A(F(i)) = B(i). Clearly, individual de­
pendences on atomic elements of A and Β will be the result
of the execution of function F, which cannot be predicted by
the compiler.
1080
IEEE T R A N S A C T I O N S O N C O M P U T E R S , V O L . C-34,
2) Limitation of array operations to atomic elements since
the von Neumann arrays are indeed oriented towards an ele­
mentary level of operation.
Β. Array Implementation
on HDFM
For the purpose of describing array-handling mechanisms,
we are mainly interested in the distribution of the memory
within the PE's. The memory space is chiefly divided into
two parts, which can be accessed by different units of the
PE's as shown in Fig. 12.
1) The TM can be directly accessed by the T M C . It stores
operation templates, i.e., the opcode of the operation, as well
as up to one operand of the instruction. Also, it is used for the
storage of some arrays. The TM can also be under control of
the destination tagger under certain conditions which will be
explained later.
2) The DM contains the second part of the instruction
template: it is the storage for the destination pointers of the
various actors. Like the TM, it is also the repository for
another category of arrays.
At the low-language level, there are two kinds of arrays.
1) Confined arrays are meant to be accessible only within
one PE by rudimentary instructions that need only know of
their starting address. They are stored in the DM for direct
accessing by instructions in process in the ALU.
2) Global arrays are also sometimes referred to as distrib­
uted arrays. They are allocated to several PE's in the ma­
chine. The form of distribution is stored in a special word
called the array description code (ADC) that accompanies
the identifier of the array when the latter is sent as an operand
to an array accessing operation. These structures are located
in the TM of each PE. The mode of accessing was described
in the second section of this paper. Note that the main differ­
ence between the two kinds of arrays is in the distribution:
while confined arrays are located in only one PE and can be
accessed merely with the knowledge of their base address,
distributed arrays are spread across many processors; ac­
cessing them requires the knowledge not only of their base
address, but also of the distribution mode (contained in
the ADC word).
Note that in addition to the definitions made above, the
access to a distributed array may be local when the element
searched is in the processor where the issuing instruction is
allocated. Conversely, the access would be remote when the
element is located in a different processor. However, an ac­
cess to a confined array will always be local.
C. Accessing
Arrays
Confined arrays are stored in the DM. When a confined
array instruction is processed by the ALU, the element within
the array is reached directly by the ALU. These arrays will be
accessed by only two instructions.
1) GETL is a "read" instruction. It requires the base ad­
dress of the array and the index of the elerrient to read (its
relative position within the array). The result of the operation
is the element read from the array.
2) PUTL is a "write" instruction. It requires the base ad­
dress of the array, the index of the element, and the value to
(Dennis [8], Arvind and Thomas [27], Gaudiot et al. [12],
McGraw and Skedzielewski [20], Catto et al. [7], Calu-
12,
-I
COM
DEST I N A T I O N
MEMORY
DESTINATION
STATE
NO.
1985
TEMPLATE
MEMORY
TEMPLATE
MEMORY
C O N T R O L L E R (TMC)
WORDS
DECEMBER
CONFINED
TEMPLATE
STORAGE
GLOBAL
ARRAY
STORAGE
ARRAY
STORAGE
DESTINATION
TAGGER
Fig. 12.
FIRING
QUEUE
Array memory organization.
store in its place. It only produces acknowledgments to indi­
cate the completion of the operation.
Distributed arrays are identified by both their base address
and their ADC and cannot be accessed by simple PUTL and
GETL operations. In addition, distributed arrays are stored in
the TM as opposed to the DM for confined arrays, and thus
are not easily accessed by the ALU. More sophisticated ac­
tors have been designed for reading and writing purposes.
We know that accesses to global arrays are of two kinds.
1) Local accesses are detected once an array access opera­
tion has been decoded in the ALU. The determination of the
locality of an access is a function of the array, its distribution,
and the index of the requested element. This calculation,
called effective address calculation, is described later. For
performance improvement, it has been decided that the ALU
would be allowed to directly access the TM if a local access
to a distributed array were to be made. In this mode, the ALU
would disable the TMC and "steal" a TM cycle from it.
2) Remote accesses are needed when it is determined by
the effective address calculation mechanism that the re­
quested array element is in a different PE from the one where
the array operation is being executed.
In the case of a SELECT operation, a fetch packet is sent
to the PE that contains the desired element. When this latter
PE receives the packet in question, the TMC performs the
access and sends t h e required data through t h e ALU/
destination tagger (ALU/DT). The data is sent to the address
that accompanied the fetch packet. This address was the RDA
input of the originating SELECT. It was passed on by the
TMC to the ALU/DT. This is schematically represented in
Fig. 13.
In the case of an APPEND, the mechanism is similar: a
"store packet" is sent to the TMC that is local to the array.
This packet contains the address of the location to be ac­
cessed, as well as the data that must be written in. The TMC
performs the store and processes the next packet. Note that in
this scheme, the acknowledgment from the APPEND, if any,
must be sent by the A P P E N D , immediately after it was pro­
cessed, i . e . , after the "store packet" has been sent to the
remote PE. This is not a guarantee that the actual store has
been performed, as this will happen only later in that TMC.
1081
GAUDIOT et al.: A DISTRIBUTED VLSI ARCHITECTURE
I N D E X
A D C
scheme is that single-dimension arrays can be allocated only
linearly.
V. ALLOCATION
F E T C H
P A C K E T
R D A
1
I
I
, ' R E A D
A R R A Y
ITEM
1
^
'
7
I
I
τ
A R R A Y
ITEM
R D A
N O R M A L
T O K E N
P A C K E T
Fig. 13. SELECT operation mechanism for distributed arrays.
This is why the third state word in the state of the APPEND
actors contains an address to which an acknowledgment must
be sent. The ack address is incorporated in a fetch packet that
is directed to the very same array element that was just up­
dated by the previous "store packet." In other words, this
second packet actually performs a fetch to the array and sends
the updated value to the acknowledgment address. Note that
the second fetch is not explicitly necessary: the two packets
cannot be processed out of order since there is only one data
path in the PE and single-path routing among the PE's is
enforced. Note in addition that this acknowledgment function
is optional.
D. Multidimensional
Arrays
Multidimensional arrays can be directly handled at the
lowest language level. Arrays are assumed to have a fixed
dimension, known at compile time. Array bounds checking is
not a lower language function and must be performed, if
needed, by program constructs provided by the compiler.
Multidimensional arrays are usually considered at the lower
level to be distributed arrays since confined arrays would
require linearization of the indexes. In the mapped array
scheme, we first note that most arrays that must be consid­
ered within the framework of this system will be uni-, bi-, or
tridimensional. These kinds of structures lend themselves
very naturally to a mapping onto our cubic architecture. In
other words, a three-dimensional matrix will be mapped with
its first dimension along the rows of the multiprocessor, its
second dimension along its columns, and its third dimension
along its planes. Higher-dimensional arrays are treated as
arrays of three dimensional arrays. A disadvantage of this
The assignment of actors to PE's can have a large impact
on the performance of the multiprocessor. For example, since
each PE is a sequential computer, actors that potentially can
fire in parallel cannot do so if they are assigned to the same
PE. Performance can also be affected by data communication
delays in the inter-PE communication network. It takes many
more clock cycles to transmit a token from one PE to another
than it does to transmit a token from one actor to another in
the same PE, which bypasses the communication network
completely.
In order to achieve the allocation goal mentioned in
Section II-D3) the allocation algorithms have been imple­
mented in close cooperation with the development of the
HDFM architecture and high-level language compiler. A
simple prototype was used to test the original approach. It
accepts a data-flow graph from the data-flow assembler pro­
gram and a description of the configuration of the data-flow
multiprocessor to be allocated onto, specifying how many
PE's there are in each dimension.
The local allocator assigns the actors of the graph one at a
time to the PE's of the machine, guided by several heuristics
to be described below. The output specifies the mapping of
each actor of the graph to a memory location in one of the
PE's of the machine. This file can then be read into the
architectural simulator to initialize the simulated machine.
To detect parallelism, the local allocator uses a transitive
closure computation applied to the data flow graph. Two
actors can fire in parallel unless there is a directed path from
one to the other in the graph. Thus, two actors can fire in
parallel unless they are related in the transitive closure of the
graph. This fact is used to determine which actors should be
assigned to separate PE's in order to maximize the paral­
lelism of the allocated graph.
The local allocator has been applied to several data-flow
graphs coded for embedded applications. As reported earlier,
good results were obtained with this simple prototype; how­
ever, it becomes impractical for graphs with more than about
400 actors because it requires over 3 CPU hours on a VAX.
Most of this time is spent computing the transitive closure of
the graph. Therefore, we began the implementation of a full
scale version, called the global allocator, which avoids this
bottleneck by using a divide-and-conquer strategy.
The global allocator combines the heuristic approach from
the local allocator with a divide-and-conquer strategy, en­
abling it to operate on large graphs. Like the local allocator
it accepts a data-flow graph and the dimensions of the ma­
chine. It also accepts a hierarchical representation of the
syntactic parse tree from the first pass of the compiler. This
syntactic information guides the allocator as it attempts to
partition the graph into parallel modules. It allows the allo­
cator to take advantage of the way the high-level language
programmer chose to partition the writing of the program into
functions, subfunctions, and expressions.
Our divide-and-conquer strategy breaks the allocation pro-
1082
cess down into two related tasks: partitioning the input graph
into a set of smaller, more tractable modules, and assigning
each module to a physical resource. This algorithm proceeds
hierarchically — partitioning the graph into several modules
and assigning each module to some portion of the PE's of the
data-flow machine. Then recursively, it further partitions
each module into submodules and assigns each of them to a
set of PE's within that module's previously assigned portion
of the machine. This partition-and-assign process is repeated
until the individual submodules are small enough to be effi­
ciently allocated, actor by actor, to individual PE's using a
version of the local allocator.
The nodes of the parse tree from the HDFL compiler corre­
spond to the syntactic constructs of the program such as
functions, subfunctions, loop bodies, etc. The tree is con­
nected by pointers to the data-flow graph so that the actors of
the graph become the leaves of the tree, as depicted in
Fig. 14. Thus, the set of actors below a given node of the tree
form the module of the data-flow graph that computes the
value of the expression corresponding to that node.
By following these pointers down the tree, the global allo­
cator partitions a module corresponding to an expression into
a set of submodules corresponding to the subexpressions of
the expression. In our functional data-flow language, expres­
sions and functions can generally be computed in parallel
because there are no side-effects. Therefore, most of these
syntactic constructs are ideal candidates for modules in the
partition of the corresponding data-flow graph. This is espe­
cially true at the lower level of the tree, where parallelism at
the individual instruction level is found.
After partitioning a module into a set of submodules, the
global allocator uses the data dependence arcs between actors
to infer the data dependences between the submodules. From
these modules and dependences it constructs a "module
graph," which resembles a data-flow graph. The nodes of this
graph correspond to submodules of the data-flow graph, and
the arcs of this graph indicate the data dependences between
submodules.
The nodes (submodules) of this graph can then be assigned
to groups of PE's of the machine using a variant of the local
allocator. The first step is to compute the transitive closure of
the module graph. In this way we never have to compute the
transitive closure of the entire graph at once, and we avoid the
inefficiency of the local allocator for large graphs.
The assignment of modules and individual actors to PE's is
guided by heuristic functions applied to the candidate PE's.
Currently, we have only two basic heuristics implemented:
communication cost and parallel processing cost. They corre­
spond to our first two goals for efficient allocation: minimize
interprocessor communication and maximize parallelism.
The communication cost measures the traffic through the
network resulting from assigning the given module to the
given PE(s). For example, if the module only receives input
arcs from other modules assigned to the same PE(s), then the
communication cost would be zero. If the module receives
input arcs from modules assigned to other PE's that are on a
separate bus, the communication cost would be much higher.
In general, when two modules are connected, the farther
IEEE TRANSACTIONS O N C O M P U T E R S , V O L . C-34,
Fig. 14.
N O . 12,
DECEMBER
1985
Graph partitioning step.
apart they are allocated, the higher the communication cost.
The allocator uses a distance function to determine how far
apart PE's are in the communication network. For example,
if two PE's are on a common bus, then the distance between
them is one "hop" because a token will have to traverse one
bus to travel from one to the other. The distance between a PE
and itself is zero hops because a token can be transmitted
within a PE bypassing the communication network. By
changing this distance function, we can easily adapt the allo­
cator to different processor interconnection topologies, mak­
ing it quite architecture independent.
The parallel processing cost uses the transitive closure of
the module graph to determine which modules should be
assigned to separate sets of PE's in order to maximize the
parallelism of the allocated graph. It returns a higher cost
when potentially parallel modules are assigned to the same
PE(s).
VI.
SIMULATION
There are two HDFM simulators, an instruction level
simulator, and a hardware level simulator. The instruction
level simulator assumes an infinite number of PE's, a unary
processing time, and zero communication time. It is useful in
determining the general characteristics of data flow pro­
grams,, for example, the average number of actors that may
fire in parallel and the average actor fan-out. The hardware
level simulator simulates the HDFM architecture described in
an earlier section. It is useful in gathering performance statis­
tics on the architecture. Both simulators functionally execute
actual data-flow programs. This paper is concerned primarily
with the hardware level simulator and the statistical results
produced by it for several data-flow programs.
The hardware level simulator was designed so that the
following goals could be met:
1) The effects of different allocation algorithms and archi­
tectural features upon the throughput and latency of data­
flow programs could be measured.
2) Better allocation algorithms and architectures could be
developed.
GAUDIOT et al.:
1083
A DISTRIBUTED VLSI ARCHITECTURE
3) Empirical models could be developed that would pre­
dict from the features of the allocator, architecture, and
graph:
• the optimal number of PE's and their configuration;
and
• throughput, latency, and other architectural measures
of performance.
In designing a simulator it is necessary to trade off the
accuracy of the simulation against the speed of the simu­
lation. The HDFM is capable of over a billion operations per
second. If one chose to simulate such an architecture using a
very accurate low-level simulation, one could easily spend a
year waiting for the simulation results (assuming a 1 MIPS
(e.g., VAX 780) machine was used for the simulation). On
the other hand, if the simulation is too abstract, the validity
of the results might be very low.
In designing the HDFM hardware level simulator a com­
promise was reached. An event-driven multilevel simulation
environment was used so that some objects (e.g., queues)
could be simulated at a high level and other objects (e.g., bus
arbitration logic) could be simulated at a low level. The result
was a simulator that was accurate almost to the cycle yet was
able to simulate small programs (less than 200 actor firings)
within a CPU minute and large programs (tens of thousands
of actor firings) within a few CPU hours.
The simulation language was ADLIB/SABLE as described
by Hill [17], [18].
A. Allocation Algorithms
Used
Several different variations of the local allocator have been
used for the HDFM,
1) Transitive Closure, which uses a modified transitive
closure of the data-flow graph to identify which actors may
potentially fire in parallel and tries to minimize the number
of actors that may fire in parallel and are allocated to the same
PE (tends to distribute the actors to many PE's) and minimize
the communication overhead between actors (tends to distrib­
ute the actors to few PE's).
2) Nontransitive Closure, which attempts to balance the
processing load (tends to distribute the actors to many PE's)
and minimize communication overhead (tends to distribute
the actors to few PE's). This algorithm was developed pri­
marily because computing the transitive closure takes order
of η cubed time where η is the number of actors, which
makes using the transitive closure algorithm impractical for
graphs with more than about 100 actors.
3) Random, which randomly allocates actors to PE's.
Both the Transitive Closure and Nontransitive Closure al­
gorithms have conflicting goals. To reach the best com­
promise between the conflicting goals both algorithms use
heuristic cost functions and allocate each actor to the PE with
the least "cost" where cost : = processing weight x process­
ing latency + communication weight x communication
latency.
In order to estimate the performance of these different
allocation algorithms several data-flow graphs have been
simulated using each of the algorithms. The results of a few
of these simulations are described next.
B. Simulation
Results
The two programs that have been simulated most exten­
sively are real time radar signal processing applications. Both
programs have been tested under a variety of allocation algo­
rithms and PE configurations.
The Radar3na program has 96 actors, 17 constants,
152 a r c s , an a v e r a g e a c t o r A L U e x e c u t i o n t i m e of
7.19 cycles (50 ns cycle time), an average actor fan-out of
1.58 arcs, and a degree of parallelism of 21.14 actor firings
per cycle (the average number of actors that can fire in paral­
lel on the instruction level simulator).
The Radarb program uses a 16-point fast Fourier trans­
form (FFT) with complex arithmetic. It has 415 actors,
71 constants, 615 arcs, an average actor ALU execution time
of 4.92 cycles, an average actor fan-out of 1.56 arcs, and a
degree of parallelism of 80.63 actor firings per cycle.
Both programs were simulated on l x l x l , 2 x l x l ,
2 X 2 X 1, 2 X 3 X 1, 2 x 2 x 2 ,
2 x 2 x 3 ,
3 x 3 x 2 , 3 x 3 x 3 , 4 x 3 x 3 , and 4 x 4 x 4 con­
figurations of P E ' s . R a d a r b was also simulated on an
8 x 4 x 4 configuration. Both of these programs were de­
signed to be continuously processing incoming data. In the
simulations, eight sets of data were used for each program
execution. Each input actor grabbed successive data as soon
as it could fire. Thus, the programs were processing several
sets of input data simultaneously. No explicit pipeline stages
existed, nor were any acknowledgment tokens used to pre­
vent sets of input data from interfering with each other. In­
stead, operand queues were used to guarantee safety (Finn
[10], Montz [21]). All three allocation algorithms were used
for Radar3na, but only the Nontransitive Closure and Ran­
dom algorithms were used for Radarb because the Transitive
Closure algorithm would have consumed excessive CPU
time. All told, over 300 simulations were performed using
these programs.
Figs. 15 and 16 show how throughput varies with respect
to the number of PE's for Radar3na and Radarb using dif­
ferent allocation algorithms. Both Radar3na and Radarb
have significantly better throughput using the nonrandom
allocations than the random allocations. The Transitive Clo­
sure algorithm yields about the same maximum throughput as
the Nontransitive Closure algorithm, but uses fewer PE's
because it is more likely than the Nontransitive Closure algo­
rithm to place two actors into the same PE when they fire
sequentially.
Figs. 17 and 18 show how the percentage of time the ALU
is busy (executing actors) varies with respect to the number
of PE's for Radar3na and Radarb. Notice that the more PE's
that are used, the less time the average ALU is busy. This is
due primarily to the fact that each PE has fewer actors that are
ready to fire at any given time. It is not due to an increase in
packet (token) latency (see Fig. 19). In addition, Fig. 17
shows that the Transitive Closure and Nontransitive Closure
allocation algorithms have similar performance (the portion
of the Nontransitive Closure graph beyond about 20 PE's is
uninteresting because the throughput does not increase when
more than 20 PE's are used).
1084
IEEE T R A N S A C T I O N S O N C O M P U T E R S , V O L . C-34,
NO.
12,
DECEMBER
1985
NON TRANSITIVF Π OSURF M A X
TRANSITIVF
TLOSUPF
AVG
Fig. 15. Throughput of Radar3na
\
%
/
NON TRANSITIVF T I O S U R F A V G
Fig. 17. Percent ALU busy time for Radar3na.
N O N TRANSITIVF Π O S U P F
#PEs
Fig. 16. Throughput of Radarb.
Fig. 19 shows how Figs. 15-18 imply that there is a trade­
off between maximizing throughput and efficiently using
PE's. For configurations with very few P E ' s , the average
ALU is very busy, but the program throughput is significantly
less than the maximum that may be obtained because not all
of the parallelism of the program is being used. As more PE's
are added, the program throughput increases, but the percent
time that the average ALU is busy decreases. This is not to
say that allocations of programs may not be found that
achieve high throughput and keep the average ALU very
busy. The relationship between maximizing throughput and
efficiency using PE's is dependent upon the parallelism in the
graph and the allocation as well as the hardware. For exam­
ple, an application may have only five actors that can fire in
parallel on the average, but ten actors that can fire in parallel
at one point in time. In this case, achieving maximum
throughput could imply using ten PE's even though only five
are needed on the average.
Fig. 20 shows how the percentage of packet commu­
nication that is local (intra-PE as opposed to inter-PE) varies
with respect to the number of PE's for Radarb (see Fig. 10
for depiction of the queue). As one would expect, as the
number of PE's increases, less of the packet communication
is local. Fig. 21 shows the average and maximum length of
the RQ for the Nontransitive Closure allocation of Radarb.
Note that the average queue length decreases rapidly beyond
a few PE's and that for eight or more P E ' s , the average queue
length is less than one packet. This is characteristic of the
other queues in the C O M and PROC chip and indicates that
the queue lengths may be limited to a few words per queue as
long as a queue overflow area is provided or other methods
are used to prevent deadlocks.
One of the first payoffs of the simulator was in detecting
that the architecture was easily deadlockable. Simulation of
Radar3na on one PE resulted in intra-PE deadlock when the
FQ, R Q , and F B Q filled up. Simulation of Radarb on two
PE's resulted in inter-PE deadlock when the R Q ' s , F Q ' s ,
TCQ's, F C Q ' s , PRIQ's, P R O Q ' s , PLIQ's, and PLOQ's filled
up. All of the queues on the PROC and C O M chip were
GAUDIOT et al.:
1085
A DISTRIBUTED VLSI ARCHrrECTURE
>(Λ
D
m
„ N O N TRANSITIVE C L O S U R E
5
50
ο
χ
c
DC
NON TRANSITIVE CLOSURE
AVERAGE
4
o
" ^ \ / l 2 8
40
60
#PEs
Fig. 18. Percent ALU busy time for Radarb.
Radarb2
Radar3na
Radar3na
Fig. 20.
MAXIMUM QUEUE LENGTH IS 520.
BUT ALU PAYS ADDITIONAL 6 CYCLE
DELAY IF QUEUE LENGTH >8
Ο <r
Ο
NOT SHOWN ON THIS GRAPH DUE TO SCALE:
Η PEs AVG QUEUE LENGTH MAX QUEUE LENGTH
158
103
42
Ω
ο
ι- °uj Ο
ο
<_> μ­
5
^α; δ:
<
Ratio of local packets for Radarb.
(NON TRANSITIVE CLOSURE)
(TRANSITIVE CLOSURE)
(NON TRANSITIVE CLOSURE)
5
u. <
ο
64
ο
Ο
38
00 r1
Ο
2>
Ο
LU
•
10
•
20
I
30
L_
40
50
60
70
80
90
100
PERCENT TIME A V E R A G E ALU BUSY
Fig. 19. Throughput versus ALU utilization.
limited to eight packets and could not be significantly in­
creased due to limited chip space. In order to prevent both
types of deadlock, we increased the effective length of the
RQ by allowing it to overflow into the DM at the cost of
additional ALU cycles. This explains the large average and
maximum RQ queue lengths for configurations with few
PE's.
Fig. 2 2 shows how the average communication packet
latency varies with respect to the number of PE's. This mea­
sure of latency includes the packet delays encountered in the
COM chips and in accessing the COM chips. It does not take
into account the delays encountered in the T M C , F Q , ALU,
RQ, or DT. It measures the latency from the output of the DT
#PEs
Fig. 21. Average RQ queue length of Radarb.
to the input of the T M C . It is a good measure of the efficiency
of the communication system. Note that for few PE's there is
very little COM chip activity, so the packet latency con­
tributed by the COM is low. It peaks at four P E ' s , and for
four or more PE's the communication packet latency de­
creases rapidly. For 18 or more P E ' s , the latency is almost
constant. This implies that the decrease in the rate of increase
of the throughput as more PE's are used (see Fig. 16) is due
IEEE TRANSACTIONS ON COMPUTERS, VOL. C-34, NO. 12, DECEMBER 1985
1086
#PEs
Fig. 22.
Average packet latency for Radarb.
primarily to the fact that there is a limited amount of paral­
lelism in the graph, not increased communication latency.
VII.
CONCLUSIONS A N D F U T U R E R E S E A R C H
We have presented a multiprocessor architecture for use in
a wide variety of data and signal processing applications.
Physical constraints such as size, weight, power con­
sumption, etc., have dictated the use of VLSI circuitry for
implementation of the needed functionality. It was shown
how an elementary data-flow PE could be designed using
only two high-density chips and some RAM chips. A varia­
tion of elementary data-flow principles of execution was
introduced so as to attune the model of execution to the
requirements of the architecture for maximal performance.
Resistance to failures has been demonstrated by the addition
of spares and static reconfiguration of the machine.
The overall system has been demonstrated (high-level lan­
guage compiler, allocator, architecture design, etc.) through
the use of a deterministic simulator. The hardware level
simulator of the HDFM has been used to simulate several
different programs. The statistics resulting from those simu­
lations indicate the following.
1) Each PE of the HDFM is capable of about 2 - 4 MIPS,
depending on the instruction mix. The overall throughput
may be significantly less than 2 - 4 MIPS per PE, however,
depending on the number of actors per PE and, the paral­
lelism of the graph. As a graph is allocated to more PE's, the
overall throughput increases, but the throughput of each PE
decreases. Assuming that the average HDFM program has an
average actor time of 5 cycles (250 ns) and is allocated to
maximize overall throughput (ALU busy 25 percent of the
time — see Figs. 15-18), each PE will achieve about one
MIPS. If one needs only about 60 percent of the maximum
achievable throughput, one may use fewer PE's and keep the
average PE busy 60 percent of the time.
2) When a graph is allocated to only a few PE's, the
queues grow very long. If there is insufficient queueing space
available, deadlock (intra-PE and inter-PE) may result. The
amount of queueing necessary for tokens decreases rapidly as
configurations with more PE's are used. Typically, queues of
only a few words are sufficient as long as some method is
used to prevent deadlocks such as off-chip queue overflow
area.
3) The packet (token) latency is primarily dependent upon
queue length and decreases rapidly as the number of PE's
increases. This implies that the number of PE's may be in­
creased without suffering higher communication latencies.
One would expect though that eventually, the buses in the
HDFM would become bottlenecks as too many PE's are used
per bus. Larger graphs and configurations of PE's will need
to be simulated to see what sizes of graphs and PE configu­
rations cause the buses to become bottlenecks.
4) The allocation algorithm plays a very significant role in
determining overall throughput. A nonrandom allocation al­
gorithm can increase the throughput of a graph by at least
75 percent over a random allocation. The Transitive Closure
and Nontransitive Closure allocation algorithms yield similar
maximum throughput, but the Transitive Closure allocation
algorithm uses PE's more efficiently. The Transitive Closure
allocation algorithm is restricted to graphs of a hundred PE's
or less because of its η cubed time complexity and can only
be used within a divide-and-conquer algorithm for larger
graphs (Campbell [6]).
5) About 5 0 - 7 5 percent of the token communications are
intra-PE as opposed to inter-PE in good allocations.
While the results described in this paper conclusively point
to the validity of our approach, there are several areas in
which further strides must be made so that data-flow architec­
tures may take their proper place in high-throughput process­
ing. Some of those areas of further research as they relate to
the HDFM and its tools are listed below.
1) Allocation algorithms need to be further refined
and tested, especially those that use divide-and-conquer
techniques.
2) A larger variety of programs need to be simulated on
the HDFM, especially ones with more actors.
3) Various methods of guaranteeing safety need to be ex­
plored: specifically, the use of operand queues against the
use of acknowledge tokens (Finn [10]).
4) The performance of the HDFM needs to be compared
against other architectures using a similar level of tech­
nology, especially von Neumann architectures and various
multiprocessor architectures such as macro data-flow and the
Cosmic Cube described by Seitz [23].
5) More complex data structures and methods of handling
them need to be developed. The HDFM has multidimensional
arrays, but not complex Pascal-like structured data types
(e.g., records).
6) More complex PROC chips need to be considered,
especially ones that include floating point arithmetic
capabilities.
REFERENCES
[ 1 ] W. Β. Ackerman and J.B. Dennis," V AL—A value oriented algorithmic
language, preliminary reference manual," Lab. Comput. Sci., Mass.
Inst. Technol., Cambridge, Tech. Rep. TR-218, June 1979.
[2] Arvind and R.A. Iannucci, "Two fundamental issues in multi­
processors: The data-flow solutions," Lab. Comput. Sci., Mass. Inst.
Technol., Cambridge, Tech. Rep. MIT/LCS/TM-241, Sept. 1983.
GAUDIOT et al.:
1087
A DISTRIBUTED VLSI ARCHITECTURE
[3] J. Backus, "Can programming be liberated from the von Neumann style?
A functional style and its algebra of programs," Commun. ACM, vol. 21,
no. 8, pp. 613-641, Aug. 1978.
[4] J. D. Brock and L. B. Montz, "Translation and optimization of data-flow
programs," in Proc. 1979 Int. Conf. Parallel Processing, 1979,
pp. 46-54.
[5] L. J. Caluwaerts, J. Debacker, and J. A. Peperstraete, "Implementing
streams on a data-flow computer system with paged memory," in Proc.
Tenth Int. Symp. Comput. Architecture, June 1983.
[6] M. L. Campbell, "Static allocation for a data-flow multiprocessor," in
Proc. 1985 Int. Conf. Parallel Processing, Aug. 1985.
[7] A.J. Catto and J.R. Gurd, "Resource management in data-flow," in
Proc. ACM Conf. Functional Programming Lang. Comput. Architec­
ture, Portsmouth, NH, Oct. 1981, pp. 77-84.
[8] J. B. Dennis, "First version of a data-flow procedure language," in Pro­
gramming Symp.: Proc. Colloque sur la Programmation, Paris, France,
Apr. 1974, in Lecture Notes in Computer Science, vol. 19, B. Robinet,
Ed. Springer-Verlag, New York, 1974, pp. 362-376.
[9]
, "Data-flow supercomputers," Computer, vol. 13, pp. 48-56, No­
vember 1980.
[10] D.J. Finn, "Simulation of a data-flow architecture," Master's thesis,
Dep. Comput. Sci., Univ. Calif., Los Angeles, Dec. 1984; see also,
, Proc. 1985 Int. Conf Supercomput. Syst., Dec. 1985.
[11] D.D. Gajski, D.A. Padua, D.J. Kuck, and R.H. Kuhn, "A second
opinion on data-flow machines and languages," Computer, vol. 15,
pp. 58-69, Feb. 1982.
[12] J.L. Gaudiot and M.D. Ercegovac, "A scheme for handling arrays in
data-flow systems," in Proc. Third Int. Conf Distrib. Comput. Syst.,
Fort Lauderdale, FL. Oct. 1982.
[13] J.L. Gaudiot, "Methods for handling structures in data-flow systems," in
Proc. 12th Int. Symp. Comput. Architecture, Boston, MA, June 1985.
[14]
, "Structure handling in data-flow systems," IEEE Trans. Comput.,
to be published.
[15] K.P. Gostelow and R.E. Thomas, "Performance of a simulated data­
flow computer," IEEE Trans. Comput., vol. C-29, pp. 905-919, Oct.
1980.
[16] J.R. Gurd, C.C. Kirkham, and I. Watson, "The Manchester data-flow
computer," Commun. ACM, vol. 28, no. 1, pp. 34-52, Jan. 1985.
[17] D. Hill, "ADLIB user's manual," Tech. Rep. 177, Dep. Elec. ^ng.,
Stanford Univ., Stanford, CA, Aug. 1979.
[18]
, "Language and environment for multi-level simulation," Dep.
Elec. Eng., Stanford Univ., Stanford, CA, Tech. Rep. 185, Mar. 1980.
[19] J.R. McGraw, "Data-flow computing: The VAL language, ACM
Trans. Programming Lang. Syst., vol. 4, no. 1, pp. 44-82, 1982.
[20] J.R. McGraw and S.K. Skedzielewski, "Streams and iteration in
VAL—Additions to a data-flow language," in Proc. 3rd Int. Conf.
Distrib. Comput. Syst., Fort Lauderdale, FL, Oct. 1982.
[21] L. B. Montz, "Safety and optimization transformations for data-flow
programs," Lab. Comp. Sci., Mass. Inst. Techol., Cambridge, Tech.
Rep. TR-240, Jan. 1980.
[22] T. J. Mowbray, "Language features for a static dataflow environment,"
Ph.D. dissertation, Univ. Southern Calif., Los Angeles, May 1983.
[23] C.L. Seitz, "The Cosmic Cube," Commun. ACM, vol. 28, no. 1,
pp. 22-33, Jan. 1985.
[24] G.K. Tucker, "Table generator input form (TGIF), language reference
and user's manual," Hughes Aircraft Co., Internal Doc. FR-82-75-665,
Mar. 1982.
[25] R. Vedder, M. Campbell, and G. Tucker, "The Hughes Aircraft Com­
pany data-flow project," in Proc. Fifth Int. Conf. Distrib. Comput. Syst.,
Denver, CO, May 1985.
[26] R. Vedder and D. Finn, "The Hughes data-flow multiprocessor:
Architecture for efficient signal and data processing," in Proc. 12th Int.
Symp. Comput. Architecture, Boston, MA, June 1985.
[27] Arvind and R. E. Thomas, "I structures: An efficient data type for func­
tional languages," Lab. Comput. Sci., Mass. Inst. Technol., Cambridge,
Rep. LCS/TM-178, June 1980.
tures for the TRW Technology Research Center, El Segundo, CA (1980-1982).
Since graduating in 1982, he has been on the Faculty of the Department
of Electrical Engineering-Systems, University of Southern California, Los
Angeles, where he is currently an Assistant Professor. His research interests,
include data-flow architectures, fault-tolerant multiprocessors, and parallel
logic programming systems. In addition to his academic duties, he has con­
sulted for several aerospace companies in the Southern California area.
Rex W. Vedder received the B.S.CS. degree from
£ the University of California at Riverside in 1980, and
i{ the Μ. S. C. S. degree from the University of Califor­
nia, Los Angeles, in 1982.
He is head of a processor design group in the VLSI
I Systems Department at the Electro-Optical and Data
Systems Division of Hughes Aircraft Company. He
is presently involved with design of a space-based
multiprocessor system and research and devel|; opment into parallel architectures for signal, data,
and symbolic processing. Previous responsibilities
included development of a data-flow architecture, investigation of systolic
arrays for image processing, and modeling and simulation of computer systems.
George K. Tucker received the B.S. degree in
engineering/computer science from the California
Institute of Technology, Pasadena, in 1969 and the
M.S.C.S. from the University of Southern Cali­
fornia, Los Angeles, in 1983.
Currently, he is a Staff Engineer at the ElectroOptical and Data Systems Division of Hughes Air­
craft Company. He has designed some compilers,
participated in research on data-flow and systolic
processors, and is currently designing some compo­
nents of a space-based multiprocessor system.
11
Jean-Luc Gaudiot (S'75-M'82) was born in
Nancy, France, in 1954. He received the Diplome
d'Ing6nieur from the Ecole Superieure d'Ing6nieurs
en Electrotechnique et Electronique, Paris, France,
in 1976 and the M.Sc. and Ph.D. degrees in com­
puter science from the University of California, Los
Angeles, in 1977 and 1982, respectively.
His experience includes microprocessor systems
design at Teledyne Controls, Santa Monica, CA
(1979-1980), and research in advanced architec­
Dennis Finn was born in Portland, OR, in 1958.
He received the B.A. degree from Brandeis Univer­
sity, Waltham, MA, in 1980, the M.S. degree in
computer science from the University of California,
Los Angeles, in 1984. He is studying for the Ph.D.
degree in computer science at the University of Cali­
fornia, Los Angeles, with a fellowship from Hughes
Aircraft Company.
Since joining Hughes in 1981 he has participated
in several projects, including the design of a VLSI
CAD system, a data-flow system, and an image un­
derstanding system. His research interests are in the areas of learning systems,
symbolic processing, image understanding, and parallel processing.
Michael L. Campbell was born in California. He
received the B.S. degree in mathematics from the
University of California, Riverside, in 1980, and, in
1982, the M.S. degree in computer science from the
University of California, Los Angeles, where he is
continuing toward the Ph.D. degree in computer
science with a fellowship from Hughes Aircraft
Company. The title of his M.S. thesis was "Com­
plexity of Parallel Computation."
His research interests include complexity theory,
parallel computation, data flow, object-oriented pro­
gramming, and high-level language design. Since joining Hughes in 1982, he
has participated in the Advanced Digital VLSI Architecture Project, which
seeks to develop architectural design techniques for high-performance com­
puters utilizing VLSI technology.
Mr. Campbell is a member of the Association for Computing Machinery,
SIGACT, SIGPLAN, AND SIGART.