IBM 3850-Mass storage system
by CLAYTON JOHNSON
IBM Corporation
Boulder, Colorado
SUMMARY
In this hierarchy, the best performance achievable comes
from DASD. The time a program waits for data to appear
on DASD or waits for space on DASD to store new data, is
the visible impact of the storage hierarchy.
A storage hierarchy also reduces the number of requests
for data that must be resolved at the Mass Storage
Facility. Recently active data sets can be kept on the
DASD level of the hierarchy for a period of time after
their last use. This is accomplished by providing storage
capacity at the DASD level that is greater than the sum of
the size of all concurrently active data sets.
Reuse of data dramatically reduces the number of accesses to the Mass Storage Facility, and makes it possible
to design a system that moves much less data between
MSF and DASD than that which moves between Main
Memory and DASD.
MSS uses a Least Recently Used replacement algorithm
to schedule which data on DASD should be destaged to
the MSF when space is needed for new data, or when
newly active data must be staged to DASD for access by
the CPU(s). Only those DASD cylinders which have
changed are actually destaged.
The following exemplifies the work load an MSS might
experience, and the advantages of this architecture.
IBM's 3850, a hierarchical storage system, provides
random access to stored data with capacity ranging from
35 X 1()9 to 472 X 1()9 bytes. The hierarchical architecture
achieves access times varying from Direct Access Storage
Device speeds to that of the Mass Storage Facility which
can be as low as 10 seconds. The architecture of the Mass
Storage System is examined to demonstrate its functional
and performance capability.
INTRODUCTION
The goal of the Mass Storage System (MSS) is to provide
capacities for handling all of an establishment's data. In
addition, it provides the functions of Direct Access Storage
Devices (DASD) at a cost which approximates that of
today's half-inch tape systems, and with an architecture
that allows users to migrate to MSS with minimal costs.
A hierarchy of devices provides the flexibility needed to
approach such a diverse set of goals. See Figure l.
A hierarchical arrangement allows the MSS architecture
to be hidden, or buffered, from main memory. This allows
system designs that are specific to the problem of storage
and retrieval of data, independent of how programs
manipulate data. Utilization of an existing device
eliminates the need to develop new Direct Access Devices
(adding to the complexity of the development project) and
provides a known, existing interface to data stored in the
MSS. Actually, the intermediate devices in a storage
hierarchy can be independent of the architectural interface to data. For example, in MSS the interface to data is
architecturally that of an IBM 3336 Model 1 DASD
volume, regardless of whether a 3330 Model 1, or Model
11 is being used. A new device or memory could be used in
the intermediate levels of the hierarchy without changing
the architectural interface to data itself. This allows for
the incorporation of new technologies in the storage
hierarchy with little or no disruptive impact to the users of
the storage hierarchy.
The ideal control of the external storage is to have on
DASD that data required by the programs in the CPU
when the programs need it, and to have space available on
DASD to store new data. How close we come to this ideal
is a measure of the trade-off of price versus performance.
CONTROL AND DATA FLOW WITHIN MSS
MSS, as seen by the System Control Program, has
nearly unlimited Direct Access Storage. This approach to
Mass Storage furnishes the function of DASD, and it also
provides a major new capability on IBM CPU systems
without the disruptive impact of a totally new architecture. MSS also follows the basic architecture of
System/370 channels and I/O addressing. The result
avoids a major redesign of Operating Systems, User~Ap­
plication Programs and Channels.
Today's systems gain addressability to data or storage
space by mounting a storage volume (either half-inch tape
or disk packs) on an auxiliary storage device. These
devices have specific addresses on one or more CPU channels. An Input or Output operation is performed to a
specific volume by addressing the specific device holding
that volume. The total addressable data at anyone time is
limited in this arrangement to the actual number of
509
From the collection of the Computer History Museum (www.computerhistory.org)
510
National Computer Conference, 1975
§J
! II
I ~ain
I
nstructions
Storage
I
1/0 Operations
DASD Buffer
I
Staging/Destaging Requests
1
I
j
Mass Storage
Figure I-Hierarchical store - Data movement
devices attached to a given system on which storage
volumes are mounted.
MSS resolves this limitation through a virtual device
concept. Because DASD (called "staging buffers") is used
in MSS as a buffer to the Mass Storage Facilities, the one
volume per device relationship is no longer necessary.
Only the active portion of a volume must be on DASD at a
particular time; therefore, the remaining space of the
DASD staging buffers can be used for portions of other
active volumes. A virtual to real mapping capability is added to the Storage Controllers. Each Channel connection
to a Storage Controller with MSS has the ability to address 64 unique volumes, independent of how many real
devices are being used as staging buffers.
A virtual device approach makes more data addressable
at any time without the cost of a corresponding number of
I/O devices; this in turn allows larger data bases to appear
to be on line, opening up new approaches to selected data
base applications.
On real DASD volumes (non-staging), a user must put
data on volumes that tend to be active at the same time,
even if they are unrelated. Many problems can result,
such as backup, security, etc. A virtual device concept
eases the problem of space management on DASD
volumes.
All space in MSS is managed in terms of DASD
volumes. A pair of cartridges is assigned a name, and this
name represents a specific DASD volume. This pair of
cartridges is called a Mass Storage Volume.
To gain address ability to a Mass Storage Volume, the
operating system (as a result of the active user programs)
requests a specific Mass Storage Volume be mounted at a
specific virtual I/O address. The mount command is
directed to the Mass Storage Controller (MSC). The
mount command allows MSS to translate an access to a
specific Mass Storage Volume when I/O access is made
against the virtual device. If the specific cylinder required
by the CPU (1/404th of a Mass Storage Volume) is already on DASD, an I/O operation proceeds. If not, and
data is being accessed, the MSC causes the cartridge
containing the cylinder to be placed on a Data Recording
Device (DRD), and the data contained in that cylinder to
be transferred to the DASD staging buffer. The transfer
path is to the Storage Controller, and from the Storage
Controller to the Staging Buffer (DASD device) avoiding
the extra load on channels and memory. The I/O operation then proceeds as before. If the Operating System
knows which cylinders will be accessed, it can cause the
MSC to stage only those cylinders containing the data set;
reducing the number of times cartridges need to be accessed.
It should be apparent that the total amount of DASD
Staging Buffer Space is related to the total size of the currently active data sets, and not to total number of active
volumes. DASD is being used as a buffer to the MSF, and
the actual device characteristics are architecturally unimportant.
CONFIGURING MSS
The objective of any configuration is to select the
components required to do the work load of a specific customer. Excess performance capability, either as a result of
the configuration or the design of the system, yields nothing useful to the customer and adds to the cost of the
system. Systems must be configured to handle the highest
work load case with the responsiveness required to meet
work schedules or human factors requirements in interactive environments. Availability and security may also affect a configuration but are not the subject of this paper.
In evaluating a Storage System, some terms must be established to describe performance. The basic objective of
storage is the transfer of data to or from a CPU's Channels. The total amount of data transferred is a measure of
throughput (total work) while the time to access data becomes a measure of responsiveness.
Current systems can be measured using various techniques to determine the total amount of data transferred
into or out of storage over given units of time. A basic difficulty in measuring a system is that, by definition, the
measurement is based on past performance. Past performance provides an adequate measure only if the environment is static; which is seldom. A knowledge of existing
applications growth and new application requirements
must be acquired and factored into planned configurations.
In a storage hierarchy such as MSS, the Direct Access
Storage must have sufficient performance capability to
supply all accesses required by the CPU's main memory.
Direct Access Storage is also involved when data is staged
from the Mass Storage Facility or destaged to the MSF. It
must have sufficient capacity to hold all open data sets
and additional capacity to maximize the reuse potential of
data. MSF must have transmission capability to handle
From the collection of the Computer History Museum (www.computerhistory.org)
IBM 3850-Mass Storage System
the data being staged or destaged, and the ability to
handle the cartridge movement involved. It must have capacity to contain all the data, backup, archival and active.
A work load analysis of the total system is required to
choose the proper configuration.
The following is a hypothetical work load compiled to
illustrate the demand that might be placed on the major
interfaces of a storage hierarchy such as MSS. (See
Figures 2-6.)
Figure 2 represents the I/O work load over time, and
the sustained data rate across the CPU's channels, with
tape and DASD averaged over one-hour periods. Each
vertical bar reflects the combined effect of operator mount
delays, operating system overhead, device raw data rate,
channell control unit configuration and demand for data
by the programs that were active in that hour.
At any point, the Input/ Output demand can be limited
by the speed of the CPU and the number of I/O bytes of
data required per instruction executed, or by the number
of programs that may run concurrently. Peak throughput
of the entire system is always limited by full utilization of
the CPU's processing cycles. If the I/O is configured so it
never limits the CPU to less than 100 percent utilization
(and the cost of this I/O is not prohibitive) an ideal
throughput system can result. Full utilization also requires
an unending supply of work and response time which may
be exceptionally long. Figure 2 is more typical, showing
peaks and valleys in the I/O work performed by the
system. Peaks represent 100 percent utilization and
valleys represent the lack of work to do; or a system being
limited by operations, number of available devices, or the
mix of programs in the CPU.
The total amount of data staged and destaged is determined by the capacity of DASD and by how the data is
being used. The potential for reuse of data can be understood by making an assumption about DASD capacity,
and observing the resultant staging rate between DASD
and MSF. First, assume there is only enough DASD capacity to contain the total size of concurrently active data
sets; but with no additional capacity to retain data on
DASD, then observe the demand on the MSF for data sets
400
350
~
300
..0
~ 250
Q.)
+-'
~
200
C'l
c
150
C'l
co
+-'
100
50
(J)
0
0
15
30
I
45
I
60
75
Retention Time (minutes)
Figure 3-Staging data rate and DASD capacity relationship (72 hour
example)
and data rate. Next, assume there is enough DASD capacity to contain the total size of concurrently active data
sets, and there is additional capacity to retain data on
DASD after its active use for longer and longer periods of
time. Now, if a data set is reused within that retention period, it does not have to be staged again nor is it necessary
to handle the cartridges. Figure 3 represents the results of
such an analysis. The plot shows the reduction in average
MSF data rate as the retention period on DASD is
increased.
Once a retention period has been selected, it is more accurate to look at the required average capacity over
smaller increments of time. Figure 4 shows the average capacity per hour that is required to contain all the active
data sets during the time of their activity, plus the additional time included for optimum reuse. Given the average
capacity to optimize the reuse of data, a closer analysis of
o
Total Capacity
.Tape Contribution
o Total Data Rate
• Tape Data Rate
800
511
1200
700
1000
600
2co
o
Q)
>
';;
u
~
W
E
~
>
~!
500
.~~
800
c. .....
400
(0
0
u'"C
600
~=
(1):=
>:2:
400
(I)
mo
300
«-
200
200
100
(/)
O'-----'-----'----'--'---'-----'----J'---'-----'------'-----L--'--L-..L----'----'
10
20
30
40
50
60
70
80
Hour
Figure 2-Systems work load (72 hour example)
0
10
20
30
40
50
60
70
Hour
Figure 4-DASD capacity required (72 hour example)
From the collection of the Computer History Museum (www.computerhistory.org)
80
512
National Computer Conference, 1975
o
Total Staging Required
• Tape Contribution
180
160
140
VI
---~
.0
~
120
100
CQ
a:
~
CQ
80
0
1:1
Q)
Cl
60
CQ
en
40
20
0
10
20
30
40
80
Hour
Figure 5-Data staging rate required (72 hour example)
MSF activity can be made. It must take two forms, the
first is the total amount of data staged and destaged, the
second is the total number of individual data sets handled.
Both are necessary because they represent different work
loads.
Figure 5 shows the staged data rate in ea~h hour of t~e
analysis. The Storage System must be confIgured so thIS
demand can be met, and so it relates directly to the
amount of DASD activity introduced by the storage
hierarchy. The number of paths required for staging and
destaging also result from this analysis.
The number of data sets being handled at the MSF level
of the storage hierarchy can be used to determine the work
load in terms of cartridge moves. Figure 6 shows this work
load stated in terms of cartridge moves per hour. The size
of the data sets, and the Mass Storage Volume organization must be considered to translate data set accesses to
cartridge moves.
SYSTEMS CONTROL
The preceding sections presented an analysis of the
work load relationship that could be expected on the
major interfaces of a storage hierarchy. If the system is
designed with work load relationships in mind, then
maximum reuse potential must be achieved. In addition,
the control system must function in large establishments
which typically have multiple CPU syste.'1s controlled by
separate operating systems. It is necessary to have,
therefore, a common point of intelligence to control both
the concurrent use of data and the staging buffer replacement algorithms.
The MSS has such a control system, called the Mass
Storage Control or MSC. In MSS, regardless of how many
Mass Storage Facilites exist, one and only one MSC controls the hierarchy. Some MSS models have two MSCs;
however, the second serves only to improve availability.
The MSC cannot perform the entire task without also having control over allocation of volumes to devices. Because
of this, MSS is designed to allow the MSC to control the
functions of space management, while the control of data
and allocation is within the operating system. An interface
between MSC and the operating system is a necessity.
In IBM System 370 architecture, a program gains access to data by having a volume mounted on an addressable device, as discussed previously. In a single CPU
environment, the operating system is always aware of
which volumes are currently mounted. In a multiple
operating system environment, such as illustrated in
Figure 7, the systems can take actions independent of each
other. In addition, a storage hierarchy such as MSS is attempting to maximize the reuse of data. If any residual
data remains on the staging buffer and the same using
system or another system reuses that data, the MSC must
know where that data resides to prevent unnecessary staging. Another condition which may exist is where one CPU
is currently using a data set on a specific volume, and
another CPU needs access to the same data set, or another
data set also residing on that volume.
The virtual device concept provides a convenient solu-
o Total Move Requirement
• Stage Up Contribution
180
160
140
V'>
Q)
>
0
~
~
120
100
"0
80
u
60
.~
Data Flow
40
20
0
10
20
80
Hour
Figure 6-Access rate (72 hour example)
Figure 7-Multiple systems environment
From the collection of the Computer History Museum (www.computerhistory.org)
IBM 3850-Mass Storage System
tion to the following requirements. First, assume that CPU
1 has access to a volume and it has allocated that volume
to DASD POOL 1. Now, assume CPU 2 wants access to
another data set on that same volume. If the operating
system in CPU 2 were to independently allocate that
volume, it would be pure chance if it selected a device address associated with POOL 1. In MSS, the CPU that is in
the process of allocation, uses the MSC control interface to
learn if the volume in question is currently mounted, or if
any residual data from a previous usage of the volume
exists anywhere on the staging buffers. If either condition
exists, the operating system retrieves the information from
the MSC and chooses a virtual address associated with the
DASD POOL that contains the data. The MSC then conditions the Storage Controllers associated with that DASD
POOL to access to the specific unit address maps to the
desired data. This makes it possible for CPU 1 to access a
specific volume on one virtual device address while at the
same time, CPU 2 has access to the volume on a different
virtual address. The converse is also true, CPU 1 and
CPU 2 can be accessing different volumes on the same virtual address.
The above described control makes it possible for
multiple CPU's, which may be controlled by different
operating systems (OS/VS1 and OS/VS2 for example), to
concurrently share the DASD buffer space without having
a common allocation system. As many as six different virtual addresses can be mapped to the same volume, or each
may have different volumes associated with them. Each
Storage Controller used in MSS can have up to three channel interfaces, and one or two Storage Controllers may access the same DASD POOL. Each interface can have up to
64 virtual addresses, independent of the actual number of
DASD devices in the pool. This means that 64 volumes
could be mounted in that pool with six alternate paths to
each, or that up to 384 volumes could be mounted with no
alternate paths.
To control sharing within MSS, two mechanisms are
used. A logical device reserve I release function is provided
and is functionally compatible on virtual devices with current IBM 3330 devices. A second mechanism is provided
called exclusive Mass Storage Volumes. This function
allows the volume to be defined to MSS so the MSC can
allow only one CPU access at a time. Actually, if the
data on the volume is read only, or only one user is in the
update or write mode, there is usually no reason to prevent
concurrent access.
SPACE MANAGEMENT
Managing space on DASD volumes is considerably more
complex than managing a tape library. This is due to the
capacity of DASD volumes and the function they provide.
Typically tape volumes contain data from a single data
set, while DASD volumes contain data from many data
sets. If the data sets have' different life times (expiration
dates), and are of a variety of sizes, the result can be frag-:
mentation of the space. This is further complicated by dif-
513
ferent security and backup requirements. It, therefore, is
desirable to provide new functions to ease the task of
managing space within MSS because of its capacity
(equivalent to up to 4,720 Packs of a 3336, Modell).
Again, it is necessary to divide the control between MSS
and the attached CPU operating systems. The division is
made to manage space within the MSC, and to manage the
content of that space with the existing catalogue functions
of the operating systems; also to complement the catalogue
functions with new software functions called Mass Storage
Volume Control (MSVC).
The objectives of MSVC are to aid the users of MSS in
space allocation and space reclamation and to provide a
management reporting function so that use of space within
MSS can be monitored. The main parameters of space
management are accounting, data set size, life cycles, security and backup. Because these parameters vary widely
between installations, a framework is implemented to give
customers control with as little implementation effort as
possible.
MSVC maintains an inventory of Mass Storage
Volumes. This inventory can be structurable into groups
of volumes. The MSVC maintains information about
ownership, space usage, space allocation defaults, backup
volumes, retention dates, and location of the Mass Storage
Volume if it is removed from the system. Volumes can be
grouped by data set sizes, user group, security or any other
parameter the customer chooses. Once a structure is
chosen, attributes can be applied to the entire group or to
individual volumes within the group. It should be possible
in most cases to use MSVC so that the space management
within MSS is automatic. MSVC also uses the control interface to MSC to manipulate the volumes physically or
logically. For example, volumes may be removed from
MSS for security purposes, or may be made inaccessible
within the system. Volumes may be copied within MSS
without data movement through the CPU's memory or
channels, and the copy is inventoried by MSVC, and access to the copy is controlled by the MSC.
CONCLUSION
As stated earlier, the number of times and the amount of
, this time, a program or task has to wait for data to appear
on DASD or has to wait for space on DASD to store new
data, is a measure of the visible impact of the storage
hierarchy and the overhead it introduces into the 1/0
system. The effect on total system performance can only
be understood by comparing the projected overhead with
the existing overhead.
Again, averages are used to demonstrate the potential
for improvement. Figure 8 is a comparison showing delay
times in a half-inch tape environment where tape volumes
are handled by operators; versus the delay times caused
by MSS. Figure 8 assumes MSS has been configured to
the optimum reuse DASD capacity and has a sufficient
number of access paths to handle the peak work loads demanded by the job mix. It also assumes the events of data
From the collection of the Computer History Museum (www.computerhistory.org)
514
National Computer Conference, 1975
o =MSS
RECOMMENDATIONS
EI = Operations delay % tape
962
GOO
Average Delay Due to Staging = 0.1 Minutes
500
CI>
Q)
I
Average Delay Due to Manual
: / Tape Mounts = 1.3 Minutes
400
I
I
u
c:
E300
::J
g
o
200
100
.2.3.5.7 1
.5
2
3
4
6
10
Task Delay Time (minutes)
Figure 8-Estimated task delay time (comparison example)
set usage are essentially the same in both environments.
The potential for improvement in this specific case is approximately one order of magnitude.
The preceding analyses allow the following observations
to be made relative to the MSS storage hierarchy.
1. The best possible performance of the hierarchy is
equal to that of the devices used in the top level of
storage. For MSS, it is the DASD level.
2. The total cost of the Storage System is dependent on
the size of data sets, their reuse characteristics, and
the amount of data that must be transmitted to and
from the CPU's main memory.
3. Mass Storage can be designed with considerably less
data transmission capability than devices which
must respond directly to the needs of an active
program.
4. The measurable impact of the hierarchy is the length
of time it takes to make data or space available to the
CPU channels. The total amount of storage space at
the DASD level of the hierarchy can have a greater
effect on this response time than the transmission
rates between the levels of the Mass Storage System.
5. A virtual device architecture allows the storage
hierarchy to adapt new technologies without changing or impacting the user's interface to data. The
hierarchy can be expanded to include additional
levels of storage within this basic approach.
Understanding the performance of a storage hierarchy is
difficult because of its wide range of applications. It is
relatively clear that properly configured, a storage
hierarchy performance will be comparable to manually
mounted tape/ disks in local and remote batch applications where data set sizes are within the 50 X 1()6 byte
range and where reuse characteristics are present. Storage
hierarchies are particularly interesting in remote compute
applications. The use of storage hierarchies in interactive
data base applications, however, needs considerably more
understanding in most cases.
There are a number of applications which require very
large data bases but have a low rate of access. If this rate
of access is less than delivery capability of cartridges in
MSS, then the analysis is simple. If, however, the total
work load on the storage system includes local and remote
batch, interactive compute, interactive data base and
utility type functions (backup, interchange, archive, etc.)
understanding the total system is very complex.
Usually this type of complexity can be understood by
modeling. In the case of a storage system, however, proven
modeling techniques are not available. Analytical models
require that input parameters are some sort of distributions or averages. Simulation models require large quantities of trace data in order to cause sufficient activity at
the lowest level of the storage hierarchy. The availability
of trace data assumes the application has been done
somewhere. Data organization must be included to relate
data accesses to MSF accesses. Also, clustering and sequencing of requests must be related to data organization
and understood relative to retention times in the higher
levels of storage.
The recommendation is to use caution in approaching
data base applications where there is a dependency on "80
percent" of the activity versus "20 percent" of the data (or
some other rule). Modeling techniques must be developed
which allow total systems to be analyzed. Data must be
collected and understood which can indicate the
predictability of when data will be used and how it relates
to other data: Lastly, it must be recognized that these applications will coexist within the same complex; therefore,
the analysis must encompass the old as well as the more
exciting, new applications.
From the collection of the Computer History Museum (www.computerhistory.org)