SOFTWARE—PRACTICE AND EXPERIENCE, VOL. 21(5), 507–518 (MAY 1991 )
Profiling Software Through the Use of
Metrics
PIERRE N. ROBILLARD AND DANIEL COUPAL
École Polytechnique de Montréal, Département de Génie Électrique, C. P. 6079, Succ. A,
Montréal, QC, Canada, H3C 3A7
AND
FRANCOIS COALLIER
Bell Canada, Corporate Quality Assurance, 2265 Roland Therrien, Longueuil, QC,
Canada, J4N IC5
SUMMARY
This paper presents an approach to software assessment using a new software tool that integrates most
of the known static metrics. A six-step method shows how to use metrics to obtain a picture of the
software project. The method is visual and each step provides graphical representations of the data.
Successive integration of data results in normality profiles. Examples illustrate each step. The method
is adaptable to various environments and specific applications.
KEY WORDS
Metrics Control graph Quality profile Measurements Quality assurance
INTRODUCTION
This paper pertains to the evaluation of a software product based on the static
analysis of the source code. Software needs some form of assessment like any other
product. The resulting assessment reports usually correspond to the target audience’s
priorities. Programmers, testers, maintainers, software architects and project managers are all likely to show an interest in software assessment with their own
viewpoint in mind.
1,2
Although some publications on software metrics have indicated a trend towards
looking at a specific aspect of software assessment, a single metric could hardly
3
satisfy everyone. Indeed, recent studies show that most metrics contain limited
information.
Combining metrics into a single composite metric may not be the
4 5
solution. ’ There are many ways to look at metrics, and it could be difficult to
identify the appropriate metrics for a particular aspect of software assessment.
The amount of data that can be collected in a software project is widely spread over
each phase of the software production.6 Typical measurement covers the attributes of
the cost, the quality and the product . Measurements over each one of these phases
are of great interest and should not be diminished. However, this paper considers
only source-code-based measurement.
This paper proposes a progressive assessment approach based on the data obtained
from the software tool DATRIX. First, the new software tool, which computes and
integrates up to 36 metrics, is described. The metrics result from the parsing of
0038–0644/91/050507–12$06.00
© 1991 by John Wiley & Sons, Ltd.
Received 12 February 1990
Revised 28 November 1990
508
P. N. ROBILLARD , F. COALLIER AND D. COUPAL
source code. The tool provides many options to the user. It can select metrics, define
ranges of values for metrics, draw control flow graphs, and export metric values.
Utility software can do extensive analyses on the exported data.
The new control flow representation improves visualization of program constructs
and identifies breaches of structure. Screens from DATRIX illustrate the integration
of various aspects of measurement.
Secondly, an incremental method is described to achieve software assessment
based on the metric values. Graphs and diagrams, taken from an actual project,
illustrate the results of each step.
AUTOMATING MEASUREMENTS
A university–industry joint R&D project is under way at the École Polytechnique
de Montréal and Bell Canada to design and implement a software data measurement
tool and to learn how to use these data to evaluate software.
The control flow representation is an original feature of DATRIX. It has the
following advantages: it is easy to visualize, it outlines the breaches of structure,
and it is programming language independent. Figure 1 shows a typical control graph
representation.
The graph nodes are branch instructions, branching addresses, entry points and
exit points. Arcs link origin nodes to destination nodes. A forward arc (downward)
follows the normal flow of execution (drawn as a single line). A backward arc
(upward) is counter-flow, as in the return arc of a loop (drawn as a double line).
The weight of an arc is the number of executable statements associated with the arc.
The number preceding the node identifies the corresponding line statement in the
source code.
Figure 1. Example of control graph
PROFILING SOFTWARE
509
The control graph expands on a half-plane with the nodes aligned on the left-hand
side. This half plane representation displays the crossing of arcs. A broken intersection identifies a crossing that defies the rules of structured programming,
and
7
breaches of structure are easily visualized. Schematic pseudocode defines the rules
for structured programming.
The user can toggle back and forth between the control graph and the source code
to see a representation of a specific part of the program structure. Figures 2 and 3
show these features. Figure 3 shows the programming statements indicated by the
cursor in Figure 2, corresponding to line 19.
The Appendix describes computed metrics. It lists the names of the metrics,
abbreviations and short descriptions. The list is not exhaustive, and new and composite metrics may be added.
Users may define a range of values for each metric. This range includes the usual
maximum and/or minimum metric values. All functions exceeding the range of metric
values are automatically listed. The user may recall the unusual functions and work
out the unusual aspect by working in turn from the metrics (see Figure 4 ), the source
code, and the control graph views.
Information is provided on project organization. A call graph illustrates the
relationships between each part of subroutines or functions.
EVALUATION BASED ON METRICS
The following six-step method is used for evaluating software based on metric
measurements. The objective is to show the use of static metrics to assess a software
project. A typical example illustrates each step.
Project analysis with DATRIX
The software tool, DATRIX, provides efficient static source code representation
and metric measurements. It runs in an MS-DOS/UNIX environment. A one million
Figure2. The control graph screen
510
P. N. ROBILLARD, F. COALLIER AND D. COUPAL
Figure 3. The source code screen
Figure 4. The metrics values screen
LOC project takes less than a working day to analyse.
It processes compilable source code files of most procedural languages. The step
output is the measurement file which contains the metric values for each function.
The measurement files are available in formats compatible with various tools, such
as Lotus 123, dBase IV, Prolog, SAS, etc.
Metric distributions
Commercial spreadsheet software generates metric distributions based on the
measurement file. Figure 5 shows the distribution for the number of executable
statements.
PROFILING SOFTWARE
511
Ordered Software Units
Figure 5 Distribution example
Bars on the horizontal axis are
on the vertical axis. Functions are
hand side.
The distributions show the full
then work out the usual range of
that have out-of-range values.
functions. The metric value for each function is
in decreasing order of metric value from the leftrange of metric values for the proiect. One can
values for a project, and identify any functions
Usual metric range of values
Figures 6, 7 and 8 show three typical distributions: the number of decision nodes
(Ncn), the number of independent paths (Np) and the volume of comments in
structures (Vcs), i.e. one distribution sample for each quality factor described in
step 5.
Decision nodes refer to statements that modify the control
flow. The number of
8
decision nodes plus one is the McCabe cyclomatic numbers.
The usual range of values selected is between 0 and 12. Six functions are above
this range. There are many possible explanations for these unusually large functions,
among them being a complicated algorithm, multiple functionalities, sloppy design,
etc.
Np is the number of paths that the control flow can follow from the entry nodes
to the exit nodes. For path computation, loop constructs iterate only once. The Np
metric represents the accessibility of the exit node. Figure 7 shows the Np distribution
on a log scale. The usual range selected is between 1 and 10,000 paths. Two functions
are out of range. The functions that have larger independent paths require unusual
test coverage techniques. These functions could be the result of a design problem.
The comment volume in a structure (Vcs) is the total number of alphanumeric
characters found in the comments. They can be anywhere in the function except in
the declaration section. Figure 8 shows the Vcs distribution. The usual range selected
is from 150 to 2000 characters.
The range depends on the documentation requirement, which for this example is
one comment preceding each control structure. An excessive number of comments
512
P. N. ROBILLARD, F. COALLIER AND D. COUPAL
Ordered Software Units
Figure 6. Distribution of the number of decision nodes
Ordered Software Units
Figure 7. Distribution of the number of independent paths
Ordered Software Units
Figure 8. Distribution of the volume of comments in structure
PROFILING SOFTWARE
513
may indicate lack of focus and reduce readability.
The metric distributions help to determine the usual range of values. Feedback
and comments from the project manager or team members are useful in assessing
the selected values.
Percentile profile
A percentile profile for a project shows the number of functions that fall within
the selected range of metric values, All of the functions that have unusual metric
values are identified and listed.
Figure 9 shows the percentile profiles for the 15 selected metrics. Percentile
profiles consist of two-colour columns. The black in each column represents the
percentage of unusual functions. These are below or above the usual range values.
Black in the upper part corresponds to the percentage of functions exceeding the
range’s upper bound. Black in the lower part corresponds to the percentage of
functions below the range’s lower bound. The darker the profile the more unusual
the project.
Normality profile
The groupings of the various metrics constitute the quality factors. Typical quality
factors are testability, readability, portability, correctness, structuring, etc. The
normality profile is a diagram that shows the percentage value of each quality factor.
The normality profile shown represents three quality factors: size, testability and
readability. Each quality factor groups five metrics. Table I lists the metrics that
compose each quality factor in the normality profile in Figure 10. The usual range
used in this particular analysis is indicated in parentheses.
Each column in the normality profile represents a quality factor. For example,
the column ‘Size’ shows that 81·4 per cent of the functions meet the range of the
five metrics composing this quality factor. The normality profile provides a quick
visualization of the overall project. High-level management can easily interpret such
a profile.
Metrics
Figure 9. Percentile profile for the functions
514
P. N. ROBILLARD, F. COALLIER AND D. COUPAL
Table I. Metrics defining the quality factors
Factor
Range
Definition
Metric
Size
Nbe
V
Ncn
E
K
Number of statements
Number of nodes
Number of decision nodes
Number of arcs
Number of crossing arcs
Testability
Np
NI
Nel
Nc
Psc
Number of independent paths
Number of loops
Mean nesting level
Mean decision node complexity
Mean decision node span
Readability
Rac
Vcd
Vcs
Rvc
Ls
Code arc comments ratio
Comments volume of declaration
Comments volume of structures
Comments volume ratio
Mean identifier length
(10–300)
(2–30)
(0–12)
(1–50)
(0–20)
(1–10000)
(0–5)
(0–4)
(l-7)
(0–5)
(50-100)
( 50-2000)
( 50-2000)
10-30)
(5-l0)
% of criterion met
96.4
100
88.4
90
81.4
80
70
60
50
40
30
20
10
0
Readability
Size
Testability
Quality Factors
Figure 10. Normality profile
Function interrelationships
DATRIX finds and prints all function interrelationships. Graphic tools display the
data. Figure 11 shows a new call graph representation, where the functions are
represented by columns in a three-dimensional depiction. The height of a column is
relative to the number of functions at the same location. For example, the column
at point A shows one function that is called 25 times and that calls others 10 times.
There is less function coupling for functions lying along the axes. Maintenance and
testing are likely to be easier for these functions.
PROFILING SOFTWARE
515
Figure 11. Call graph for the functions
CONCLUSION
To conclude, we outline the steps required in the assessment process, discuss what
we have learned from these analyses and how they can be improved. We also identify
areas where more research is needed.
Metrics are computed from the static analysis of source code. All, or a subset of,
the metrics are selected to perform the assessment. Metric distributions are studied
to determine the usual ranges of values. Out-of-range functions are identified.
Percentile profiles give a project overview of the percentage of out of range functions.
Quality factors are defined by grouping metrics. A normality profile gives the
percentage of success for every quality factor. Finally, function couplings are evaluated from a 3-D call graph representation.
This approach, based on graphs and profiles, is visual and, as a result, many
different types of people are afforded immediate access to information. DATRIX
automatically finds any function with a given profile. This means that significant
data, such as unusual functions, are also immediately available, and no time is
wasted in looking at the more common functions.
Function profiles can be modified at will to explore specific aspects of program
quality, and profile simulations can evaluate the impact of function redesign on
project quality. The programmer can obtain immediate feedback on any function he
has implemented.
The versatility of this approach makes it conducive to exploratory study and to
the development of custom assessment programs.
New metrics based on information
9
theory can be integrated into existing metrics. This approach is also of interest in
the maintenance and testing area where an inside picture of the software can provide
insightful clues as to where further development effort should be concentrated. We
feel that merely introducing this tool into the development process at an earlier
stage may result in increased productivity.
516
P. N. ROBILLARD, F. COALLIER AND D. COUPAL
The problem of metrics validation remains. It could be solved by an extensive
study of the relationship between the presented metrics (with their usual values) and
some data relative to error rates, development costs and reliability indicators. Such
a study could be viewed as the next step towards the validation of this model.
However, regular use of the tool will provide users with enough experience in the
analysis of metrics to be able to obtain precise information on the quality of the
system. At this time, in fact, an expert system could be designed to evaluate software
quality based on metric measurements.
We cannot yet define standard projects. We find that each project is different,
calling for different ranges of values and profiles, and to link these various ranges
and profiles to historical data in order to define standard projects will require further
studies.
ACKNOWLEDGEMENTS
We would like to thank Jean-Charles Bernard, Jean Mayrand and Mario Simoneau
for many helpful suggestions and comments. We are grateful to the anonymous
referees for their comments. This work was supported in part by Bell Canada Inc.
and the operating grant program of the Natural Sciences and Engineering Research
Council of Canada.
APPENDIX
Abbreviation
Name
E
K
Number of arcs
Number of knots
Ncn
Number of conditional nodes
Ne
Number of exit nodes
Nel
Mean nesting level
Nelmax
Maximal nesting level
Ni
Number of entry nodes
NI
Number of loop constructs
Np
Number of independent paths
Definition
Number of arcs in the control graph.
Number of arc crossings in the control graph.
Ncn is the number of nodes in the
control graph having an out-degree
greater than one.
Ne is the number of nodes in the
control graph where the control flow
stops or returns to a calling software
unit.
Nel is the arithmetic mean of the
nesting level of all arcs.
Nelmax is the maximal value of nesting level in the software unit.
Ni is the number of entry nodes in
the control graph.
NI is the number of backward arcs in
the control graph.
Np is the number of paths that the
control flow can follow from the entry
node(s) to the exit node(s). Loops
are visited only once.
517
PROFILING SOFTWARE
Abbreviation
Psc
Pscmax
V
Vg
Vp
Vs
Nbe
Ls
Nbt
Bs
Bsp
Nelw
Nr
Rlw
Vw
RVC
Vcd
Name
Mean conditional node span
Definition
Psc is the mean number of nodes that
are located within the span of conditional nodes.
Maximum conditional node Pscmax is the maximum number of
span
nodes that are located within the span
of a conditional node.
Number of nodes
V is the number of nodes in the control graph.
Cyclomatic number
Vg = E–V+2 p ; where p = 1.
Number of pending nodes
Vp is the number of nodes in the
control graph having an in-degree of
zero that are not entry nodes.
Structural volume
Vs = (E+ V–1)/6.
Number of executable state- Nbe is a count of all executable statements
ments in the software unit.
Mean length of variable Ls is the mean number of characters
names
of all variables that are declared
and/or used in the software unit.
Total number of lines
Nbt is a count of all lines in the
software unit.
Number of breaches of struc- Number of arc crossings in the conture
trol graph that violate the principles
of structured programming, i.e. the
usage of one-entry, one-exit control
structures.
Weighted number of breaches Bsp is the sum of the weights of all
of structure
pairs of arcs involved in breaches of
structure.
Weighted mean nesting level Nelw is the arithmetic mean of the
nesting level of all arcs considering
their weights.
Number of recursive nodes
Number of nodes in the control graph
where the software unit calls itself.
Loop weight ratio
Rlw is the ratio of the sum of weights
of the subgraphs delimited by backward arcs (loops) on the total sum of
weights.
Sum of weights
Vw is the sum of the weight of all
arcs in the control graph.
Comments volume ratio
RVC = Vs/(Vcd + Vcs) .
Comments volume in declar- Vcd is the total number of alphabetical
characters found in the comments
ations
located in the declaration section.
518
P. N. ROBILLARD, F. COALLIER AND D. COUPAL
Name
Definition
Vcs
Comments volume in structures
Rac
Commented arcs ratio
Rls
Loop structural ratio
Rnc
Commented nodes ratio
Nc
Mean node complexity
Ncmax
Maximal node complexity
Vcs is the total number of alphabetical characters found in the comments
located anywhere in the software unit
except in the declaration section.
Rac is the percentage of commented
arcs over the total number of arcs
with a weight greater than zero.
Rls is the ratio of the structural volsubgraphs
ume
contained in
delimited by backward arcs (loops)
on the total structural volume.
Rnc is the percentage of commented
nodes over the total number of nodes
conditionals
nodes
are
(only
considered).
Nc is the arithmetic mean of the complexity attribute for all nodes having
an out-degree greater than one. The
complexity is defined as the number
of operators and variables in the
node, minus one.
Ncmax is the maximal value of the
complexity attribute for all nodes
having an out-degree greater than
one.
Abbreviation
REFERENCES
1. V. Coté, P. Bourque, S. Oligny and N. Rivard, ‘Software metrics: an overview of recent results’,
J. Syst. Software, 8 (2), 121–131 (1988).
2. M. Sheppard, ‘A critique of cyclomatic complexity’, Software Eng. J., 3, (2), 30–36 (1988).
3. D. Coupal and P. N. Robillard, ‘Factor analysis of source code metrics’, J. Sysf. Software, 12,
263–269 (1990).
4. J. C. Munson and T. M. Khoshgoftaar, ‘The dimensionality of program complexity’, Proc. 11th
International Conference on Software Engineering, Pittsburgh, 1989, pp. 245–253.
5. H. F. Li and W. K. Cheung, ‘An empirical study of software metrics’, IEEE Trans. Software Eng.,
SE-13 (6), 697–708 (1987).
6. D. N. Card and R. L. Glass, Measuring Software Design Quality, Prentice-Hall, Englewood Cliffs,
New Jersey, 1990.
7. P. N. Robillard, ‘Schematic pseudocode for program constructs and its computer automation by
Schemacode’, Commun. ACM, 29 (11), 1072–1089 (1986).
8. T. J. McCabe, ‘A complexity measure’, IEEE Trans. Software Eng., SE-2 (4), 308–320 (1976).
9. P. N. Robillard and G. Boloix, ‘The interconnectivity metrics: a new metric showing how a program
is organized’, J. Syst. Software, 10, 29–39 (1989).