What I have researched so far II
Outline: Me from his work:
More Information about Traditional Testing Techniques
Class Testing Techniques – details
Flowgraphs
Testing Levels for Object Oriented Software – Y Labiche, P. Thevenod-Fosse, H. Waeselynck,
M.-H. Durand – lots of detail
Defining and Ordering Testing Levels - various
Future Work
The State Based Testing of Object Oriented Programs – CD Turner & DJ Robson
Finite State Automata
State-based Testing
Substate
Substate Values
Performance Testing
Algorithm for Generation of Test Cases
Integration Testing
More assessment
Miscellaneous Stuff
`
=================================================================================
I have outlined five methods for object-oriented testing. The first three come from Dechang et al’s
paper from the last summary. Those are kind of sparsely detailed in that paper. If you would like me to
further explore any of those options, they have cited the original papers which I should be able to find
and research. The other two methods were very detailed, and so their summaries are kind of long.
More Information about Traditional Testing Techniques
Labiche et al

Lubiche states that traditional unit and integration levels for procedural programs don’t fit for
OOP
o Unit testing cannot be deduced from one object’s operations because when those
methods are viewed in isolation you cannot see the relations to other operations and
the class’s own state changes. Therefore, Labiche says, the smallest unit to be tested
cannot be smaller than an object of a class
Robson & Turner

Two basic categories of traditional testing techniques:
o



Functional: black-box testing (generation of test cases based purely on functional
specifications)
o Structural: white-box (uses information about the program’s structure to generate test
cases for exercising specific portions of code)
Functional system testing views entire system as a single unit with inputs and expected outputs
Functional unit testing views each separate program unit (method) with inputs & outputs
Traditional testing techniques assume the Von-Neumann model of processing (input-processoutput)
o Adapt to OO:
Input
Params
+
Current
State
Output
Params
Process
+
New
State
Class Testing Techniques
Dechang et al
Flow-graph-based
Proposed by Zweben (‘‘Systematic Testing of Data Abstractions Based on Software Specifications’’) et al
and expanded on by Parrish et al (“Automated Flow Graph-Based Testing of Object-Oriented Software
Modules”)


Class behavior is modeled by a flow-graph
Given class C and a set O of operations, create a flow graph G. G is a directed graph with:
o A node for each operation
o Each edge (O1, O2) in graph corresponds to the fact that O2 may be invoked after O1
 The existence of any edge can be determined if what is required of O2 is
satisfied by what is ensured by O1
 This comes from model-based specification language, or
 Comes from class interface, which provides both class name and all
parameters’ types and modes of transmission (IN, OUT, or IN-OUT)
 Author gives special property to Boolean return values: label these edges T/F to
indicate that a particular edge only works for true or false returns
o Authors note that for many abstract data types (I imagine others as well) the graph is
cyclical. This makes the number of paths infinite and impossible to test all paths. Test
cases must be designed with an upper-bound on path length.

Author also states that tests for traditional data flow graphs or control flow graphs can be
adapted for Object-Oriented flow-graph
o Says that test cases can be derived based on def-use relations from data-flow graphs.
You can modify the method for data-flow graphs by using IN transmission modes for
parameters as use and OUT parameter transmission modes as def.
o From Wikipedia: “Definition-Use Chain (DU Chain), which consists of a definition, D, of a
variable and all the uses, U, reachable from that definition without any other
intervening definitions.”
 I wanted further clarification on the kind of methods that could be adapted from traditional
testing to the flow-graph based approach. However, I could not find the two papers that the
author sourced (Zweben and Parrish et al) online.
Graph-Based Approach
 Tests class behavior based on testgraph (T):
o Each node corresponds to a state of the class under testing (CUT)
o Each edge (S1, S2) corresponds to a sequence of operations that cause the transition
from S1 to S2
 State space is often large to infinite, and a limited number of representative
states are chosen to test
o A path is a sequence of nodes where each node is connected to the previous node. A
path is a sequence of state changes that an object of a class might go through that the
programmer wishes to test.
o Test case set up is systematic and is designed based on a coverage criteria such as node
coverage, path coverage, branch coverage, etc.
o Also facilitates the design of a testing oracle
ASTOOT – A Set of Tools for Object-Oriented Testing
Proposed by Doong & Frankl (“The ASTOOT Approach to Testing Object-Oriented Programs”).

Founded on algebraic specification
o From Wikipedia:
Algebraic specification[1][2][3], is a software engineering technique for formally specifying system
behavior. Algebraic specification seeks to systematically develop more efficient programs by:
1. formally defining types of data, and mathematical operations on those data types
2. abstracting implementation details, such as the size of representations (in memory) and the
efficiency of obtaining outcome of computations
3. formalizing the computations and operations on data types
4. allowing for automation by formally restricting operations to this limited set of behaviors and data
types.
An algebraic specification achieves these goals by defining one or more data types, and specifying a
collection of functions that operate on those data types. These functions can be divided into two classes:
1. constructor functions: functions that create or initialize the data elements, or construct complex
elements from simpler ones
2. additional functions: functions that operate on the data types, and are defined in terms of the
constructor functions.


The axioms of the algebraic specification define a term rewriting system with each axiom as a
rewriting rule
Definitions:
o 2 sequences of operations are equivalent if S1 can be reduced to S2 by applying the
axioms of the specifications
o 2 objects are observationally equivalent iff they are in the same “abstract state”
o If O1 and O2 are observationally equivalent and then S1 is applied to O1 and S2 is to O2,
the resulting states of O1 and O2 are observationally equivalent, then the
implementation of a class is correct.
Ex.
S1 and S2 are equivalent:
S1: Create.Insert(5).Insert(3).Delete(5)
S2: Create.Insert(3)
Dechang et al’s Comparisons of Methods
Flow-graph-based
Advantages

Can be directly obtained from specifications which facilitates automation of flow-graph creation
and testing analysis
 Can use traditional, examined, analyzed testing techniques from class-flow graphs & controlflow graph
 Facilitates systematic test data generation
Disadvantages

Does not model class state space. Effectiveness is reduced because state is not taken into
consideration. Author also mentions that “the coverage criteria may be biased.”
 May not satisfactorily characterize a class’s behavior. It may also lead to a complete graph if
there is no restriction on the order of operations, which means that graph would not
characterize the states well.
 Further studies are needed to improve the effectiveness of flow graph in modeling class
behavior.
Graph-based
Advantages

Class state and operation interactions are modeled, which means that it is more likely to be
effective.
 It may also function as an oracle for test data
Disadvantages


The number of testgraphs needed may be infinite.
To design a “good” testgraph, a “good” characterization of class behavior is required. This
develops a “chicken-egg paradox” where you have to be able to correctly model class behavior
before testing it.
 You must develop methods to analyze class behavior first
ASTOOT
Advantages
 Randomly generated test cases
 Captures class state and state transitions
 In test case, expected result is represented concisely as a Boolean oracle
Disadvantages


A class must be self-testing, some mechanism must be provided to check against tag (method to
test equivalency)
Since randomly generated, test coverage analysis is difficult
Testing Levels for Object Oriented Software – Y Labiche, P. ThevenodFosse, H. Waeselynck, M.-H. Durand




There is the general problem of complex class dependencies
o Makes it difficult to decide where to begin testing and there is no obvious order for
integration testing (individual software modules are combined and tested as a group)
Stubs – every class not included has to be simulated, increasing costs -> minimize stubs
o Not always feasible to make a stub that is simpler than the piece of code that it
simulates
o No hope for full automation of stub creation without fully understanding the functions
to be simulated
o Bug potential for stubs may be greater than the real function
Identify testing levels early in the process
“Class Firewall, test order, and regression testing of object-oriented programs” – Kung et al
o Classes under test at a given level should only depend on previously tested classes
 Therefore, no stub is required since previously tested classes can be included in
subsequent testing levels
Kung et al’s Test Order


No abstract classes
Object Relation Diagram (ORD) – captures static dependencies (compile-time)
o The ORD for program P is an edge-labeled digraph where nodes represent the classes in
P and edges inheritance, aggregation, and association relations
 An edge labeled I from C1 to C2 indicates that C1 is a child class of C2
 An edge labeled Ag from C1 to C2 indicates that C1 is an aggregated class of C2 (C1
contains one or more objects of C2)


 An edge labeled As from C1 to C2 indicates that C1 associates with C2
Class firewall – identifies effects of a class change at class level CFW(X), for a class X, is the set of
classes that could be affected by changes to class X
o shows which classes should be retested when X is changed
 Should include child classes, aggregated classes, and classes associated with X
 Can be derived using dependence binary relation R from directed edges of ORD
Test order – finding algorithm
o A desirable test order requires minimum effort for test stubs (assuming # of stubs is a
good indicator)
o Main idea is to test independent classes first and then dependents based on class
relationships
o Acyclic ORD, use topological sort to find test order. Ensures that X is tested before all
those in CFW(X)
o Cyclic ORD
 Cluster: maximum set of nodes that are mutually reachable through relation R
 Cycle breaking: removing edge from the cluster until the graph becomes acyclic
 Algorithm:
1. Transform cyclic ORD into acyclic digraph ORD’ where nodes represent
clusters; topographical sort to produce test order for classes in clusters.
2. For each cluster, break cycle and use topological sort to produce test order
for classes in cluster. To break a cycle, temporarily remove one association
edge as association relation has the weakest coupling.
Labiche et al’s Comments & Motivation on method


Tests can be derived early because ORD is similar to models provided in the design process (i.e.
UML)
Limitations:
o Dynamic – (run-time) relationships are not taken into account
 Polymorphism (F may associate with G or H)
o Does not give explicit information on classes on which test class is dependent. That
information could allow early analysis of design.
o Some test levels become infeasible with abstract classes
New Method for Defining & Ordering Testing Levels
“TOONS” - Testing level generator for Object-OrieNted Software (even though this is the name for the
testing software, I call the actual method TOONS because “new method derived from Kunal et al’s
method“ is long)
1. Class Dependency Analysis
 2 sets of classes and a Boolean function are associated with class X
o D1(X) is the set of classes on which X depends statically
o D2(X) is the set of classes on which X depends either statically or dynamically or both
o



Bd(X) indicates whether or not X may dynamically depend on at least 1 class due to
polymorphism
D1(X): class X depends on Ck iff there is a directed path from X to Ck in ORD
o Inverse relation of CFW(X)
D2(X): if X associates with or is an aggregated class of Ck – then X may depend on all child
classes of Ck due to polymorphism
o Denoted by dotted, directed edges in ORD, making it a completed ORD or C-ORD
o X depends on Ck either statically or dynamically iff directed path from X to Ck
o If there is a cyclic relation between two or more classes, then D2(X) is the same for
all classes in the cycle
Boolean function Bd(X)
o D1(X) ⊆ D2(X)
o If D1(X) is a strict subset of D2(X), it implies that X dynamically depends on some class
in D2(X)
 Reverse not true, X may depend dynamically on classes in D2(X) even if
D1(X)=D2(X)
o Bd(X) is true iff at least one class has a dotted outgoing edge in D2(X)U{X}
2. Testing levels
 Testing level T is then described by Triplet:
o T.goal – set of classes under test
o T.need – set of classes that have to be involved in test (T.goal) and all classes on
which they are dependent according to T.type
o T.type – indicates type of dependencies (static/dynamic)
 Levels deduced from class dependency analysis:
o D1(X) gives minimum set of levels (static only)
o D2(X) gives maximum set (static and dynamic)
o Bd(X) tells you T.type
 Basically:
o Testing level for each class X where:
 T.goal is the class itself
 T.need is a set including D1(X)
and the class itself ({X}UD1(X))
 T.type is static
o Additional testing level for dynamic
dependencies (Bd(X) = 1):
 T.goal is class itself
 T.need is a set including D2(X)
and the class itself ({X}UD2(X))
 T.type is dynamic
o Testing level for cycles:
 T.goal is a set of all the classes
included in the cycle
 T.need is all the classes depended on by all classes in the cycle
 T.type is dynamic
3. Partial Ordering of Testing Levels
 Find a test order taking account of:
o Dependencies between classes (class is tested after class it depends on)
o Kind of dependencies (dynamic after static)
 Class M precedes class N if
o M.type = N.type and M.need is a subset of N.need, or
o (M.type = sta and N.type = dyn) and M.need is a subset of N.need
 Properties of partial order:
1. If X statically depends on Y, then testing level for Y sta precedes X sta
2. If X necessitates 2 levels of testing (Bd(X) = 1) then static X precedes dynamic X
3. If X necessitates testing level of type dyn & Y is a member of that testing level’s
need, then Y static testing level precedes it
4. If X & Y create dynamic testing levels and X dynamically depends on Y, then TY
(dynamic testing level of Y) precedes TX (dynamic testing level of X)
 Represented Graphically – with vertices representing testing levels; edges representing
immediate precedence
Static
testing
levels
Cycle
Dynamic
4. Adding Information to Testing Levels
 Labiche et al state that representation by triplets is poorly informative in regards to classes
involved at level T
o No compact representation really can be extremely informative
 The class under test (belonging to T.goal) at a level is denoted by #
 A class that is only a parent class (in T.need) is not instantiated, written between
parentheses
 A class can be a parent and a server class. If T.type = dyn (polymorphism) indicated by *. If
T.type=sta relations are tested by parent class instantiation, uses “+”
5. Accounting for Abstract Classes
 Cannot be instantiated
 C-ORD is modified in 2 ways:
o When role of abstract class implies instantiation, testing is infeasible and is removed
from test order graph (delete vertex, merge incoming & outgoing edges). Test is
postponed until next feasible testing level (special character # is added)
o Abstract class may be both a parent & server class (“+” & “*”). In this situation,
class loses server status and will only be instantiated with children. Only role of
abstract class is to be a parent: parentheses added to “+” & “*”
Future Work
1. Modifications to include more information
a. Semantics of class
b. Reuse of classes developed in other applications
i. Already pretty complicated representation, possibly unwieldy?
2. Practical testing at the bottom of the graph (i.e. many polymorphic and other relations) have
not been well-defined
3. Further investigation into cycle breaking
The State Based Testing of Object Oriented Programs – CD Turner & DJ
Robson
Object-Oriented method place emphasis on types of data operations will be performed on, rather than
on operations themselves
Finite State Automata


Main emphasis of state-based testing is the values stored in an object, its state.
o Draws similarities with Finite State Automata
Responses to state & stimulus combination:
1. Change object’s state to the appropriate new state
2. Leave object’s state as is
3. Change to undefined state (error)
4. Change to defined, inappropriate state (error)
5. Leave as is when it’s supposed to change (error)
(validate 1 & 2, and detect 3,4,5)
State-based Testing






Each method is considered as a mapping from its starting (input) states, to its resultant (output)
states affected by stimuli
Notation:
o σ = set of valid states for object
o ϵIi = set of states that the ith method is expected to accept (both by action or inaction)
o ϵOi = set of states that the ith method is expected to be able to generate as output
o αIi = set of states that the ith method actually accepts as input
o αOi = set of states that the ith method is actually able to generate as output
o If a class is valid (for state-based testing) then αIi = ϵIi and αOi = ϵOi for all of the
methods of the Finite State Automata
The more data members, the more time it takes to design and implement the tests
Data members are used for data-storage (storing & recalling, little processing) and controlmembers (used to trigger events).
The level of control-information and hence interaction between methods of the class also
affects the time to design & generate state-based tests
Of course, complexity of code also affects complexity of testing
Substates



Object’s state = combined values from all data members at current time
o or combination of all substates at the current time
Substate = the value of a particular data-member at a specific point in time
Partial state = combination of values of a subset of the state at one time
Substate values

Introduce 2 types of substate-values:
o Specific substate values: tested directly within code or are described in design as having
a special significance
o


General substate values: a group of substate values considered in the same manner
 No need to distinguish between for state-based testing
 Assumed that all attribute values that are part of same general group will
behave the same
 Used to significantly reduce the number of test cases
 Assumption is not used during structural testing and so the class will still
be fully tested
 A substate values are a collection of specific- and general- substate values
iValue with design stating
-1 has special meaning
Value is always >= -1
And all numbers >-1 are tereated the same
Then iValue has substate values
==-1 (specific) and >-1 (general)
Parameters can be analyzed in a similar way to substate-values.
If the data member is a pointer, then it will most likely only have 2 substate-values, one specific and
one general. The specific is probably NULL and generic for all other possibilities. It may also have
address or default object, possibly.
Performing Testing



Ii and Oi must be determined from class design, not code as code may contain errors.
Additions to Class:
o In order to determine object’s current state, a new version of the class must be added
with at least one new method per substate. These methods enable tester to inspect
substates’ values.
o If there is a change between 2 values which were both considered to be part of the
same general group, you get an undetectable change. Rectify by adding data members
to mirror original values and a feature for each mirror to check for change.
o Recommended to add statements to the beginning & end of methods to report value of
parameters passed & data returned.
Data Scenarios
o This can be used for more complex data structures, such as linked lists. For dynamic
data structures, you must know when and which changes can occur.
o 2 sets of data-scenarios:
 Scenarios used by methods that act upon elements
 Scenarios used by methods that act upon links between elements
o Used as additional test cases to appropriate groups of methods
o Additional substates are required for detecting the state of various parts of the dynamic
data structure.
Algorithm for Generation of Test Cases
1. Allocate one substate per data-member
2. Determine the data scenarios from the design of the class
3. Allocate any extra substates required for the data scenarios to function properly
4. Determine the specific and general-values for these substates. Include invalid values that are
required when dereferencing a pointer as mentioned previously.
5. Add the methods to test for substate values to the class.
6. Determine substates for which a test for a change of value is required. Add methods for
detection. Also add data-members required for monitoring.
7. From the design, determine which states are Ii and Oi for each method. Ii should be the same
because there should be no implied order for methods. These sets of states form the basis for
test cases.
8. Analyze the design for the call graph of intermethod calls within the class.
9. Start with features at bottom of call graph, especially those not called by any other method
10. For each state that the method is expected to handle as input, calculate the state that the
method should leave the object in. This is done for all significant values that can be passed as
parameters.
11. Generate code to create case’s initial state. Follow this with the test for validating starting state.
Then add code for test case, including call to test feature. Then, add a test for final state of
object.
12. Go back to 10 and continue until all features are tested.
Integration Testing


Emphasis on interaction between features of different classes, rather than within classes
o Implies that it involves creation of states for main object and objects passed as params
A test case for integration testing of A & B
1. Create starting state for object of A
2. Create starting state for object of B
3. Invoke method under test of A
4. Validate resultant state for B
5. Validate resultant state for A
Turner & Robson’s Assessment





The more features of a class interact via the data representation of a class; the more effective
state-based testing is likely to be.
If class is only repository or non-dynamic data storage, then technique will have limited effect
Number of testing cases that can be generated is related to number of different substate-values
for each substate
Not a substitute for functional, nor structural testing
o Complementary technique for testing interactions with state
Should test cases concentrate on individual substates or combinations of substates?
o Individual: number of test cases increases, easier/faster to generate when automated,
uses more resources (time/space in disk)
o Combination: difficult to generate, more thorough
o Compromise must be decided upon by tester
Miscellaneous Stuff




The state-based technique proposed by Turner and Robson is interesting. However, it only tests
to make sure that the state of the class is correct, which is important. That means that you need
to use other tests to make sure that actual methods themselves are inputting and outputting
the correct information.
I think the two techniques TOONS and State-based try to cram too much information with the
modifications near the end. Especially with the TOONS method, there is too much information
crammed into the triplets in the first place; adding in the information for abstract data types just
further complicates the graphs. This makes it difficult to understand, and adding symbols to the
already abstracted version of the C-ORD makes it worse. It is important to keep as much data as
possible to make testing more complete and easier to generate. However, it must be
compressed and presented in a fairly straight-forward, and easy to manage format, especially
when there are directional arrows already going all over the place.
The State-based method attempts to rectify the problems with only testing state by checking for
data changes near the end. They want to do this by adding more variables to check and report
problems with variables. It seems to me like these variables and there methods would also have
to be tested for accuracy, which further complicates testing.
Most of the testing for these methods can be derived from specifications or designs. This is
useful because it means that testing can be done while the designs are being implemented. A
lot of them talk about using UML diagrams to help determine which classes do what and how
methods affect those relationships.
Sources
Dechang Gu, Yin Zhong, and Sarwar Ali. 1994. On testing of classes in object-oriented programs. In
Proceedings of the 1994 conference of the Centre for Advanced Studies on Collaborative
research (CASCON '94), John Botsford, Ann Gawman, Morven Gentleman, Evelyn Kidd, Kelly
Lyons, Jacob Slonim, and Howard Johnson (Eds.). IBM Press 22-.
Labiche, Y.; Thevenod-Fosse, P.; Waeselynck, H.; Durand, M.-H.; , "Testing levels for object-oriented
software," Software Engineering, 2000. Proceedings of the 2000 International Conference
on , vol., no., pp.136-145, 2000
Turner, C.D.; Robson, D.J.; , "The state-based testing of object-oriented programs," Software
Maintenance ,1993. CSM-93, Proceedings., Conference on , vol., no., pp.302-310, 27-30 Sep
1993