White-Box Testing
( “FÅP”: First-year Project Course, ITU, Denmark )
Claus Brabrand
[ [email protected] ]
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
Feb 26, 2008
Outline

Motivation


What is Testing?


Methodology: ”coverage testing”
Automation


Relation to other programming-related tasks?
White-box Testing


Why bother with testing?
How to automate testing
Exercises

Training for the exam
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[2]
Feb 26, 2008
Learning & Exam Goals

”Product”:

”Oral Exam”:
 Today, we’ll ”train for the exam”… :-)
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[3]
Feb 26, 2008
Software Errors

Mobile Phones

’00-…


Cruise Control System Model

’86 (Grady Booch)



Accellerated after car ignition  car crashes
Baggage Handling System

’94-’95 (at Denver Int’l Airport)

Claus Brabrand, ITU, Denmark
Freeze and odd behaviors (really annoying)!
$ 360,000,000 USD
WHITE-BOX TESTING
[4]
Feb 26, 2008
Software Errors (cont’d)
z
zzz

Train Control System

’98 (Berlin)


Mars Pathfinder

July ’97


Periodic resets ...on mars!
Win95/98 w/ 3rd-Party Device Drivers

late ’90es

Claus Brabrand, ITU, Denmark
Train cancellations
Dysfunction (“blue screen of death”)!
WHITE-BOX TESTING
[5]
Feb 26, 2008
Software Errors (cont’d2)

Therac-25 Radiation Therapy

’85-’87


Patriot Missile Guidance System

’91 (Gulf War 1.0)


Accumulating rounding errors  deaths
Ariane V

’96 (one of the most expensive bugs, ever)

Claus Brabrand, ITU, Denmark
Massive overdoses (6 deaths / amputations)!
Conversion from 64-bit float to 16-bit signed int
WHITE-BOX TESTING
[6]
Feb 26, 2008
…and what about?!


Surgical Laser Control System


Air Plane Control System


Dysfunction (plane crash)!
Nuclear Powerplant Control System

Claus Brabrand, ITU, Denmark
Oops…!
Core melt-down (“China-syndrome”)!
WHITE-BOX TESTING
[7]
Feb 26, 2008
Outline

Motivation


What is Testing?


Methodology: ”coverage testing”
Automation


Relation to other programming-related tasks?
White-box Testing


Why bother with testing?
How to automate testing
Exercises

Training for the exam
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[8]
Feb 26, 2008
Errors!

Syntactic errors:


(different kinds)
Mal-formed program:
Semantic errors:
int square(int x) {
return x*x
*** syntax error
}
’;’ expected
at line 2
int square(int x) {
return n*n;
*** symbol error at line 2
}
undefined variable ”n”
Symbol errors
 Type errors
int square(float x) {
 Other semantic errors:
return x*x;
error at line 2
(e.g. uninitialized vars) }*** type
function returns float,


Logical errors:

Compiler: ”no errors”
Claus Brabrand, ITU, Denmark
int square(int x) {
return x+x;
}
no errors
WHITE-BOX TESTING
[9]
not int
found!!!
Feb 26, 2008
Int’l standard for evaluation of:
Software Quality
ISO 9126
How easy is the SW to understand?
…and use?
Claus Brabrand, ITU, Denmark
ISO
How robust is the SW wrt. incorrect
9126 inputs,
’^C’, external netwk failures, ...?
WHITE-BOX TESTING
[ 10 ]
Feb 26, 2008
Testing vs. Debugging?

Testing vs. Debugging:
Regarding:
Functionality:
Efficiency:
Quality Assurance:
(Functionality)
Testing
(Performance)
Testing
Diagnosis:
Debugging
Profiling
Purpose:
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 11 ]
Feb 26, 2008
Testing vs. Debugging (cont’d)
Program:
0110
1021
(confidence?!?)
SYSTEMATIC
Quality assurance:
Functionality
Testing
Is program ok?
0110
1011
Fix problem
(reprogram)
Evaluate test results
Diagnosis:
Debugging


0110
1011
Document test results
(greater confidence!)
Determine problem?
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 12 ]
Feb 26, 2008
Performance Testing vs. Profiling
Program:
0110
1021
(confidence?!?)
SYSTEMATIC
Quality assurance:
Performance
Testing
Efficient enough?
Improve program
0110 (reprogram)
1011
Evaluate test results
Diagnosis:
Profiling


0110
1011
Document test results
(greater confidence!)
Determine problem?
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 13 ]
Feb 26, 2008
Testing…:

”Testing is easy”
(e.g., ”random experimentation”)

”Testing well is not easy”
 Requires SYSTEMATIC approach; test-case
 production
 evaluation
 documentation

”Testing can never prove error absence”
(i.e., testing is an ”incomplete process”)
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 14 ]
Feb 26, 2008
Appropriate Test Sampling?

Representative?

Comprehensive?

Quality?

Quantity?

…?
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING




[ 15 ]
Feb 26, 2008
White-box vs. Black-box Test

White-box Testing:



(aka., ”structural testing”)
(aka., ”internal testing”)
Test focus:

Black-box Testing:




(aka., ”behavioral testing”)
(aka., ”external testing”)
Test focus:
source code

specification (manual)
n = in();
?
odd(n)
tt
n = n/2;
~
ff
n = 3*n+1;
program
out(n);
spec
Complementary Approaches!!!
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 16 ]
Feb 26, 2008
”Software Testing” (R. Patton)

Background reading:
”Software Testing”, Ron Patton, Sams Publishing, 2006
 Part II (pp. 53 – 123); ”Testing Fundamentals”:

Type:
-testing
-testing
Static
(before runtime)
”Examining
the Code”
(chapter 6)
”Examining
the Spec.”
(chapter 4)
Dynamic
(at runtime)
”Testing w/
X-ray Glasses”
(chapter 7)
”Testing w/
Blinders On”
(chapter 5)
Time:
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 17 ]
Feb 26, 2008
Outline

Motivation


What is Testing?


Methodology: ”coverage testing”
Automation


Relation to other programming-related tasks?
White-box Testing


Why bother with testing?
How to automate testing
Exercises

Training for the exam
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 18 ]
Feb 26, 2008
Test Coverage?

Method coverage:


Statement coverage:


Does every statement run (at least once)?
Branch coverage:


Does every method run (at least once)?
Does every branch run (at least once)?
Path coverage:

Does every path run (at least once)?
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 19 ]
Feb 26, 2008
Statement coverage

Branch coverage:


Does every branch run (at least once)?
-Box ”Branch Coverage Testing” is:
Efficient (fast) !
 Effective (thorough) !


Claus Brabrand, ITU, Denmark
Good for complicated program logic
(esp. ”initialization errors”)
WHITE-BOX TESTING
[ 20 ]
Feb 26, 2008
Control Structures

Control Structures:


Statements (or Expr’s) that affect ”flow of control”:
if-else:
true
[syntax] if ( Exp ) Stm1 else Stm2
if:
Exp
false
Stm
[semantics]  The expression must be of type boolean;
if it evaluates to true, the given statement
is executed, otherwise not.
Claus Brabrand, ITU, Denmark
confluence
true
[syntax] if ( Exp ) Stm
false
Stm2
Stm1
[semantics]  The expression must be of type boolean; if
it evaluates to true, Statement-1 is executed,
otherwise Statement-2 is executed.

Exp
WHITE-BOX TESTING
confluence
[ 21 ]
Feb 26, 2008
Control Structures (cont’d)

confluence
while:
[syntax]  while ( Exp ) Stm
[semantics]  The expression must be of type boolean;
if it evaluates to false, the given statement is
skipped, otherwise it is executed and
afterwards the expression is evaluated
again. If it is still true, the statement is
executed again. This is continued until the
expression evaluates to false.

Exp
true
false
Stm
Exp1;
for:
confluence
[syntax] for (Exp1 ; Exp2 ; Exp3) Stm
Exp2
false
Stm
[semantics]  Equivalent to:
{ Exp1;
while ( Exp2 ) { Stm
}
Claus Brabrand, ITU, Denmark
true
Exp3; }
WHITE-BOX TESTING
Exp3;
[ 22 ]
Feb 26, 2008
Stm/Branch Coverage Testing

if:


if-else:


TEST condition true and false
while:


TEST condition true and false
TEST zero, one, more-than-one iterations in loop
for:

TEST zero, one, more-than-one iterations in loop
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 23 ]
Feb 26, 2008
Example 1
Choice points?
public static void main ( String[] args ) {
int mi, ma;
if (args.length == 0)
/* 1if-else
System.out.println("No numbers");
else {
mi = ma = Integer.parseInt(args[0]);
for (int i=1; i < args.length; i++) { /* 2for
int obs = Integer.parseInt(args[i]);
if (obs > ma)
/* 3if-else
ma = obs;
else
if (mi < obs) mi = obs;
/* 4if
}
System.out.println(”min=" + mi + "," +
"max=" + ma);
}}
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 24 ]
*/
*/
*/
*/
Feb 26, 2008
Control-Flow Graph

int mi, ma;
CFG:
true
args.length
1 == 0
System.out.println
("No numbers");
false
mi = ma = Integer.parseInt(args[0]);
int i=1;
true
i < args.length
2
false
int obs = Integer.parseInt(args[i]);
obs 3> ma
true
ma = obs;
true
false
mi <4 obs
mi = obs;
false
i++;
System.out.println(”min=" + mi + "," + "max=" + ma);
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 25 ]
Feb 26, 2008
Coverage Table

”Coverage Table”:
Choice
1ife true
false
2for zero-times
once
more-than-once
3ife true
false
4if true
false
Claus Brabrand, ITU, Denmark
Input property
Data set
No numbers
A
At least one number
B
Exactly one number
B
Exactly two numbers
C
At least three numbers
E
N > current max
C
N  current max
D
N  cur max & N > cur min E (3rd num)
N  cur max & N  cur min E (2nd num)
WHITE-BOX TESTING
[ 26 ]
Feb 26, 2008
Expectancy Table
 ”Expectancy Table”:
Data set Input
Expected output
A
”no numbers”

B
[17]
”min=17,max=17”
C
[27,29]
”min=27,max=29”
D
[39,37]
”min=37,max=39”
E
[49,47,48] ”min=47,max=49”
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
=





Actual output
”no numbers”
”min=17,max=17”
”min=27,max=29”
”min=39,max=39”
”min=49,max=49”
[ 27 ]
Feb 26, 2008
Debugging ’ D ’
then reveals…
public static void main ( String[] args ) {
int mi, ma;
if (args.length == 0)
/* 1if-else
System.out.println("No numbers");
else {
mi = ma = Integer.parseInt(args[0]);
for (int i=1; i < args.length; i++) { /* 2for
int obs = Integer.parseInt(args[i]);
if (obs > ma)
/* 3if-else
ma = obs;
else
if (mi < obs) mi = obs;
/* 4if
}
System.out.println(”min=" + mi + "," +
"max=" + ma);
}}
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 28 ]
*/
*/
*/
*/
Feb 26, 2008
Re-Test !

…as debugging often
introduces new errors !
Fixed Program:
Coverage Table:
Expectancy Table:

Recall: no
guarantee!
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 29 ]
Feb 26, 2008
Example 2
public static void main ( String[] args ) {
int mi1 = 0, mi2 = 0;
if (args.length == 0)
/*
System.out.println("No numbers");
else {
mi1 = Integer.parseInt(args[0]);
if (args.length == 1)
/*
System.out.println("Smallest = " + mi1);
else {
int obs = Integer.parseInt(args[1]);
if (obs < mi1)
{ mi2 = mi1; mi1 = obs; }
for (int i = 2; i < args.length; i++) {
obs = Integer.parseInt(args[i]);
if (obs < mi1)
/*
{ mi2 = mi1; mi1 = obs; }
else
if (obs < mi2)
mi2 = obs;
}
System.out.println("The two smallest are: " +
mi1 + " and " + mi2);
}
}
}
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
1if-else */
2if-else */
/* 3if */
/* 4for */
5if-else */
/* 6if */
[ 30 ]
Feb 26, 2008
Coverage Table (Ex. 2)
Choice
1ife true
1ife false
2ife true
2ife false
3if true
3if false
4for zero-times
4for once
4for more-than-once
5ife true
5ife false
6if true
6if false
Claus Brabrand, ITU, Denmark
Input property
Data set
No numbers
A
At least one number
B
Exactly one number
B
At least two numbers
C
2nd number ≥ 1st number
C
2nd number < 1st number
D
Exactly two numbers
D
Exactly three numbers
E
At least four numbers
H
3rd number < current min
E
3rd number ≥ current min
F
3rd ≥ cur min & 3rd < 2nd least
F
3rd ≥ cur min & 3rd ≥ 2nd least
G
WHITE-BOX TESTING
[ 31 ]
Feb 26, 2008
Expectancy Table (Ex. 2)
Data set Input
A

B
[17]
C
[27,29]
D
[39,37]
E
[49,48,47]
F
[59,57,58]
G
[67,68,69]
H
[77,78,79,76]

Expected output
”no numbers”
”17”
”27 and 29”
”37 and 39”
”47 and 48”
”57 and 58”
”67 and 68”
”76 and 77”
=








Actual output
”no numbers”
”17”
”27 and 0”
”37 and 39”
”47 and 48”
”57 and 58”
”67 and 0”
”76 and 77”
Debugging reveals that variable
”mi2” erroneously retains initialization (0).
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 32 ]
Feb 26, 2008
Debugging (Ex. 2)
public static void main ( String[] args ) {
int mi1 = 0, mi2 = 0;
if (args.length == 0)
/* 1if-else */
System.out.println("No numbers");
else {
mi1 = Integer.parseInt(args[0]);
if (args.length == 1)
/* 2if-else */
System.out.println("Smallest = " + mi1);
else {
int obs = Integer.parseInt(args[1]);
mi2 = obs; /* 3 */
if (obs < mi1)
if
{ mi2 = mi1; mi1 = obs; }
for (int i = 2; i < args.length; i++) {
/* 4for */
obs = Integer.parseInt(args[i]);
if (obs < mi1)
/* 5if-else */
{ mi2 = mi1; mi1 = obs; }
else
if (obs < mi2)
/* 6if */
mi2 = obs;
}
System.out.println("The two smallest are: " +
mi1 + " and " + mi2);
}
}
}
Re-Test: 
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 33 ]
Feb 26, 2008
Control Structures (cont’d2)

switch:


Choice points?
do Stm while ( Exp );
Exp1 ? Exp2 : Exp3
”&&”; ”lazy conjunction” (aka., ”short-cut ”):


default : Stm* break;
”?:”; ”conditional expression”:


switch ( Exp ) { Swb* }
do-while:


Swb:
case Exp : Stm* break;
Exp1 && Exp2
”||”; ”lazy disjunction” (aka., ”short-cut ”):

Exp1 || Exp2
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 34 ]
Feb 26, 2008
Control Structures (cont’d3)

try-catch-finally (exceptions):


return / break / continue:


return Exp ;
break ;
continue ;
e.g.; f(x)
”recursive method invocation”:


return ;
”method invocation”:


try Stm1 catch ( Exp ) Stm2 finally Stm3
e.g.;
f(x)
”virtual dispatching”:

e.g.;
Claus Brabrand, ITU, Denmark
f(x)
WHITE-BOX TESTING
[ 35 ]
Feb 26, 2008
Outline

Motivation


What is Testing?


Methodology: ”coverage testing”
Automation


Relation to other programming-related tasks?
White-box Testing


Why bother with testing?
How to automate testing
Exercises

Training for the exam
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 36 ]
Feb 26, 2008
Test Automation

(Re-)Running tests is boooring (& error prone)


Thus, automate them ”once-and-for-all”
JUnit:
 public class MyTestCase extends TestCase {
@Test // Testing if 3*2=6:
public void testMultiplication() {
assertEquals("Multiplication", 6, 3*2);
}
/* ...other tests... */
}

Can be run from Eclipse/JUnit:

Claus Brabrand, ITU, Denmark
(if appropriately ”subclassing” TestCase)
WHITE-BOX TESTING
[ 37 ]
Feb 26, 2008
Outline

Motivation


What is Testing?


Methodology: ”coverage testing”
Automation


Relation to other programming-related tasks?
White-box Testing


Why bother with testing?
How to automate testing
Exercises

Training for the exam
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 38 ]
Feb 26, 2008
Exercise:

Part I:
Warm up exercise:
Draw a control-flow diagram
for the ”do-while” construction
 Program merge (in Java):
List<Integer> merge(List<Integer> list1,
List<Integer> list2);
(produce)
 Test your merge method:
a) Label choice points
b) Build ”coverage table” & make data set (test suite)
c) Build ”expectancy table”
d) Run test suite (upon failure: fix and retest program)
 Introduce subtle bug
 Run test to document presence of bug
 Submit “erroneous program” to class program pool

Part II:
(consume)
Claus Brabrand, ITU, Denmark




Pick “erroneous program” from class program pool
Test merge program (and debug to find bug)
Re-Test fixed merge program
Write report (and send it to the teaching assistant)
WHITE-BOX TESTING
[ 39 ]
Feb 26, 2008
Specification (merge)

Interface (for ”merge”):
 List<Integer> merge(List<Integer> list1,
List<Integer> list2);

I/O assumptions:

Input: both lists are sorted:


Output: list must be sorted:


…and, numbers occur maximum once (in each list)
…and, numbers occur maximum once
Programming constraints:
no recursion !
 no java.util.Iterator’s !

Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 40 ]
Feb 26, 2008
The Testing Report

Report (ca. 3 pages):

Must explain how you tested, (debugged), & re-tested
the erroneous ”merge” (it must include, at least):





i) The erroneous program
ii) a ”Control-Flow Graph” for the program [hand-drawn ok]
(incl. labelled ”choice points”)
iii) ”Coverage Tables”
iv) ”Expectancy Tables”
Submit the testing report to the T.A. (Anders)
(deadline: Tuesday; March, 11 at 09:00 CET)
 Note: the report is only on Part II


Claus Brabrand, ITU, Denmark
(i.e., the program you didn’t write)
WHITE-BOX TESTING
[ 41 ]
Feb 26, 2008
Example 1
public static void main ( String[] args ) {
int mi, ma;
if (args.length == 0)
System.out.println("No numbers");
else {
mi = ma = Integer.parseInt(args[0]);
for (int i=1; i < args.length; i++) {
int obs = Integer.parseInt(args[i]);
if (obs > ma)
ma = obs;
else
if (mi < obs) mi = obs;
}
System.out.println(”min=" + mi + "," +
"max=" + ma);
}}
Claus Brabrand, ITU, Denmark
WHITE-BOX TESTING
[ 42 ]
Feb 26, 2008