David Janzen
PhD Candidate
University of Kansas
Lawrence, KS USA
Software Architecture Improvement through Test-Driven Development
4
Solution: Conduct empirical studies that focus on design and internal quality
• Conduct formal experiments in academic and professional settings.
• Isolate TDD as independent variable.
• Experiment group develops software with Test-First approach.
• Control group develops software with Test-Last approach.
Iterative
Iterative
vs.
Test-First
Test-Last
• Does TDD produce code with better internal quality (i.e. better designs)?
•
•
Is code simpler, easier to reuse, easier to maintain?
Does code have lower complexity, smaller classes/methods, lower coupling?
• Does TDD produce code with better external quality (i.e. fewer defects)?
• Does TDD take longer to develop the same functionality?
2
TDD Instruction
Problem 2: How do we teach TDD?
Project Phase 1
Test-First
CS1
Test-Driven
Learning
Test-Last
CS2
From the beginning (CS1)?
After students know some basics (CS2)?
Once students have some maturity (Software Engineering)?
In the field (Industry training)?
3
Author(Year)
Programmers
54% fewer defects
n/a
Kaufmann(2003)
8
improved information flow
50% improvement
Muller(2002)
19
no change, but better reuse no change
Pancur(2003)
38
no change
no change
Erdogmus(2005)
35
no change
improved productivity
Author(Year)
Quality Effects
Productivity
Individual
Profile
Features Completed
Intermediate
metrics
Final
metrics
14
4
Software Metrics
2
0
Test-First
5.00
4.50
4.00
3.50
4.00
3.00
3.00
2.00
2.00
1.50
0.50
0.00
No-Tests
Test-Last
Tested code was simpler
Test-Last
Test-First
No-Tests
600
400
200
0
Test-Last
506
182
Test-First
No-Tests
Test-Last
Student Perceptions
1.00
1.00
No-Tests
1000
800
•Test-First spent 88% less effort/feature than No-Tests
•Test-First spent 57% less effort/feature than Test-Last
•Only Test-First completed both phases
Coupling between Objects
5.00
1424
1400
1200
6
6.00
Measure external
quality with
defect density.
1600
12
8
7.00
Test-First
Effort Per Feature
10
Cyclomatic Complexity
Test-Last
Higher coupling in Test-First, but…
Test-First
Test-Last
Opinion of Test-Last (Pre-Experiment)
Tested vs. Untested Code in Test-First Project
Information Flow/module
5.00
3.00
4.50
2.56
2.50
4.00
3.50
2.00
2.50
5.00
4.00
3.00
2.00
1.00
0.00
3.00
SE
(grad)
TDD Training
1.00
Untested Code
Test-First
Test-Last
Productivity Effects
Companies Programmers
George(2004)
3
24
TDD passed 18% more tests
TDD took 16% longer
Maximilien(2003)
1
9
50% reduction in defect density
minimal impact
Williams(2003)
1
9
40% reduction in defect density
no change
Related experiments focus on external quality (defect density),
and indicate potential benefits of TDD with negligible productivity effects.
Test-Driven Learning (TDL) integrates automated unit-tests into
traditional pedagogical examples, thus modeling good testing
throughout the curriculum.
Traditional Example
TDL Example
#include <iostream>
using namespace std;
int sum(int min, int max) {
int sum = 0;
for(int i=min;i<=max;i++)
sum += i;
return sum;
}
int main() {
cout << sum(3,7) << endl;
// above should print 25
}
#include <cassert>
int sum(int min, int max) {
int sum = 0;
for(int i=min;i<=max;i++)
sum += i;
return sum;
}
int main() {
assert(sum(3,7)==25);
}
Test-Last
0.50
Complexity
0 Information Flow indicates
procedural design in NoTests and Test-Last teams
All academic programmers will develop the same application within a class.
Industry programmers will complete different projects so we will compare
Test-First projects with previous Test-Last projects by same team.
•
•
•
•
•
•
•
No-Tests
Test-Last
0.00
Test-First
Test-First
6
0.00
No-Tests
Team
1.00
0.00
0.00
Test-Last
Test-First
1.50
0.50
Test-First
DEFECTS
# of
Team 3: Test-Last
Test-First
Productivity Effects
59
# of
Team 2: No-Tests
Individual
Profile
Measure internal
design quality with
size, complexity,
structural, and
OO metrics.
Post-experiment
survey
2.00
Edwards(2003)
Study
Team 1: Test-First
TDD
Training
Pre-experiment
survey
0.00
Industry
TDD in Industry
•Undergraduate Software Engineering survey course
•Semester-long team programming project
•Phase 1: text UI, Phase 2: Graphical UI
•Summer 2005
Tested Code
TDD in Academia
Quality Effects
Context
HTML Pretty Print Project
1.50
Related Work: Focus on defects and external quality
# of
Measure reuse
From one phase/
Iteration to another.
Test-Last
SE
(undergrad)
Test-Driven Learning (TDL) is an approach to teach with tests
• Is TDL effective?
Study
Experiment Design
2.50
• Can tests be integrated throughout the curriculum?
•
Initial Results: Experiment in Undergraduate Software Engineering Course
Measure productivity
with total hours and
hours per feature.
Test-First
• Can undergraduates learn TDD?
• Where is the optimal level in the curriculum to teach TDD?
•
•
•
•
Project Phase 2
5
Minutes
Problem 1: Does TDD improve Software Quality?
Favor -
1
Coupling
Code covered by tests had
lower complexity and
coupling in Test-First team
Testing Metrics
Test LOC % Classes Tested
Test-First
168
Assertions/SLOC
38.46%
0.086
Test Coverage (lines)
31.00%
Opinions of TF improved
Test Coverage (branches)
43.00%
No-Tests
0
0.00%
0.000
0.00%
0.00%
Test-Last
38
25.00%
0.045
29.00%
23.00%
Threats to Validity
•Testing metrics from Phase 1 (text UI)
•Test-First wrote more tests and covered more code with tests
•No-Tests team was assigned to be Test-Last but “ran out of time
for tests”
Conclusions
Test-First team was more productive
Test-First team produced more Object-Oriented Design
Test-First team wrote more tests
Tested code was less complex and more decoupled
Test-Last team produced more decoupled code
Test-First perceptions improved with use while Test-Last perceptions worsened
Additional industry and academic studies are warranted
Opinions of TL declined
•Small sample size (ten students)
•Test-First team had slightly more Java experience
•Majority of core code implemented by one developer
on each team
7
You can contribute
Industry Professionals
Academic Scholars
•Contribute project data
•Email [email protected]
•Adopt TDL
•Replicate studies
8
Acknowledgements
•Dr. Hossein Saiedian, advisor