Open source and empirical studies
An Empirical Study of Open-Source and Closed-Source
Software Products
by Paulson, J.W.; Succi, G.; Eberlein, A.
04.04.2007
Anna Ruokonen/TUT
Empirical studies (1)
• Empirical research
– Empirical observations or experiments are run in order to answer
particular research questions
• A research design: the logical sequence connecting the empirical data
to initial research questions and conclusions
1. The research questions (who, what, why, how, where)
2. Propositions (->what should be studied)
3. Unit(s) of analysis
4. The logic linking the data to the propositions
5. The criteria for interpreting the findings
[Jin2003]
04.04.2007
Anna Ruokonen/TUT
Empirical studies (2)
Construct validity
1. Select specific types of changes that are to be studied
2. Demonstrate that selected measures of the changes do reflect the
selected types of changes
Internal validity
•
•
Address rival explanations
Are the results due to circumstances only (the null hypotheses)?
[Jin2003]
04.04.2007
Anna Ruokonen/TUT
The Beliefs
• The 5 common beliefs of OS:
– Defects are found and fixed more rapidly in OS projects
– Hypothesis of more prevalent creativity, faster project growth, modularity
and simplicity
• The beliefs often lack empirical evidence
• Switch from CS to OS an attempt to win market share, expand
products growth, improve market penetration
• This study investigates claims of OS SW community
• Chosen metrics compare OS and CS project characteristics
04.04.2007
Anna Ruokonen/TUT
Project characteristics
Some factors affecting a software growth (OS or CS):
•
•
•
•
•
•
the application domain
the SW development process
the type of product
the management
marketing dynamics
organizational dynamics
04.04.2007
Anna Ruokonen/TUT
Analyzed projects
The most well-known and succesful OS projects
• Linux
• Apache
• GCC, GNU Compailer Collection
Three CS projects
• Representative of particular CS domain
• SW protocol stacks of wireless telecommunications products
04.04.2007
Anna Ruokonen/TUT
GROWTH
HYPOTHESIS: OS development fosters faster system growth
METRICS:
• Overall project growth in functions over time
• Overall project growth in LOC over time
FINDINGS:
• Growth rates over time were similar and constant for all analyzed
projects
• Does not support hypothesis
04.04.2007
Anna Ruokonen/TUT
Linear approximation of total project size
04.04.2007
Anna Ruokonen/TUT
SIMPLICITY
HYPOTHESIS: OS projects succeed because of their simplicity
METRICS:
• Overall project complexity
• Average complexity of all the functions
• Average complexity of the functions added
FINDINGS:
• All 3 metrics showed that OS projects are more complex
• Possible reasons: embedded SW, personal style, language
• Does not support hypothesis
04.04.2007
Anna Ruokonen/TUT
Complexity
04.04.2007
Anna Ruokonen/TUT
DEFECTS
HYPOTHESIS: OS projects have fewer defects as they are found and fixed more
rapidly
METRICS:
•
•
Functions modified over time
Modified functions as percentage of total functions
FINDINGS:
•
•
•
•
In OS more function was initially changed
In OS more defects are found and the product becomes more stable over time
OS showed significantly higher percentage of modified functions
Supports hypothesis
04.04.2007
Anna Ruokonen/TUT
Functions modified
04.04.2007
Anna Ruokonen/TUT
CREATIVITY
HYPOTHESIS: OS foster more creativity
METRICS:
• Functions added over time
FINDINGS:
• No commercial project showed any positive growth rate over time
• The growing rate of OS projects was greater than the CS projects
• Supports hypothesis
04.04.2007
Anna Ruokonen/TUT
Growing rate over time
04.04.2007
Anna Ruokonen/TUT
MODULARITY
HYPOTHESIS: OS projects are more modular
METRICS:
• Correlation between functions added and functions modified
FINDINGS:
• There was a strong corrrelation between the growing rate and changing
rate in OS projects
• No correlation in CS projects
• Does not support hypothesis
04.04.2007
Anna Ruokonen/TUT
Corrrelation between the growing rate and
changing rate
04.04.2007
Anna Ruokonen/TUT
Summary
• OS foster more creativity
• OS projects have fewer defects as they are found and fixed more
rapidly
• External global factors and feedback mechanisms have a greater
impact on the SW evolution than the development metod
• What are the global factors shared by both OS and CS projects?
• Before move to use OS development method, all of the goals to be
achieved should carefully considered
• Ensure that proper metrics are collected and monitored
04.04.2007
Anna Ruokonen/TUT
References
• [Jin2003] Case Study Research: Design and Methods Third Edition
(Applied Social Research Methods): Robert K. Yin. 2003, Sage
Publications.
04.04.2007
Anna Ruokonen/TUT