REMOS 2014
30 May 2014
Prof. Dr. Lee Sai Peck, University of Malaya
Conducting Software Engineering
Research …
 Need well thought out planning about your research
topic.
 Realistic about duration of your MSE research program

plan for 2 semesters for your research
 Focus on one research problem

Go for the depth of research
 Need also to think about validation of your research.
 Validation method chosen is dependent on the type of
research undertaken.
Prof. Dr. Lee Sai Peck, University of Malaya
Questions to be answered in your
dissertation …
•
What, precisely, is your contribution?
•
•
•
•
what questions did you answer?
why the reader should care?
what larger question your research addresses
What is your new result?
•
•
•
•
What new knowledge you have contributed
On what previous work your research is built
How is your result different from and better than the prior
work?
Why should readers believe your result?
•
•
Evaluation method used to evaluate your claim
Concrete evidence showing your result justifying your claim
Prof. Dr. Lee Sai Peck, University of Malaya
Types of Software Engineering
Research
 method/means of development (synthetic activities)
 i.e. creating and modifying software, including code, design
documents, etc
 method for analysis or evaluation (analytic activities)
 i.e. predict, determine, and estimate software properties
 design/evaluation/analysis of a particular instance
 E.g. what is the property of X of artifact/method Y?
 E.g. what is a better design, implementation, maintenance, or
adaptation for application X?
 generalization or characterization
 feasibility study or exploration
A clear statement on the specific problem you solved and how it
will scale.
Prof. Dr. Lee Sai Peck, University of Malaya
Research Results
Explanation/Examples
Method or
technique
New/better way of doing some task, such as design,
implementation, maintenance, measurement,
evaluation, etc
Qualitative/
descriptive model
e.g., a taxonomy for a problem area; architectural
style or a framework
Empirical model
E.g. empirical predictive model based on observed
data
Analytic model
E.g. structural model permitting formal analysis or
automatic manipulation
Tool or notation
Implemented tool that embodies a technique; formal
language to support a technique or model
Specific solution,
prototype, answer,
or judgment
Solution to application problem showing application
of SE principles.
Prof. Dr. Lee Sai Peck, University of Malaya
Explain your Research Result …
 Explain your result in such a way that someone else could
use your ideas.
 Precisely tell what’s new: the idea, the application of the idea,
the implementation, the analysis, etc

Example:





Is the contribution, the technique that is embedded in the
tool?
Effectiveness of the tool when compared to others
Applicability of the tool?
Does the tool simply support the main contribution, or is
itself a principal contribution?
Can the idea be applied without the tool?
Prof. Dr. Lee Sai Peck, University of Malaya
Validate your Research Result …
 Develop appropriate evidence to validate research
results by performing validation
 Show evidence that your result is valid

i.e. actually helps to solve the problem you set out to solve.
 Select a form of validation that is appropriate for the
type of research result and the method used to obtain
the result.
Prof. Dr. Lee Sai Peck, University of Malaya
Empirical Validation
Survey
•
interviews or questionnaires
Controlled Experiment
•
in the laboratory, involves manipulation of variables
Case Study
•
observational
8
Empirical Approach: Survey
 Pose questions via interviews or questionnaires
 Process
 select variables and choose sample
 frame questions that relate to variables
 collect data
 analyze and generalize from data
 Uses
 descriptive (assert characteristics)
 explanatory (assess why)
 exploratory (pre-study)
Prof. Dr. Lee Sai Peck, University of Malaya
Empirical Approach:
Controlled Experiment
 Manipulate independent variables and
measure effects on dependent variables.
 Requires randomization over subjects and
objects.
 Relies on controlled environment (fix
factors not being manipulated).
 Often involves a baseline (control group)
 Supports use of statistical analysis
Prof. Dr. Lee Sai Peck, University of Malaya
Empirical Approach: Case
Study
 Study a phenomenon (process, technique, device) in a specific setting
 A certain attribute is studied (e.g. reliability, cost)
 Data is collected over time to measure the attribute
 Easier to plan than controlled experiments
 Can involve comparison between projects to build a baseline
 E.g. as the organization’s standard process for software development
 Uses include larger investigations such as:
 industrial study
 longitudinal study: a correlational research study involving repeated
observations of the same variables over long periods of time
 psychology - study developmental trends across life span
 sociology - study life events throughout lifetimes/generations.
Prof. Dr. Lee Sai Peck, University of Malaya
Types of Software
Engineering Validations [Shaw 2003]
 Analysis
 Evaluation
 Experience
 Example
 Persuasion
 Assertion
Prof. Dr. Lee Sai Peck, University of Malaya
Type of Validation – Analysis
 Static analysis (classified under Historical method by
[Zelkowitz 1997])
 E.g. used in software complexity and data flow research.
 analyze the structure of the completed product to
determine characteristics about it.

E.g. examine the product to learn if its complexity value is
lower because of the development method used.
Prof. Dr. Lee Sai Peck, University of Malaya
Type of Validation – Analysis
(Cont…)
 Rigorous analysis


For a formal model, perform rigorous derivation and proof
 Formal analysis can only be applied to very limited domain
 A large portion of the SE techniques and practices cannot be
formalized
For an empirical model, rigorous analysis of data on use in
controlled situation
Prof. Dr. Lee Sai Peck, University of Malaya
Type of Validation – Analysis
(Cont…)

For a controlled experiment, develop a rigorous experiment
design with statistically significant results
 Disadvantage:
 Deficiency in the experiment design could make the whole
experiment useless
 Collecting data that is statistically significant costs a lot of
money, time and other resources.
Prof. Dr. Lee Sai Peck, University of Malaya
Type of Validation – Analysis (Cont…)
 For a controlled experiment, explain the
experimental design:








Hypothesis: null & alternative/research hypothesis
Treatments
Subjects and objects
what is being controlled
How data is collected
how to ensure internal validity?
how it was analyzed
significance of results
whether the conclusions follows rigorously from the
experimental data
Prof. Dr. Lee Sai Peck, University of Malaya
Type of Validation – Analysis
(Cont…)
 Validity of the experiment: both internal and external


Internal validity: extent to which effects of extraneous
variables are controlled
 E.g. Subject variety, environment effects, history effects,
selection bias, Hawthorne effects, Experimenter effects &
instrument effects.
External validity: extent to which the results of an experiment
can be generalized to different setting, persons and times.
 E.g. can the result be generalized to a large population?
 Can the result be generalized to other software maintenance
problems?
Prof. Dr. Lee Sai Peck, University of Malaya
Sample
Experimental Evaluation Setup
18
Sample
19
Sample
20
21
Type of Validation – Analysis
(Cont…)
 Dynamic analysis (specific case of controlled method)
 A product is either modified/executed under carefully
controlled situations in order to extract information on
using the product.

Techniques that employ scripts of specific scenarios or which
modify the source program of the product itself in order to be
able to extract information while the program executes.
 Benchmarking occurs when a common script is used to
evaluate several products that have the same
functionality in order to compare the performance of
each.
Prof. Dr. Lee Sai Peck, University of Malaya
Type of Validation – Analysis
(Cont…)
 Simulation (specific case of controlled method)
 evaluate a technology by executing the product using a
model of the real environment.
 Used to predict how the real environment will react to
the new technology.
 Involve modeling the behavior of the environment for
certain variables, and ignore other harder-to-obtain
variables
 Using a simulated environment is often easier, faster,
and less expensive to run to obtain results than the full
product in the real environment (i.e., the dynamic
analysis method).
Prof. Dr. Lee Sai Peck, University of Malaya
Type of Validation - Evaluation
 Evaluate based on stated criteria to judge a technique



For a descriptive model, it adequately describes phenomena of
interest
For a qualitative model, it accounts for the phenomena of
interest
For an empirical model, it is able to predict … or generates
results that fit actual data.
 However some evaluations are subjective, hence lack
of scientific rigor.
 Successful evaluation are usually using small sample
spaces.
 Very hard to generalize the results to larger domains.
Prof. Dr. Lee Sai Peck, University of Malaya
Type of Validation - Experience
 Demonstrate that the research result has been used on real
examples by someone.
 Often measure benefits of applying a certain technique in
practice to justify it.
 Results come from real-world settings, hence are more convincing.
 Evidence of its correctness/usefulness/effectiveness is:
 shown narratively for a qualitative model
 shown usually in the form of statistical data, or on practice, for an
empirical model, or tool
 the comparison of systems in actual use, for a notation/technique.
 Lessons-learned documents can be produced after a large
industrial project is completed.
 can be used to improve future developments.
Prof. Dr. Lee Sai Peck, University of Malaya
Type of Validation - Example
 For a technique or procedure, convincingly show how
it works on …
 a “slice of life” example based on a real system,
accompanied by explanation of why the simplified
example retains the essence of the problem being
solved.
 a system being developed.
Prof. Dr. Lee Sai Peck, University of Malaya
Type of Validation - Persuasion
 Validation purely by persuasion is rarely sufficient
 Highly biased, can hardly be called a validation at all.
 Arguments based on common sense or intuition.

OK for motivating research ideas or help focus on the central
idea of a technique at early stage.
 Example:
 For a technique, recommend to do a certain way
 For a system, recommend to construct a system in a
certain way
 For a model, give an example showing how the idea
works
Prof. Dr. Lee Sai Peck, University of Malaya
Type of Validation - Assertion
 No serious attempt to evaluate the research result
 Highly unlikely to be acceptable, potentially biased
 E.g. developer of a technology tends to show the newly
developed technology is more effective or superior.
 Sometimes this may be a preliminary test before a
more formal validation of the effectiveness of the
technology.
 Classified under Observational method by [Zelkowitz
1997] together with project monitoring, case study and
field study.
Prof. Dr. Lee Sai Peck, University of Malaya
Concluding Remark
 Triangulation method can be used to improve the reliability of
the research, esp when the sample size is small.
 In a triangulation technique/method, a phenomenon is viewed
from different perspectives, with each perspective detected by
using a different method.
 Each method is directed to the same destination and focuses on
the same event or phenomenon.
 To study a phenomenon (e.g. to measure one concept, i.e. a
characteristic of the phenomenon), data can be collected
through 3 different aspects, which are:
 Time – data is collected at different times
 Space – data is collected in different settings and locations
 Individual – data is collected by different individuals.
Prof. Dr. Lee Sai Peck, University of Malaya
References









M. Shaw. Writing Good Software Engineering Research Papers. Proceedings of the 25th
International Conference on Software Engineering, IEEE Computer Society, 2003, pp.
726-736.
M. V. Zelkowitzayb, D. Wallace. Experimental validation in software engineering.
Information and Software Technology 39 (1997) 735-743.
J. Hu. What We Really Should Do in Software Engineering Research Validation. Project
Report of 15-839 Spring 2000.
Aarom Sloman, “Types of research in computing science, software engineering and
artificial intelligence”, School of Computer Science, University of Birmingham.
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/cs-research.html
Chris Johnson, “Basic Research Skills in Computing Science”, Department of Computer
Science, Glasgow University, Glasgow, G12 8QQ.
Chris Johnson, “What is Research in Computing Science?”, Department of Computer
Science, Glasgow University, Glasgow, G12 8QQ.
'Tunj O. DEJOBI, “Research Methodology in Computer Science & Engineering”
Chua Yan Piaw, “Mastering Research Methods”, McGraw Hill, 2012.
C. Wohlin et al., Experimentation in Software Engineering, DOI 10.1007/978-3-64229044-2 7, © Springer-Verlag Berlin Heidelberg 2012.
Prof. Dr. Lee Sai Peck, University of Malaya