CE902 Lecture 1:
Research Methods and Experimental Design
Annie Louis
January 16, 2017
CE902 Professional Practice and Research Methodology, Spring 2017
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
CE902-7-FY: Course basics
Instructor. Annie Louis, 1NW.3.19, [email protected]
Time/Location. EBS.2.34, Monday 3 to 5 pm
Course content. Research methodology, philosophy of science,
basic statistics, practical research tips, writing and presentation
skills
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Course structure
I
5 lectures covering weeks 16 to 20
I
A progress test in week 21 (7% of the module grade)
I
A project proposal due end of term (40%) [Includes classroom
exercises]
I
An exam (30%)
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
What you should be doing starting today
Work on your project and proposal!
Every week, you should be spending at least 4 hours towards your
proposal report. That amount of work should be evident from your
report.
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
In this lecture, you will learn
I
What is a scientific question?
I
What methods are commonly used for answering scientific
questions?
I
What errors and other factors influence these methods? How
to deal with these issues?
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
What is science?
Knowledge or a system of knowledge covering general
truths or the operation of general laws especially as
obtained and tested through scientific method
- Merriam Webster dictionary of English
I
What types of knowledge?
I
What sorts of methods?
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Scientists often work with testable questions
Testable questions can be answered with specific activities
I
Experiments, analyses, measurements
‘Is the sky more blue on a clear summer than on a winter day?’
I
Take digital photographs of the sky at noon on many summer
and winter days, keeping a number of things constant (which
camera you use, location, humidity, ...)
I
Compare the pixels
Not testable
I
How can I grow the biggest possible tomatoes?
I
Why is the sky blue?
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Research methods in science: A couple of directions
I
Hypothesis-driven research
I
Measure a value
I
Measure a function or relationship
I
Construct a model
I
Observational and exploratory research
I
Industrial and applied research
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
1. Hypothesis-driven research
A Hypothesis is a statement about the world, whose truth is being
tested
Briar’s Aspirin cures headaches faster than RCS Aspirin
Eating two ounces of olive oil a day decreases the odds of
contracting heart disease
Water boils at 100 degrees Celsius under the pressure of
1 atmosphere
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
A hypothesis may be true or false but must be valid
Be a precise statement and be testable
I
What is the best fertilizer to use to get large and tasty
tomatoes? – not a statement
I
If electrons were 10% less massive, no life would exist – not
testable for all forms of life
Must be falsifiable (Popper 1959)
I
You can imagine evidence for the hypothesis being false
I
Macs are better than PCs cant give enough evidence for an
enthusiastic PC user
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Conducting hypothesis-driven research
1. Frame a hypothesis
2. Design and conduct experiments to test the hypothesis
3. See if the hypothesis was correct or disproved
[More about the method later]
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Appropriate when you have a small number of cases to
decide between
Is medical treatment A more effective than B for arthritis?
Do electric power lines cause cancer?
Is this computer program more efficient/accurate for my task?
Not suitable for a vast number of possibilities
I What is the density of this rock? – in a continuous range of
values
I
I
Picking a single value for a hypothesis seems almost impossible
to get right
No use doing density is 0.1gm/cm3, density is 0.2 gm/cm3,
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
2. Measure a value
A number or a few numbers
I
Important in itself or as a starting point for further research
I
Sometimes measurements are long-term expensive projects
involving thousands of scientists
Distance of the nearest star to the Earth (other than the Sun)
I
Proxima Centauri is 4.22 light years or 39,900,000,000,000 km
away
Mass of the electron neutrino
I
Almost invisible to us. Current experiment in Germany has a
budget of 33 million Euros
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
3. Measure a function or relationship
“Measure the stiffness of a sample of rubber a function of the
amount of cross-linking agent used to process it”
Between control and response variables
I Response variable: the quantity which is measured as the
outcome
I
I
Control variable: the quantity the experimenter wants to vary
and know how the changes affect the response variable
I
I
Stiffness of the sample of rubber
The amount of cross linking agent used
Other variables should be kept constant
I
Temperature, pressure
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
One type of relationship: Correlation
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
4. Construct a model
Mathematics, and many theoretical directions in physics and
biology, computer science
Eg: Spelling correction systems in Word Processing tools
I
I
Non-native speakers make similar types of errors
Why do these systematic errors occur?
I
The native language of the speaker, the layout of the
keyboard, singular/plural in English
I
A model (mathematical, statistical) that can predict these
types of errors
I
Test the model on a corpus of correct and misspelled words.
How accurate is it? Does it give a lot of false positives?
I
If the model is good, we can use it to identify new spelling
mistakes
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Other examples
Predict London’s weather for next week
I
Based on data from weather stations, and math related to air
temperature, motion, pressure and humidity
Calculate the distance a projectile fired near the Earth’s surface at
known speed will travel as a function of angle
I
Newton’s laws of motion
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
5. Observational or exploratory research
Related to the curiosity side of science and very important in
practice
I
A starting point for many finer research questions
Careful observations of wildlife in the Galapagos islands
I
Darwin’s theory of evolution
Chemical composition of Greenland ice cores as a function of depth
I
As much as 3km long cores have been studied
I
Used for studying climate changes
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
6. Industrial or Applied Research
In response to a market need for some product or facility
I
Iterative prototype building and analysing its merits
I
Companies engage in research closely related to their product
and market
Aircraft design and testing
I
Boeing 777 was designed using a computer program called
CATIA by Dassult
Programs that control scientific instruments without using knobs
I
Labview from National Instruments
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Research methods in science: Review
I
Hypothesis-driven research
I
Measure a value
I
Measure a function or relationship
I
Construct a model
I
Observational and exploratory research
I
Industrial and applied research
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
How to design and analyse experiments?
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
How to design and analyze experiments?
Components of an experiment
Errors you may encounter and have to guard against
Analysing the results of your experiment
We will focus on
I
Hypothesis-driven research
I
Measurements
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Hypothesis-based experiments
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Step 1: Identify independent and dependant variables
Analogous to control and response variables
I
Independent: changes during the experiment (control)
I
Dependent: what you aim to measure (response)
Briars Aspirin cures headaches faster than RCS Aspirin
Dependent: What will you measure?
I
number of people reporting continued headaches
Independent: What are you going to vary?
I
type of aspirin given to the subjects (Brians or RCS)
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Step 2: Frame the null and alternative hypotheses
Null hypothesis: Changes in independent variable has no effect
I
Briars Aspirin and RCS Aspirin cure headaches at the same
rate
Alternative hypothesis: Independent variable does lead to
changes in the dependent variable
I
Briars Aspirin cures headaches faster than RCS Aspirin
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Accepting or rejecting the null hypothesis
You have a coin. You flip it 100 times, it comes up tails 54 times
and head 46 times. Is the coin fair?
Null hypothesis: The coin is fair
Alternative hypothesis: The coin is not fair
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Analyze your data
N = 100 flips
Represent heads by 1 and tail by 0
I
46 1s and 54 0s
Get average value of a flip (46 * 1 + 54 * 0)/ N = 0.46
Standard deviation of the values, s = 0.501
Standard
error (how much uncertainty is there in the average)
√
s/ N = 0.05
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Plot mean and standard error
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
The error bar pretty much overlaps the expected fraction of heads
Cannot reject the null hypothesis
I
The result of slightly more tails is only by random chance
based on these 100 flips
Therefore the coin is fair
A statistical test can be used to get a precise value for how likely is
it for the result to have occurred by chance
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Where can things go wrong?
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Errors in hypothesis testing
Type I: Researcher rejects the null hypothesis even when it
actually is correct
I
Incorrectly predict that there is an effect of the independent
variable
I
Vitamin C helps cure colds. Linus Pauling (a very influential
scientist) maintained this hypothesis although numerous
experiments never found any evidence.
Type II: Researcher accepts the null hypothesis even when it
actually is false
I
Ignore an actual effect
I
Many studies before finally a relationship between smoking
and lung cancer was accepted
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Bias
Actions that either wilfully or otherwise lower the validity of an
experiment
In medicine, double-blind treatment and control groups are used to
control bias
I Those who receive a new treatment (treatment) and those
who get the old one (control)
I Neither the patients nor the doctors know which group one
belongs to
I In a single-blind experiment, the doctors know the group (still
biased)
Placebos
I While testing if a new drug is useful, the control group is
given a placebo
I A placebo should be identical to the drug in appearance and
taste
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Bias
In data-oriented experiments
I
Looking at test data while designing the model and
experiments
I
Using test data to set some parameters of your model
I
Collecting data only as long as they fit your expectations or
results
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Carrying out measurements
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Random error
Small noise in your measurements
I
Some measurements are correct, some are a bit off
I
Eg: the last digit of your weighing scale is a bit inaccurate
due to external factors such as electric currents
Can typically be resolved by taking a large number of
measurements and averaging the values
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Systematic error
Every measurement has a specific type of distortion
I
Eg: always is higher by 10%. Cannot be solved by averaging
repeated measurements
Best solution is to calibrate your equipment
I
Make sure your instrument is working correctly
I
Check measurements of known values (oxygen in the general
atmosphere, density of water, acceleration of gravity)
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Part 2: Writing and presenting
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Part 2: Writing and presenting
Communication is an important part of research
I
Writing and giving talks are an important aspect of jobs both
in research and industry
I
A difficult skill to learn
Gain confidence
I
You become better with practice
I
We will do small exercises towards this goal
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
CE902 Proposal vs. CE901 project
Criterion
Output
When
Mark (in module)
Activities
Supervisor
CE902
Proposal
Spring term
40%
Choosing a topic
Reading
Planning
Individual supervisor
Annie Louis
CE901
Dissertation
Summer term
100%
Conducting experiments
Writing code
Evaluating your system
Individual supervisor
CE902 Lecture 1: Research Methods and Experimental Design
What should be clear from any report
I
What is the problem area?
I
Your research question
I
Why is this problem important?
I
What methodology you aim to use?
I
What you expect to find
If a reader cannot figure these out, you will get very low scores.
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Purpose of the proposal: Preparing for your MSc
dissertation
An opportunity to think carefully about your project topic and
execution
I
Ensure you are defining a real problem
I
Clearly define your objectives and methods
I
Allows your supervisor to provide useful feedback
Identify areas that you should read about, implementation details
to think about, start collecting data and software modules
I
Minimizes problems and setbacks later in the project execution
Gain practice in effective and clear writing about research
I
Important in our fields and hugely important for your
dissertation
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
What should a proposal contain?
Title
Abstract
Table of Contents
Main Body (the following are NOT section headings)
I
Introduction
I
Related work
I
Background
I
Research Questions
I
Planned approach, Methodology
I
Evaluation
I
Project Plan
References
Appendices (opt.)
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Title: express the essence of your project
Framing the title as a question:
I
Can ASP.NET be used effectively to build a social networking
site?
Describing the subject of the study:
I
A social networking site using ASP.NET
A transition from the general to the specific:
I
Developing large-scale social networks: A case study using
ASP.NET
A transition from the specific to the general:
I
Net Links: An social networking website created with
ASP.NET
Should help readers find your proposal if there were thinking of a
similar problem!
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Abstract: summarize what you will to do and expected
contributions
About 300-words
Give the reader a first impression
I
Why is this report important and interesting?
Don’t include too many details
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
What should be present in an abstract?
I
What is the problem? Why is it important/useful?
I
What is the key challenge?
I
Why is this problem hard?
I
What did you do? How did you tackle the challenge?
I
How did you test/evaluate? What were the key findings?
In the proposal, results are what you expect to find
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Keywords and related terms
Keywords
I
A list of terms that set your report in context
I
What domains of expertise benefit from your report
I
Examples: Software design; mobile robotics; optoelectronics;
...
Related terms (optional)
I
What subjects are related to the report
I
Which domains of expertise are needed to understand your
report
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
A 5-point introduction
Similar to abstract but in more detail.
You also talk about how you differ from prior work
And specifically enumerate your contributions
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Related work: Place your project within the field
Summarize in your own words publications related your project
I
Academic journals
I
Conference proceedings
I
Academic textbooks
I
Online resources (use with caution!)
Say explicitly how each publication is related to your project and
how your project is based on it/ differs from it
I
Important, dont just summarize the related work!
Also include work from the industry, current technology
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Background
Material necessary for a reader to understand your report
A refresher describing the mathematical/conceptual details, any
notation you will be using
Details of the algorithm/system that you will build off of
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Research questions
State the research question/topic you are addressing
I
Attempting to fill a gap in the current knowledge
I
If implementation or simulation, you may be trying a different
approach
Write down your hypotheses clearly and precisely
Also say what are the limitations of your work
I
Important but you will not address
I
Beyond the scope of your project
I
Will be interesting future work
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Objectives
The subtasks that need to be done to complete your project
I
Build a software module for doing X
I
Use X to study the structure of Y
I
Collect statistics regarding the efficiency of system Z
I
Design a website that shows real-time performance of the
system
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Methodology
Identify/discuss the research methods you intend to use
I
What software/hardware/system you will build
I
What experiments you will carry out
Examples
I
Details of the experiment measuring the conductivity of two
materials
I
Specifics of a survey examining the ease of use of you software
I
Plans for collecting data
I
Description of the software you will build using ASP.NET
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Evaluation: How you will measure the success of your
project
Tests: test cases, unit + integration testing
I
Suitable for most kinds of software
I
Present the results of tests using metrics
Experiments
Surveys
I
A good survey takes time; plan for time to conduct it
I
Include statistical analysis of the results
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Project plan
You must include:
I
A Work Breakdown Structure
I
A Gantt Chart
The plan must be specific to your project, and detailed
You should continually be reviewing and revising the plan
I
A tool for you and your supervisor to track your progress
[More details next week]
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Miscellaneous: You may also consider
Resource requirements (e.g. computing resources).
An assessment of potential risks to the success of the project, and
possible ways to avoid or deal with them.
High level diagram(s) showing the architecture of the system you
intend to produce or investigate
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Conclusions
Summarize the main ideas
State expected contributions and significant outcomes
Describe future work that will be enabled by your study
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Format: Follow the guidelines
Use the template provided
I
Use 12 point, Times New Roman (or similar) font
I
Use single spacing
I
Body text should be fully justified
Make it obvious that your proposal meets the criteria of the
marking scheme
I
Read the marking criteria carefully
Proof-read your proposal
I
Print it out to review it
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Simple techniques for better writing
Use figures and examples to illustrate your idea, method, results
I
Much stronger than lengthy explanations
I
Will also help you write better by referring to the figure or
example
Start collecting your examples now!!
Re-read your proposal a couple of times
I
Will a competent undergraduate student understand your
main ideas?
I
Spelling/grammar errors will put people off your main points
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Writing clearly is hard! Comes with practice
Break down what you have to write into parts
I
Write each one and then combine into a coherent text
Best writing is rewriting start early!
I
Even most experienced writers revise their text many times
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
When do we start? Right now.
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Task 1
Meet your supervisors. Set up a meeting at least once in two
weeks.
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Task 2
The PageRank Citation Ranking: Bringing Order to the
Web. http://ilpubs.stanford.edu:8090/422/
Page, Lawrence and Brin, Sergey and Motwani, Rajeev and
Winograd, Terry (1999). Technical Report. Stanford InfoLab.
What you should do
I
Read it (a general technology, don’t worry about
understanding all the technical details in depth)
I
With the view of writing a good paper or explaining an idea
I
We will do an exercise next week, so make sure you read it!!
I
Will be part of your proposal marks (2%), so do not miss the
lecture
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
Doing well in this course
I
Concepts related to research, data analytics and technical
writing will be introduced
I
I
I
Attend the lectures
Do the readings and classroom exercises
Start early on the proposal writing. Use input from the module
early on and be reflective when you write
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design
References and acknowledgements
[1] Book: The Research Methods Knowledge Base. William M.K
Trochim, James P. Donnelly, 2008
[2] Book: Research Methods for Science. Michael P. Marder, 2011
Annie Louis
CE902 Lecture 1: Research Methods and Experimental Design