The Scientific Method
DA2205
September 22, 2014
Recap of previous lecture
• Do computer science research as scientific research because
1. Computer science can be viewed as a science: the
transformation of artificial and natural information processes
and/or
2. It forces you to have good research methods
• Research is hard because of
1. intrinsic difficulty of discovering the unknown,
2. personal failings - self-deception,
3. bias and limitations imposed by the community.
• But the scientific method has allowed for rapid and not too
many misguided declarations of progress!
The computer scientific method
Obsolete Scientific Method
Computer Scientific Method
• Hypothesis
• Hunch
• Sequence of experiments.
• 1 experiment &
• Change 1 parameter per
experiment.
• Prove/Disprove Hypothesis.
• Document for others to reproduce results.
change all parameters
• Discard if it doesn’t support
hunch
• Why waste time?
We know this.
Source: “How to have a bad career in research/academia” by David Patterson, Feb 2002.
Good research methods help you
• Identify interesting new questions you hadn’t thought of.
(Exploratory data analysis)
• Ask questions that are scientifically meaningful.
(Falsifiable hypotheses)
• Avoid fixating on one favored hypothesis.
(Multiple working hypotheses)
• Devise evaluations that maximize what you can learn.
(Experimental design)
• Avoid seeing patterns in randomness.
(Hypothesis tests)
• Understand when you have collected sufficient evidence.
(Statistical power)
• Formulate broader theories. (Modeling)
It discourages this scenario
The Significance of Research
9-Oct-10
2
Today’s lecture
•
Review of the scientific method
- Definition of Scientific Theory, Hypothesis and Law.
- Review of reasoning in research
? Deduction, Induction, Abduction
- Expanding Scientific knowledge and falsification.
- The Hypothetic–deductive scheme.
•
Scientific method and computer science
•
The variety of research endeavour
Review of the scientific method
Scientific method
• The scientific method helps us do this
Observation
Theory
World
Validation
Important functions of the scientific method
• Validation
? Can a general statement be judged as true, false or probable?
• Analysis of hierarchical structure
? Reductionism:
Understand complex system by reducing them to the
interactions of their parts.
? Emergence:
Produce complex, interesting high-level function by combining
simple low-level mechanisms in simple ways.
• Causality
? Define and explore the connection between actions.
Encoding of scientific knowledge or premises
• Scientific Hypothesis:
- Proposed explanation for an observable phenomena.
• Scientific Model:
- A physical, conceptual, or computer-based
representation of a system.
• Scientific Theory:
- A comprehensive set of ideas that explains a
phenomenon.
• Scientific Law:
- Describes a phenomena, often mathematically.
Theory building should be the goal because
• Theories lie at the heart of what it means to do science.
? Production of generalizable knowledge
? Serve to explain and predict
? Concepts, relationships, causal inferences
• Theory provides orientation for data collection
? Theoretical perspective allows efficient observation of the world
• Theories allow us to compare similar work
? Theories include precise definition for the key terms
? Theories provide a rationale for which phenomena to measure
• Theories support analytical generalization
? Provide a deeper understanding of our empirical results
? ...and hence how they apply more generally
? Much more powerful than statistical generalization
Reasoning within the scientific method
Deduction explained
• Deduction is
Arguing from the general to the particular
• Example
All Frenchmen like red wine.
Pierre is a Frenchman. =⇒ Pierre likes red wine.
• Terminology of the example:
All Frenchmen like red wine. ← premise
Pierre is a Frenchman. =⇒ Pierre likes red wine.
↑
↑
premise
conclusion
Virtues of deduction
• A deduced conclusion is definitely true if
? premises (axioms) are true and
? reasoning carried out correctly.
• An example is Pythagoras’ Theorem can be deduced from
Euclid’s axioms.
• For deductive logic:
discovery process ≡ justification process
• However, most mathematical theorems are not discovered
solely by the exercise of deductive reasoning.
• Use deductive reasoning to justify or falsify an inspiration or
an intuitive belief.
Limitations of deduction
• Deductive reasoning uncovers what is implicit in our premises.
• Unfortunately, deductive reasoning cannot bring us
knowledge of the world beyond our premises.
• It can however bring us awareness about our premises.
•
But how can we expand our knowledge???
Limitations of deduction
• Deductive reasoning uncovers what is implicit in our premises.
• Unfortunately, deductive reasoning cannot bring us
knowledge of the world beyond our premises.
• It can however bring us awareness about our premises.
•
But how can we expand our knowledge???
Reasoning to potential new knowledge:
Induction
Induction explained
• Induction is
Arguing from the particular to the general
• Example of inductive inference:
The first five eggs in the box are rotten.
All six eggs have the same best-before date stamped on them.
Infer the sixth egg will be rotten too.
• Inductive inference does not guarantee the conclusion is
true.
An inductive argument
• Start with observations or experimental results.
• On the basis of these, start framing general principles that
take the observations into account.
• Illustration of its limitation (enumerative induction)
? Someone from Europe, having seen many swans, all of them
white, comes to the conclusion “All swans are white”.
? He anticipates the next swan to appear will also be white.
? The generalization is confirmed with every new swan that is
seen.
? Then, visiting Australia, the person comes across a black swan
and has to think again.
An inductive argument
• Start with observations or experimental results.
• On the basis of these, start framing general principles that
take the observations into account.
• Illustration of its limitation (enumerative induction)
? Someone from Europe, having seen many swans, all of them
white, comes to the conclusion “All swans are white”.
? He anticipates the next swan to appear will also be white.
? The generalization is confirmed with every new swan that is
seen.
? Then, visiting Australia, the person comes across a black swan
and has to think again.
The inductive scientific method
• Generation of a possible hypothesis
? Gather evidence – if possible eliminate irrelevant factors.
(somewhat contradictory as need a hypothesis to define relevancy.)
? Conclusions inferred from the evidence leads to a hypothesis.
• Refinement of the hypothesis
? Experiments are devised to test out the hypothesis, based on
its predictions.
? If necessary the hypothesis is modified to take into account the
results of the later experiments.
? A general theory is framed from the hypothesis and its
related experimental data.
• Verify or falsify hypothesis
? Use this theory to make predictions. On the basis of these
can confirm or disprove the theory.
Reasoning to potential new knowledge:
Abduction
Abduction
• Infer a as an explanation of b.
• Abduction allows the precondition a to be inferred from the
consequence b.
• Deduction and abduction differ in the direction in which a rule
like “a entails b” is used for inference.
• Truth of the assumptions do not guarantee the truth of the
conclusion.
What is abduction?
“...a method of reasoning in which
one chooses the hypothesis that
would, if true, best explain the relevant evidence. Abductive reasoning
starts from a set of accepted facts and
infers their most likely, or best, explanations.”
Example of abductive reasoning
• You notice “The lawn is wet”.
• If “it rained last night”, it would be unsurprising that “the
lawn is wet.”
• By abductive reasoning, the possibility that it rained last night
is reasonable.
Example of abductive reasoning
• You notice “The lawn is wet”.
• If “it rained last night”, it would be unsurprising that “the
lawn is wet.”
• By abductive reasoning, the possibility that it rained last night
is reasonable.
Note
• Abducing rain last night from the observation of the wet lawn
can lead to a false conclusion.
• Maybe dew, lawn sprinklers, or some other process could have
resulted in the wet lawn.
Abductive reasoning in science
• Abduction selects, among the hypotheses that are being
considered, the one that best accounts for the evidence.
• Abductive reasoning is closely related to the statistical
method of maximum likelihood estimation.
• There exist several obvious threats to validity
? Small hypotheses spaces.
? Small amounts of evidence to explain.
Both are key challenges to the scientific practice.
Challenges of abduction
• Creating hypothesis spaces likely to contain the “true”
hypothesis
? Approach – Create large hypothesis spaces.
• Knowing when more valid hypotheses are missing from your
hypothesis space
? Approach – Constantly evaluate your hypothesis space, and
expand the hypotheses space whenever the data become extremely
unlikely, given any current hypothesis.
• Creating good sets of evidence to explain
? Approach – Seek diverse and independent evidence with which to
evaluate hypotheses.
Reasoning to potential new knowledge:
The hypothetic–deductive reasoning scheme
Scientific reasoning: Hypothetic–deductive scheme
• Science in its discovery mode is not propelled by logic.
• Scientific reasoning is a dialogue between the possible and the
actual.
• There are two thought process which alternate and interact
the imaginative ←→ the critical
• Imaginative process: Formation of a hypothesis
? Form an opinion, take a view, make an informed guess, which
may explain the phenomena under investigation
• Critical process: Expose the hypothesis to criticism
? Empirical testing of logical consequences of the hypothesis.
? If predictions accurate =⇒ gain confidence in the hypothesis.
? If predictions not accurate =⇒ hypothesis wrong.
Scientific reasoning: Hypothetic–deductive scheme
• Science in its discovery mode is not propelled by logic.
• Scientific reasoning is a dialogue between the possible and the
actual.
• There are two thought process which alternate and interact
the imaginative ←→ the critical
• Imaginative process: Formation of a hypothesis
? Form an opinion, take a view, make an informed guess, which
may explain the phenomena under investigation
• Critical process: Expose the hypothesis to criticism
? Empirical testing of logical consequences of the hypothesis.
? If predictions accurate =⇒ gain confidence in the hypothesis.
? If predictions not accurate =⇒ hypothesis wrong.
Hypothesis
• A supposition or proposed explanation made on the basis of
limited evidence as a starting point for further investigation.
• Also – theory, theorem, thesis, conjecture, supposition,
postulation, postulate, proposition, premise, assumption
• Origin – late 16th century: via late Latin from Greek
? hupothesis ≈ “foundation”,
? from hupo (≈ “upon”) + thesis (≈ “placing”).
Types of hypotheses
• Existential
An entity or phenomenon exists.
Atoms contain uncharged subatomic particle (neutrons)
• Compositional
An entity or phenomenon consists of a number of components
(perhaps with a specified frequency).
Atoms consists of protons, neutrons and electrons
• Correlational
Two measurable quantities have a specified association.
An element’s atomic weight and its properties are correlated
• Causal
A given behaviour has a specified causal mechanism.
The low reactivity of the noble gases is caused by their full outer
shell of valence elements.
Good property of a hypothesis:
Falsifiability
Karl Popper’s “conjectures and refutations”
• Confirmations of a theory are typically easy to find when
sought.
• True confirmations should be surprising.
? If we didn’t know the theory, we should have expected an
event which was incompatible with it.
• Good scientific theories are prohibitions.
? They forbid certain things from happening. The more a theory
forbids, the better it is.
• A theory which is not refutable by any conceivable event is
non-scientific. Irrefutability is not a virtue but a vice.
“No amount of experimentation can ever
prove me right; a single experiment can
prove me wrong”
– Albert Einstein
“A theory which cannot be mortally
endangered cannot be said to be alive.”
– W.A.H. Rushton
Falsifiability
In The Logic of Scientific Discovery (1934) Popper argued:
• Falsifiability is the logical possibility that an assertion can be
shown to be false by evidence.
• “Falsifiable” does not imply “false”. Instead, if a falsifiable
proposition is false, then its falsehood can be shown by
experiment, theorem or simulation.
• There are degrees of falsifiability – some theories are more
falsifiable than others.
• Falsifies theories can be rescued by introducing some ad hoc
change, but only by lowering their apparent validity.
Examples of non-falsifiable statements
• An alien spaceship crashed in Roswell New Mexico.
• A giant white gorilla lives in the Himalayan mountains.
• Loch Ness contains a giant reptile.
In each case, if the statement happens to be wrong, all you will
ever find is an absence of evidence.
Examples of falsifiable statements
• No alien spaceships have ever landed in Roswell New Mexico.
- Find just one spaceship and the statement is disproven.
- Do not need an exhaustive elimination of all possibilities.
- Just one spaceship will do it!
• All humans live forever.
- Find just one dead person and the statement is disproven.
- Do not need an exhaustive elimination of all people.
What makes hypotheses unfalsifiable?
• Vagueness
The theory that does not predict any particular experimental
outcome.
• Complexity
A theory that can “explain” any experimental result.
• Special pleading
Traditional experimental methods are claimed not to apply.
Degree of falsifiability
• Number of experiments.
• Probability that a given experimental result would be
produced given that the theory is false
? How many other outcomes were possible?
? In the absence of the theory, what is the probability
distribution over those outcomes?
• Diversity of attempts.
? Potential unknown causal factors
? Potential experimental flaws
“Science is what we have learned about
how to keep from fooling ourselves”
– Richard Feynman
Scientific Enquiry
Prior Knowledge
Initial Hypothesis
Observe
what is wrong
with the
current theory?
Experiment
Theorize
manipulate
refine/create a
the variables
better theory
Design
Design empirical
tests of the theory
Reasoning to potential new knowledge:
Multiple Working Hypotheses
Why use multiple working hypotheses?
• Personal investment – Helps to separate you from your
hypotheses; Shifts your personal investment from the
hypothesis to the hypothesis test.
• Focus – Reinforces a focus on falsification rather than
confirmation.
• Efficiency – Allows experiments and proofs to be designed to
distinguish among several competing hypotheses rather than
evaluating a single one.
• Harmony – Limits the potential for professional conflict and
rejection because all hypotheses are considered rather than
only one.
Alternatives
• Method of the ruling theory – Data are sought to confirm
the one theory held by a researcher.
• Method of the working hypothesis – Data are sought to
falsify the current hypothesis of a researcher.
• Method of multiple working hypotheses – Data are sought
and multiple hypotheses are evaluated based on those data,
disconfirmed hypotheses are abandoned, and a new
(potentially modified) set of hypotheses are evaluated in the
next round.
“The story of a theory’s failure often strikes readers as sad
and unsatisfying. Since science thrives on self-correction,
we who practice this most challenging of human arts do
not share such a feeling. We may be unhappy if a favoured
hypothesis loses or chagrined if theories that we proposed
prove inadequate. But refutation almost always contains
positive lessons that overwhelm disappointment, even
when... no new and comprehensive theory has yet filled
the void.”
– Stephen Jay Gould
Some myths about science
• “Scientists follow the scientific method”
? There is no one method
? Many methods available...
? ...and all of them have known flaws
? Scientists use imagination, creativity, prior knowledge,
perseverance.
• “Scientific knowledge is general and absolute”
? Empirical Induction used to build evidence
? Scientists often get it wrong...
? ...but Science (as a process) is self-correcting
Source: Steve Easterbrook. ASE’ 07, Tutorial T2 : Empirical Research Methods for SE, 2007
Relevance for Computer Science Research
Scientific method and the computer science researcher
Some questions we should consider
• When does the CS researcher need the scientific method?
• How does the CS researcher use the scientific method?
First let’s think about what a CS researcher does?
Scientific method and the computer science researcher
Some questions we should consider
• When does the CS researcher need the scientific method?
• How does the CS researcher use the scientific method?
First let’s think about what a CS researcher does?
Core technologies of computing
Core Technologies
Algorithms
Artificial Intelligence
Computational Science
Computer Architecture
Data Mining
Data Security
Data Structures
Databases
Decision Support Systems
Distributed Computation
e-Commerce
Graphics
Human-Computer Interaction
Information Retrieval
Management Information Systems
Natural-Language Processing
Networks
Operating Systems
Parallel Computation
Programming
Languages
Real-Time Systems
Robots
Scientific Computation
Software Engineering
Supercomputers
Virtual Reality
Vision
Visualization
Workflow
Have we missed anything?
Compilers
The Great Principles of Computing
At a high-level within all these technologies dealing with
• Principles of Mechanics
• Principles of Design
Computing: Principles of Mechanics
The fundamental laws and recurrences of computation
1. Computation
What can be computed and what are the limits of computing?
2. Communication
How can the transmission of messages occur from one point to
another?
3. Coordination
How can multiple entities cooperate to achieve a single goal?
4. Recollection
How can information be stored and retrieved?
5. Automation
What cognitive tasks, ordinarily associated with human intelligence,
can be automated?
Computing: Principles of Design
Architecture: How to arrange the components of a system?
Process: How we arrange our work load and thoughts?
1. Simplicity
How can we avoid complexity?
2. Performance
How can we predict performance of complex systems?
3. Reliability
How do we keep devices and information available when needed?
4. Evolvability
Can our design be adaptable to changes of function or scale?
5. Security
How do we allow or limit access as needed?
Addressing these challenges requires the Scientific Method
1. Information systems are often complex
=⇒ only understand, validate and improve them by
? performing experiments,
? collecting data, and
? interpreting the data.
2. This requires Scientific Method in conjunction with
mathematical models, theory and statistics
? formulate hypotheses, design experiments,
? test hypotheses, validate system and predict performance
Addressing these challenges requires the Scientific Method
1. Information systems are often complex
=⇒ only understand, validate and improve them by
? performing experiments,
? collecting data, and
? interpreting the data.
2. This requires Scientific Method in conjunction with
mathematical models, theory and statistics
? formulate hypotheses, design experiments,
? test hypotheses, validate system and predict performance
A framework for computer science research problems
“Unlike other scientists, who study chemical reactions,
processes in cells, bridges under stress, animals in mazes,
and so on, we study computer programs that perform
tasks in environments.”
– Paul Cohen, 1995
Whether your subject is a rat or a computer program, the task of
science, to provide theories to answer 3 basic research questions:
How will a change in the agent’s
• system affect its behavior given a task and an environment?
• task affect its behavior in a particular environment?
• environment affect its behaviour on a particular task?
Source: Paul Cohen, Empirical Methods for Artificial Intelligence, MIT Press, 1995.
A framework for computer science research problems
System
Task
Behaviour
Environment
Explanation of elements
• System – Aspects influenced by a system designer
? Specific algorithm used, system architecture, data structures,
parameter settings, etc.
• Task – Aspects influenced by a prospective user
? Specific queries, requests, input data, etc.
• Environment – Aspects influenced neither by designer nor
user
? Network environment, available cycles or memory, etc.
• Behaviour – Performance of the task by the system within
the environment.
Framework allows generation of many hypotheses
• Compositional
System
Task
An entity or phenomenon
consists of a number of
components (perhaps with a
specified frequency)
• Correlational
Behaviour
Environment
Two measurable quantities
have a specified association
• Causal
A given behaviour has a
specified causal mechanism
The Varieties of Research Endeavour
Scientists cannot be pigeon-holed
“Scientists are people of very dissimilar temperaments
doing different things in very different ways. Among
scientists are collectors, classifiers and compulsive
tidiers-up; many are detectives by temperament and many
are explorers; some are artists and others artisans. There
are poet-scientists and philosopher-scientists and even a
few mystics. ...and most people who are in fact scientists
could easily have been something else instead.”
– Peter Medawar (1915-1987)
Source: Peter Medawar, ’Hypothesis and Imagination’ in the The Art of the Soluble, 1967
What do researchers spend most of their time doing?
Construct a
general theory
Identify an
important exception
to a theory
Find flaws
in a previous
experiment
Use a theory to explain
an observation
Devise a research
question
Gather data
Compare
results from
theory and
experiment
Design an
algorithm
Design an
experiment
Run an
experiment
Devise a new test of
a hypothesis
Devise a new measurement
or technique
Identify an
existence proof
Make a
conjecture
Construct a
theoretical proof
Unify disparate
theories
Establish a
relationship
between
variables
Not everything
• These are a researcher’s core activities.
• Not everything a researcher typically does. Time also spent
? Reading the literature
? Getting resources
? Building infrastructure (software, labs, etc.)
? Teaching students
? Building professional relationships
? Reviewing papers
• These auxiliary tasks enhance and is scaffolding for the core
activities.
Construct an algorithm or system
• Much “doing computer science” is about building the
infrastructure to do computer science
• Examples include building new...
? Compilers
? Garbage collectors
? Networking protocols
? Machine learning algorithms
• But, of course, this isn’t everything we do (or should do)...
Construct an algorithm or system
• Much “doing computer science” is about building the
infrastructure to do computer science
• Examples include building new...
? Compilers
? Garbage collectors
? Networking protocols
? Machine learning algorithms
• But, of course, this isn’t everything we do (or should do)...
Identify a research question
• Identify a research question about which a hypothesis can be
formulated
• These hypotheses are typically about
? Algorithms
? Tasks, or
? Environments
• Questions about
? Individual elements (e.g., existence proofs)
? How changes in one element affect another
? Comparisons of two more more elements holding others
constant
• Often iterative and done by multiple researchers
Make a conjecture
• Formulate a hypothesis or predict an experimental result
• Types
? Existential (e.g., phase transitions)
? Compositional (e.g., behavior regions of CSPs)
? Correlational (e.g., Training set size/tree size)
? Causal (e.g., bias propagation)
• Want multiple hypotheses that explain existing observations,
rather than just a single one.
Make a conjecture
• What makes a good conjecture?
? Falsifiable (Popper)
? Important, if true (or false)
? Consistent with existing observations
? Not obviously true
• Examples
? Structure of DNA
? Mendeleev’s periodic table
Gather data
• Naturalistic observations
(e.g., router performance in actual networks)
• Carefully selected observations
(i.e., quasi-experimental design)
• Experiments
• Sets of previously published experiments
(i.e., Meta-analysis)
Identify a relationship between variables
• The basis for both correlational and causal hypotheses
• Many variations
? Most common is two continuous variables (e.g., scatterplot),
but also...
? Non-linear relationships
? More than two variables
? Discrete variables
• Emphasizes the importance of exploratory data analysis
Unify previously separate theories
• Synthesize multiple theories into one
• Why is this useful?
? Simplicity – Why have two theories when you can have one?
? Expanded evidence and methods – Evidence and methods
from both areas can be used to examine the theory
? Faster error correction– If one theory is falsified or modified,
well know that both should be
Identify an important exception
• Results in
? Assignment of exception to a trivial cause (e.g., experimental
error), or
? Recognition of a misclassification of previously observed
phenomena, or
? Recognition of new subclass of situation that requires a
different theory, or
? A wholescale revision in the theory.
Exciting when the unexpected happens
“The most exciting phrase to hear in
science, the one that heralds new
discoveries, is not ’Eureka!’ (I found it!)
but ’That’s funny ...”
– Isaac Asimov
Discover an existence proof
• Only slightly different from identifying an important exception,
but often about phenomena about which no theory exists
• Goals
? Describe new example accurately and as completely as possible
? Find other examples that share characteristics
? Attempt to generalize to a description of the class
Determine a new test of a hypothesis
• Goal – Maximize probability of falsification (assuming
hypothesis is false)
• Valuable when...
? Independent of prior attempts
• Different experimental or theoretical methods
• Different data
? Similar to, but better than, prior attempts
• Higher accuracy
• Rule out alternative explanations (better control)
Design an experiment
• Specify algorithm, task, and environment
• Identify key dependent variables and appropriate measurement
techniques
• Use experimental methods
? Control or randomize key independent variables (potential sources
of variation)
? Use replication, blocking, and modeling to maximize value of data
collection
Formulate a measurement or technique
• Surprisingly common key to important scientific discoveries
• Examples
? Watts & Strogatz
Clustering coefficient & path length
? Kirkpatrick & Selman
Measures of problem difficulty
? Friedman (and others)
Bias-variance analysis
Other Varieties (theory and experiment)
• Devise a theoretical proof
• Run an experiment
• Compare results from theory & experiment
• Find flaws in existing experiments/proofs
• Use a theory to explain an observation
Other Varieties (large theories)
• Formulate a general theoretical framework
• Examples
? Quantum mechanics
? Plate tectonics
? Evolution by natural selection
? Germ theory of disease
? Gravitation