Robustness analysis versus reliable process
reasoning
Chiara Lisciandra
Metascience
ISSN 0815-0796
Metascience
DOI 10.1007/s11016-014-9927-2
1 23
Your article is protected by copyright and all
rights are held exclusively by Springer Science
+Business Media Dordrecht. This e-offprint
is for personal use only and shall not be selfarchived in electronic repositories. If you wish
to self-archive your article, please use the
accepted manuscript version for posting on
your own website. You may further deposit
the accepted manuscript version in any
repository, provided it is only made publicly
available 12 months after official publication
or later and provided acknowledgement is
given to the original source of publication
and a link is inserted to the published article
on Springer's website. The link must be
accompanied by the following text: "The final
publication is available at link.springer.com”.
1 23
Author's personal copy
Metascience
DOI 10.1007/s11016-014-9927-2
ESSAY REVIEW
Robustness analysis versus reliable process reasoning
Robert Hudson: Seeing things: The philosophy of reliable
observation. Oxford: Oxford University Press, 2014, xii+274pp,
£41.99, $58.50 HB
Chiara Lisciandra
Ó Springer Science+Business Media Dordrecht 2014
Robert Hudson’s book is a contribution to the recent debate on robustness analysis
in scientific practice, with a specific focus on the empirical sciences. In this context,
robustness analysis is defined as a way to increase the probability of a certain
hypothesis by showing that the same result is obtained from several, alternative
methods. The rationale underlying this practice is that it would be highly unlikely if
different, independent means of observation provided the same wrong outcome.We
do not believe in miracles; hence, the probability of the initial hypothesis being true
increases if the same result occurs across conditions.
Simple as this notion sounds, according to Hudson, the various attempts that have
been offered in the literature to formalize it have proved to be unsuccessful.
Hudson’s skeptical argument is formulated at the outset of the book and illustrated
throughout by means of several episodes from the recent history of science, mainly
from physics and biology. Meticulous reconstructions of the discovery of atoms and
molecules, the controversy about the existence of the mesosome, and the debate
around dark matter and dark energy are offered.
The main argument that Hudson presents unfolds as follows. Imagine that a group
of scientists, or different scientific teams, had several different experimental
procedures at their disposal, all of uncertain reliability. Suppose that the observations
from the various instrumental procedures were to converge on the same result. It might
be argued that it is more probable that different, independent means of observation
detect the same result if that result is true, than if that result is false. However, Hudson
claims that if the reliability of the various different procedures is uncertain, then the
epistemic advantage in switching from one procedure to the other is questionable. In
such situations, scientists should first worry about making the instruments more
reliable, rather than pointing out that the same result is consistent across conditions.
C. Lisciandra (&)
Social and Moral Philosophy, Department of Political and Economic Studies, University of
Helsinki, PO Box 24, 00014 Helsinki, Finland
e-mail: [email protected]
123
Author's personal copy
Metascience
There have been numerous cases in the history of science—some of which are
presented by Hudson in the central chapters of the book—in which convergence on
the same outcome turned out to be a blunder; progress was made once that better,
more reliable instruments were devised and became available to scientists. Since
robustness arguments have been used in support of both successful and unsuccessful
results, relying on this strategy appears weak after all.
Furthermore, intuitive as it sounds, the argument for the value of robustness
analysis conceals fundamental difficulties. To start with, Hudson draws a distinction
between robustness analysis as (a) a method of adopting various inferential
machineries, whose error term is known, so as to aggregate their outcomes in such a
way that the error entailed by each procedure cancels out across conditions (e.g.,
when the measurements of two balances with different error terms are compared
with one another); and (b) as a method for discovering new physical entities or
phenomena. In the first case, the reliability of different instruments is known and
their independence is given. In the latter case, the reliability of the various
instruments should be deduced from their convergent result.
Hudson claims that neither case is immune to problems. In the first case, the
nature of the difficulties is mainly pragmatic, insofar as the aggregation of results
coming from various instruments is far from trivial. Also—but this is my addition—
it could be asked why we should rely on results deriving from a less reliable
procedure to confirm results deriving from a more reliable one. The less accurate
procedure will give estimates that are more biased than the original one; thus, it is
not clear how such practice can increase confirmation. In the second case, the
problems are mainly epistemic and concern the significance of results given by
various procedures whose reliability and independence cannot be easily assessed. In
new frontiers of science especially, convergence on the same result (or the lack of it)
is not always a knockdown argument in favor of (or against) the hypothesis under
scrutiny. Suppose the various instruments did not detect the same effect. Should
scientists question the initial result, or should they exclude those instruments that
did not detect that result? The main problem, which echoes the experimenter’s
regress argument, is that if the ability to capture a certain phenomenon is the
criterion for establishing the reliability of an instrument, then negative results might
be excluded at the outset. If so, the robust results will ultimately not stand as
particularly significant.
The previous issues are presented in the first chapter of the book, where Hudson
reviews the most important philosophical arguments in support of robustness
analysis, respectively, pragmatic, probabilistic, and epistemic arguments. He aims
to rebut each of them and to replace robustness analysis with ‘a reliable reasoning
process’, i.e., a procedure to assess the reliability of our scientific tools. In the
following chapters, his strategy is to analyze whether robustness analysis in
scientific practice has been successful. If it were so, then—even if the epistemological arguments have not been decisive so far—the results could speak for
themselves. Notwithstanding the supporters of robustness analysis, however,
Hudson argues that the main cases usually presented in the literature have been
mistakenly interpreted as successful examples of robustness analysis.
123
Author's personal copy
Metascience
The first case that Hudson discusses is the debate on the existence of the
mesosome, which engaged biologists in the sixties and the seventies of the last
century. In the philosophical literature, the discovery that the mesosome was a byproduct of a specific experimental apparatus has been defended as a successful case
of robustness analysis (see e.g., Culp 1994). According to the defenders of this view,
it was because independent means of observation produced contrasting results that
scientists concluded that the mesosome was in fact an experimental artifact. At the
same time, they disregarded convergent results coming from practices whose
independence could not be similarly assessed. To this interpretation, Hudson objects
that the key factor in denying the existence of the mesosome was simply an
acknowledgment that the procedures through which the mesosome was revealed
were unreliable. It was only when it became clear that chemical fixation damages
bacterial morphology, and new cryofixation techniques were developed, that the
hypothesis about the mesosome was disproved. According to Hudson, robustness
analysis did not play the role that was attributed to it, since the discovery of the
mesosome was ultimately just the result of a reliable reasoning process.
The episode of the mesosome is mainly known to philosophers working in the
experimental robustness niche; but Hudson proceeds to address one of the most
famous cases in the philosophy of science literature, namely Perrin’s discovery of
atoms and molecules. The story usually goes that the consistency of the result
through different and independent methods of measurement was decisive in ruling
out the possibility of the result being the effect of one specific measurement tool.
Once again, according to Hudson, the previous reconstruction is misleading. In
support of his claim, Hudson calls the reader’s attention to many excerpts from
Perrin’s works (Perrin 1910, 1916, 1926). He attempts to understand what was
going on in the mind of the scientist, whether it was a form of robustness analysis or
a different way of reasoning. Although Perrin himself admits his astonishment at the
agreement of the results obtained by different physical processes, according to
Hudson this was not what convinced the scientist that he had finally proved the
atomic nature of matter. As Hudson reconstructs the story, it was one specific
experimental apparatus, the experiment on Brownian motion with emulsions, that
possessed for Perrin a higher degree of epistemic authority, which made it the
standard by which to calibrate other methods. In other words, the key issue was not
independent convergence on that result, but rather the fact that it was derived from
the most reliable approach available.
Finally, with the discussion on dark matter and dark energy, Hudson brings us to
one of the most exciting enterprises in modern science. In the interest of space, I
will mainly refer to the case of dark matter and its composition. The focus of
Hudson’s argument is on three different research groups, based, respectively, in
Italy, the United States, and France, each of which devised different procedures for
their purpose. Hudson’s point is to show how rarely it happens in science that
different groups rely on robust results in support of their respective arguments. Even
when agreement is achieved between different laboratories, as in the case of some of
the results of the dark matter project, scientists tend to base their arguments in favor
of their own analysis on the virtues it has with respect to alternative ones; or on the
ways in which it remedies the deficiencies of other approaches. Scientists usually do
123
Author's personal copy
Metascience
not refer to other groups’ results to strengthen their own position. The underlying
rationale is that, as long as their instruments are reliable, they can trust their
outcomes without the need to refer to the results derived from different methods,
which would make their achievements superfluous or unnecessary.
Both because scientific practice does not rely on robustness analysis and because
this method is epistemically dubious, Hudson’s conclusion is to favor the skeptics.
The last part of the book is dedicated to the consequences of the debate on
robustness analysis for scientific realism. The aspect of the book I appreciated the
most is that the arguments for and against robustness are embedded in actual
scientific practice; at times, however, Hudson risks losing the reader in the details of
historical reconstruction.
Through the exposition of the argument, several questions may come to the mind.
One of the main issues is that previous treatments of robustness analysis did not
include the notion of reliability in the picture—and for good reason. It is indeed
when we do not have information about the reliability of our instruments that
convergence on the same outcome becomes relevant (Kuorikoski et al. 2010). The
process that leads to scientific accomplishments is rarely the one where reliability is
a given, or where a single experiment settles the conflicts between competing
hypotheses once and for all. It is most likely the case that a process of continuous
adjustment between the experiments we devise and the results we get from them is
what fosters progress in science. If we rebut robustness analysis on the grounds that
we are not justified in deriving conclusions unless a certain reliability of our tools is
proved, then we should suspend judgment about the outcomes of our experiments
across the board.
Ironically enough, Hudson’s argument is developed and supported via different
case studies: Is he himself relying on robustness analysis in support of his position?
Would it not be enough, according to his view, to have one decisive case study to
support his argument? To this objection, Hudson may reply that he is not supporting
his argument via robustness analysis, as he is listing a number of cases to show that
scientists do not apply this method in their work. In other words—he may say—he is
simply reasoning by induction. The differences between induction and robustness,
and between the confirmations we get from repeating the same experiment versus
changing it, are discussed in one of the most interesting and thought-provoking
sections of the book.
On a different note, the definition of robustness analysis stated at the outset of the
book strongly recalls the notion of coherence in epistemology. It would be a further
contribution to the debate to clarify whether the two notions overlap or not. If they
do, then a greater attention should have been given to the large body of literature
dedicated to coherence in epistemology (Bovens and Hartmann 2000; Schupbach
2011). If they do not, it would have been interesting to see how they differ. Finally,
given his focus on the experimental sciences, Hudson’s analysis takes no notice of a
highly contentious area of the debate, which investigates robustness analysis as a
method of testing our hypotheses precisely when empirical results are not available.
Nonetheless, Seeing Things addresses a number of cutting-edge issues in the domain
of experimental robustness, which makes this book an important contribution to our
understanding of the notion of robustness. This is one of the first monographs to
123
Author's personal copy
Metascience
present different accounts of robustness analysis in a unified picture. It is
particularly welcome in an area of research which is dispersed over various
subfields; even more so because, despite his focus on experimental robustness,
Hudson opens new perspectives on the topic, which will most likely have a bearing
also on the non-empirical side of the debate.
Acknowledgments With thanks to Alessandra Basso, Aki Lehtinen and Caterina Marchionni who took
part in the reading group on Seeing Things at the University of Helsinki.
References
Bovens, L., and S. Hartmann. 2000. Coherence, belief expansion and Bayesian networks. In Proceedings
of the 8th International Workshop on Non-Monotonic Reasoning, NMR’2000, eds. Baral C.,
Truszczynski M, Breckenridge, Colorado.
Culp, S. 1994. Defending robustness: The bacterial mesosome as a test case. In PSA 1994: Proceedings of
the 1994 biennial meeting of the philosophy of science association, vol. 1, ed. D. Hull, et al., 46–57.
East Lansing, MI: Philosophy of Science Association.
Kuorikoski, J., A. Lehtinen, and C. Marchionni. 2010. Economic modelling as robustness analysis. British
Journal of Philosophy of Science 61: 541–567.
Perrin, J. 1910. Brownian Movement and Molecular Reality. Trans. Soddy F. London: Taylor and Francis.
Perrin, J. 1916. Atoms. Trans. Hammick D. L. London: Constable.
Perrin, J. 1926. Discontinuous structure of matter. In Nobel lectures: Physics, 1922–1941, ed. Nobel
Foundation, 138–164. New York: Elsevier.
Schupbach, J. 2011. New hope for Shogenji’s coherence measure. British Journal of Philosophy of
Science 62: 125–142.
123