1
A review of the co-citation based Science Foresight
process
Dr David J Holland Smith
Alison Mathew
Dr Nick M Kemp
Dr J Sylvan Katz, Katz Competitive Intelligence Ltd
Dstl/CR10342 1.0
23 March, 2004
© Crown copyright 2004 Dstl
© Crown copyright
Defence Science and Technology Laboratory UK
Executive summary
Overview
This report was commissioned by the UK MOD’s Research Acquisition Organisation as
part of a wider investigation aimed at informing the selection of an appropriate process
for ‘horizon scanning’. Though the stakeholders in the output of such scanning extend
throughout MOD and into its wider networks within government and commercial
organisations, the process must primarily serve Output 5 of MOD’s research programme.
In this context, horizon scanning can be seen as the means for identifying significant
developments in civil and foreign defence science and technology that may not otherwise
be noted through MOD’s own technical programmes.
Through the ‘Science Foresight’ Project, which reported in 2001, MOD/Dstl, aided by
SPRU, designed and piloted a process that addresses at least some of the aspects required
for horizon scanning as defined above. Specifically, this process provides a means for
‘targeting’ influential scientific authors and collecting their views as to the future of their
domain.
Aim. This report aims to provide an objective assessment of the Science Foresight
process, as it might be cast within MOD’s new Output-based framework for research
delivery.
The Science Foresight process
The Science Foresight process is designed to provide a systematic and efficient means
for collecting a body of scientifically informed opinion regarding the future of a field or
domain of interest, drawn directly from those who are most influencing its development.
•
‘Influential’ authors are determined through a statistical analysis of the patterns of cocitation within the domain (The publisher, Thomson-ISI, offers the bulk of this
analysis as a commercial service). This first identifies the key groups that are
pioneering particular advances, and then selects one or a small number of key authors
from each group.
•
Key authors are then invited to express their views as to the likely future for their
areas of work. A systematic on-line questionnaire technique was pioneered for this
aspect of the process.
•
The opinions thus gathered are analysed, summarised and reported for consideration
within MOD.
Findings
Through horizon scanning, RAO particularly wishes to identify relevant developments in
domains not addressed by MOD’s current research programme. Identifying such areas is
non-trivial, and may in some cases fall outside the coverage of the ISI database currently
used for co-citation analysis.
Dstl/CR10342 1.0
Page 3 of 30
Co-citation analysis is one member of a class of bibliometric techniques that form an
established and internationally respected means for gaining insight into the structure of
research and its key contributors. Its use within the Science Foresight process appears to
be appropriate. For practical reasons, however, this form of analysis is currently limited
to the ISI database whose contributing authors will provide a mainly science-centric
perspective. Furthermore, the current process cannot draw on material that is not indexed
by ISI. This includes much of the World Wide Web, and other technical literature such as
books and reports.
The process of advising on the content of MOD’s research portfolio may begin with the
collection of academic and other scientific opinion, but can only be completed once
matched with knowledge of potential military applications. Whether explicitly or
otherwise at this stage, key items on the relevant technology roadmap need to be
understood.
The availability of military impact assessment would considerably increase the potential
readership within MOD and enable the participation of a much wider group. Even with
impact assessment, however, a paper (or electronic) report is still likely to be difficult to
navigate and to extract the information that is likely to be most useful.
Conclusions and recommendations
Taken overall, Science Foresight provides a practical and cost-effective means for
sampling opinion from a wide range of leading scientific authors. If the process is to be
incorporated into MOD’s future horizon-scanning activity, additional analyses will be
required. In particular:
•
MOD will need to devise a means for identifying fields of interest within which the
process of identifying leading scientific authors can be focused;
•
safeguards should be incorporated where possible to alert the readership to views
which are judged to be extremely atypical or potentially erroneous;
•
a means will need to be identified for ‘mapping’ scientific opinion into an assessment
of potential military impact;
•
if RAO wishes its horizon-scanning activity to include inputs from the technology
community in addition to those from the authors of science papers, as obtained by the
current process, an appropriate method will need to be developed1, and
•
a revised means for disseminating the results of the analysis should be introduced.
With these additional measures, the process could contribute significantly towards
satisfying RAO’s requirement for horizon scanning from open sources.
Appendix A provides further insight into the original Science Foresight Project and
describes possible enhancements towards the realisation of ‘dynamic foresight’.
1
The implementation of co-citation analysis on databases other than the ISI Research Fronts database may
be problematic.
Page 4 of 30
Dstl/CR10342 1.0
Table of contents
1
Introduction
7
1.1
Background
7
1.2
Aim of study
8
2
The Science Foresight process
9
2.1
The Science Foresight concept
9
2.2
The 2001 Science Foresight process
9
3
Assessment of the Science Foresight process
11
3.1
Domain selection
11
3.2
Identification of authors
11
3.3
Likely strengths of the author-selection method
12
3.4
Possible weaknesses of the author-selection method
13
3.5
Questionnaire considerations
14
3.6
Analysis of returns
15
4
Conclusions
17
4.1
General findings
17
4.2
Strengths of the Science Foresight method
17
4.3
Weaknesses of the Science Foresight method
17
5
Recommendations
19
5.2
General issues
19
5.3
Domain identification
19
5.4
Author identification
19
5.5
Information gathering
19
5.6
Reporting
20
6
Acknowledgements
21
Dstl/CR10342 1.0
Page 5 of 30
7
List of references
22
8
List of abbreviations
24
APPENDIX A From Foresight to Dynamic Foresight
25
A-1
Introduction
25
A-2
Expert Selection
25
A-3
Validation
26
A-4
Dissemination and Visibility
26
A-5
Dynamic Foresight
28
A-6
Appendix summary
29
Page 6 of 30
Dstl/CR10342 1.0
1
Introduction
1.1
Background
1.1.1
The UK MOD has long sustained a deep interest in relevant scientific and technical
developments within the civil sector. Knowledge of current and likely future advances is
used to underpin and complement MOD’s own research programme. Access to civil
technologies enhances the overall effectiveness with which MOD sustains the means for
defence. ‘The first step in acquiring access is understanding what exists, requiring, in
turn, that MOD maintains awareness of the developments in the civil sector world-wide’
(DG(R&T), 1999).
1.1.2
In practice, the evolution of technology has long been recognised as a complex sociotechnical phenomenon that is very difficult to monitor, to track and to forecast. Not only
is the range of information sources vast2, the estimation of relevance, possibly projecting
many years into the future, often requires deep human expertise and insight.
1.1.3
Since the Second World War, a number of authors have attempted to provide a
theoretical foundation for analyses of technology evolution. In parallel, a number of
programmes have implemented pragmatic solutions. Figure 1-1 charts a selection of
some of the more significant contributions and events.
Theoretical development
Information
economy
Skunk works
(LockheedMartin)
Knowledge
economy
Product
life-cycle
Technology
trajectories
Technology
forecast
surveys
(Japan)
Value
innovation
Virtual
organisations
UK Government
Foresight programme
EU Framework
Programme
’50
?
Disruptive technologies
Paradigm
shift
’40
Much to be done
on the
underlying theoretical
drivers
’60
’70
’80
’90
’00
Punctuated equilibria,
S-curves
’10
’20
’30
year
NB. curve is illustrative only
Figure 1-1: A timeline of relevant developments in technology evolution (Holland Smith & Kemp, 2003)
2
Broadly, every minute sees the publication of forty new pages of scholarly text
(http://www.sims.berkeley.edu/research/projects/how-much-info-2003), at least one new patent application
(http://swpat.ffii.org/news/04/oecd0130/index.en.html), and a new Web site
(http://wcp.oclc.org/stats/size.html).
Dstl/CR10342 1.0
Page 7 of 30
1.1.4
Considering MOD’s particular requirement for horizon-scanning in support of Output 5,
the definition of technology watch, as accepted by the Science and Technology (S&T)
Board, has strong relevance (especially if extended to science and technology watch3).
“monitoring developments in civil and foreign defence technology to identify
potentially disruptive technologies4 and inform balance of investment in our
own research programme” (see Henderson & Edwards, 2003)
1.1.5
Notably, this describes a process that works in addition to the literature search and
analysis that is conducted as part of the normal research process. Here, emphasis is
placed on assessing the current portfolio of research and supporting the management of
its content through time.
1.1.6
In 2001, MOD/Dstl, with the aid of SPRU5, designed and piloted a process that might be
adopted within Output 5 to provide a horizon-scanning service. This offers a ‘…simple,
objective and cost effective technique to gather information about emerging short and
long-term research developments…’ (Katz et al., 2001). Known as ‘Science Foresight’,
this process provides a means for ‘targeting’ influential authors and collecting their views
as to the future of their domain.
1.2
Aim of study
1.2.1
This report aims to provide an objective assessment of Science Foresight, as it might be
cast within MOD’s new ‘Output’-based framework for research delivery.
3
The process may also need to be aware of relevant market and economic forces, availability of funding,
relevant legislation and so on (Holland Smith et al., 2001).
4
Holland Smith & Kemp (2003) suggest that ‘disruptive’ here refers to ‘… technologies which simultaneously
change the way in which a capability is realised whilst displacing current solutions’. For the military,
disruption will typically require changes to force structure; but the phenomenon is equally at play in a
commercial context where market leaders can be overturned and supply chains radically re-shaped.
5
SPRU’s original title was the Science Policy Research Unit and hence the acronym. Its current title is
Science and Technology Policy Research.
Page 8 of 30
Dstl/CR10342 1.0
2
The Science Foresight process
2.1
The Science Foresight concept
2.1.1
Science Foresight, described in detail in Katz et al. (2001), essentially comprises four
sequential steps as shown in Figure 2-1.
Choose
technical
domain(s) for
study
Identify key
authors using
co-citation
method
Survey selected
authors’ views
by
questionnaire
Collate, edit and
report views
within MOD
Figure 2-1: The components of Science Foresight
2.1.2
The approach, once the domain(s) for investigation are known, is to automatically select
key academic authorities based on their published output, and then to collect their views
and forecasts via questionnaire. Through this means, the technique is relatively objective
and auditable, yet it has the potential to deliver significant insights at relatively low cost.
2.2
The 2001 Science Foresight process
2.2.1
Domain selection. The 2001 Foresight project focused on the physical and engineering
sciences. This choice was based on a study of the MOD Technology Strategy, the
Strategy for Applied Research 1999-2019, and the MOD Technology Taxonomy Issue 5
(1999) (Katz et al, 2001).
2.2.2
Identification of authors6. This was carried out using a form of bibliometric analysis
based on citation and co-citation statistics. ‘Co-citation’ occurs where two documents or
authors appear within a single reference list (i.e. are ‘co-cited’) in a subsequent paper7.
Documents that are co-cited are assumed to be closely related in terms of topic; the more
frequent the co-citation, the stronger the inferred relationship. In co-citation analysis,
statistics are compiled based on counts of the number of times that two works are cocited in later publications. Using these statistics for documents within a particular field, it
is possible to produce a representation or ‘map’ describing the field’s intellectual
‘structure’. Specifically, the ‘groups’ or ‘clusters’ of papers identified in this way
generally correspond to subspecialty areas or ‘research fronts’8, and may point to sociotechnical networks engaged in the process of generating important new knowledge.
2.2.3
Thomson ISI, which offers a commercial co-citation service known as the Research
Fronts database, was used as the source of published literature.
6
7
8
A more detailed account appears in (Katz et al, 2001)
The analysis is typically associated with formal technical documents.
A research front is a collection of between two and fifty co-cited papers that are clustered together.
Dstl/CR10342 1.0
Page 9 of 30
2.2.4
Science and engineering fields of potential interest were identified and approximately
500 of the most representative clusters were determined using a well-defined set of rules
(Katz et al., 2001). The most highly cited paper from the pair of most highly co-cited
papers published in the 1990s was then chosen from each cluster. The authors of these
papers were used to prepare the list of experts. In total, 483 highly co-cited papers were
selected, involving 2120 authors and representing research activity in 66 physical science
and engineering fields.
2.2.5
Information gathering. Authors, judged by the co-citation method as being leading
authorities within a particular research front, were contacted by letter and invited to
provide views as to the future of their specialisms. Access was provided to a secure
online questionnaire that covered background information, emerging developments in
science, factors affecting developments, and the driving forces in science.
2.2.6
Analysis of returns. Each author was asked to describe up to two short-term predictions
(covering the next five years) and one longer-term prediction (for the period of five to ten
years from now). Up to 300 words of free text was available for each prediction.
2.2.7
Analysis of these returns was limited to extracting one or two key sentences, or, if none
existed, providing a short summary. In all cases, care was taken to use the original
authors’ words and to remain faithful to their text. ‘No attempt was made to judge,
criticise or interpret a predicted emerging development. This task is left strictly to the
reader.’ (Katz et al., 2001).
Page 10 of 30
Dstl/CR10342 1.0
3
Assessment of the Science Foresight process
3.1
Domain selection
3.1.1
When used in conjunction with the Thomson ISI database, the technique is able to cover
a broad swath of science and engineering at relatively low cost9. The starting assumption
is that one or a small number of individuals can represent the scientific views of the
potentially large communities in which they work. To some extent, this reduces the need
to pre-specify the areas of interest10. However, there remains a trade-off between depth of
analysis and breadth of coverage, and the method may be best suited to the identification
of radical innovations as opposed to the much greater number of smaller, incremental
ones.
3.2
Identification of authors
3.2.1
A number of different strategies might be employed for author selection. One of the
simplest, given access to existing domain expertise, is the simple nomination or conomination11 of knowledgeable ‘colleagues’ or scientific peers. Such methods, based on
human judgement, are open to a range of biases and will not be equally effective in all
cases. One way to avoid such bias is to work with the full text of documents by studying
the words and their co-occurrence patterns. Callon and his colleagues’ co-word analysis
(Callon et al, 1991) and Kostoff’s database tomography (Kostoff, 2001) are examples of
such techniques: the former has been used to identify socio-technical networks, whilst
the latter can guide the production of domain taxonomies. Either can be used to draw
attention to a ‘balanced’ sample of a domain’s authors, but again, each requires human
input and can be quite labour-intensive.
3.2.2
Bibliographic databases store pre-extracted data about collections of published literature
and are able to underpin a range of automated ‘bibliometric’ analyses. Naturally, these
analyses can only be applied to data furnished by the database provider.
3.2.3
Publication counts simply indicate which authors are most prolific, not necessarily most
informative or influential. Citation counts say more about the influence of scientific
authors, but offer little information regarding the structure of a scientific community or
scientific advances. How can we know, for example, when two prolific and highly cited
authors are writing about significantly different advances? Science Foresight uses cocitation to identify and analyse research fronts and the authors working within them12.
The authors of later (citing) papers effectively contribute the measure, and its objectivity
reflects the fact that some degree of consensus has been established across many
independent authors.
9
The use of other data sources may significantly increase the costs of the process.
In fact, co-citation links have been shown to cross established disciplinary boundaries (Small, 1999), and
the analysis may lead to new and interesting developments outside the prescribed area of interest.
11
Co-nomination is an iterative process. It starts with a list of recommended experts and converges to a list
of experts who have multiple nominations by different experts (Katz et al., 2001).
12
Citation analysis then targets authors within research fronts.
10
Dstl/CR10342 1.0
Page 11 of 30
3.2.4
Ideally, we would wish to find authors who have just published papers that will later
prove to be influential. However, for co-citation to operate, not only must the original
papers have been published, but also a significant number of later papers that cite them.
Since a conventional publication cycle typically takes between 1 and 3 years13, the
authors identified through co-citation may no longer be ‘at the front’ of their research
area. In practice, a mixture of bibliometric and other techniques will undoubtedly deliver
the best results.
3.2.5
Before leaving the time-lag issue, it is worth noting that an alternative approach, known
as bibliographic coupling, simply assesses two document’s similarity in terms of the
overlap between their reference lists (see Morris et al, 2003 for a recent implementation).
This metric does not rely on later authors and has been shown to be capable of
identifying how a particular field has advanced and its structure has evolved over recent
years. Whilst bibliographic coupling is capable of generating groupings that may be used
alongside or in place of ISI’s Research Fronts, the lack of a commercial supplier for this
form of analysis may render it prohibitively expensive to implement on a large scale.
3.2.6
The co-citation analysis tended to identify senior scientists of long standing in their
fields, who were actively publishing in prestigious international journals14. To
complement this group, there may be value in targeting the authors of influential ‘review’
articles, which explicitly set out to draw conclusions from the work of a number of
researchers.
3.2.7
The ultimate assessment of the author selection technique used in Science Foresight is
whether the predictions gathered prove to be accurate. To a limited extent, it would be
useful to compare the predictions made using this technique with others for the same
field, but arrived at using different methods. The only true arbiter, however, is time.
3.2.8
In the absence of a practical means for validating the predictions, the following bulleted
list summarises a range of the more general strengths and weaknesses of co-citation
based techniques.
3.3
Likely strengths of the author-selection method
•
The metric is readily available through the ISI Research Fronts Database. Whilst it is
possible to analyse other sources using information extraction techniques, these can
be expensive to establish. Moreover, since automated processes are likely to perform
less robustly than a human analyst when working with written text, additional errors
may be introduced.
•
The assessment of science and technology outputs by citation counts is both
quantitative and relatively objective (see 3.2.3).
•
Co-citation is relatively straightforward, relies on few assumptions, and can be quite
cost-effective.
13
The consequence for this time-lag of the rapidly growing trend towards e-publication needs to be
investigated.
14
Since Thomson ISI takes relatively few journals in non-English, generally, these authors (or their coauthors) will have some familiarity with the language.
Page 12 of 30
Dstl/CR10342 1.0
3.4
•
‘Since only highly cited papers are used as the source for co-citation statistics, the
analysis is relatively robust to biases otherwise caused by excessive self-citation’
(Katz et al, 2001).
•
A single, large database, such as the ISI Research Fronts, does capture the bulk of
formally published ‘mainstream’ scientific literature. Studies indicate broadly that,
20% of scientific journals publish 80% of all articles (see for example, Geisler,
2000).
Possible weaknesses of the author-selection method
•
The time lag between the inception of a new line of research and the formation of a
co-citation cluster will usually be measured in years. For incremental
developments15, the average age of cited papers is five years although this may be
considerably less in some fast moving areas. There is therefore an issue over how far
back in time the citations should reach. (Geisler, 2000).
•
Typically, there may be an over-representation of theoretical as opposed to
experimental papers, due to the former’s relative ease of production (King, 1987).
Citation patterns may also vary between these two.
•
The ISI database covers some engineering and technology, but is strongly focused
towards science. The ‘science-push’ view (Irvine & Martin, 1984), which will
typically emerge from its scientific authors, may not, on its own, satisfy MOD’s
needs for both science and technology perspectives16.
•
The Science Foresight process ultimately requires the identification of individual
authors. The papers that are highlighted through co-citation, however, may be written
by more than one author. The statistics produced do not inform the choice of which
of several joint authors to approach17.
•
Citations may reflect either agreement or disagreement with the cited papers. While
the general pattern is overwhelmingly towards positive citation, notable exceptions
do exist.
•
The ‘obliteration phenomenon’ (King, 1987) refers to a known tendency for
breakthrough advances to be cited less frequently over time. This effect is domain
dependent but usually applies to papers that are decades old. The effect on co-citation
analysis is, therefore, small at this time although if the current trends toward
increasing volume and speed of publication continue, the phenomenon of obliteration
may eventually need to be taken into consideration.
15
As opposed to those which have more radical effect and whose time-scales may be measured in
decades.
16
US patents are required to cite the prior art in their subject, and the practice is not uncommon elsewhere.
Patent citation analysis may offer a quite different view of technical advances, here emphasising technology
over science.
17
There may be merit in considering author co-citation which accounts both for individual seminal papers,
and authors whose importance is due to a body of work (White & Griffith, 1981; McCain, 1990).
Dstl/CR10342 1.0
Page 13 of 30
•
The output of scientific endeavour, through scholarly journals, is typically focused
through small, self-selected groups of scholars who form an elite or ‘invisible
college’ within a given discipline (e.g. Geisler, 2000). Important developments
occurring outside these invisible colleges may not be cited as frequently as those
within. To the authors’ knowledge, the impact of this phenomenon on co-citation
analysis has not been systematically studied.
3.5
Questionnaire considerations
3.5.1
The Science Foresight Project specifically set out to identify an effective but low-cost
means for gathering the views of an international collection of leading specialists. Since
the specialists were not directly funded for their participation, the challenge was to reach
a geographically diverse group of senior people while providing them with the flexibility
to engage on their own terms.
3.5.2
Given the travel implications, formal interviews could only have been undertaken by
telephone or with some other on-line mediation. To judge from the current growth of
tele-marketing, this approach may warrant consideration, although the cost of analysts
who are able to ‘interview’ world-class scientists and engineers is likely to be high. In the
2001 exercise, a self-administered questionnaire method was chosen such that
individuals could provide an input at a time of their own choosing.
3.5.3
When one uses self-administered questionnaires, it is imperative to minimise the scope
for uncertainty or ambiguity in the interpretation of questions. This issue is particularly
acute when dealing with an international audience. In this case most respondents may be
assumed to at least be familiar with the language (see Paragraph 3.2.6). However, unlike
with researcher-administered questionnaires, additional indicators such as the
respondent’s apparent confidence and the time taken to complete each question cannot be
gathered18 (Peterson, 2000).
3.5.4
In the Science Foresight Project, open-ended questions were used to capture respondents’
views of the future in their domain. Katz et al. (2001) note the variability of responses
and that some predictions were much more difficult to summarise for reporting than
others. Whether this variation might be reduced by refinement of the questionnaire is
unclear.
3.5.5
To illustrate the complexities, consider the following question taken from the Science
Foresight Questionnaire:
Over the next five years do you expect the availability of funding in your research area
will?
Increase
Decrease
Stay the same
Cannot Predict
Table 3-1: Example question (Katz et al., 2001)
18
The time taken to complete an unsupervised online questionnaire cannot be accurately gauged, since it is
not possible to determine the cause of any interruptions (e.g. thinking time or tea time).
Page 14 of 30
Dstl/CR10342 1.0
3.5.6
This question might reasonably be answered from either a ‘local’ (i.e. personal,
organisational or national) or a ‘global’ perspective; the response given could therefore
differ according to the interpretation adopted. Furthermore, the ‘availability’ of funding
might be taken to refer to the total anticipated investment in a domain, or simply to the
number of independent awards likely to be available.
3.5.7
The Science Foresight Project broke new ground in the use of ‘secure’ questionnaires
which were hosted at SPRU’s web-site and completed on-line. Doubtless this technique
offers greater scope than traditional paper forms for ensuring the accuracy of gathered
information. For a future evolution of the Foresight process, MOD’s previous investment
in human factors research, and especially in interface design, might be reviewed to shed
light on such issues as:
•
the balance between open and closed questions19;
•
the ordering of questions;
•
respondent motivation and providing appropriate incentives to respond;
•
assessing the robustness of responses, and
•
matching the questionnaire with the subsequent analysis.
3.5.8
Respondents’ motivation to invest time and effort in co-operating with the survey needs
to be considered. In general, for non-funded returns, a good starting point for such
considerations is that respondents are seeking to influence the conclusions from the
exercise (Peterson, 2000). In the case of Science Foresight it was noted that respondents
may be gratified to have been ‘singled out’ by an objective process as being the
individuals whose opinion is worth eliciting.
3.5.9
In the 2001 Science Foresight, respondents were predominantly upbeat when predicting
that funding and collaboration would increase in the future (Katz et al., 2001). In reality,
this is unlikely to be the case for all areas, and may suggest a bias towards optimism20.
Furthermore, there was less optimism regarding the number of graduate students who
would be available to provide the increased research effort.
3.5.10
Drawing these observations together, it is seen that the robustness of the process is likely
to benefit from formal review or debate by peers. This, however, brings significant
implications for the cost of the exercise.
3.6
Analysis of returns
3.6.1
The identification of domain experts based on co-citation analysis assumes a correlation
between technical expertise and influence, on the one hand, and the ability to predict
19
In closed questions the respondent selects an answer from a fixed set of options, whereas in open
questions the respondent’s answer is not limited in this way and is usually discursive in nature.
20
Leading scientists will typically have devoted their working lives to the pursuit of research in their chosen
area. In general, they will regard this as the most important research area and tend to believe that others will
eventually come to see that too, and hence that this will therefore eventually be reflected in more funding.
Dstl/CR10342 1.0
Page 15 of 30
future developments, on the other. There are no known reasons to doubt this assumption,
although strictly speaking, there is little evidence to support it either. For MOD’s
purposes, ideal respondents would also have a clear view as to the potential military
relevance of their science.
3.6.2
Katz et al. (2001) acknowledge that analysis and presentation should be undertaken
differently in future iterations of their process. There are at least two levels at which this
thinking needs to be applied.
•
Summarisation. In the 2001 Science Foresight report, ‘…no attempt was made to
create a general overview of the emerging developments…’ (Katz et al., 2001). For
presentational purposes, respondents’ narratives were summarised into (typically)
two sentences, taking care to maintain the meaning of their words.
•
Abstraction. The assessment of military relevance and impact was not attempted in
the 2001 Science Foresight study. Given summarised information regarding the
future of science in particular domains, it was left to the report’s readers to determine
its military implications. Thus the original format for the report will achieve value
only within a subset of its potential readership. No attempt was made to collect
readers’ own analyses of impact and to facilitate their dissemination.
3.6.3
Whether explicit or otherwise at this stage, the analysis must develop an understanding of
the key items on the relevant technology roadmap that describes the passage(s) of the
science or technology through to its ultimate use(s).
3.6.4
In summary, Katz et al. (2001) describe Science Foresight as ‘…the act of looking
forward in an effort to anticipate the nature and significance of scientific
developments…’. For the RAO, in support of Output 5, this implies not only the need to
identify important science, but also to assess its particular relevance to the MOD. The
latter implies the need for a further stage in the analysis that might be termed ‘impact
assessment’.
Page 16 of 30
Dstl/CR10342 1.0
4
Conclusions
4.1
General findings
4.1.1
The 2001 Science Foresight study represents an interesting and welcome development in
MOD’s approach to maintaining an understanding of world-wide science and
technology. The approach is innovative with genuine novelty, especially in the use of online questionnaire completion. As designed, the process is squarely aimed at gathering
opinion across a broad and influential multi-national group, at low cost. Inevitably, some
compromises are required to meet these competing aims and the process may need to be
extended if it is to be adopted within Output 5. Most notably, these include impact
assessment and improved reporting.
4.1.2
Accepting that it is too soon to judge the accuracy of the Science Foresight output, it is
nevertheless possible to summarise some of the salient strengths and weaknesses of the
method.
4.2
Strengths of the Science Foresight method
4.3
•
Relative to other methods, the technique provides broad coverage at low cost,
especially in cases where one is seeking to identify innovations that are more radical
in their potential impact.
•
The approach generates a number of possible futures rather than consensus on just
one outcome.
•
The process is consistent with the MOD’s technology awareness ‘level’ of
Technology Watch21 (Henderson & Edwards, 2003).
•
The online questionnaire approach facilitates the engagement of senior and busy
scientists.
•
Information collected on limiting factors and driving forces can be used to support
and qualify forecasts.
Weaknesses of the Science Foresight method
•
Domain selection. The technique is subject to a trade-off between depth of analysis
and breadth of coverage and may be poorly suited to the identification of incremental
innovations. This will be the case, even in a narrowly specified domain, where papers
reporting ‘lesser’ innovations may fall beneath the thresholds required for inclusion
in the Research Fronts database.
21
This level identifies potentially interesting new technologies but does not involve a formal assessment of
MOD significance.
Dstl/CR10342 1.0
Page 17 of 30
•
Identification of authors
− Uses published findings that are, typically, several years old22.
− Is limited to broadly scientific texts. Respondents hence may know little about
translating their science into technology and systems.
•
Information gathering
− The questionnaire method may be susceptible to subjectivity and to respondent
bias23 and lacks some of the quality assurance provided by peer review.
− The approach offers little incentive for respondents to share new and particularly
innovative concepts.
− Information is returned as a set of scientific opinions, in narrative form, from
which it may be difficult for a non-domain specialist to abstract common themes
and principles.
•
Reporting
− The lack of military ‘impact assessment’ reduces the scope of the readership
within MOD and hence the size of the potential audience that will be interested.
− The gathering of a rich set of information from all respondents in a single
technical report, even in summarised form, was acknowledged to result in poor
‘navigability’.
22
If this is a significant issue in practice, it will apply more to the identification of incremental than radical
advances.
23
i.e. those who respond may have systematically different views from those who do not respond.
Page 18 of 30
Dstl/CR10342 1.0
5
Recommendations
5.1.1
A series of recommendations follow from the foregoing analysis and conclusions. Where
possible, these are grouped under the headings of the conceptual process. In support of
this, Appendix A provides further insight into the original Science Foresight Project and
describes possible enhancements that would move towards the realisation of ‘dynamic
foresight’.
5.2
General issues
5.2.1
There is an issue over ‘ownership’ of the study output. MOD staff who have the power to
act on the results, should be engaged as appropriate in the process design and in the
analysis.
5.3
Domain identification
5.3.1
The Science Foresight process as described does not include a specific means for
identifying new and potentially ‘interesting’ domains. MOD does, however, need to
continually re-assess its current understanding in this respect. This process should,
therefore, either be considered as one of a range of techniques for ‘horizon scanning’, or
a separate process for domain identification should be designed and implemented.
5.4
Author identification
5.4.1
The following points should be addressed when re-designing a future Science Foresight
process.
•
The possible virtues of bibliometric coupling should be considered, along with issues
for its operationalisation if appropriate.
•
The respective merits of citation databases, the World Wide Web and e-publications
as sources of author identification should be re-assessed at intervals.
•
It may be possible to incorporate the opinions of ‘review’ paper authors to
complement those of the leading specialists captured by the present method.
•
The importance to Science Foresight of incorporating a technology-centric view
should be considered in detail. If required, a means should be established for
collecting this input alongside the current science-centric views.
5.5
Information gathering
5.5.1
The questionnaire should be appraised for the possibility of capturing domain-specific
jargon used to describe key developments. This will enable the retrieval of further
supporting material and may be used to draw attention to extreme or potentially
Dstl/CR10342 1.0
Page 19 of 30
erroneous views. Alternatively, it may be possible to achieve this aim through textual
analysis (see 5.5.3. below).
5.5.2
The timing of the questionnaire survey should be matched to the availability of the
scientific target audience. Mid-way through an academic term is probably the optimum
for academic scientists.
5.5.3
Textual (or other) analyses (see, for example, Kostoff, 2000) should be considered for the
papers within each cluster. Such analyses could generate hypotheses upon which the
selected experts could be asked to comment.
5.6
Reporting
5.6.1
If the Science Foresight process is to complement MOD’s Technology Watch
programme at levels two and three, then a process for analysing questionnaire responses
in order to assess the military impact of scientific advances should be designed and
costed.
5.6.2
The format of the final report should be reviewed. It may be preferable to replace this
entirely with a well-indexed, subject-searchable database (possibly using MOD
definitions/taxonomies). Appendix A describes an alternative, and potentially powerful
approach, which treats the process as dynamic.
Page 20 of 30
Dstl/CR10342 1.0
6
Acknowledgements
6.1
The authors acknowledge the kind assistance of Sally Stewart, formerly of Dstl, who was
instrumental in pioneering the Science Foresight process within MOD. Our thanks are
also due to Professor Ben Martin of the University of Sussex for his detailed technical
review and kind suggestions for improvement of the text.
Dstl/CR10342 1.0
Page 21 of 30
7
List of references
7.1.
Callon, M., Courtial, J-P. & Laville, F. (1991) Co-word analysis as a tool for describing
the network of interactions between basic and technological research: The case of
polymer chemistry. Scientometrics, 22, 1, pp.155-205.
7.2.
DG(R&T) (1999) MOD strategy for keeping track of world-wide civil sector R&T.
D/DG(R&T)/9/22, June 1999.
7.3.
Geisler, E. (2000) The Metrics of Science and Technology, Quorum Books, Westport,
C.T.
7.4.
Henderson, J. I. & Edwards, B. (2003) Improving technology watch in the Corporate
Research Programme (CRP). Distillation, Issue 04.
7.5.
Holland Smith, D. J. & Kemp, N. M. (2003) Text Analysis for S&T Intelligence,
Innovation and Creativity: Final Customer Report. QinetiQ Customer Report.
QINETIQ/FST/CR031323/1.0.
7.6.
Holland Smith, D. J., McOwat, D. M. & Marks, L. M. (2001). Development of a costeffective science and technology intelligence process. DERA Technical Report.
DERA/TD/TR000004/1.0, March.
7.7.
Irvine, J., & Martin, B.R. (1984). Foresight in science: Picking the winners. London:
Frances.
7.8.
Katz, J. S., Stewart, S., Gow, T.A.K. & Martin, B. (2001) Science Foresight Project:
Final Report. Volume 1. Dstl Technical Report. DSTL/TR01697.
7.9.
King, J. (1987) A review of bibliometric and other science indicators and their role in
research evaluation. Journal of Information Science, 13, 5, pp.261-76.
7.10.
Kostoff, R. N. K. (2000) Fullerene data mining using bibliometrics and database
tomography. Journal of Chemical Information and Computer Science, 49, 8, pp.674-85.
7.11.
Kostoff, R. N. K. (2001) The role of open source intelligence in global technology watch.
Military Information Technology.
7.12.
McCain, K.W. (1990). Mapping authors in intellectual space: A technical overview.
Journal of the American Society for Information Science, 41, pp.433-443.
7.13.
Morris S.A., Yen G., Wu Z. & Asnake, B. (2003) Time line visualization of research
fronts. Journal of the American Society for Information Science and Technology. 54, 5,
pp.413-422.
7.14.
Peterson, R.A. (2000). Constructing Effective Questionnaires. Sage Publications Inc.,
Thousand Oaks, California, USA.
Page 22 of 30
Dstl/CR10342 1.0
7.15.
Small, H. (1999) Passage through science: crossing disciplinary boundaries. Library
Trends, 48, 1, pp.72-108.
7.16.
White, H.D. & Griffith, B. (1981). Author co-citation: A literature measure of intellectual
structure. Journal of the American Society for Information Science, 32, pp.163-71.
Dstl/CR10342 1.0
Page 23 of 30
8
List of abbreviations
IRAP Industrial Research Assistance Programme
NRC National Research Council [of Canada]
ISI
Thomson ISI publisher and database provider
S&T
Science and Technology
SCI
Science Citation Index
SPRU (formerly) Science Policy Research Unit [University of Sussex]
(now) Science and Technology Policy Research
Page 24 of 30
Dstl/CR10342 1.0
APPENDIX A
From Foresight to Dynamic Foresight
This appendix has been contributed by Dr J. Sylvan Katz of Katz Competitive
Intelligence Ltd, Saskatoon, Canada and part time Senior Fellow at SPRU,
University of Sussex. Dr Katz was a manager and principal author of the Science
Foresight study undertaken in collaboration with MOD and Dstl during 2000/1.
A-1
Introduction
A-1.1.1
The Science Foresight Project was apparently the first international on-line foresight
activity. The aim of the project was to design and assess a simple, objective and costeffective technique to gather information about emerging short and long-term
research developments, primarily in the physical and engineering sciences. During
the course of the project, and through subsequent involvement in two focused
horizon-scanning activities, new ideas have emerged that could improve the original
Science Foresight process. This appendix discusses possible ways:
•
to improve expert selection;
•
to validate predictions of emerging developments;
•
to increase the visibility and dissemination of foresight information, and
•
to add more value to expert predictions by moving from conventional foresight
to dynamic foresight.
A-2
Expert Selection
A-2.1.1
Experts were selected from the ISI Research Fronts database by starting with a list of
Science Citation Index (SCI) subject categories24 from which highly co-cited papers
were to be selected. Currently, the approximately 8,500 journals covered within the
Science, Social Science, and Arts and Humanities Citation Indexes, are classified
into 170 subject categories. Of these, 66 science and engineering categories were
selected for the original study, based on information derived from the MOD
Technology Strategy, the Strategy for Applied Research 1999-2019, and the MOD
Technology Taxonomy Issue 5 (1999) (Katz et al., 2001). About 500 highly co-cited
papers were chosen from the Research Fronts database25 using queries (in the SQL
language) and custom scripts (written in Perl). Some manual manipulation of the
data was required when, for example, Access tables were transformed into formats
suitable for processing by the Perl programs and vice versa.
24
Examples of subject categories are acoustics, energy and fuels, applied mathematics and
telecommunications.
25
Supplied by ISI as a Microsoft Access database.
Dstl/CR10342 1.0
Page 25 of 30
A-2.1.2
If the Science Foresight approach is to be operationalised for ongoing use, then
further thought should be given to ways of improving expert selection. In particular,
a user-friendly interface could be developed that allows non-programmers to select
highly co-cited papers based on a set of well-defined selection parameters.
A-2.1.3
Perhaps some consideration could be given to examining the way highly co-cited
papers are selected according to the subject categories of interest that are being
scanned. For example, customised subject categories could be developed that are
more closely aligned to the fields that the MOD/RAO is interested in scanning.
However, it must be pointed out that this could be costly and time-consuming since
it requires the participation of people knowledgeable about the contents of the
journals and the fields of interest. An alternative approach to selecting research
fronts of interest might be to run a keyword search over the descriptive labels given
to each cluster in the research front database. The difficulty with this approach is that
the keywords can be quite specific and it could be difficult or time consuming to
develop an effective keyword search strategy.
A-3
Validation
A-3.1.1
The problem of validating expert predictions was one of the major issues
encountered in the original project. In the original project, experts were asked to
describe two short-term and one long-term emerging development in their research
fields. Unfortunately, the only way to confirm the validity of their predictions would
be to wait for time to pass and then to examine the literature for supporting
information. This would be a costly and time-consuming process that might not be
worth pursuing.
A-3.1.2
There is a novel approach that could be used to gather predictive information that
can be at least partly confirmed in the present. Instead of asking experts to predict
emerging developments, they could be asked to describe important research
problems that are currently being investigated in their disciplines and specialities.
Furthermore, they could be asked to provide one or two references to peer-reviewed
papers that discuss the problems and their significance. In addition, the experts might
be asked to hypothesise about the nature of potential radical and incremental
developments that might emerge once those research problems had been solved.
A-3.1.3
The implication of using this approach is that the validity of the research problems
can be established immediately by examining the reference publications and crossreferencing responses from other experts in the same disciplines and specialities.
While the potential technical developments cannot be confirmed until later, they can
be examined by knowledgeable people with interests in emerging technologies
relevant to the military.
A-4
Dissemination and Visibility
A-4.1.1
The findings from the original Science Foresight Project were disseminated through
a printed report and an interactive web site. The printed report was valuable for
documenting the objectives, methodology and findings; however, it was not
Page 26 of 30
Dstl/CR10342 1.0
particularly satisfactory for presenting the richness and diversity of the predicted
emerging developments. In the second phase of the project an interactive web site
was created to overcome this limitation. The web site contained:
•
a downloadable PDF version of the printed report;
•
HTML versions of the abstract, executive summary and the observations and
recommendation; and
•
searchable databases of the highly co-cited papers from which the experts were
chosen and of the predicted short- and long-term emerging developments.
A-4.1.2
Generally speaking, the dissemination process did not draw as much attention from
the wider foresight community as might have been hoped. This was probably due to
the lack of time and funds needed to present the project findings at international
conferences and to write peer-reviewed publications.
A-4.1.3
The interactive web site was located within the SPRU web site at the University of
Sussex. Unfortunately, the university does not have a method for easily tracking the
amount of traffic that the web site generated or the types of queries that were being
made against the databases. However, a Google search using the phrase “Science
Foresight Project” indicates that the web site has a reasonably good Google page
rank and about 30 web sites have links to the project web site.
A-4.1.4
There are a number of ways to improve the dissemination of the findings of future
foresight projects. One of the significant ways that the visibility of a future foresight
project might be increased is to involve members of the international foresight
community. However, this introduces questions like “what role, if any, should these
organisations play in shaping future projects?” On the other hand, there is little doubt
that these organisations could provide valuable inputs into the interpretative and
dissemination stages. By way of example, the unit for Science and Technology
Foresight in the European Commission’s Research DG for the knowledge-based
society and economy could be approached. This unit is responsible for monitoring
S&T Foresight activity around the world and it is already familiar with the Science
Foresight Project. In addition, the leader of the Office of Technology Foresight at the
National Research Council of Canada has expressed an interest in future foresight
projects. This group receives foresight input from Defence R&D Canada, who have a
‘disruptive technologies’ working group and a ‘technology watch’ function, and the
Natural Sciences and Engineering Research Council of Canada whose interests lie
mainly in the realm of emergent and convergent shifts in disciplines and domains.
A-4.1.5
The web site for the original project should be considered as a prototype that can be
improved for future foresight activities. Consider, for example, the possibilities that
would be available if the information solicited from the expert community was
adapted to facilitate the new approach outlined in section A-3. In this case, not only
would experts provide brief descriptions of the important research problems that are
being investigated but they would also provide references to publications supporting
their claims. A bibliographic database of these references, perhaps including
abstracts, could be included with the other data about the highly cited papers and the
important research problems identified experts. Furthermore, hyperlinks could be
Dstl/CR10342 1.0
Page 27 of 30
provided to the full text for those documents that can be found on the Internet. These
reference documents would not only provide support information but they could also
be used as the starting point for other types of bibliographic and bibliometric
investigations.
A-4.1.6
It might be worth reconsidering the location of the web site. Perhaps greater Internet
visibility and ‘product branding’ can be achieved by locating it on its own server
with its own domain name. The affiliation with SPRU, Dstl and the MoD can easily
be maintained through careful design of the graphical presentation on the web pages.
This approach would have a number of advantages. For example, the server can be
configured to track the amount of traffic and general location of visitors to the web
site. It can monitor the types of searches that are made of the databases and the
number of times reference documents are accessed. In addition, a discussion forum
could be included on the server to gather comments and suggestions from the
broader international community about the value of the project so they could make
contributions to the information that has been gathered.
A-5
Dynamic Foresight
A-5.1.1
A foresight activity requires a significant investment of time, money and effort to
gather and organise expert opinions so that they can be explored and digested
relatively easily. Furthermore, other than through the preparation of formal reports
and briefs, relatively few simple techniques are available for users to record and
exchange views dynamically as they explore a foresight database. If such techniques
were available, then it would be possible to move from the current rather static
foresight activities to a more dynamic foresight process.
A-5.1.2
In the autumn of 2001, the Industrial Research Assistance Program of the National
Research Council of Canada (NRC-IRAP) asked Dr Katz to assist in a short-term,
narrowly focused foresight activity. This centred on the potential uses of prairie
crops for enhancing health. Using experience gained from the Science Foresight
Project, a dynamic foresight system was prototyped, built on a web-based knowledge
system. This project was successful and a second project was undertaken to
prototype a dynamic foresight system with expanded capabilities that could be used
in strategic planning foresight activity for the NRC’s Plant Biotechnology Institute.
The second prototype was constructed in collaboration with QCC Communications
Inc, a publicly traded Canadian company that specialises in e-business and network
applications development. This project was completed successfully and it laid the
groundwork for the development of a fully functional foresight system.
A-5.1.3
Generally speaking, dynamic foresight is a technique that overcomes some of the
shortfalls of conventional static foresight activities. It uses Internet-based methods
to disseminate and capture value-added information in a cost-effective and
continuous manner. It allows users to view and maintain a foresight knowledge base
with a browser through a secure public, private or semi-private portal. Most
importantly, the accuracy and integrity of the information in the knowledge base is
maintained through collaborative and peer-reviewed processes.
Page 28 of 30
Dstl/CR10342 1.0
A-5.1.4
Dynamic foresight has three main ingredients: a focus, a moderator and an interest
group. The focus is any well-defined area that is being monitored. The moderator is
an individual or group of individuals responsible for ensuring that the information in
the knowledge base is current and accurately reflects the emerging developments in a
focus area. An interest group is composed of the users of foresight information. The
users are the human knowledge base and reviewers of the content of the foresight
knowledge base.
A-5.1.5
A dynamic foresight system contains the tools that allow web-naive individuals to
create and manage information in a knowledge base. Initially, the knowledge base is
constructed by experts with knowledge about a focus area. A knowledge base can
contain information for many different but perhaps related focus areas. Users of the
foresight information are asked to advise the moderator and other users about other
emerging developments and innovations that they find as they explore the
knowledge base.
A-5.1.6
Conceptually, dynamic foresight uses a table of contents to organise information. A
table of contents has a hierarchical structure and it contains hyperlinks to information
located on the Internet. The information could be located anywhere including on
secure servers requiring special authorisation. The source of information could be
anything that can be viewed with a browser such as (but not limited to) Microsoft
Word documents, searchable databases and media files. As users explore the
knowledge base and other resources, they may locate information that is relevant to
the focus area. Alternatively, it may be that the organisation of the table of contents
would more accurately reflect the emerging activities in a focus area if the structure
or entries in the table of contents were moved, removed or modified. Users provide
such feedback to the moderator and other users by email or through an on-line
discussion system. The moderator maintains the integrity of the knowledge base by
looking for user consensus before authorising a modification to the table of contents.
Over time the table of contents and hence the knowledge base will change to reflect
the emerging developments in the focus area. The dynamic foresight system is
designed to allow certain groups of users to view only selected portions of the table
of contents and hence limit their access to portions of the knowledge base. This
feature facilitates gathering feedback from the broader foresight community as well
as from restricted and confidential groups of users.
A-5.1.7
A dynamic foresight system may not only extend the useful lifetime of a foresight
activity but it may also be used for other purposes. For example, it can be employed
as the starting point for future foresight activities. Experts could be asked to reshape
and update a knowledge base. Furthermore, indicators of emerging developments in
a focus area can be constructed by tracking the changes in the structure of the table
of contents over time.
A-6
Appendix summary
A-6.1.1
This appendix outlines a variety of ways that the original format of the Science
Foresight Project could be modified to enhance the process of selecting experts, to
validate information about research from which radical or incremental technologies
could emerge, to increase the visibility and dissemination of the information that is
Dstl/CR10342 1.0
Page 29 of 30
collected and to extend the useful life-time of a foresight activity through the
addition of value-added information. The implementation of these ideas will depend
on such things as the possibility of integrating foresight activities into broader
horizon-scanning activities as well as the funds and resources available for future
foresight activities.
Page 30 of 30
Dstl/CR10342 1.0