Subject Knowledge, Thesaurus-assisted Query Expansion
and Search Success
Anne Sihvonen & Pertti Vakkari
Department of Information Studies
University of Tampere
FIN-33014, Finland
{Anne.Sihvonen, Pertti.Vakkari}@uta.fi
Abstract
This study explored how experts and novices in pedagogics expanded queries supported by the ERIC thesaurus,
and how this was connected to the search success in an easy and a difficult search task. The expert group
consisted of 15 undergraduates in pedagogy and the novice group of 15 students with no studies in this field.
Their search logs were recorded and a pre- and post-search interview was conducted. The results show that the
number and type of terms selected from the thesaurus for expansion by experts improved search effectiveness,
whereas there were no connections between the use of thesaurus and improvement of effectiveness among
novices. Thus, a vital condition for benefiting a thesaurus in query expansion to improve search results is
sufficient familiarity with the search topic. The results suggest also that it is not in the first place the number of
terms used in expansion, but their type and quality which is crucial for search success.
Introduction
Query formulation, especially the selection of terms, is crucial for search success. Searchers may
have difficulties in expressing their information needs in query terms leading to short and simple
queries with poor results (Sutcliffe & al. 2000).
Thesaurus is one of the major means of providing searchers with terminological support in the query
formulation. A thesaurus consists typically of controlled vocabulary, which represents the semantic
relations between the terms. A searching thesaurus is developed for supporting users in query
formulation and expansion by suggesting additional terms, synonyms and narrower terms, in
particular. (Aitchinson & al.1997). Its strength lies in its ability to provide searchers with information
about the equivalence and hierarchical relations of query terms. Thus, the thesaurus does not only
support in term selection, it also helps searchers to structure and articulate their information needs
within a query concept (Vakkari 2002).
Electronic thesaurus versions have strengthened its role as a search aid (Pollard 1993). Many
operational systems accessible via the Internet have incorporated thesauri in their interface as a part of
their browsing and searching facilities (Shiri &al 2002). Despite the increased provision of
commercial thesaurus-enhanced interfaces, no empirical studies have been reported, which evaluate
these interfaces (Shiri &al 2002).
Cognitive structures reflecting the subject knowledge of searchers, in their part steer information
searching (Ingwersen 1999). Subject knowledge is mediated by two mechanisms to term selection and
search success. First, the level of subject knowledge co-varies with individuals’ ability to articulate
information needs and consequently, with the identification of search terms. Secondly, it affects the
relevance assessment of documents retrieved. (Vakkari 2002). We can expect, that subject knowledge
has an influence on how searchers utilize a thesaurus in searching.
The aim of this study is to explore how experts and novices in pedagogy select search terms in
reformulating their queries supported by a thesaurus, and how this is related to the search outcome.
Earlier empirical results
Connections between subject knowledge and thesaurus use are an unexplored terrain. Therefore firstly
we will briefly introduce studies on the relations between subject knowledge and searching, and
secondly on thesaurus use in interactive query formulation.
Subject knowledge and searching
Relations between subject knowledge and searching have been explored in longitudinal studies on
students’ learning task performance and searching (Wang 1997; Vakkari 2001), and in studies on
expert-novice setting within a search session (Hsieh-Yee 1993; Shute & Smith 1993).
Hsieh-Yee (1993) showed that subject expertise had an influence on search tactics and term selection,
if the searcher had sufficient searching skills. On this condition, subject experts used more of their
own search terms than novices, whereas novices turned more to thesaurus terms, and put more effort
into query formulation experimenting with various term combinations.
By comparing searches by two intermediaries, one of whom was a subject expert and the other not,
Shute & Smith (1993) found that the expert used more terms than the novice. The former’s ability to
understand the semantic context of the terms contributed in finding more appropriate terms for query
expansion.
The growth of knowledge in the task performance process by subjects is reflected in the search
process. During task performance the vocabulary grows and becomes more specific. Especially the
number of narrower and related terms grows (Vakkari 2001; Wang 1997), as well as the number of
tactics (Vakkari 2001). Also the relevance assessments become more selective (Vakkari 2001).
The role of thesaurus in term selection and query expansion
Studies have explored the use of thesaurus either from the angle of search process or outcome. It has
been found that about half of the terms used by the search intermediaries (Fidel 1991) or end-users
(Nielsen 2002) come from the thesaurus.
The utility of using a thesaurus has been measured by user satisfaction and by search effectiveness.
Users seem to experience the thesaurus as useful for finding new search terms (Jones & al 1995; Shiri
& Revie 2001). Users tend to regard the thesaurus as beneficial as a source of more specific
expressions, especially for finding synonyms and parallel terms (Nielsen 2002).
Experimental results demonstrate that a thesaurus can be beneficial for query expansion (Beaulieu
1997). Jones & al. (1995) found that thesaurus-based interactive query expansion for users own search
topics increased recall and brought new relevant documents in the top of the list in a best match
system. These documents, however, often usurped the places of those from the original query, leaving
fairly constantly the overall proportion of useful material which users are likely to see.
In all, although the thesaurus is considered to be an important aid for selecting terms in query
formulation, findings on its contribution to the search process and outcome are scarce and scattered.
The satisfaction experienced by users and search effectiveness hint that the thesaurus is beneficial in
query expansion.
Research questions and design
Research questions
The aim of this study is to analyse how domain experts and novices search information assisted by a
thesaurus and to what extent they find relevant references? This main problem is divided to the
following sub-problems:
• What terms experts and novices select for original and thesaurus-aided, expanded queries, and
how the thesaurus helps in this selection?
•
•
How many relevant references experts and novices retrieve in the process?
How the selection of expansion terms is connected to the search success in these two groups?
Research design
We compared thesaurus-aided searches of 15 students in pedagogics (experts) and 15 students with no
credits in pedagogics (novices).
Fifteen experts were recruited from the course on “Open learning environments and multiform
teaching”. It is an advanced course at the end of undergraduate studies in pedagogics. The experts had
approximately 25 credits of the required 40 for a BA in pedagogics. The group of novices consisted of
fifteen students with no previous study of pedagogics.
The experiment was integrated in the course so that the participants could utilize the results of the
searches in their course assignment. In this way we tried to increase the realism of the research design
(cf. Borlund 2000) at least in the part of the experts. The novices did not had a similar incentive.
However, they spent approximately as much time in searching as the experts. They also used as many
tactics in the easy task as experts (10.1 vs. 9.6) and more in the difficult task (10.6 vs. 14.4). Thus,
novices seemed to put as much effort in searching as experts.
Neither of the groups had credits in information studies. The novices rated their searching experience
slightly better than the experts on a five-point scale from weak to excellent, but the difference was not
statistically significant. There were no self reported differences between the groups in the ability to use
Boolean operators. Thus, the variation in search skills between the groups was controlled.
The system used was the ERIC database with ERIC Wizard interface and thesaurus
(http://ericae.net/scripts/ewiz/amain5.asp). It is a Boolean system. Terms provided by users are
commented by the system if these do not match with the thesaurus and alternatives are suggested from
the ERIC thesaurus.
Search tasks All participants performed two search tasks. The tasks were connected to the contents of
the course from which the experts were recruited. They were designed in cooperation with the course
teacher. We used simulated search tasks consisting of a short cover story, which gave the scenario of
the use situation of the search results, and the actual search task (Borlund 2000):
•
•
Task 1. Find documents about “email as a forum for teaching critical thinking/argumentation” for
writing an assignment for a course in pedagogics on Open learning environments and multiform
teaching.
Task 2. You are acting as a member of a team designing a leaning environment. For that purpose
you are searching for documents about “Relations between learning conceptions and virtual
learning environments”.
The first search task was designed to be easy and the second one to be difficult. White & Iivonen
(2002) suggest that a difficult search task is general, complex and unclear, whereas an easy search task
is simple, specific and clear. It also includes fewer concepts than a difficult search task. Therefore, for
the first task we selected specific and familiar concepts like “email” and “teaching”, but for the second
task terminology which required more familiarity with pedagogics like “learning conceptions” and
“learning environment” was selected. Our aim was also to vary the number of query concepts (facets)
between the two tasks, but this was restricted by the interface allowing a maximum of three concepts
per query.
The facets of the first, easier task were: 1) email, 2) critical thinking or argumentation, 3) teaching.
The facets of the second, more difficult task were 1) learning conceptions, 2) virtuality, 3) learning
environment. Thus, the difference in the difficulty between the tasks was based on the fact that the
easier task consisted of more specific and familiar concepts than the difficult task.
The test participants’ comments in thinking aloud during searching indicated that they felt more
difficulties in selecting terms for the second task. This corroborates that the second task was more
difficult than the first one as experienced by the searchers.
Search sessions The search sessions included the following elements. 1) A structured pre-interview
eliciting background information, 2) A brief introduction to the search system and thesaurus, 3)
Forming the initial query plan. The plan was sketched in a form by defining the major concepts and
expressing them by query terms, which were asked to be placed in boxes representing each concept
(facet), 3) the expansion of the initial query plan by using the thesaurus and its execution, 4) relevance
assessments of the first 20 references from the expansion, and an assessment of the overall satisfaction
with the references retrieved, 5) a structured post-interview measuring the experienced utility of the
thesaurus by users.
The search processes were recorded in the search logs of a separate ERIC Wizard web-site created for
this study by ERIC. The transactions were also recorded from the computer screen by using video
capture. Search terms were identified by using this data.
The students were asked to think aloud in the sessions, which were observed and recorded by the
researcher.
Query formulation and expansion A facet is an exclusive aspect of a search topic. A facet includes
query terms that play a similar semantic role or are interchangeable in a query. Terms within a facet
are combined by Boolean disjunctions. The facets in their part are typically combined by Boolean
conjunction (Sormunen 2000).
The ERIC interface supports query formulation by providing boxes representing facets, in which users
can place appropriate terms. The terms within boxes are combined automatically by OR operators and
between boxes by AND operators.
The users expanded the original query plan as it was expressed in the forms by consulting the
thesaurus. They might add or delete their own terms or thesaurus terms. The system commented on
terms provided by the users and suggested alternatives from the ERIC thesaurus.
Query formulation has an impact on the structure of the query and consequently, on the search results.
The structure of a query consists of three dimensions. Exhaustivity consists of the number of facets in
a query. The extent of the query refers to the number of terms used to express a facet. The specificity
of the query depends on how specific terms are used to express the facets it contains. Manipulation of
the dimensions affects precision and recall of search results (Sormunen 2000).
Search success The users were asked to assess the topical relevance of the first 20 references retrieved
by the expanded search. They were asked to classify the references as relevant, if the search topic was
the main theme of the reference, or as relevant to a certain extent, if the search topic was a minor
theme of the reference. References off target were non-relevant.
Those references judged as relevant or to some extent relevant by the users in expanded searches were
sent to two external experts in pedagogics to form a recall base. In the first search task a total of 61
relevant and to some extent relevant references were retrieved, and in the second task 235 references.
From the initial queries there were no relevance assessments by the users available due to the time
limitations. Our pre-test showed that including relevance judgements of the initial queries would have
prolonged the session too much. Therefore the initial queries were replicated by a researcher and the
top 20 references were compared to the sets of relevant references from the expanded queries. If a
reference was not included in these sets it was added to the list to be sent to the expert judges. Thus,
the list assessed by the external judges consisted of relevant and to a certain extent relevant references
and from the replicated initial queries those references not having a relevance assessment.
References from the difficult search task were judged by the teacher of the “Open learning
environments” course, and from the easier task a doctoral student in pedagogics familiar with the
topic. They assessed the references based on the same three-point scaling of relevance as the
searchers.
Search success was measured by counting the number of those references retrieved which were judged
as relevant by external experts. We excluded the references they considered as only being relevant to a
certain extent. By excluding the less relevant items, our aim was to ascertain that most of the
references in the recall base were in principle possible to assess reliably by the novices. Studies have
shown that less knowledgeable users in a field tend to select documents by looser relevance criteria
and assess information as partially relevant more often than experts (Vakkari 2001). Vakkari &
Sormunen (2004) have also shown that users’ likelihood of identifying a document as relevant
increases when the degree of the relevance of documents increases. Thus, excluding the less relevant
documents enhanced the reliability of search success.
The final recall base in the easy task consisted of 22 relevant references, and in the difficult task of 80
relevant references. Search success was measured by the number of relevant items retrieved in
expansions at the top 20 documents.
Subject knowledge Subject knowledge refers to the degree of knowledge a person has in a particular
field. The more knowledgeable a person is the more comprehensive, differentiated and integrated is
his conceptual structure concerning the field. An expert’s knowledge structure consists of more
concepts, which are more specific, and more interrelated than a novice’s knowledge structure
(Robertson 2001). Thus, an expert is more able to articulate query terms and to identify them in the
thesaurus (Vakkari 2002).
Subject knowledge was operationalized by using expert-novice comparison. This procedure has been
used in several studies on searching (e.g. Hsieh-Yee 1993; Shute & Smith 1993). The difference
between the groups has been described in the beginning of section 3.2. Research design.
Results
Selection of terms
First we will explore the number of terms used and the exhaustivity, extent and specificity of queries
in expert and novice groups. Thereafter we will analyse the role of users’ own and thesaurus terms in
expansion, and explore the selection process of thesaurus terms and the type of terms selected.
In the easy task we found in the expert group an outlier with values in term variables much greater
than in the rest of the group. We removed this case from the data.
Terms and facets in the initial and expanded queries In both groups more terms were used in the
initial and expanded queries in the easy task compared to the difficult task (Table 1). The experts
formulated the initial queries of the easy task by using about one term more than the novices, whereas
in the difficult task there were no differences in the number of initial terms. In the easy task the
novices expanded the queries by 2.9 terms and the experts by 1.8 terms. In the difficult task the
experts expanded the queries by 2.4 terms, and the novices by 1.8 terms. As the consequence of the
expansions the number of terms between the groups was about equal in the easy task, whereas in the
difficult task experts used somewhat more terms than novices. The difference in the number of terms
in favour of experts grew with expansion in the difficult task, whereas it diminished in favour of
novices in the easy task. However, the differences were not statistically significant.
In the initial queries of both tasks there were no differences in the number of facets between the
groups. The exhaustivity of the queries stayed quite constant in the expansion of both tasks. However,
the reformulation impaired the exhaustivity of the queries by novices in the difficult task to a certain
degree. Although the difference was almost statistically significant (p=0.07), the difference of 0.3
facets is not practically meaningful. Thus, the expansion did not increase the exhaustivity of the
queries.
The specificity of the queries was measured by the number of narrower terms (NT) used. There was no
significant difference in the specificity of the queries between the user groups.
Terms and facets
Initial query: terms
Expanded query: terms
- NTs
Initial query facets
Expanded query facets
Initial query: terms/facet
Expanded query: terms/facet
Easy search task
Experts Novices
(14)
(15)
4.8
3.9
6.6
6.8
0.3
0.1
2.86
2.87
2.86
2.93
1.7
1.4
2.3
2.3
p.
0.10
0.80
0.41
0.94
0.52
0.13
0.90
Difficult search task
Experts
Novices
(15)
(15)
3.5
3.5
5.9
5.3
0.3
0.4
2.7
2.7
2.7
2.4
1.3
1.3
2.2
2.1
p.
1.00
0.45
0.77
1.00
0.07
0.81
0.79
Table 1. The average number of terms and facets in the initial and expanded queries.
The number of terms used expressing a facet represents the extent of a query. The extent of the initial
queries by experts in the easy task was larger, but this difference diminished in the expansion due to
the greater number of terms added by novices. There were no differences in the extent of the
articulations of the difficult task.
In all, the structural dimensions of the queries did not differ much between novices and experts in both
of the tasks before or after expansions. The only remarkable difference was the greater exhaustivity of
expanded queries by the experts (p = 0.07). It seemed also that in the easier task the expansion
diminished the difference in the extent of the queries in favour of the novices. Over both tasks and
groups, the expansion improved the extent and to some degree the specificity of the queries, but not
exhaustivity. This latter finding is probably explained by the fact that the conceptual query plans of the
searchers were quite exhaustive and that the interface limited the number of facet options into three.
The role of thesaurus terms in expansion The expanded queries of both groups contained more of
their own terms than the thesaurus terms in both tasks (table 2). Experts especially turned more to
their own vocabulary. However, there were no differences between the groups in the share of own
terms, which matched with the thesaurus.
In both tasks the majority of expansion terms originated in the thesaurus. In the easy task the
proportion of thesaurus terms was 62 % (1.7) among experts and 69 % (2.4) among novices. In the
difficult task the corresponding figures were 75 % (2.6) and 63 % (2.3).
The type of terms
Thesaurus terms
Own terms
- matching with thesaurus
Easy search task
Experts
Novices
(14)
(15)
1.7
2.4
4.9
4.4
3.3
2.8
p.
0.29
0.34
0.34
Difficult search task
Experts
Novices
(15)
(15)
2.6
2.3
3.3
2.9
1.6
1.3
p.
0.60
0.29
0.29
Table 2. The average number of own and thesaurus terms in the expanded queries.
Novices picked more terms from the thesaurus than experts in the easy task (1.7 vs. 2.4, p=0.29). In
the difficult task experts used more of both their own and thesaurus terms. Both experts and novices
used in the expanded queries relatively more thesaurus terms in the difficult task. The proportion of
thesaurus terms in the easy task was for the experts 26 % and for the novices 35 %, in the difficult task
it was 44 % and 43 %. It seems that in the easy task experts in particular coped with their own
vocabulary, whereas in the difficult task their need for terminological support also increased.
In the easy task experts entered 1.2 terms more in the thesaurus and saw 0.8 terms more than novices
(table 3). However, they selected somewhat fewer terms for expansion compared to novices. Novices
selected from thesaurus related terms in particular. Also experts favoured them, but to a lesser extent.
The role of terms
Own terms entered
Terms seen
Terms chosen
• RT/synonyms
• NT
• BT
• System suggested
Easy search task
Experts
Novices
(14)
(15)
5.9
4.7
7.0
6.2
2.4
1.7
1.6
1.0
0.1
0.3
0.2
0.1
0.5
0.4
p.
0.11
0.64
0.29
0.27
0.41
0.33
0.61
Difficult search task
Experts
Novices
(15)
(15)
5.1
5.1
7.4
5.9
2.3
2.6
0.9
1.2
0.4
0.3
0.0
0.3
1.0
0.8
p.
0.93
0.27
0.60
0.35
0.77
0.04
0.50
Table 3. Thesaurus navigation in term selection (average number of terms).
In the difficult task both groups fed in and selected about an equal number of terms in the thesaurus,
although experts saw more terms. In this task system-suggested terms were as popular as RTs (related
terms). Novices especially selected terms suggested by the system. Their share from all thesaurus
terms was for novices 44 % and for experts 31 %. It seems that the greater difficulty of the second task
increased the need for terminological support by the system especially among novices.
The only statistically significant difference in thesaurus navigation was the greater number of BTs
(broader terms) selected by experts compared to novices (p=0.04). Although the absolute difference
was small (0.3), a more thorough analysis revealed that some experts got hold of the proper expression
of the second facet “learning theories” (BT) through a narrower term, a particular learning theory, e.g.
“constructivism”, which they knew.
In all, it seems that experts explored the thesaurus more, although only one difference between the
groups was statistically significant.
The outcome of the searches
Search success was measured by the number of clearly topically relevant references in the top twenty
items retrieved. The utility of the thesaurus as the source of expansion terms is analysed by correlating
the increase in the number of relevant references with the number and type of terms used in query
reformulation.
Search success in the initial and expanded queries The expansions improved search effectiveness
significantly in both tasks within both groups. In the easy task the increase was among experts 3.0
references (p=0.009) and among novices 1.6 references (p=0.06). In the difficult task the
corresponding figures were 3.5 (p=0.002) and 2.7 (p=0.002).
In the easy task the initial queries of the novices were slightly more successful than those by experts.
The former retrieved 3.9 relevant references, and the latter 3.4 references (table 4). The expansions of
the experts were more effective producing an increase of 3.0 relevant items compared to 1.6 by
novices (p=0.27). In all, experts found slightly more references than novices (6.4 vs. 5.5, p=0.28).
In the difficult task experts retrieved already in the initial search remarkably more relevant references
than novices (2.2 vs 0.7, p=0.09). It seems that they were better able to formulate their initial query
plan than novices. The difference increased by the expansion so that experts’ result was 5.7 items
compared to novices' 3.4 items (p=0.001).
The experts found more relevant items by expansion in both tasks than novices. They were more
successful especially in the expansion of the easy task.
The number of relevant Easy search task
references retrieved
Experts
Novices
(14)
(15)
Initial query
3.4
3.9
Expanded query
6.4
5.5
Increase
3.0
1.6
p.
0.69
0.28
0.27
Difficult search task
Experts
Novices
(15)
(15)
2.2
0.7
5.7
3.4
3.5
2.7
p.
0.09
0.001
0.46
Table 4. The average number of relevant references retrieved.
Expansion terms and the increase in the number of relevant items In the following, associations
between the terms used and the increase in the number of retrieved relevant items is analysed by using
Pearson correlation.
In the easy task the number of own terms added in expansion correlated slightly in both groups (r=.33
vs r=.36) with the increase in the number of relevant references retrieved (table 5). This refers to the
utility of own terms as query terms.
Also the number of thesaurus terms selected in expansion was associated to some extent with the
search success among experts (r=.40, p=.16), but not among novices (r=-.12). In particular, RTs and
synonyms identified from the thesaurus by experts were effective query terms producing nearly
statistically significant correlation with the increase in search result (r=.51, p=.07). The low correlation
indicates that the selection of other types of terms in the thesaurus did not contribute to a better search
result.
Also in the difficult task experts seemed to benefit from the thesaurus more than novices. The number
of thesaurus terms (r=.36, p=.19), NTs in particular (r=.50, p=.06), correlated nearly statistically
significantly with the increase in the number of relevant items in expansion by experts. The use of
other types of terms from the thesaurus was not associated with the search result.
Increase in the # of relevant Increase in the # of relevant
The number of expansion items in the easy task
items in the difficult task
terms
Experts
Novices
Experts
Novices
(14)
(15)
(15)
(15)
Own terms
.33
.36
.17
.11
.08
Thesaurus terms
.36
-.12
.40
.18
.03
-.19
.51
• RT/synonyms
-.05
.50
.31
.07
• NT
-.01
-.16
.24
• BT
-.03
.12
.10
.00
• System suggested
Table 5. Correlation between the increase in the number of retrieved relevant items and of new terms
in expansion (n=15; r>.51; p<.05 & n=14; r>.53; p<.05).
In all, in both tasks, the thesaurus supported experts in identifying pertinent terms for query expansion.
In the easy task RTs and synonyms and in the difficult task NTs were pertinent query terms leading to
a considerable increase in the number of new relevant items in the search result. Neither the number
nor the type of thesaurus terms selected by novices was associated with the search result. It seems that
the utilization of the thesaurus in term selection requires, at least to some extent, subject expertise. If
one does not have it, the search results remain weaker regardless of thesaurus use.
In both groups over both tasks, terms suggested by the system used in expansion were not associated
with search success.
Discussion
This study explored how experts and novices in pedagogics expanded queries supported by the ERIC
thesaurus, and how this was connected to the search success in an easy and a difficult search task.
Terms
Over the user groups and search tasks, expanded queries included significantly more terms than initial
queries. Both groups used more thesaurus terms than their own terms in reformulation, especially in
the difficult task. About two thirds of the expansion terms originated in the thesaurus. In the easy task
the proportion of thesaurus terms was greater by novices, whereas in the difficult task by experts. The
finding in the easy task is consistent with Hsieh-Yee (1993) that experts use own terms more than
novices. Experts were able to draw more on their own vocabulary in the easy task, but also they
needed more terminological support in the difficult task. The former conclusion is in line with
findings that expertise helps in identifying terms for expansion (Shute and Smith 1993; Vakkari 2001).
Experts navigated more intensively than novices in the thesaurus in both tasks. They entered more of
their own terms, saw more terms, but selected fewer terms in the easy and about equal number of
terms in the difficult task than novices. Both user groups selected most frequently RTs in both tasks.
In the difficult task both groups also frequently picked system suggested terms. Over both tasks
novices used statistically significantly more terms suggested by the system. They seemed to need
more terminological support from the system due to their more limited vocabulary.
Experts were more selective in their term choices, whereas novices experimented more with terms in
reformulation. Novices tried a more trial and error method than experts. This conclusion is supported
by Hsieh-Yee's (1993) finding that novices experimented more with various term combinations.
Search success
Subject knowledge not only affects the selection of search terms, but also the ability to identify
relevant documents. Thus, it may be that the connection between the number of expansion terms used
and the increase in the number of relevant items found is caused by the subject knowledge. There are
two factors that speak against this possible spurious association.
Firstly, studies show that novices tend to accept references as being relevant more easily than experts,
who use stricter criteria (Vakkari 2001). Thus, novices seem to more reliably identify clearly topical
than less topical information. Moreover, searchers are more likely to identify highly relevant items
than marginally relevant ones (Vakkari & Sormunen 2004). For increasing the reliability of
identifying relevant items, we only used clearly relevant documents in forming the recall base.
Secondly, for the significance of term selection for search success speaks the fact that in both tasks a
more detailed analysis revealed that the expression of certain facets, especially of the most difficult
ones, had a crucial influence on the success of the searches. If differences in identifying relevant
references would have been crucial for search success, then the associations between term selection
and success should have been distributed more evenly over the facets. We may conclude that term
selection had a vital role in search success.
Terms and search success
The query reformulation improved search success significantly within both groups. The increase in
the number of relevant references was greatest among the experts. The difference between the groups
was statistically significant in the difficult task.
About two thirds of expansion terms were selected from the thesaurus. A major finding was that the
increase in search success was associated considerably with the number and type of thesaurus terms
used in expansion mainly among experts, but hardly among novices. The findings hint that at least in
the difficult task novices tend to use trial and error methods in expansion, which were not patterned.
The number of expansion terms used by experts were considerably associated with the increase in the
number of relevant items retrieved. In both tasks the thesaurus contributed remarkably in the search
success. In the easy task the number of RTs and synonyms and in the difficult task the number of NTs
was associated with the increase in the number of relevant items retrieved in expansions. In the latter,
query concepts were more abstract like "learning conception" needing specifications like naming
particular learning theories. Therefore hyponyms improved search success. In the easy task concepts
were less abstract like "email" or "teaching" requiring synonyms for improving success in expansions.
The results suggest that the increase in the extent and in the specificity of the queries produced by
thesaurus terms contributed to the improvement of search effectiveness in the expert group. The
exhaustivity of the queries was not associated with the success due to its invariability caused by the
limited number of facets in the interface.
The contribution of the thesaurus to search success among experts corroborates partly the findings by
Jones & al (1995), that a thesaurus increased recall of the searches. They also found that search
success was associated with the intensity of thesaurus navigation, which corresponds with our findings
that experts tend to select their terms from a greater number of thesaurus terms seen.
The role of searchers' own expansion terms was small. They were slightly associated with the increase
in search effectiveness in the easy task in both groups. Interestingly, in neither group was the number
of terms suggested by the system associated with the search success, although they were used second
commonly by the searchers. Most of those terms were surrogates to user terms not found in the
thesaurus. The system provided a list of suggestions, typically phrases in alphabetical order. In some
cases the list did not contain relevant terms, but some searchers used the anyway, or in other cases the
list was very long hiding the correct term. Lead-in terms providing one synonym to a user term were
the most useful suggestions. In all, the major reason to the ineffectiveness of terms suggested was that
they did not linked the user to terms representing a query concept pertinently. This is in line with the
finding by Jones & al. (1995) that success is correlated with finding good matches for one's query in
thesaurus to start with. One means for increasing this match is to introduce a more comprehensive
entry vocabulary in thesaurus (Bates1986).
Conclusions
It seems that a vital condition for benefiting from a thesaurus in query expansion to improve search
results is sufficient familiarity with the search topic. In our study the number and type of thesaurus
terms used among novices in education were not associated with the search success. Only experts
benefited from it in this respect. The initial query formulation and selection of expansion terms from
the thesaurus was easier for experts due to their more comprehensive subject knowledge and
vocabulary (cf. Robertson 2001). They were able to exploit the terminological information provided
by the structure of the thesaurus more effectively as their more selective thesaurus navigation also
showed.
If subject novices are not able to exploit interactively terminological resources of thesaurus, would
automatic query expansion based on thesaurus work better? The success of expansion by thesaurus
terms matching with query terms depend on the selection of query terms by novices and on the
comprehensiveness of entry vocabulary. Our study showed, that system suggested terms used in
expansions both by novices and experts did not contribute to search success mainly due to the poor
match. Thus, it seems that without considerable increase in the number of entry terms, automation
would not improve search results compared to the interactive mode.
This study seems to show that it is not in the first place the number of terms used in expansion, but
their type and quality which is crucial for search success. First, not all thesaurus terms used, but the
particular type of terms improved search effectiveness. Second, although novices used more expansion
terms in the easy task and about as many in the difficult task than experts, their search success was
weaker. Thirdly, the number of search terms selected by novices was not associated with the search
success, whereas the number of specific types of terms chosen by experts was significantly associated
with improvement of search results.
Although the importance of analysing the semantic content of query terms in relation to search
effectiveness has been touched in some studies (Nielsen 2002), it is still an unexplored terrain in
interactive studies.
Acknowledgements
Warm thanks to Larry Rudner of ERIC for providing us a separate ERIC Wizard web site for
performing and recording the searches. We wish also to thank the members of the FIRE research
group for their useful comments.
References
Aitchinson, J., Gilchrist, A., Bawden, D. (1997). Thesaurus construction and use: a practical manual.
London: Aslib.
Bates, M.J. (1986). Subject access to online catalogues: a design model. Journal of the American
Society for Information Science 37(6): 357-376.
Beaulieu, M. (1997). Experiments of interfaces to support query expansion. Journal of Documentation
53(1):8-19.
Borlund, P. (2000). Evaluation of interactive information retrieval systems. Doctoral Dissertation.
Åbo: Åbo Akademi University Press.
Fidel, R. (1991). Searchers' selection of search keys. Journal of the American Society for Information
Science, 42: 490-527.
Hsieh-Yee, I. (1993). Effects of search experience and subject knowledge on the search tactics of
novice and experienced searchers. Journal of the American Society for Information Science 44: 161174.
Ingwersen, P. (1999). Cognitive information retrieval. In: M. Williams (ed) Annual Review of
Information Science and Technology, vol. 34. (pp. 3-52). Medford: Information Today.
Jones, S., Gatford, M., Robertson, S., Hancock-Beaulieu, M., Secker, J. (1995). Interactive thesaurus
navigation: Intelligence rules OK? Journal of the American Society for Information Science 46 (1):
52-59.
Nielsen, M. L. (2002). The word association method: a gateway to work-task based retrieval.
Doctoral dissertation. Åbo: Åbo Academy University Press.
Pollard, R. (1993). A hypertext-based thesaurus as a subject browsing aid for bibliographic databases.
Information Processing & Management 29(3): 345-357.
Robertson, S.I. (2001). Problem solving. Hove: Psychology Press.
Shiri, A., Revie, C. (2001). User-thesaurus interaction in a web-based database. In: Proceedings of the
Infotech Oulu. International workshop in information retrieval. (pp.23-32). Oulu.
Shiri, A., Revie, C., Chowdhury, G. (2002). Thesaurus-enhanced search interfaces. Journal of
Information Science 28(2): 111-122.
Shute,S. , Smith, P. (1993). Knowledge-based search tactics. Information Processing & Management
29(1): 29-45.
Sormunen, E. (2000). A method of measuring wide range performance of Boolean queries in full-text
databases. Acta Universitatis Tamperensis 748. Doctoral Dissertation. Tampere: Tampere University
Press.
Sutcliffe, A. G., Ennis, M., Watkinson, S. J. (2000). Empirical studies of end-user information
searching. Journal of the American Society for Information Science, 51: 1211-1231.
Vakkari, P. (2001). A Theory of the Task-based Information Retrieval. Journal of Documentation 57
(1): 44-60
Vakkari, P. (2002). Subject knowledge, source of terms and term selection in query expansion. In:
Crestani & al. (eds) Advances in Information Retrieval. Proceedings of the 24th ECIR. (pp. 110-123).
Berlin & Heidelberg: Springer.
Vakkari, P. , Sormunen, E. (2004) The influence of relevance levels on the effectiveness of interactive
information retrieval. Journal of the American Society for Information Science 55. To be appear.
Wang, P. (1997). User's information needs at different stages of a research project: a cognitive view.
In P. Vakkari, R. Savolainen, & B., Dervin (Eds.), Information Seeking in Context (pp. 307-318).
London & Los Angeles: Taylor.
White, M., Iivonen, M. (2002). Assessing level of difficulty in web search questions. Library
Quarterly 72(2): 205-233.