FOnline Query Refinement on Information Retrieval Systems:
A Process Model of SearchedSystem Interactions
Hsinchun Chen', MIS Dcpament. University of Arizona
Vusanr Dha?, IS Dcpanment. New Yo& University
Abstract
-@=
'83 +I-81.
2- 1036.
-
rnen university. Jan
651.1984.
his anicle reports findings of empirical research that investigated information searchen' online query reirnement process. Prior studies have recognized h e information specialists' role in helping sevchen aniculate
yld refine queries. Using a scmantic network and a Problem Behavior Graph to represent the online search
our study revealed that searchers also refined their own queries in an online task environment. The
,,,formation retrieval system played a passive role in assisting onl~ncquery refinement, whch was. however, one
hatcodirmcd Taylor's four-level query formulation model. Based on our empirical findings, we proposed using
process model to facilitatc and improve query refinement in an online environment. We believe incorpordting
fiis model into retrieval systems can result in Ihe design of more "intelligent" and useful information reuieval
Introduction
Electronic text-based information storage and retrieval systems in the form of online caralogs, online bibliogaphc
databases. and videotex that Can store huge amounts of data and allow access via a terminal or a television set
3re changing the way we gather, process. and retrieve information. These systems provide a wide variety of
lnformalion and services, ranging from daily updates of foreign and national news, movie reviews, law cases,
yld finmcial data on companies 10 joumal anicles. books, uademarks, and statistics. While archival information
sourccs such as libraries are becoming increasingly computerized. access ID such information is often difficult,
due in large pan 10 the indeterminate nature of the process by whch documenu arc indexed and the latitude
searchers have in expressing a query. For inexperienced searchers. the problem of finding documents that are
relevant to a query can be difficult for three reasons:
I . i t can require a significant amount of knowledge of h e subject area in which information is sought,
2. i t rcquires knowledge about the functionality of the information storage and retrieval system, and
3. it requires knowledge about the classification scheme employed in the information storage and retrieval
syslcm.
Searchcrs generally have limited knowledge about the classification scheme and the retrieval system. Since
the purposc of the search itself is oficn to acquire knowledge about the subject area. searchers may not be clear
about the subject area for which answers are being sought Searchers may have only a felt or conscious need
whch requires to be formalized and articulated. A human information specialist such as a reference librarian
ofien assumes an active role in helping searchers refine and articulate their queries.
The focus of our research was to identify empirically how searchers nfined their queries when using an online
catalog. Wc compared the results with prior studies in which librarians' assistance was present. Based on our
empirical findings. we proposed a process model for facilitating onlinc query refinement.
A-Gna, Tucson. AZ 85721. USA
'[email protected].
MIS, UNVCISII~of
'[email protected].
IS, NYU. NY. N Y 10003.USA
-_
Permission to copy without fee all part of this material is granted provided
that the copies are not made or distributed for direct commercial advantage,
the ACM copyright notice and the title of the publication and its date appear,
and notice is given that copying is by permission of the Association for
Computing Machinery. To copy otherwise, or to republish, requires a fee and/
or specific permission.
(C)
1990
ACM
0-89791-408-2
115
90
0009
115
$1.50
KMwledye
Componuus
I
2.3 hdexing and
,,,dexing uncertaint!
t.3.1
Indexing Unc
oeprocess of index
in an indexing
terms in Figure 1). 1
different times [lS] [
2.3.2
Figure
A framework for information storage and rcuieval
In Section 2, we prcsent an information reuicval framework as the context for our research. We then descnk
our research design and method in Section 3. In Section 4, we discuss how scarchen refined their queries in
onhne environment based on Taylor's theory of query formulation. In Section 5 . we present a process model
aimed at facilitating online query refinement. We draw conclusion concerning our research in the final section.
Search uncer
Search uncemkty R
(se
h e boxes on thi
1. Search Strate
Search strateg
two strategies
search terms
Another study
search. Two I
and a decisior
approach. one
combines a l l :
"raw wi&
retrieve initia
added in the
6
2 Information Relrieval Framework
In this section, we present the information rctneval framework we developed. Thls framework, as shown 111
Figure 1 , examines b e human agenb involved in the. onllnc information stonge and retrieval environment. %,e
typcs of knowledge these agents possess, and the unique chanctenstics of their indexmg and search behavior
2.1
Knowledge Components
Three types of knowledge are required for the information sloragc and retrieval process. Fint. classi6caUon
scheme knowledge is used for indexing documents. Second, subject area knowledge is required for expressing
a query. Lastly, system knowledge helps a user perform a fruitful and effective search in an online ~ u i e v a l
system. These Wee knowledge componenls are presented on h e left hand side of figure 1.
116
2. Search Term
A high d e g m
search terms
any two p a p
2.3.2 Search uncertainty
tricval
r our research. We then descnk
%hers refined their queries in an
we present a process model thai
ur research in the final sation
This framework. as shown in
e and retrieval environment, h e
indexing and x v c h behavior.
Scab uncertahty refers to thc latitude searchers have in adopting search strategies and choosing search terms
,see
boxes
on the right side of Figure I ) during the information reuieval p m s s .
1. Search Strategies:
Search srraregy is ohcn used to describe the plan or approach
u) the whole search. In a card catalog study,
two straugics for scarchng have been identified: a “self-Rliant” style where searchers genera& their own
search terms and a “caialog-oriented” style where searchers use the terms found in the card catalog [26].
Another study classifies thc search strategy in terms of the critical decision points faced during the online
search. n o types of decision p o i n ~occur
~
during the search: a decision to rcact to unfavorable nsults
and a decision to revise search logic [la]. Bourne identifies two search strategies. In the “building-block”
approach. one enters various Erms as separate search statements. After the search results a n derived, one
combines all search statements into a single final statement using the Boolean operator. AND. T h i s strategy
conuasts with the “pearl-growing” strategy. in which one starts by searching on a few specific terms to
retrieve initial citations. These citations arc then examined carefully for new candidau search terms to be
added in the subsequent searches [le].
2. Search Terms:
process. First, classification
*ledge is required for expressing
~ v esearch in an online rcuicval
of Figure 1.
val
A high degree of uncertainty with regard to search terms has been observed. Searchen tend to use differcnt
search terms for the same information sought. Studies have revealed that, on avenge. the probabiliiy of
any two pcople using the same term to describe an object is about 10 percent [13] (121. This fundmental
117
propeny of language limits the succcss of various design methodologies for controlled vocabulary-& "ell
interaction (121.
Due to the uncertainty in choosmg the index and the search terms. generating an exact march betwRn
searcher's terms and those of the indexer becomes difficuk Bates [2] argues that for a successful macuch
the scarchef must somehow generate as much "variety" (in the cybemtuc sense. as defined by [I]) in lh;
search as is produced by the indexers in their indexing.
while i d e x e n usc the rule of specificity for indexing, searchers tend to approach a search by specify
broader terms fiW. There may be Several reasons for h s . One hypothesis is &tal searchers often do
have "queries." but what Bclkin calls an "anomalous state of knowledge" [4). Searchers ofun e x p ~ ~
refine hs anomalous state into a query chrough an inuractive process. The organization of a catalog or
system docs not always facililatc this type of query refinement. however. In c o n m t . reference libma
appear to be particularly adept at performing rhrs function.
,,,
Taylor suggests Lhat a searcher's queries sLan from an actual but uncxprtsscd need (visceral need). 9
visceral need is refined to a conscious description of the need (conscious necd). This nccd is bur
formalized as a statement (formalized need). However, the actual query p r e s e n d to the informatlo,
syslcm may be compromised by the searcher's expectation of the system (compromised need) [28]. Based
on Taylor's model. a similar model for describing query refinement during the pre-search interview between
rrference librarians and online searchers was developed by Markey [17]. We also used Taylor's l h e q lo
model the online query refinement process. Details are discussed in Section 4.
The imponance of query refinement during the information retrieval process and the rrference librariansi
rolc in assisting h s process are well recognizcd in earlier research Nevenhcless, prior studies did no,
investigate the funcuonality of the retrieval sysums in assisting query refinement. In our research, we
investigated o n h e query refinement process in detail, and our findings were used to develop a pmss
model for aiding online information retrieval.
3 Research Design
A field study was conducted in 1988 at New York University. Data collection techniques including h&.
aloud protocols, tape-recordings, interviews. and questionnaires were uscd. By studying the interactions b e t w ~ ~
information searchers and an online catalog, we were able to identify the query refinement proccss whch OCEIJ~
The online c a d o g system we studied, Bobcat. listed over 600,ooOcatalog records including all new maten&
acquired aftcr 1973 and many older items previously listed in the card catalog. Joumals were not listed.
system provided seven search options: tide search. author search. combination of author and title search, subFc:
search, call number search, keyword search, and Boolean search. These options arc considered standard in mor:
online catalog systems.
Thrny business school students ranging from Ph.D. canddates to freshmen pYricipated in the study. Thcv
subjccu were asked to perform a search for documents within a subject area of their own choosing. In geneni
the most frcquenlly chosen option was subject search, followed by keyword search using index term (one UOT:
only). Before beginning their searches, subjccu were q u e s t e d to explain what they were looking for. They w::
also asked LO think aloud during their searches. Their verbal protocols were tape-recorded, and the inuncuord
between thc searchers and the system were loggcd. The interactions lasted between 5 and 40 minutes. Ah::
the inunction. subjects were requested to state their queries again. A few follow-up questions pertaining 10 IhC
search process and the problems encounund during the search also were asked.
Our study was based on information processing theory [19]. The technique we uscd to analyze the COUCCK
118
6313 was prorocol analysis.
~ o g n i t i v ePsycho~ogycdmm
4
Levels of Query Refir
B a g d on the representation
dcSCnbe rhe online query ref
differences between the
such help.
4.1
Representation of
S
The representation scheme
,,erwork and a Problem Beh
Intelligence. The semantic r
on Lhc other hand. depicu 114.1.1
Semantic Network
me concept of a Semanuc I!
the m c m g of wc
cpresent ObJects. rhrngs. cor
mn used UI different applic
&a have been applied 10 tk
for retrieval systems. have t
Knowledge Of a SUbJcct
where I t n k S are Of nu0 type
general and s ~ ~ ~ lterms
fic(
Subject Headings (LCSH) h
3 poruon of the semanuc ne
4.1.2
Problem Behavior
f
In order to make the analys,
process in tcrms of a Rob1
in a time sequence, from a~
be needs). Thrs dcrallcd re
inccraction logs and verbal
operator elements [ 191 [ 3 1
t ~ The
k
operator elemcnr
SWLC of knowledge as outp
PEG. w h c h was grounded
10 study the human problem
analysis [51, mathematlcal 1
useful to rcpnsent the onhn
raring an exact match k t w
fie
lrgues that for a successful -nmatch,
uc sense. as defined by [l]) .
Qthe
o approach a search by spec'
%ng
icsis is that searchers often do not
gc" (41. Searchers often e x w
tb
Thc organizauon of a catalog or
a
:r. In contrast. reference lib
manas
ne
pressed nccd (visceral need).
cious need). T ~ I need
S
is final]
ucry prcsentcd to the informationY
n (compromised need) (281. Bared
'g the prc-search interview k t w e c n
1. we also used Taylor's theory
:tion 4.
'OCCSS and ihc reference libmanas
Vcvcnhele~~,
prior studies did noI
i refinement. In our research, y e
:S were used to develop a process
:&on techniques including h&.
J studying the mteracuons b e t w e n
y refinement process whch O C C U ~
ccords including all new maten&
og Journals were not listed. The
1 of author and title search. SubJeci
'nsan considered standard in moSl
:n pmcipatcd in the study. These
of their own choosing. In gencd,
search using index term (one uord
II they were looking for. They wcrc
tape-carded, and the interactions
between 5 and 40 minutes. Aher
Alow-up questions pertaining to the
id.
4e we used to analyze Ihe collected
4.1
Representation of Search Process
rePCsentation scheme we uscd for the analysis of the query r e h e m e n process is b a d on a semnric
and a Problem Behavior Graph (PBG). Both representations are widely wd in the area of Anificial
bLcuigence. Thc semantic network is u x d to represent the semantic contents of searchers' queries. Thc PBG.
other hand, depicts the flow of ihc online information rcmcval process.
yhc
atwork
on
,.l,l Semantic Network
conapt of a semantic network was first introduced by Quillian (211 as a general association mechanism for
the meaning Of words. A semantic network Epnsenu; knowledge by mof
a d links: nodes
Eprescntobjects. h n g s , concepts. facts. etc.: links reprcsnt relationships among h m . Sematic n e t w o h have
wn in different applications, mainly in Ihe area of natural language processing [23] (331 [a]. Recently, they
have k n applied to the design of information revieval systems [ E ] [9]. In panicular, the online thesauri
Tor nttieval systems, have bccn represented using a semantic network structure [lo] [25].
uowlcdge of a subject area can be captured and represented by a large semantic network of terms (concepts)
links are Of N O types: relaU0n.s between non-index and index terms (USE links) and nlaaons bctwccn
and specific terms (NT/BT links). These links exist in most thesauri. including the Library of Congress
SubJCCt Headings &csH)handbook (the classification scheme of the online catalog we studied). Figure 2 shows
Pmon of the semantic network corresponding to the LCSH classification scheme.
J.I.~
Problem Behavior Graph
In order to make the analysis
Of
rhc online query refinement process more meaningful. we summarize the search
p m s s in terms of a Problem Behavior Graph (PBG). This representation describes problem-solving activities
In a time sequence. from an initial state (a vague description of needs) to a goal state (a solution that satisfies
he needs). This detailed representation of the problem-solving process is derived by firs1 spliaing up the user's
Inlcraction logs and verbal protocols into their semanric clemerus. which consists of knowledge elements and
operator elemenu [ 191 [31 I. The knowledge clemrnrr spccify the kinds of knowledge the subject has about the
ink. The operutor elemenrr are a finite set of actions that take a state of knowledge as input and produce a new
swu of knowledge as output. Users typically use a finite number of operators to change the knowledge states.
PBG.whch was grounded on the information processing theory 1191, has bccn used in a large body of research
io siudy the human problem solving process. Examples of the domains that have been studicd include: hnancial
andysis [51. mathematical programming 1201. scheduling [14]. and conceptual data modeling (31. We found it
useful to rcpresent the online information retrieval process in terms of PBG.
119
I,
ST:The USE link
searchef may follob
For example. by fc
or * ' E ~ C C U OBra1
~IC
2. BT or NT: The Ir
link. Following
e x a ~ ~ ~ pby
l e follou
.
&an "Computers."
Systems" IS more
may decide to bm
3. AT: Terms which
tor (broader term)
dctcrmine the WIT
two terms in rhe I
tivcly. For examp
reached by follou
Dam Pnxcssing"
Intelligence." Sc;
4.
Figure 2: A sample nctwork of LCSH terms
DT: This operato
( i t . . one ~ c r mca
query from one t~
Bascd on these pni
through h e problem SC
and each arc npresenls
semantlc network. we 1
two query processes U!
4.2
Taylor's Theoi
In a seminal anicle 0,
clienthnformauon spec
formalized need. and
3 . The librarian's role
120
I
e information reuieval
the essential bowledge elements are the LCN used IOexpress thc query.
’,ator elements are the moves or actions that change the content of the query. we refer to thcm as the
Ope
AS shown in Table 1. we observed five semantic operarors used by searchers to move from
j,f@[ic
to the next They are: ST (synonymous term operator. moves from a non-index term to its synonymous
BT (broader term o p e w r . from SpCCifiC gexral term). NT (nanuwcr term operator, from general
AT (adjacent term operator. from one term to another term which pmially overlaps in meaning),
“
(disjointed term operator. from O w term to mother S ” c a l l y different term). These operators were
Pd
b m our empirical studies. Details of how searchers used these operators in n b n g their
an
jerlvcd
Gsnrd
in Section 5.
P In rhrs subsection. we explain these operators in the context of rhc semantic n~hvorkm C t u E we proposed.
lnJcxcidc
c
S
ST:me USE link in OUT semantic network leads from a non-index term to its synonymous index em. A
may follow this link tO identify Synonyms. These links an listcd explicitly in the LCSH Handbook.
For example, by following the USE link in Figure 2, a searcher can identify “Computers” ar, a synonym
of *‘Electronic Brain.”
2,
or NT: The NT link leads from a broader term to a narrower term. BT link is the reverse of NT
l i d . Following the NT (or BT) links we can determine the level of specificity of different terms. For
by following thc NT links in Figure 2. wc know that “Electronic Data Processing” is more specific
*‘Computers,”“Artificial IntcUigence” is more specific than “Elecuonic Data *sing;*
and **&pen
Systems” is more specific than “Arrificial Intelligence.” Both N T and BT links are Uansitivc. Searchers
may decide to broaden or narrow their queries by following these links.
3, AT: Terms which are not directly related via the USE or NTBT links but have thc same common ancestor (broader term) or descendant (narrower term) in the networlr are considered adjwenr terms. We can
determine the common ancestor or descendant of any two terms by activating all paths leading to/from the
~ W Oterms in the network. The intersection of these paths is thc common ancestor or descendant respectively. For example, “Fuzzy A l g o r i h s ” and “Machine Theory” have a common ancestor. “Algorithms.”
xached by following the reverse direction of NT links (BT links) in Figure 2 for both terms. “Electronic
Data Processing” and “Machine Theory" an adjacent because they have a common descendant, “Artificial
htclligcncc.” Searchen may change their topic of interest from an initial term to its “adjacent” terms.
4.
DT: This operator represents the m i t i o n from one term to another term whch is semantically disjointed
(i.c.. one urm cannot reach another via rhc links in the network). Th~sgenerally represents a change of
query from one topic to another d i s j o i n d topic.
Bascd on thcsc primitives. we can construct a PBG to describe the progression of the state of the query
thmugh the problem space. Each node of the graph represents a particular state of knowledge for the searcher.
and each arc represents an operator that was applied to transit to the next state. By superimposing the PBG on the
scmanuc network. we get an interesting visual picture of the trajectory of the query. Figures 4 and 5 dcmonsuate
IWO query p w s s e s using this scheme. We discuss them in the next subsection.
4.2 Taylor’s Theory for Online Query Refinement
In a seminal anicle on query refinement Taylor proposed four levels of query refinement that p e m n to the
.Iicnuinfonnalron specialist interview session (271 [28] These four levels are. visceral need. conscious n d .
lormaliud need. and compromised need. Changes of needs from one level to another are mdicated m Figure
3 The libranan’s role in assisting query refinemcnt has been well recoyzed in thc pnor studies We postulate
121
Scmanuc Elemem
Knowledge Element:
TERM
Openuor Element:
ST
BT
NT
AT
DT
Table
kscnpuon
d
d a y COkctiOn
~ [ c f l l c w s , w e n Uscful
examples of qucr]
of Ihe subjet% ch.
significantly dif
”f?igh[
Terms used by the -her.
(Going from one term to m k r )
Synonymous Term
Broader Term
Narrower Term
Adjacent Term
Dispinted Term
‘Yf
‘Tcm
refinement (43-44
could recognize and
;&ribe
chesc two types of
w f o ~ a l i z c dneed. Q3-@
Semantic elemuus for query rehncment
j f . ~
Formalize the Con
For eight of the SUbFCtS wc
(in wmsof contents). The
different from wh
and M n
Figure 3: Taylor’s theory of query rehnemuu
that online infomation retrieval systems also assist searchers in refining or articulaing queries. but possibly
more passive way.
We represented the 30 scarchcr/system interactions in terms of the PBG and then analyzed rhesc PBGs &
on Taylor’s model. Although our study was unable to identify the visceral necd, it revealed the remaining
levels of n d s . (However, the very existence of the searcher’s query indicates the presence of the viscenl w ,
Definitions for the four types of needs in Lhc online task environment an as follows [28]:
1. Q1. Visceral need: a vague son of dissatisfaction. or a cenain incompleteness in the searcher’spi-01
the world.
2. Q2. Conscious need: the conscious, within-brain description of h e wed. In our study, h s was revelled
in the searchers’ hnk-aloud protocols during their onlme searches.
3. Q3. Formalized need: the formal statement of nced. This is a searcher’s query staument. In our srud!
we elicited bolh pnscarch and postscarch query statemenu, using interviews.
4. Q4. Compromised need: the query as presented to thc infomation system. This is disclosed by thc actll~
search terms used by a searcher on the system. They were recorded in our log Gle.
122
included in their
FiguE 4 illustrates how
c o m ~ n c (terms)
n~
involvl
&er (see the ovals and I
p6G (follow the h V y arr
I” figure 4. The concepts
postsearch query statemenl
s i x groups of terms wc
wc d e r to this as a c o w
concept a n c o d via
p u p s w e n involved: Grr
nght hand comer of Rgun
“data,” “data cnOy.” “ d a
Zacr - management.” ar
factors” ( c o ~ c c t c dby an
query statement was very
by &IC rectangular box in
6). Afwr the search. the :
11 contained concepts in (
own brainstorming and th
similar to the one obscrv
suppon from the system,
I
Formalize the Conscious Needs: (Q2-QJ)
of thc subJccts we observed. their presearch and poswarch query Statements were si@ficmdy dLFfcrcnt
of conunts). The subjects' think-aloud p r o t ~ l revealed
s
their conscious n&, whch involved needs
(in
different from what had been s w d in their prescarch statcmcnts. These conscious needs WCR f o m d m d
Mn included in their postscarch w e m e n t s .
~iprt
4 illusuata how the content of one subject's query changed during the scarch process. Thc knowledge
c o m p (terms)
~ ~ involved in tlus interaction arc represented using the semantic network s m c u r e we described
d e r (scc he ovals and the light arrows). The flow of the search process (in time sequence) is captured by a
p ~ (follow
c
the b v y arrows from Srurt IO End). The terms involved in the search are represented by the ovals
figure 4. Thc concepts involved in the subject's prcsearch query statement (rectangular boxes) as well as the
p w c h query Statement (shaded ovals) arc also shown in Figure 4.
six groups of terms w e n mentioned in this query. Each group consisted of terms that comprised a conccpr.
wc d e r to h s as a conccpr group. In terms of semantic network representaiion, words that address a similar
concept are c
o d via some links (the ST or NT links as shown in Figure 4). In this example. six concept
were involved: Group 1 - "error" and "error - psychology" (connected by an NT link as shown in the upper
right hand comer of figure 4). Group 2 - "routine work" and "motor skdls" (COMCCU~
by an NT link). Group 3 -data." "data e W . " "data editing." "data prowsing," "data processing service center." "data processing service
xrucr - management." and "data reduction" (connected by rhe NT links). Group 4 - "ergonomics" and "human
fmm" (COMWted by an ST link). Group 5 - "learning." and Group 6 - "performance." The subject's pnscarch
query staicment was very f u u y . "how pcoplc make m."
It mentioned only one concept in Group 1 (ideated
by the rectanflar box in Figure 4). Five new concepts were introduced during the search (Groups 2, 3 . 4 , 5, and
6). After h e search, the searcher's query statim" was: "the human f;lrrprs in h e putinc wo& of data cnuy."
11 contained concepts in Groups 2. 3. and 4 (see the shaded ovals in Figure 4). We postulate thar h e searcher's
own brainstorming and the incidental information displayed by the system resulted in a query refinement process
similar to the one observed in the clicnt/librarian interview session. However, because h e searchers had little
support from the system. this query refmemcnt process was often ume-consuming and ineffective.
For
and then analyzed these PBGs b w
need, it revealed the remaining bee
tCS the presence of the visceral "ecd.)
s follows [28]:
iplcuness in rhe searcher's picture of
need. In our study. this was revealed
%her'squery statement. In our study
[crviews.
,ystem. This is disclosed by the actual
in our log me.
123
124
L
0 FmlQucrr
A
A
A
h-r
B
4B
.B U
I
~
W
Q"of
A
: mD*t from A io IU w o w a
tam
B.
A+
:
A B*
;
1r0m A
mm
yl-
Yrm
I
m e from A Io YI drrpa Yrm B.
Figure 5 : An example of compromising the formalized needs
4.2.2
Compromise the Formalized Needs: (Q3-Q4)
The contents of 21 subjects' presearch query statements had the same or similar meaning as their postsearch
query statements. However, the search urms they used during IIK
search vaned. Thc search terms input to the
dormation system (the compromised needs) were vanations of their formalized needs. We posit that ttus was
the
KSdt
I
I
I
of the searchers' attempts to adjust to the system's vocabulary.
h terms Of semantic network representation. searchers may move from one term to its synonymous terms
(ST) in order to generate matches; to its broader terms (BT) in order to broaden Ihc scope of their queries; to its
narrower terms (NT) in order to focus their searches; or switch from one topic to another topic which overlaps
panially in meaning (AT) in order U) cover dilTcrent aspects of their queries.
Figure 5 shows an example of how one searcher chose different search terms to CXPKSS an unchanged (in
tcrms of its meaning) query. Only one concept group was involved in h s scarch. but h s concept group contains
a number of terms which arc relaid to "planning" (see the ovals in Figure 5 ) . The subject's prescarch statement
and postscarch statement addressed the same mformation need. "hemchical planning" (see thc shaded oval and
the rectangular box in Figure 5 ) . although different terms were chosen to CXPKSS rhe query. After fading initially.
the searcher tried other semantically close unns to generate matches. These terms included: "business planning."
"multi-level planning." "large scale planning." and "organizational planning." (Follow thc dark arrows from Start
io End in Figure 5 . )
There may be several hypotheses to explain this process. First, bccause of rhc controlled vocabulary used for
document indexing, search terms used initially may not have produced any matches. So searchers were often
forced to use different terms. Second, information displayed by the system may have suggested new clues for
125
1
I
i
I
I
3
h e search. Lastly. searchers may have found it more prudent to approach their queries from different angles (b
I
using synonymous, broader. m w c r , or adjacent terms).
4.3
/
Q u e r y Refinement: L i b r a r i a n vs Retrieval System
In earlier studies. the librarian's function in refining searcher's queries has been well documented. In h s %Qon
we discuss the differences between the way librarians assist in refining queries and the way online remeral
systems provide such help.
1. The Role: Librarians assume a more active role in refining queries than retrieval systems do. Lib-*
attempt u) understand the pamn's underlying information needs (conscious needs) in a very early stage,
This active role in query rcfincment is useful for achieving an efficient (minimal wasted effon) %arch,
Current online retrieval systems are too passive in this respect. requiring searchers u) do too much.
Experienced librarians, in particular, arc good at idcntifylng a pauon's conscious needs by detecting in@,,.
sistency or incompleteness in the pauon's queries. They play an imponant role in prompdng patmu lo
formalize their conscious needs. searchers of online systems. on the other hand, ranly have rtUs support,
2. Sources of Knowledge: In swcherfiibrarian interactions. both the searcher's subject area knowledge a
d
the librarian's subject area, classification scheme. and system knowledge contribute to the generation of
new cum for the search. During searcher/system interactions. on the other hand, searchers can n l y only on
their own knowledge in the subject area (searchers in general have limited classification scheme and system
knowledge) and incidental information displayed on the scrccn. For searchers who have litlle subject area
knowledge (e.g., freshmen), this process is often unproductive.
Librarians' classification scheme knowledgc is often used to assist patrons in generating queries du in.
formation systems can recognize (compromised needs). Their knowledge about the system's conmud
vocabulary, the cross-refcnncing structure. and the index principles employed helps pamns obtain a good
surrogate for their formalized needs. In the searchcdsystem interactions. this process is essentially one of
uial and error.
3. User Modeling: One capability which we found to exist only in the librarians was their ability to perfom
a user modeling function. For example, librarians may c x p t vagueness in the queries from inexperienced
and less sophsticatcd searchers (e.g., freshmen). Librarians generally spend a considerable amount of time
in sharpening queries from these clients. But when dealing with more sophisticated searchers (e&, Ph.D
students), they may assume the searchers know exactly what they what. This user modeling function o i
the reference librarians has been reported in [7].
It
We summarize these differences between librarian and online system assistance in refining queries in Table 2.
is clear that in order to design more effective and useful online reuieval systems, query refinement functionality
n d s to be incorporated into Lhe retrieval systems.
5
Process Model of O n l i n e Q u e r y Refinement
We derived some interesting empirical findings penaining to how searchcrs refine their queries when using hc
reuieval system. These findings were used io develop a process model that can facilitate online query rrfinemcn[.
126
T w o Approaches l o Or
5.1
w e observed that searchers nfu
browsing and retrieval by inst;
5.1.1
Semantic-Based Bmwsi
Searchers may obtain their sea!
swdics we identified a typology
included synonymous term (ST)
(DT). The searcher's query refir
of concepls (terms. topics, etc.)
We discuss these semantic opcr
1. Synonymous Term (ST):
terms were self-generated
as a consequence of havL
term. "ergonomics," to ex
2. Broader Term (BT): Sea
are more general. This c
under the specific terms s
looking for books about *
to "statistics" after obtain
3. Narrower Term (NT) SO
terms. This o c c u d w
expectation of what a re:
huge set of matched citai
"measurement." Searcher
4. Adjacenl Term (AT): SC
initial terms in content. "
These adjacent terms ma
2
i.s from different
angles
by
f
I
Differences
The Role
SOWCCS
of
11 SearchcrLibrarian
I Searciur/Svsmn i
/I o Active
1 o Passive
1
o Subjca A m . Syslcm, o Subject Area
~
Knowledge
Classification Scheme
User Modelinc
o Yes
o No
Table 2: Librarian vs. system in query rcfinunent
5.1 Two Approaches to O n l i n e Query Refinement
We observed that searchers refined their queries using two approaches to which we referred as semantic-based
browsing and retrieval by instantiation.
5.1.1 Semantic-Based Browsing
in generating queries thc inbout h e system's conmllcd
I helps pamns obtain a good
process is essentially one of
Searchers may obtain their search terms by browsing thc semannc network of concepts. From our empirical
studies we identified a typology of semantic operotors that s a c h e r s used for refining heir queries. This typology
included synonymous term (ST), broader term (BT). narrower term (NT), adjacent term (AT), and disjointed tcnn
(DT). The searcher's query refinement process can be viewed as a ttaversal (browsing) in this semantic network
of concepls (terms, topics. etc.) using these five operators. Figures 4 and 5 illustrate rhis t p of query refinement.
We discuss these semantic operators below.
I . Synonymous Term (ST):In the scarckdsystem interactions, we observed a proccss when the synonymous
terms were self-generated by the searchers when their initial terms matched too few citations. For example.
as a consequence of having found no matches by using "human factors." a user next used a synonymous
term. "ergonomics." to express hs query.
was their ability to perform
e queries from inexperienced
considerable amount of time
ticaud searchers (e.& Ph.D.
is user modeling function of
2. Broader Term (BT): Searchers may change from terms whxh an more specific in meaning to terms whch
are more general. T h ~ schange may be due in part to a searcher's misconception thaI citations classified
n refining queries in Table 2.
under the specific terms should also be classified under broader terms. For example. a searcher who was
looking for books about "statistical power" immediately changed her scarch term from "statistical power"
to "statistics" afwr obtaining no matches f" the first term.
j
luery refinement functionality
their qucnes when using thc
lute online query reiiiemcni.
3. Narrower Term (NT) Sometimes. searchers may decide to narrow their queries by choosing more specific
terms. This occurnd when a set of matched citations was too large. Searchers ofvn had their own
expectation of what a reasonable number of matched citations should be. For example. after deriving a
huge set of matched citations under "statistics," a searcher immediately retumed to a more specific term.
"measurement." Searchers also used namwer terms when h e initial term generated jmlevant citations.
(AT):Searchers may change their search topic to one which partially overlaps with their
initial terms in content. This o c c u ~ ~ ewhen
d
few relevant citations were derived from their initial us".
These adjacent terms may capture aspects of he query which were not represented in the initial terms.
4. Adjacent Term
127
For example. a searcher switched from "economics of informdon" to "game theory" in two c o ~ c u t i v ,
search stages in an attempt to explore the hfferent aspects of the "bargaining problem."
5 . Disjointed Term (DT): Searchers may change their search topic to another Xmandcally disjoinred
This occumd when multiple topics were involved in a query and whcn previously ignored c o n s c i o ~"eeds
were revealed during the search. h an o d k retrieval sysum that has hrll Boolean capability. h s %
of query can be expressed using Boolean operators. Searchers. however. may have problems using these
operaton. In our study. four subjects attempted to use a Boolean search. while most subjects searched w,
single wrm (not using the Boolean search option) even though they had more than one topic in their que9.
For i m m c e . a scarchcr used "hyperunsion" and "salt substitute" (two dsjointed topics) in two XPsteps in an effort to search for materials about " h e effect of salt substitutes on hyperunsion" T h i s qu
e9
can be expressed by the Boolean logic: "hyperunsion AND salt substitute."
These five operators correspond to the various links of the semantic rework ~tll)cturcwe have proposd.
ST operators follow the USE links in the semantic network (sct Figure 2). ' h e NT and BT operators follow,
in either direction, the NT or BT links. Terms which have a common ancestor or descendant (but not 0"
hierarchy of NT or BT links) arc considered to have an AT relationship (going from one such term to mhcr
is considered as an AT operation). Terms which are not llnked in thc network are considered to have an DT
relationship (likewise. going from one such term to another is considered as an DT operation). We believe hat
an online thesaurus that is based on IIUS
semantic network suucture can help searchers refine their queries.
5.12
Retrieval by Instantiation
Detailed citation information can also help searchers obtain new CUCS for search. This includes: the ude.
author, thc publisher, and the index tcrm of a book. In our studics. we frcquendy heard statements such as:
Eprcscnt the searchers' informatic
"ius book is exactly what I am looking for
terms.
I am looking for books similar to this one.
Second, the search terms elicit
the LCSH Handbook). Non-index
By using information associated with the citations that are right on target, searchers can obtain other relevml
citations. Searchers can use the index terms derived from the matched citations to perform a subject search. New
relevant citations can be obtained. Searchers can examine the titles of the matched citations in order to elicit new
search terms (tide also reflects the content of a book). Searchers can find all books written by a particular a u h r
in the area (most authors work in a few specific areas). Searchers can sometimes find all new books published by
a particular publisher (many publishers specialize in cenain subject m a s ) . This nmeval mechanism. which we
referred to as retrieval by instantiation. has also been found useful in the design of other informadon reuievd
systems (291 [30)1321.
5.2 A Process Model
USE links in this thesaurus. T:
opcraton) whch represent the sea
the
5.2.2
Terms Grouping Stage
The Terms Grouping Stage CON
derived from the previous stage acc
network) and generating new canc
rank these newly derived terms.
1 . Grouping: Grouping of tem
Grounded on our empirical findings, we have developed a proccss model for assisting query refincmenr u s q
online information retrieval systems. This process model consists of five stages as shown in Figure 6. They
arc: Initial Query Stage, Terms Grouping Stage. Relevance Evaluation Stage, Terms Solicitation Stage, and
Citations Instantiation Stage. We discuss each of these stages below.
lead to/from these terms. Tivia some paths arc classificc
relating to a dissertation top
An imponant by-product oft
p a h . These new terms can
5.2.1
Initial Query Stage
A search session starts with the Initial Query Stage (see the box at the top of Figure 6). l b o activities aft
involved in this stage. First. searchers express their queries by using a few search terms. These search urms
128
2. Ranking: The concept grou
terms (originators) in each g!
than a group with fewer onh
:'game thcory" in two c o " s e ~ ~ t i ~ ,
:aining problem."
I----
other semantically disjointed 'epic,
prrviously ignored ~ ~ i o n uq ss
s full Boolean capability, this 'ype
cr, may havc problems using these
while most subjects searched with
more than one topic in thcir queq,
I disjointed topics) in two scpmlc
i t u m on hypcrunsion" Thls que9
.
NU."
t-k svucturc we have proposed. The
T ~ NT
c and BT operators follow,
xstor or d w n d a n t (but not on a
ing from OW such term to another
~ r arc
k considend to have an DT
an DT operation). we belicvc that
~ a ~ krefine
r s their queries.
Figurc 6: A proccss model for query rcfmemcnt
:arch. This includes: the tide,
endy kard statements such as:
leprescnt the searchers' information n d s . The searchers can use Boolean operaton to combinc thcir search
lCrmS.
scarchen can obtain othcr relevat
subject search. New
ched citations in order to elicit r e w
books writun by a panicular a u h r
ncs find all ocw books published by
his retrieval mechanism. which we
!csign of other information retricvd
IS to perform a
or assisting query refinement using
;rages as shown in Figurc 6. Thcj
age, Terms Solicitation Stage, and
Second, the search terms elicited arc used to consult a semantic network-based online thesaurus (based on
LCSH Handbook). Non-index terms supplied by the scarchcrs can be translad to indcx terms by tracing
USE links in h s thesaunrs. Thc output of this stage is a list of index terms (possibly combined by Boolcan
operators) which rcprcsent the searchers' qucrics.
5.1.2 Terms Grouping Stage
Terms Grouping Stage consists of two proccsscs. The first proccss focuses on gmuping the index terms
derived from the previous stage according 10 their contcnts (relationships to each other with respcct to the semantic
nelwork) and generaung new candidau: terms from each group. The sccond process applies a few heuristics to
rank these ncwly derived terms.
1. Grouping: Grouping of terms is accomplished by first instantiating all paths in the scmantic network which
lcad to/from lhcse terms. This opcrauon is rcfcmd to as spreading ocnvorion. Tcrms which arc connccUd
via some paths are classified into the same concept group. Complex qucrics (e.g., a h . D . student's query
relating to a dissertation topic) may often involve morc than one concept group.
An important by-product of rhis spreading activation proccss is a set of llcw terms found dong the activated
palhs. Thcsc new terms can become good candidate terms for the searchers' qucncs.
)p of Figurc 6). Two activities arc
search terms. These search urms
2. Ranking: The concept groups derived arc first ranked according to the total number of searcher-supplied
terms (originators) in each group. A group with many originators is considered more imponant to the query
than a group with fewer originators. We d e r to this heuristic of ranking as the principle of origination.
129
we rank the newly dcnved terms w i h n each group Thc d i n g IS bascd on
specificiry A term which IS more specific reveals the searcher's underlying mformmon
more general counterparls (an observauon from our empincal studies) and thus should be
Next.
5.2.3
* prosss
Prlncrpk
was
passive
Y
hnt,
med
h% Ih
LhcC , L a ~ olevel
n using author
on IO capture the Subject
b i l c v c , can serve as a ba
r f ~ smodel in
Relevance Evaluation Stage
n e Terms Grouping Stage produces groups of ranked candidate terms. Thcsc terms need to be evaluated
searchers, however. During the Relevance Evaluation Stage (see the box in the center of Figure 6),
-htn
select and rank those terms they hnk arc relevant to Ihcir queries. This process is intended IO reveal
scarcht,,,
conscious needs. A list of ranked citations can Lhcn be generated from the r d e d candidate terms.
ur
p ,rerence
5.2.4
Terms Solicitation Stage
I1J
shown in Figure 6, if searchers are satisfied with thc reuicved citations (both in terms of h e nmh
01
mauhed citations and the relevance of h e citations), the search terminates successfully. On the other
ill
searchen arc not satisfied with the results. thc search process continues. New information nccds to bc supplied
The search moves on to the Terms Solicitation Stage if the searchers have new search terms to provide. wh,
no W h terms can be supplied by the searchers. thc interaction proceeds to the Citations Instantiation sh
ge
Th~sstage involves the use of the detailed information in Ihc reuicvcd citations.
At the Terms Solicitahon Stage searchers supply new search terms. These new tcms arc first translaa lnlo
index terms. A spreading activation process using these new terms and the previous terms then follows. N~~
candidate terms and citations can be generated. This iterative process ends when no new terms can k supplied
by the searchers. This stage attempts to simulate the searchers' semantic-based browsing approach descnh
earlier.
AS
"
5.2.5
Citations Instantiation Stage
While the Terms Solicitation Stage aims at soliciting new terms from the searchers, Ihc Citations INtantbtion
Stage instantiates the information embedded in the citations that have been selected by the searchers. It attemp&
to simulate the retrieval by instantiation approach for query rcfmment. The index terms assigned IOthe xlccted
citations, in particular, provide good clues for the search. By performing a spreodrng activation p w s s using
the old terms and the index terms just derived, new relevant terms and citations can be obtained.
As shown in Figure 6. OUT proposed process model for online query refinement is iterative in narurc. By
applying the heuristics and the refining mechanisms in this model repeatedly, a retrieval system can better assist
searchers in aniculating and refining their queries.
w. ROSS
Ashby. An Infrodun
Marcia J . Bates. Subject acu
'*I Informution Science, 37(6):35:
D,Baua and J. G. Davis. A SIU.
13' betwan expen and novice de
System (/CIS-89). pages 9 1- 1
p ~ J., Belkin. R. N. Oddy. and
/ournal of Documenlolion, 38
J.
Marinus Bouwman. Huma
15' Management Science. 29:653(61 R.J. Brachman. w h a t ' s in a c
Man-Machine Studies. 9. 197
[7] Hsinchun Chen and Vasant Dt
interaction In Proceedings o
(81 Hsinchun Chcn and Vasaru I
systems. In Proceedingr of I /
R. Cohen and Rick Kji
networks. Information Proce
'91 Paul
101 Timothy C. Craven. Thcsa
6 Conclusion
permuted index displays. Inj
Electronic information storage and retrieval systems have changed the way people retrieve information. lnvcston
access financial data of companies via their terminal at home. Lawyers consult law cases by online browsing of
databases. Researchers obtain information about relevant studies by using online catalogs and online bibliographic
databases. W l e the amount of information available for online access increases dramatically, searches arc often
problematic. Th~smay be due in large part to the difficulty involved in aniculating and r c h n g the searchen'
underlying dormation needs. Human information specialists' role in refining queries has been well r c c o p u d
in prior studies. 1n.our research, we observed an online query refinement process consistent with Taylor's theory.
130
I I ] K. Andcrs Ericsson and He
Cambridge, Massachusetts, '
121 G. W.Fumas.T. K. Landau
communication. Communlcc
[I31 M. D. Good. J. A. Whitesid(
cations of [he ACM, 27( IO)::
ists
hiring is based on h e principle of
ying informdon nccd better than it.,
and thus should bc ranked higher.
Ol-3
@,,Mentation to capture the subject arta knowledge and developed a p m s s model for q u c y refinemem. This
,,,del, we believe, can serve as a basis for designing more “intelligent” and useful infomation nvieval syste~
ns.
\VC have incorporated this model into the design of a knowledge-based docwnenr r t h e v d system. Readen aIe
;e urms need to
be c v a l u d by
n the center of F i y r t 6). xarchers
IS is intended to reveal thc seearcher’s
anked candidate urns.
(both in terms of the number of
successfully. On the othcr hand. if
w infomation nccds to bc supplied.
new search terms to provide. When
t thc Citations Instantiation Stage.
IS
(21 Marcia J. Bates. Subject access in online d o g : a design model.
Information Science, 37(6):357-376, November 1986.
JOUrMl
of rhe American Sociery of
131 D. Batra and I. G.Davis. A study of conceptual dala modeling in daub= design: similarities and differences
bctwcen expen and novice designen. In Proceedings of the 10th IIuerMnOMl CorSfcrence on informrun
System (ICIS-89), pages 91-100. Boston. MA, December 1989.
OILS.
c new terms are first uanslated into
previous terms then follows. New
when no TYW terms can be supplied
lased browsing approach dcscnbed
141
N. J. Bekin, R. N. Oddy, and H. M. Brooks. Ask for information revicval: Pan I. background and theory.
Journal of Docmenration. 38(2):61-71, Junc 1982.
[ 5 ] J. Marinus Bouwmm. Human diagnostic reasoning by computes: an illusration from financial analysis.
Management Science. 29:653472, Junc 1983.
161 R.J. Brachman. what’s in a concept: StNCIUd foundations for semantic network. Intermdona1 J o w d of
Man-Machine S t u d m . 9. 1977.
archers, ihe Citations Instantiation
elected by the searchers. It attempts
index terms assigned to the selected
l spreadrng activation process using
ION CM be obtained.
:ftnemcnt is iterative in nature. By
, a retrieval systcm can bencr assist
171 Hsinchun Chcn and Vasant Dhar. Reducing indeterminism in consultation: a cognitive model of wrflibrarian
interaction In Proceedings of the 6th National Conference on Arnficial lnrelligence ( M - 8 7 ) . 1987.
[SI Hsinchun Chen and Vasant Dhar. A knowledge-based approach to the design of document-based retrieval
systems. In Proceedings of the j l h Conference on office Informanon Sysrem, Cambridge, MA 1990.
191 Paul R. Cohcn and Rick Kjeldsen. Information reuicval by consmined spreading activation in semantic
networks. Informalion Processing and Managemenr, 23(4):255-268. 1987.
[IO] Timothy C. Craven. Thesaural relations in a concept-network management system for customizing of
permuted index displays. Information Processing and Management. 2 0 ( 5 / 6 ) : 6 3 3 410, 1984.
eoplc retrieve information. Investors
;ult law cases by o&ne browsing of
ine catalogs and online bibliogmphc
ascs dramatically, searches arc often
iculating and r e h n g the searches’
ng queries has been well rccognizcd
a s consistent with Taylor’s t h c ~
[ I l l K. Andcrs Ericsson and Hcrben A. Simon. Protocol anaiysis: verbal reporr as &a.
Cambridge, Massachusetts. 1984.
The MIT Rcss.
[I21 C . W. Fumas. T. K. Landauer. L. M. Comer, and S.T. Dumais. The vocabulary problem in human-system
communication. Communicafions of rhe A C M . 30(11):964-97 1, November 1987.
~ .
1131 M. D. Good, J. A. Whiteside. D. R. Wixon. and S. 1. Jones. Building a user-derived interface. Communicotions of rhe ACM. 27(10):1032-1043, October 1984.
131
M. J. metula, and F. J. Lerch. Pcrformancc f behavior: a study in the fragility d e x p euse.
Procee&ngs of the 10th lnternarional Coderence on IrJormation s~stcmr(ICIS-89). pages 101-1 17, B
Q'or,
MA, December 4-6 1989.
MiChad Davld Willi$"s
351. 1984.
[ 141 8. Huguenard,
[ 151 J. Jacoby and V. Slamccka. Indexer Consistency Under Minimal Conditions. Documenlation, Inc., B
W.A. Woods. What's in
(331 Representation and Undc
MD, 1962.
I
[ 161
F. W. Lancastcr.
Informution Retrieval Systems. John Wiley and Sons. h.,
1979.
[ 171 K a n Makey. Lcveb Of qUCStiOn formulation in negotiation of information nccd during the online prC
imemicw: a proposed model. Infomution Processing and M a ~ g e m e n r .17(5):215-225. 1981.
Oth
1181 K a n M A c y and Pauline Athcmn. Online Training and Pracricc M a w 1 for ERlC Data Base semche,,
Syrac~se,NY:ERlC Clcaringhousc on Information Resources. W C : ED 160109. 1978.
[19] A. Ncwcll and H. A. Simon. H m n Problem Solving. Renticc-Hall. Englcwood Cliffs. NJ. 1972.
[20] Wanda Orlikowski and Vasant Dhar. Imposing svucture on linear programming: an empirical uysu
of expen and novice models. In Proceedings of the Filth National Conference on Artificial INelligewc,
( M I - 8 6 ) .Philadclpha PA, August 1986.
1211 M. R. Quillian. Semantic memory. ln Semantic IrJormarion Processing, 1968.
[22] Perru Shoval. Principles, procedures and rules in an expcn system for information retrieval. I n f o ~ i o ,
Processing and MaMgement, 21(6):475487. 1985.
1231 R.F. Simmons. Semantic Nerwork: Their Compuration and Use for Understanding English Sentences, In
Schank. R.C. and Colby, K M . (eds.). Computer Models of Thought and Language, Freeman. 1973.
[24] Mary Elizabelh Stevens. Automatic Indcring: A State-of-the-art Reporr. U.S. Government Printing Offic(
Washington. DC. 1965.
1251 Gary W. Strong and M. Carl Drou. A thesaurus for end-user indexing and retrieval. Information Procejjlnq
and MUMgerPIent, 22(6):487492. 1986.
[26] R. Tagliacozm and M. Kochcn. Information-seeking behavior of catalog users. Information Storage p~
Retrieval. 6:363-38 I . 1970.
[27] Rober S. Taylor. The process of asking questions. Am. Documen.. 13:391-396. 1962.
128) Rober S. Taylor. Qucsuon-negotiation and information seeking in libraries. College and Research kbraritf,
29:178-194, May 1968.
[29] Bemd Teufel. Natural language documenb-indexing and nuicval in an information system. In /CIS'&
Conference Proceedings, Minneapolis. MN, December 1988.
[30] F. N. Tou. M. D. Williams. R. Fikcs, A. Henderson. and T. Malone. Rabbit: An intchgent databav
assistant. In Proceedings of the National Conference on Artificial Intelligence. 1982.
1311 D.A. Waterman and A. Ncwcll. Prolocol analysis as a task for anificial intelligence. Arrificial Intelligence
2:285-318. 1971.
132
in the fragility of e x p e a
(]CIS-89).pages 101-1 17. Bos,o,kl
spdy
JIIJ.
*.
Documentation. hc.. Be&
sda,
($1
~ i c h a c lDavid Williams. What mikes rabbit run? lnrcrnaiional Journal of Man-Mochinc Srudies. 21:333352. 1984.
W.A. Woods. What's in (1 fink: Foundationsfor Semantic Networks. In Bobror, G. and Collins. A. (eds.),
1331 Representation and Understanding: Studies in Cogniive Science. Academic Pnss. N e w York. NY, 1975.
h.,
1979.
ion necd during h e online p
, 17(5):215-225. 1981.
~ r
*h
d for ERIC Data Base Scorcherr,
3D 160109, 1978.
bglewood Cliffs, NJ. 1972.
rogramming: an empirical analrsis
Ionfercnce on Arnfcial I ~ c l l i g c n c ,
:. 1968.
r information reuicval. l n f o m i o n
ndcrsranding English Sentences.
Id Language, Freeman. 1973.
1.
u.S. Govcmmcnt Printing Offie.
dog usen.
information Storage
391-396, 1962.
rics. College and Research Lbrartcs.
an information system. In ICIS'88
Rabbit: An inichgent databaw
kgence. 1982.
le.
d intelligence. Artificial lntcllige~c.
133