Ingrid Nilsson
BTH & Lund University
Sidan 1
Last printed 4/19/2006 8:27 AM
NETLEARNING2006
Do you really know where your students’ attention is?
A bilingual eye-tracking experiment combining listening and picture
viewing .
Introduction
In computer assisted teaching, as well as in newspaper reading or TV/computer
advertising, a combination of message types is utilized, which, ideally, produces the
intended reaction in a viewer. In advertising, a large part is name recognition through
auditory and visual highly attention-grabbing means; in newspaper presentations, it is
often a ranking- and importance-weighing presentation emphasized through font size
and hierarchical positioning on a page; in computer assisted teaching, it is a pedagogic
segment of informative text presenting a process with or without an illustrating picture
sequence. Of the three, distance learning is the most interactive, even if the immediate
contact between teacher-student is replaced by course content being transmitted as
visual and/or auditory material and written interaction later following. This allows
greater flexibility in time and space, but it removes the direct feedback. So, when
students are sitting in front of their computer – is there any way of knowing how they
listen to, and perceive, the information received? Is there any means of deciding how
or where they look at the computer screen to search for information?
This paper describes an eye-tracking experiment where a spoken series of sentences
illustrating motion verbs have been presented to test subjects, each followed by a
picture illustrating the same motion event. Of interest has been to document whether
gaze fixations followed a specific language pattern, either in words used in the actual
material heard or according to native language, or to see whether not so much actual
language use as semantic content guides the immediate visual focus of attention.
General background
Do we really see what we hear?
The visual segment of a message can either be used as an unimportant backdrop, or it
can be used to illustrate the brand, event, or process being described. But, without
seeing the recipient, can we really decide whether the information presented visually is
effective or not; whether the attention directed towards the image part influences, or is
being influenced by, the transmission of the message? In order to examine this
process, eye-tracking experiments can indicate when and where visual attention is
channeled as demonstrated through gaze fixation patterns. By examining fixation
patterns, indicating when and where visual attention is directed, and whether this is
done as a response to a spoken input, we can study the connection picture-text and its
influence on a viewer/listener.
Ingrid Nilsson
BTH & Lund University
Sidan 2
Last printed 4/19/2006 8:27 AM
Eye-tracking studies
Experiments with eye-tracking are based on a camera recording and measuring the
movements of the eye of a viewer while watching a stimulus in front of her. The
stimulus watched is photographed as well, and the gaze pattern is overlaid on that
picture. Eye-tracking experiments supply precise information, in intervals of 20
milliseconds, on where viewers are looking, how long they are fixating on a particular
location, the order of their fixations, and the number of repetitions within a specific area
in a picture 1 . Thus, eye-tracking methodology provides precise insight into how an
individual viewer perceives a visual stimulus presented. Furthermore, many of the
viewing reactions to a spoken stimulus in terms of initial gaze shifts – called saccades
– and fixations are unconscious or involuntary unless the test subject has been told to
observe his viewing behavior, as proven by the fact that both the rapid gaze shift
patterns and the fixation patterns are present even if a screen turns blank after the
initial stimulus presentation. The gaze focus still falls of the part of the screen where
the stimulus section mentioned was first shown 2 .
Previous eye-tracking studies have also been performed to examine how, f.i.,
newspapers or net papers are read 3 , or how effective advertising banners on the
Internet are 4 . For newspaper reading, it was found that fixation patterns for text and
pictures were dependent on whether the page was scanned for information or read for
contents. Readers spent more time looking at the pictures when scanning, and very
little time fixating on the pictures when reading for contents (Holmberg). Newspapers
were read in 55% of the cases, and scanned in 45%. Net papers, on the other hand,
were scanned (56%) more than read (44%) (Holmqvist). The answer concerning
banners is that in a majority of cases they are not noticed at all, or even actively
avoided, by the viewer, even when they contain information relevant to a given task
(Pagendarm), or that only if they are positioned centrally in the screen, and inside the
borders of the text being read, are they noticed by the viewer (Dren).
These studies, among others, are indications of the relations between words and
picture, demonstrating both unintentional and intentional focusing of the attention on
details or sections within a picture. The following study is an attempt to examine
whether specific formulations in sentences presented to test subjects can steer the
focus of attention to particular areas in a picture.
A linguistic area particularly suited for such studies is spatiality. Fundamental to the
possibility of expressing the concept of space is the fact that it is relational 5 . There has
to be an agent or an otherwise deictic center – a reference point in relation to which a
spatial coordinate or change in coordinates can be indicated. That means that there
have to be at least two areas of interest both in a picture and in a verbal message: an
action and a place. There are various ways of expressing this relation through lexicon,
syntax, or semantics in different languages, some focusing more on the manner, some
on the location or direction, and some on the process – which could all influence where
a gaze is primarily focused, and thus be of importance to the directing of focus in any
1
For more information on eye-tracking methodology, visit the web page of the Eye Tracking
Laboratory at the SOL Centre of Lund’s University,
http://www.sol.lu.se/humlab/eyetracking/index.html?expand_menu=6
2
Johansson et al. 2006
3
Holmqvist et al. 2004; Holmberg et al. 2005
4
Pagendarm et al 2001, Dren 2005
5
Frawley 1992
Ingrid Nilsson
BTH & Lund University
Sidan 3
Last printed 4/19/2006 8:27 AM
message combining visual and verbal means, as well as to deciding whether the
structures of different languages induce us to focus visually on different aspects of a
picture.
Expressions for spatial directions and locations are present in all languages, but the
level of specificity often varies. If a language utilizes only a few basic spatial
expressions, where the level of detail is not very specific, it would follow that a more
detailed way of describing spatial positions is unimportant, or is replaced by some
other means of indicating positions relative to a deictic center. If a language has a
multitude of spatial expressions with more subtle distinctions, positional information is
of higher communicative importance, and presumably of higher visual importance.
A grammatical theory giving expression to such distinctions between languages is the
Markedness Theory. Originally formulated within the Prague school of linguistics by
N.S.Trubetzkoy as a phonological theory 6 , it has later been applied also to syntax and
semantics.
The Markedness Theory
According to the theories of markedness and language universals, certain basic
occurrences (such as vowel, consonant, negation, agent, verb, object, place, etc.) are
found in most languages, and are therefore considered common, or unmarked. Other,
often more specific, elements are more uncommon and are therefore considered to be
especially marked 7 . Thus, whereas all languages can express location when
describing an action taking place 8 , most can also express some form of direction, but
many, however, may lack specific words to do so. Normally ‘in loco’ adverbs occur with
situative verbs, and ‘ad locum’ adverbs, when they exist, with motion verbs 9 . In
English, however, the ‘ad locum’ adverbs have mostly fallen out of use (e.g., hither and
thither are now considered archaic), even if the means to express directed motion
through rewriting a translated expression, or “circumstranslating” 10 , remains. (For
instance, you can easily (even if not in one-word expressions) describe: walking
towards/from a place, or walking in a place; as well as whether this action is taking
place close to the speaker, close to the listener of a communicated message, or far
away from both speaker and listener; whether the goal of the action is included, or only
the process in action; whether the manner of action is included in the description, or
only the place/direction.)
Must any of these choices be included/excluded in a communication in a specific
language? 11 If so, how does that influence the directing of our visual attention, or the
choices made by a translator? 12 Does it also influence the interpretation or the visual
focus by a reader/viewer? Different languages have different degrees of illustrative
expressions relating to such descriptions, and the closer in typology the two languages
to be compared, the more similar such expressions are expected to be, and the more
specific any discrepancies have to be. Thus, through eye-tracking experiments nonintended discrepancies in gaze focus and linguistic interpretation between two
6
Lyons 1977
Comrie 1981, Jakobson 1968, Sobkowiak 1997
8
Comrie 1981
9
Hyman 2003
10
Nilsson 2006 (to be published)
11
For a discussion, see Mesterton 1979.
12
For a discussion of this issue, see Catford 1965.
7
Ingrid Nilsson
BTH & Lund University
Sidan 4
Last printed 4/19/2006 8:27 AM
languages could be indicated, and when a visual/auditory combined message is being
used in, for instance, advertising or communication, either an adjusted picture or an
adjusted sentence could be inserted into a message to re-direct gaze and linguistic
focus towards the intended goal.
Concerning spatiality, Leonard Talmy 13 has coined the terms verb-framed, and
satellite-framed in language typology to describe how languages construe such
events. The descriptions are verb-framed if they are expressed mainly through a
combination of one finite verb expressing either motion, manner, or path 14 , plus one or
more complementary non-finite verb forms expressing the other aspects of the event
(for example Spanish: ‘salió de la camara rodando’ (He left the room rolling); or
satellite-framed if they are expressed through one (motion) verb, plus
adverbials/particles more or less tightly bound to the verb, expressing one or both of
the other aspects (for example English: ‘he rolled out of the room’). Typical examples
of these two structural types are Romance languages (verb-framed) and Germanic
languages (satellite-framed). In Swedish (Germanic), a manner verb + a directional
particle/adverbial (Jag sprang in i huset. = I ran into the house) is the preferred way of
formulating such an event, while in English (even though Germanic) there can be a
choice depending on whether the main verb is based on Germanic or Latin influenced
vocabulary (I ran into the house/I entered the house running.).
Furthermore, there seem to be languages which will not quite conform to the pattern
for either one of these two types, though, such as Thai 15 , and a third framing type has
therefore been proposed by several linguists. Dan Slobin 16 , and Sven Strömqvist 17 ,
among others, have suggested the term equipollently-framed– which can also be
thought of as multi-(or serial-) finite verb framed – languages. Hopefully, a more
extensive study, using eye-tracking equipment, could relate the three types of
language and their different manners of expressing motion, manner, and direction to
each other, further clarifying attention focus and visual patterns, and facilitating
communication as well as translation.
Teaching students through a second language, the question of the translative validity
of corresponding or equivalent words in different languages is eventually raised 18 ,
even between languages as similar in structure as Swedish and English, and an
experiment such as the one performed here could help resolve the problem as to
whether a sentence such as the Swedish ‘han cyklade dit’, or ‘han cyklade där’, would
both – or just one of them – be interpreted as being synonymous with the English
sentence ‘he was biking there’. (Possibly because of just such inherent ambiguity,
English also employs fewer spatial words overall than does Swedish 19 . (For an
illustration of English verb complements, see Svartvik & Sager 1996.)) Since Swedish
situative adverbs/particles, especially in their directive function (hit/dit (≈ hither/thither),
etc.) have no overt manifestation in English, this functional contrast would then have to
be resolved, or ‘disambiguated’, in English, through other means to which an eyetracking experiment could give some indications.
13
Talmy 2000a and b.
Talmy 2000b.
15
Zlatev 2004a, 2004b
16
Slobin 2004
17
Strömqvist 2005, 2006 (in press)
18
Jakobson 1959
19
Jespersen 1964
14
Ingrid Nilsson
BTH & Lund University
Sidan 5
Last printed 4/19/2006 8:27 AM
But can it be demonstrated that even such a small difference from a typological point
of view as is to be found between Swedish and English would result in different
patterns in gaze focusing? And if so, can this difference be generalized to other more
dissimilar languages?
Previous background studies
In order to first of all find out whether any difference really existed between English
and Swedish concerning how directive/situative expressions are used when translating
between the two languages, all material containing the Swedish words hit or dit either
in original or translated texts was extracted from a parallel translational corpus 20 , the
English Swedish Parallel Corpus (2001), a 2.8 million word corpus consisting of a
number of Swedish and English original texts and their official translations (Link to the
opening page of ESPC - http://www.englund.lu.se/research/corpus/access.phtml).
Among the 450 examples of hit/dit originally used in Swedish, and the 450 examples
used in translations from original English texts, three different translational equivalents
were found 21 :
In the first case, for a Swedish original translated into English:
o Either the translator uses the closest equivalent (here/there) (~39%), or
o Rewrites the expression in English (hit = into this room, etc) (~31%),
o Or ignores it completely (hit -> Ø) (~25%)
For a translation into Swedish:
o An English here/there is being translated as either hit/dit (directional) or
här/där (locational), depending on whether the verb expresses directed
motion or not (here/there -> hit/dit, ~32%)
o Longer expressions in English, expressing the goal for a directed motion,
are replaced by the shorter expressions hit/dit in Swedish (~27%)
o It is inserted (Ø -> hit) (~33%)
Such results, and such similar results regardless of into which language the translation
is performed, point to a difference in how location versus direction is indicated which
could be significant in terms of communication, as well as translation and teaching.
Present experiment
Methodology
Material
•
•
•
•
20
21
Pictures illustrating both an action and a possible goal
Short spoken sentences in Swedish as well as in English illustrating actions
shown in accompanying pictures
Eye-tracking equipment, complete with video, sound, and recording facilities
English, as well as Swedish, L1 speakers with knowledge in the other language
as well.
Nilsson 2005
Nilsson 2003
Ingrid Nilsson
BTH & Lund University
Sidan 6
Last printed 4/19/2006 8:27 AM
Visual material: 20 pictures from various type of illustrative material – photos,
children’s book drawings – illustrating both an action and a possible goal, with
protagonists moving either away from the camera/onlooker or towards a goal further
away or in the background. For each picture, a goal area and an action area are
posited as mutually exclusive Areas-of-Interest, within the borders of which the fixation
patterns will be analyzed.
Ex. This picture could illustrate the following sentences:
1) He was biking there.
2) He biked there.
3) Han cyklade där.
4) Han cyklade dit.
The goal area would cover the gate, and the immediate
area behind it, and the action area would cover the biking boy.
Auditory material: 80 sentences – 40 in Swedish and 40 in English illustrating an action
corresponding to that in a picture.
o Action: Simple past tense + goal (Swedish dit) 20
o Action: Simple past + location (Swedish där) 20
o Action: Simple past tense + non-specific spatial expression (English there) 20
o Action: Past progressive tense + non-specific spatial expression (English there)
20
Sentence types: To keep the material comparable to the spatial sentences previously
analyzed from the English Swedish Parallel Corpus (a bidirectional translation corpus),
verbs and usage ratios were adapted from that corpus. The four types of sentences
were each applied once to every picture.
Technical equipment: Using a 19” monitor screen, pictures and spoken sentences
were shown and coordinated through the program e-Prime 22 , using a random order of
configuration. Eye-movements and fixations were recorded with the help of a headmounted eye-tracker with head tracking (SMI iView X HT) mounted on a bicycle
helmet. A scene camera, also mounted on the helmet, recorded eye-movements as
well as fixation patterns overlaid on the screen picture shown to the test subject. The
gaze coordinates were given in terms of fixations per 20 milliseconds intervals, within a
radius of 10 pixels.
Test subjects: 19 adult speakers of Germanic languages, 10 Swedish-speaking
persons and 9 English-speaking persons took part in the experiment. All had moderate
to good knowledge in the other language as well.
22
http://www.pstnet.com/products/e-prime/
Ingrid Nilsson
BTH & Lund University
Sidan 7
Last printed 4/19/2006 8:27 AM
Table showing sentence types and conditions included in experiment
Test subjects
English
Swedish
Native Engl. &
2nd lg Engl.
Native Swed. &
2nd lg.Swed.
Tense
Location
Direction
Simple past, -ed
There
Progr.past, was -ing
Simple past, -ade
Där
Dit
Simple past, -ade
Procedure
The test subjects were presented with an empty monitor screen containing a central
dot for starting eye-fixation purposes. A short sentence either in English or Swedish
was played through head-phones, whereupon a picture was immediately shown. The
picture was displayed for 5 seconds, and all eye-movements were recorded.
Test subjects who were native English speakers heard the English sentences first in
one block, followed by the Swedish sentences. For Swedish native speakers, the
reverse order was followed. The pattern according to which the sentences in each
language block were administered to each test subject was randomized through ePrime.
The sentences consisted of a maximum of 8-10 words, often shorter, with as little intrasentence variation both concerning content as well as type as possible, ex. They were
arriving there – they arrived there; de anlände där – de anlände dit. In order to
eliminate as many linguistic distractors as possible, pronouns were used for the
protagonists. Some control or distraction sentences/pictures were utilized, where, f.i.,
the pronoun did not coincide with the gender of the person shown, or where the picture
was turned 90 degrees, or where the screen showed non-relevant material, ex. a
pencil showing outside the actual picture, in order to counteract the repetitiveness of
the task. None of the control sentences will be included in the present account.
Hypotheses
A) The sentences heard will not influence the gaze fixation patterns, so there will be
no measurable difference in where fixations are concentrated, neither between
English and Swedish, nor between how the sentences are patterned.
B) The auditory input will influence fixation patterns, and sentences containing English
–ing, or Swedish där, will show less gaze fixations in the area of interest containing
the goal, and more in the action area; and sentences containing English –ed, or
Swedish dit, will show less fixations in the area of interest containing the action,
and more in the goal area.
Since it is hypothesized that verb mode, directional expression, or language – or a
combination of the three – could influence fixation lengths and focus when tracking eye
movements of test subjects, the following possibilities might present themselves:
Ingrid Nilsson
BTH & Lund University
Sidan 8
Last printed 4/19/2006 8:27 AM
1. The goal AoI will receive a higher percentage of fixations in sentences with
the Swedish dit – i.e. the gaze will remain away longer from the action AoI
than in sentences with the Swedish där.
2. The goal AoI will receive a higher percentage of fixations in sentences with
the English simple tense –ed – than in sentences with the English
progressive tense –ing.
3. The action AoI will be fixated longer in sentences with the Swedish där than
in sentences with the Swedish dit.
4. The action AoI will be fixated longer in sentences with the English
progressive tense than in sentences with a simple tense.
5. The fixation percentages in the action AoI will be smaller in sentences
containing the simple past: English –ed there, Swedish –ade dit, than for
sentences containing English –ing there, or Swedish –ade där.
Results of the type mentioned under points 1 and 3 will be considered an indication
that the goal/place distinction in Swedish is decisive.
Results of the type mentioned under point 2 and 4 will be considered an indication that
the simple tense/progressive tense distinction in English is decisive.
Results of the type mentioned under point 5 will be considered an indication that the
syntactic distinction simple vs. progressive tenses in English may fill the same function
fulfilled by the goal/place distinction in Swedish, thus disambiguating the wider
semantic field represented by there in English.
Possible results
1. If the zero-hypothesis is correct, the viewing patterns, in terms of the five
operational hypotheses, should be the same for English and Swedish listeners
regardless of which type of sentence is being heard.
2. If hypothesis 1 is correct, a native English speaker will look more towards the
center/action being performed than towards the periphery/goal for the action,
maybe regardless of whether the sentence being heard is in English or
Swedish. A native Swedish speaker will look more towards the periphery/goal
of the action than towards the center/action being performed, again maybe
regardless of the actual language of the sentence being heard, and both will
look more towards the center/action when no goal-oriented motion is indicated
(Han cyklade där.). This would indicate a typological difference in the treatment
of directional expressions between the two languages.
3. If hypothesis 2 is correct, there is no difference which depends on the language
of the sentence being heard as to where the English vs. the Swedish native
speakers focus their gaze when hearing a directive motion being described, but
only depending on the situation being described. If a specifically directed
motion is described, the gaze is directed towards the goal.
4. There is a difference in where the fixations are directed, but it is language
dependant, and both native and non-native speakers follow the same pattern,
i.e., there is a combinatorial effect, but it is the result of syntactic and/or
semantic combinations used in that particular sentence and language, not on a
difference in focus based on typology or situation specificity.
Ingrid Nilsson
BTH & Lund University
Sidan 9
Last printed 4/19/2006 8:27 AM
Hypothetical results of picture viewing with auditory input.
Type
1
Native lang.
English
Swedish
2
English
Swedish
3
English
Swedish
Lang. version
heard
English
Swedish
English
Swedish
Action
described
X
X
X
X
English
Swedish
English
Swedish
English
Swedish
English
Swedish
Goal
described
X
X
X
X
X
X
X
X
Factors influencing
focus of attention
Typology of native lg.
decides focus.
Ø spec. language,
≠ approach to
directive expressions
Situation described
decides focus.
Ø spec. language,
= approach to
directive expressions
Language specificity
decides focus.
9 spec. language
Ø approach to
directive expressions
Ideally, the outcome would support the schematics of the expected results in the table
above. So far, with some of the material not yet analyzed, there is some indication that
type 1 – a different approach to spatial directive expressions – could be the case, even
if variations have been found. Also, semantics seem to play a larger role than a slightly
different if related typology, or type of situation, which indicates that some version of
type 3 could be true.
RESULTS
Fixation averages
To illustrate the results of this experiment, two pictures with accompanying sentences
have been chosen for detailed analysis. The verbs are ‘arrive/anlända’ and ’go/gå’,
each given with two variables – ‘arriving there’ vs. ‘arrived there’, and ‘gick där’ vs.
‘gick dit’. Looking at the averages for the amount of fixations for the two Areas-ofInterest (Goal and Action), as related to the different conditions and variables, the
following results are of interest:
Highest percentages of Area-of-Interest (AoI) fixations per verb total, regardless of
verb subtype:
All fixations, all subjects:
Arrive, goal -43.63%.
All fixations, all subjects:
Go, action -50.24%.
All fixations, all subjects:
Anlända, goal -- 48.66%.
All fixations, all subjects:
Gå, action -54.82%.
Ingrid Nilsson
BTH & Lund University
Sidan 10
Last printed 4/19/2006 8:27 AM
The fixations on the action AoI for ‘arrive’ or ‘anlända’, and on the goal AoI for ‘go’ or
‘gå’, were approximately half as numerous as the ones mentioned above.
Concerning arrive – combining the –ing and –ed samples, the goal area of interest
received most attention.
Concerning anlända – combining där and dit samples, the goal area of interest overall
received most attention.
Concerning arrive, per condition, the goal -ed area of interest received most attention.
Concerning anlända, per condition, the goal dit area of interest received most
attention.
These results support the research hypothesis.
Percentages of Area-of-Interest fixations per condition, total:
Arrive, goal, -ing
40.58%.
Arrive, goal, -ed
46.68%.
Anlända, goal, där
45.37%
Anlända, goal, dit
51.95%.
Go, action, -ing
Go, action, -ed
Gå, action, där
Gå, action, dit
49.63%.
50.84%.
61.11%.
48.53%
Concerning both go and gå, the action areas of interest received most attention.
Concerning go, the action -ed area of interest received most attention, which does not
support the research hypothesis in respect to the combination of conditions. If H1 had
been supported, action + -ing would have shown the highest percentages.
Concerning gå, the action där area of interest, received most attention, supporting H1.
In English, the difference between the percentages for the two conditions used is not
very large -- only around 1.2%. The difference is considerably larger for Swedish -almost 12%, which could indicate a stronger distinction between direction versus
location. This supports the theory that spatial indications are of less importance than
process mode indications in English, especially when combined with the more
inclusive semantic field of there. Seen in terms of the Markedness Theory, the
Swedish distinction between direction and location is thus clearly marked, resulting in
higher communicative importance of a semantic combination of characteristics which
supports this interpretation. With an unmarked spatial expression such as there a
combination of a verb seen more as a manner verb than a direction indicating verb is
experienced to be quite all right in English, whereas in Swedish this combination is not
favored.
Combined per semantic type instead of per language, results strongly indicate that the
same basic semantic interpretation is evident in both languages.
Percentages of Area-of-Interest fixations per lexical equivalent total:
Arrive and anlända, goal
Go and gå, action
46.14%.
52.53%.
In terms of Talmy’s discussion of motion verbs and languages which favor either a
primary indication of manner, or languages which favor a primary indication of path, or
Ingrid Nilsson
BTH & Lund University
Sidan 11
Last printed 4/19/2006 8:27 AM
direction, both English and Swedish are Germanic satellite framed languages with a
preference for manner over path. The difference between Talmy’s two language types
can be demonstrated through the verbs here chosen to illustrate this experiment.
Arrive is a Latin based verb originally including the path (to the shore), which form has
been kept in the Germanic translation as well. Such a verb stresses path, and
consequently should show a preference for goal fixations in listeners – as is the case;
whereas go/gå is a verb of manner, channeling attention towards the action more than
the goal.
When spatial directive particles/adverbs, such as the Swedish words hit and dit, are
combined with a manner verb stressing action (ex. springa dit /run to there/, cykla hit
/bike to here/), the spatial motion interpretation of the verb is shifted to the satellites,
whereas for a path verb, such as arrive or anlända, the satellite emphasizes and
supports the goal function in the verb, but is not by itself the indicator of a directive
implication. Since it is ambiguous, the English there would consequently not be able to
specifically carry a spatial directive motion interpretation in combinations with manner
verbs, but would still be able to intensify the goal function of a path verb. The above
results supports this theory, since no marked preference for one or the other modes of
the English manner verb form in combination with a spatial adverb is shown
concerning go, but this type of preference is clear in Swedish.
In terms of expected results, there should be agreement between English and Swedish
as concerns path verbs, and their use together with either –ed (English), or dit
(Swedish). The gaze pattern should focus on the goal AoI to a higher extent than when
–ing and där are used. This was also the case.
As demonstrated through the eye-tracking experiment, for manner verbs there should
be a difference between English and Swedish: while English should not show any
preference for mode used, Swedish will show a preference for the use of dit in
sentences indicating a goal and a preference for the use of där in sentences indicating
an action as demonstrated through gaze patterns. This was also the case.
This indicates that – from a Swedish point of view – there is no possibility to
immediately disambiguate the meaning of there in combination with manner verbs,
lending support to the theory that the process, not the goal, is more important in
English than in Swedish, at least for manner type of verbs.
Per verb comparison.
Arrive/anlända
There were five instances in English of the action getting longer fixations when –ed
was used (TS2E37, TS3E28, TS14E22, TS16E22, and TS20E20). All other cases for
this verb support the research hypothesis. Thus English seems to show a weak
tendency towards hypothesis 1 for sentences with – ed, i.e., the goal AoI receives the
most attention, whereas Swedish shows a strong tendency towards hypothesis 1 for
sentences with dit. There was only one single instance in Swedish of a stronger focus
on the action AoI when dit was heard (TS9S57) than when där was heard, which
supports the research hypothesis that dit should direct the focus of attention more
towards the goal AoI.
Ingrid Nilsson
BTH & Lund University
Sidan 12
Last printed 4/19/2006 8:27 AM
For English sentences with – ing the difference in focus is generally not as strong
between goal and action Areas-of-Interest as in sentences with –ed. Sometimes one,
sometimes the other, will have longer fixations. If the goal AoI receives a longer
fixation, however, it is not with a large margin of difference.
When där is used in Swedish, there are only three instances of the action AoI receiving
the most attention; and in English, when -ing is used, there are only four instances of
the action getting the most attention. This would seem to contradict the research
hypothesis. However, in Swedish sentences with där, when the goal AoI receives the
strongest focus in sentences, it is generally with a smaller margin over the action AoI
than what is the case when dit is used.
The difference between the lengths of fixation on the goal versus on the action, in
percent, is generally not as large when -ing, or där is being used, as when -ed or dit is
used. This, again, could point to a higher importance of the goal AoI in sentences
using -ed or dit than in sentences using -ing or där.
Go/gå.
In English, under the condition –ed, there were only three instances of the goal
receiving more attention than the action; and in Swedish there were just three as well
when dit was being used; in all other instances the highest fixation rates were found
within the action area of interest.
There was only one instance in English (TS6E29), where the goal area of interest was
getting longer fixation times than the action area of interest when -ing was being used,
but the results of that particular set of coordinates seem somewhat uncertain (test
subject number 6). In Swedish, there were no such examples. For all test subjects,
the action area of interest receives the most fixations in sentences with där. In a
substantial portion of the cases (14), though, the difference in percentage between the
two areas of interest is less when dit is part of the sentence heard, and in two cases it
is roughly the same.
Concerning the difference between the per lexical equivalence totals, it is evident that
the semantic content of arrive and anlända directs the attention towards the goal,
whereas the semantic content of go and gå, with their emphasis on manner, on the
other hand directs the attention towards the action. While still verbs of motion, both
arrive and anlända, then, can also be considered as verbs of change of state – moving
from one condition to another, while go and gå must be considered verbs of manner
and thereby of a more locative action.
By not channeling the attention towards the manner -- and thereby the action -- but
more towards the change of state, verbs such as arrive and anlända adopt more of a
spatial connotation, allowing the goal to take on a larger role semantically. In contrast
to this, verbs such as go and gå, by their emphasis on manner, and thus on the action,
thereby diminish the semantic impact of even very strong directive indicators, such as
dit; or of tense markers, such as the simple past, which both should denote the action
as finished, and the goal as having been reached.
Ingrid Nilsson
BTH & Lund University
Sidan 13
Last printed 4/19/2006 8:27 AM
GENERAL DISCUSSION
As has been demonstrated through earlier studies, there is a strong connection
between the pattern of fixations in a picture, and both a text previously presented 23 ,
and a text consequently formulated for speaking 24 . The intent in this study has been to
examine if any such fixation patterns were based on purely semantic reasons,
regardless of language (null hypothesis); or whether language specific expressions,
such as spatial directives (Swedish), or syntactic mode (English), could influence these
patterns, and also whether there would prove to be any differences between the
fixation patterns which might be due to the test subject’s native language (research
hypothesis).
Overall, the goal AoI receives more and longer fixations in Swedish than in English, at
least for the for arrive/anlända (change-of-state verb), regardless of spatial
adverb/particle used in Swedish. This could indicate a cognitive typological variation
between the two languages. The two verbs used are semantically equivalent and both
historically incorporate a spatial directive preposition/adverb in their modern verb stem.
For the verbs go/gå, overall, the action AoI by a large margin receives the highest
percentage of fixations in both languages. Both verbs are lexically alike, even if the
English verb has broadened its semantic area considerably, to the extent of also being
used in composite tenses as a marker of future tense. The Swedish verb gå still
retains its strong emphasis on manner (which in the English verb go is very weak). The
English go also carries the connotation of intent, which the Swedish verb gå does not.
Whether it is the strong, connotative emphasis on intent in go which focuses attention
on the action AoI in English, or whether it is the week, but existing, emphasis on
manner, is at this moment, uncertain.
For Swedish, the results are somewhat more unexpected. Gå is a motion verb of
manner, which together with a strongly directive particle/adverb ought to channel the
attention towards the goal, but which has not been found to do so here. On the
contrary, the action, by far, receives the most attention. There is however, often less of
a difference in the fixation percentages when dit it is used as a spatial denominator
than when där is used, which might still point to some support for the research
hypothesis. But that the action AoI consistently receives more attention when a
directional manner- motion verb is used in conjunction with a strong, spatial- directive
particle/adverb (dit) is more unexpected.
Adapted to screen communication this implies that the semantic content of a message
has to be taken into special consideration when the intent is to guide a viewer’s
attention towards a special section of a visual stimulus. For verbs expressing a change
of state, or path, the languages react similarly, but for verbs of manner or action
special care has to be taken to assure that any goal indication is clearly marked
through a special expression.
As was found in the previous translation studies of English and Swedish concerning
the words hit and dit and their translations, around 30% of the cases were not
translated by or from here and there, but either ‘circumstranslated’ (rewritten mostly
23
24
Johansson et al. 2006
Strömqvist 2006, in press
Ingrid Nilsson
BTH & Lund University
Sidan 14
Last printed 4/19/2006 8:27 AM
with the help of a prepositional phrase) or left untranslated. The ~30% of rewritings
found in connection with these expressions now has a better explanation. While it is
enough in Swedish to add either one of these expressions (hit or dit) to clarify the goal
aspect of the verb-meaning of a manner-motion verb, there cannot fill this function
since it can only support the goal aspect of the verb, not supply it. A different means of
expressing the goal function has to be found and prepositional phrases often take its
place.
Areas of future interest
General research questions connected to this area of study
1. What are the effects on the gaze patterns of the viewer/listener/translator of the
different spatial expressions used?
2. What lay-out version of text and/or picture avoids evoking a non-desirable
focus effect in the viewer/listener? (Such as going continuously back and forth
between the two areas of interest specified, but not fixating on either one of
them; or being distracted by either the text or the picture, and ignoring the other
communicative channel.)
3. Does it seem more important in Swedish or in English, based on eyemovement patterns, to include the goal for an action performed?
4. Is there a difference in focus between English and Swedish? Does English
generally focus on the process, whereas Swedish does not?
5. How can this difference, if any such is found, be utilized in attention-directing
situations such as picture viewing or auditory illustrations?
~~~~~~~~~~~~~~~~~
Ingrid Nilsson
BTH & Lund University
Sidan 15
Last printed 4/19/2006 8:27 AM
References:
Catford, J.C. 1965. A Linguistic Theory of Translation. London: Oxford University
Press.
Comrie, Bernard. 1981. Language universals and linguistic typology. Chicago:
University of Chicago Press.
Cruse, D.A. 1986. Lexical Semantics. (Cambridge textbooks in linguistics). Cambridge:
Cambridge University Press.
Dren, Andreas. 2005. ”Annonser i nättidningar: Kvalitet och placering i målorienterad
sökning”. Unpublished student paper:
http://www.sol.lu.se/humlab/eyetracking/Studentpapers/AndreasDren.pdf
Frawley, William. 1992. Linguistic Semantics. Hillsdale, NJ: Lawrence Erlbaum
Associates, Inc.
Holmberg, N. and Holmqvist, K. (2005) “Tidningsläsares visuella beteende eyetracking på Norrköpings tidningar”.
http://www.sol.lu.se/humlab/eyetracking/Projectpapers/EtNT2Rapport.pdf
Holmqvist, K. and Wartenberg, C. (2004) “The role of local design factors for
newspaper reading behaviour - an eye-tracking perspective”, submitted.
http://www.sol.lu.se/humlab/eyetracking/Projectpapers/SNDSa.pdf
Holmqvist, K., Johansson, V., Strömqvist, S., Wengelin, Å. 2002. “Analysing reading
and writing online”. In S. Strömqvist (editor) The Diversity of Languages and
Language Learning. Lund: Lund University Press.
Hyman, Malcolm D. 2003. “Greek and Roman Grammarians on Motion Verbs and
Place Adverbials.” Web article from Harvard University,
http://archimedes.fas.harvard.edu/mdh/motion.pdf
Ingo, Rune. 1991. Från källspråk till målspråk. Introduktion i översättningsvetenskap.
Lund: Studentlitteratur. ISBN 91-44-33441-9.
Jakobson, Roman. 1968. Child Language, Aphasia and Linguistic Universals. The
Hague: Mouton.
Jakobson, Roman. 1959. ‘On linguistic aspects of translation’. In Andrew Chesterman
(ed.), Readings in translation theory, 53-60. Oy Finn Lectura. 1989
Jespersen, Otto. 1964. Essentials of English grammar. Forge Village: University of
Alabama Press.¨
Johansson, Roger, Jana Holsanova and Kenneth Holmqvist. 2006. ”Pictorial and
spoken elicitation yield the same imagery eye movements, both in daylight and
in complete darkness.” Department of Cognitive Science, Lund University.
Ingrid Nilsson
BTH & Lund University
Sidan 16
Last printed 4/19/2006 8:27 AM
Kleberg, Lars, red. 1998. Med andra ord. Texter om litterär översättning. Stockholm :
Natur och kultur.
Lyons, John. 1977. Semantics I. Cambridge: Cambridge University Press.
Mesterton, Erik. 1979. Om möjligheten och omöjligheten att översätta. I: Kleberg, Lars,
red. 1998. Med andra ord. Texter om litterär översättning. Sid. 172-188.
Nilsson, Ingrid V. 2003. “Engelska översättnignsmotsvarigheter till de svenska orden
hit/dit.” I: Andréasson, Maia och Susanna Karlsson (red). langue.doc.
Göteborg University Open Archive.
<https://guoa.ub.gu.se/dspace/bitstream/2077/22/6/Nilsson_I_2003.pdf>.
Nilsson, Ingrid V. 2005. “Hit and dit in translations between Swedish and English.” In:
Lund University, Dept. of Linguistics, Working Papers 51, 143–155.
Nilsson, Ingrid V. 2006?. “Translative Strategies in the ESPCorpus and in a pilot test:
Hit & Dit = Here & There?” In: Proceedings from NAES2004, Århus, Denmark,
May 2004. (To be published.)
Pagendarm,Magnus and Heike Schaumburg. 2001. “Why are Users Banner‐blind?
The Impact of Navigation Style on the Perception of Web Banners.” Journal of
Digital Information, Volume 2 Issue 1, Article No. 47, 2001-08-24
http://jodi.ecs.soton.ac.uk/Articles/v02/i01/Pagendarm/
Svartvik, Jan & Olof Sager. 1996. Engelsk universitetsgrammatik. 2nd ed. Stockholm:
Almqvist & Wiksell.
Slobin, Dan. 2004. The Many Ways to Search for a Frog. Linguistic Typology and the
Expression of Motion Events. In: Strömqvist, Sven & Ludo Verhoeven, eds.
2004. Relating Events in Narrative.
Sobkowiak, Włodzimierz. 1997. Logic of markedness arguments. Paper presented at
the 30th Poznań Linguistic Meeting in May 1997.
Strömqvist, Sven & Ludo Verhoeven, eds. 2004. Relating Events in Narrative:
Typological and Contextual Perspectives. Vols.I and II. Mahwath, NJ:
Lawrence Erlbaum Associates Publishers. ISBN 0-8058-5071-6, and 0-80584672-7.
Strömqvist, Sven, Kenneth Holmqvist, & Richard Andersson. 2006, to be published.
Thinking for speaking and channeling of attention – a case for eye-tracking
research? In: J. Guo et al. (Eds.) Festschrift till Dan Slobin.
Talmy, Leonard. 2000a. Toward a Cognitive Semantics, Vol.I. Cambridge, MA: The
MIT Press. ISBN 0-262-20120-8.
Talmy, Leonard. 2000b. Toward a Cognitive Semantics, Vol.II. Cambridge, MA: The
MIT Press. ISBN 0-262-20120-8.
Ingrid Nilsson
BTH & Lund University
Sidan 17
Last printed 4/19/2006 8:27 AM
Zlatev, Jordan & Caroline David. 2004a. Motion Event Construction in Swedish,
French and Thai: Three Different Language Types? In: Manusya. Journal of
Humanities.
Zlatev, Jordan & Peerapat Yangklang. 2004b. A Third Way to Travel. The Place of
Thai in Motion-Event Typology. In: Strömqvist, Sven & Ludo Verhoeven, eds.
2004. Relating Events in Narrative. Pp.159-190.
Attachments
Excel sheets demonstrating:
1. Individual fixation percentages per test subject – arrive & anlända
2. Individual fixation percentages per test subject – go & gå
3. total and average percentages of fixations for all conditions