A Comparison of Visual and Haptic Object Representations Based on Similarity
Theresa Cooke, Christian Wallraven, Heinrich H. Bülthoff
Max Planck Institute for Biological Cybernetics, Tübingen, Germany
{firstname.lastname}@tuebingen.mpg.de
Abstract
Do we judge similarity between two objects to be the
same using touch and vision? We investigated this using
psychophysical experiments in which subjects rated the
similarity between objects presented either visually or
haptically. The stimuli were a family of novel, threedimensional objects whose microgeometry (“texture”) and
macrogeometry (“shape”) were parametrically varied.
Multidimensional scaling of the similarity data was used to
reconstruct haptic and visual perceptual spaces. For both
modalities, a two-dimensional perceptual space was found
whose dimensions clearly corresponded to shape and
texture. Interestingly, shape dominated in visual space,
whereas both shape and texture were important in haptic
space. Furthermore, stimuli clusters were observed in this
space, suggesting the emergence of category structure
based on similarity relationships. The same category
boundaries were confirmed in a visual free sorting
experiment. This study reveals differences in object
processing across modality and demonstrates an approach
for analyzing such differences in multisensory
visualizations.
Keywords--- similarity, categorization, shape, texture,
touch/haptic, vision, MDS, psychophysics
1. Introduction
Does touching an object give rise to a different percept
than seeing it? Although the visual and haptic sensory
systems are able to extract many of the same properties of
an object, there are fundamental differences in the way
these features are extracted by the two modalities. Touch
and vision operate on different input dimensionalities:
touch operates on input from tactile receptors in threedimensional (3D) space, while vision operates on twodimensional (2D) retinal input. The two modalities have
different view preferences: the haptic system has a bias
towards encoding the back of objects, whereas the visual
system prefers the front view of objects [1]. The haptic
sensory system operates in much closer proximity to the
motor system than the visual sensory system does, both in
the central and in the peripheral nervous system [2].
Several of the differences between the two modalities
are related to spatial scale and sampling properties.
Visual perception has a large spatial extent and functions
simultaneously at several scales, while haptic perception is
limited to near-body space and is markedly affected by the
scale at which a feature evolves (e.g., in curvature
perception [3]). The haptic system is capable of directly
extracting features at the level of the most peripheral
receptors (e.g., for pressure or temperature), whereas visual
object feature extraction requires processing at higher
levels beyond the retina [3]. Moreover, object shape and
other global object features can be sampled in parallel by
the visual system, while they generally have to be sampled
in serial by the haptic system due to its more limited
effective field of view [4].
Although both modalities are capable of extracting
geometric object properties and their spatial relationships,
it is not surprising given the above differences that largescale or “global” spatial processing should be more
accessible to the visual system, whereas properties of an
object which can be “locally” extracted (i.e., by a single
finger or small group of receptors) should be more easily
accessible to the haptic system [3]. A fundamental
question is how such preferences for scales and features
affect object representation and behaviour. Not only is this
question important for elucidating the basis of high-level
cognitive competences (such as similarity judgments and
categorization), but it is also critical for understanding how
the brain integrates different modalities for the purpose of
recognition.
In this study, we chose to investigate this question by
having subjects make similarity judgments between pairs
of objects within an object family which varied both on a
local scale (easily accessible to the haptic system) and on a
global scale (easily accessible to the visual system). The
objects were presented visually in a first experiment and
haptically in a second experiment. Similarity ratings and
perceptual maps derived from these ratings using MDS
(multidimensional scaling) were then compared across
modality.
Relationships between similarity and
categorization were explored by comparing similarity maps
with category boundaries determined in a visual free
sorting task. These findings shed light on how local and
global variations in object properties affect object
descriptions, similarity relationships, and categorization,
and how these effects vary as a function of perceptual
modality. The methodology of our study also offers a new
approach for comparing different visualizations and
identifying important perceptual dimensions for effective
multisensory visualization.
2. Methods
In this section, we describe the stimuli used in the
experiments, as well as the experimental design for the
visual and haptic similarity ratings and free sorting
experiments.
2.1. Stimuli
The stimuli consist of a family of novel, 3D objects
(Figure 1), created in the graphics package 3D Studio Max
6.0. This package provides full control of object properties
such as size, shape, and texture, allowing them to be varied
in defined steps. The family begins with a “base object”
(see Figure 1, object #5), which consists of three parts
connected to a centre sphere and a texture map which is
applied to the 3D mesh. The other family members are
generated using two manipulations. The first manipulation
alters the object’s microgeometry (or “texture”) by
changing the amount of mesh displacement which the
texture map is allowed to cause. The second manipulation
alters the object’s macrogeometry (or “shape”) by moving
mesh vertices towards a local average, essentially
removing sharp angles in the global shape. Objects created
using these variations can be plotted in a 2D creation
parameter space whose dimension correspond to
“microgeometry” and “macrogeometry”.
The objects were then printed in 3D, layer-by-layer, by
depositing filaments of heated plastic (Dimension 3D
Printer, Stratasys, Minneapolis, USA). The printed objects
are hard, white, and opaque, measuring 9.0 ± 0.1 cm wide,
8.3 ± 0.2 cm high, and 3.7 ± 0.1 cm deep and weighing
approximately 40 g each.
2.2. Visual similarity ratings and free sorting
Ten subjects with normal or corrected-to-normal
vision were paid 8 EUR per hour to rate the similarities
between photographs of the objects presented at 75 Hz on a
Sony Trinitron 21” monitor with a resolution of 1024 x 768
pixels. None of the subjects had touched or seen the
objects before. Photographs of the objects were displayed
Figure 1. Stimuli varied parametrically
in terms of microgeometry (texture)
and microgeometry (shape).
using the Psychtoolbox extension for MATLAB [5, 6] on a
Macintosh G4 computer. The image size was 7.6 x 7.6
degrees of visual angle (as if the object was held at arm’s
length). Subjects were seated 60cm from the monitor in a
dimly-lit room. A fixation cross appeared for 500ms and
then each of the objects appeared for 500ms, separated by a
500ms interstimulus interval. At the end of each trial,
subjects rated the similarity of the objects on a scale
between 1 (low similarity) and 7 (high similarity).
Response time was unlimited. There were six experimental
blocks of 325 randomized trials (each object was compared
once with itself and once with every other object, i.e., 25 +
(25*24)/2 = 325). The total experiment lasted about two
hours. At the end of the experiment, subjects filled out a
debriefing questionnaire which asked them to describe the
objects’ appearance, to explain how they had made their
similarity judgments, and to explain how they would
categorize the objects. In addition, subjects were given a
randomized set of printouts of the images they had just
seen (3 cm x 3 cm) and were asked to sort them into any
number of categories (free sorting task).
2.3. Haptic similarity ratings
Ten right-handed subjects were paid 8 EUR per hour
to rate the similarities between the objects after exploring
them haptically. None of the subjects had touched or seen
the objects before. Subjects sat in front of a table, facing an
opaque curtain. Behind the curtain, the experimenter
presented two objects, one after the other. The objects were
always presented in the same fixed position, face up on the
table. Subjects were given up to 10 seconds to trace the
contour of each object with their right hand. The contour-
Figure 2. Mean similarity ratings for objects explored visually (left) or haptically (right). The matrix is
symmetric since ratings were averaged over all trials in which a given pair was shown, regardless of
presentation order. Mean standard deviation was 1.3 for visual similarity ratings and 1.2 for haptic
similarity ratings.
consisted of three blocks of 325 randomized trials spread
out over five two-hour sessions on consecutive days. At
the end of the experiment, subjects filled out a debriefing
questionnaire, asking them to describe how the objects felt,
to explain how they had made their similarity judgments,
and to explain how they would categorize the objects.
following procedure was chosen because it has been shown
to allow for haptic extraction of a wide range of object
properties, including both local texture and global shape
[7]. Unimanual exploration was chosen over bimanual
exploration for simplicity in this intial set of experiments.
In the ten seconds provided, even untrained subjects had
sufficient time to trace the object’s contour twice. Subjects
rated the similarity between the objects on a seven point
scale from 1 (low similarity) to 7 (high similarity). They
could respond at any time after the second object was
presented. They occasionally asked to repeat the trial and
this was allowed. Subjects were instructed to keep their
eyes closed during the experiment. The full experiment
0.4
visual
haptic
0.35
3. Results and discussion
3.1. Visual similarity ratings and free sorting
Mean visual similarity ratings for the twenty-five
objects are shown in Figure 2. The most striking patterns
Shape dominant
Texture dominant
0.3
Stress
0.25
0.2
stress threshold
haptic subjects
visual subjects
0.15
0.1
-0.5
0.05
0
1
2
3
4
Number of dimensions
5
0
Shape/Texture Tradeoff
0.5
(b)
(a)
Figure 3. MDS stress plot (using Young’s stress formula 1) (a) and shape/texture tradeoff values for
individual subjects (b).
and analyzed using the ALSCAL algorithm in SPSS
version 12.0.1 [8], while individual subject data was
analyzed using the INDSCAL algorithm [9], which has the
advantage of providing individual and mean weights for
the output dimensions.
The stress plot for the MDS analysis of mean visual
similarity data is shown in Figure 3. Stress values below
0.2 are generally accepted as an indication that the
dimensionality of the output space is sufficient to faithfully
represent the input distance information [10]. Thus the
stress values obtained here show that one perceptual
dimension is sufficient to explain the similarity data (stress
= 0.05 for one dimension).
Plotting the two-dimensional stimulus configuration
(Figure 4) confirmed that this first dimension corresponded
to shape variation, while the second dimension
corresponded to texture variation. Thus shape was the
dominant perceptual dimension for visual similarity
judgments, while texture played only a minor role. Note
that the scaling and orientation of the map is not
determined by MDS; in all cases, we fitted the maps
obtained from MDS to a uniform grid with five shape and
five texture levels, which we refer to as an “ordinal map.”
When the perceptual map is overlaid on the ordinal map,
the dominance of the shape dimension in the visual map is
particularly striking.
The dominant role of shape in visual judgments can
also be seen from the relative shape/texture tradeoff values
for individual subjects (Figure 3). These tradeoff values
were derived from the individual dimension weights
provided by INDSCAL. Almost all subjects in the visual
in the matrix are the large box patterns, which arise due to
sharp changes in similarity ratings. A closer look reveals
that this sharp change is directly related to shape changes
in the stimuli. Stimulus 1 is perceptually very similar to 6
and 11 (mean ratings of 6.5 and 6.0, respectively), but
suddenly much less similar to stimuli 16 and 21 (mean
ratings of 3.2 and 2.8, respectively). Note that this pattern
holds regardless of texture level. Texture-induced effects
on similarity ratings are most visible in regions where the
mean similarity is high, e.g., stimulus 1 is decreasingly
similar to stimuli 2, 3, and 4, with mean ratings of 5.4, 5.1,
4.7, 4.2, respectively, or a total change of 1.2. In contrast,
texture effects are muted in regions where the similarity is
low due to large shape differences, e.g., there is little
change in similarity between stimulus 1 versus 22, 23, 24,
or 25, with mean ratings of 2.5, 2.4, 2.2, 2.2, respectively,
or a total difference of 0.3. This pattern indicates that
texture was used to differentiate stimuli when they closely
resembled one another, but was used less when objects
were perceived to be very different from one another.
To see these effects more clearly, the similarity ratings
were analyzed using multidimensional scaling (MDS).
MDS takes a matrix of pair-wise distances as input and
returns a stress plot from which the number of dimensions
needed to represent the objects can be determined (similar
to principle components), together with the coordinates of
each object in the output space. MDS does not, however,
provide an interpretation of the axes labels: these must be
interpreted based on the output map. Similarity data was
analyzed using two versions of MDS: mean similarity data
across subjects were transformed to Euclidean distances
visual map
ordinal map
5
4
3
1112 1415
6
13
1 7 28 410
3 9 5
2
1
21 17
4
2
3
4
microgeometry
5
24
25
20
19
16
12
3
14
13
15
11
2
6 7
2
1
1
1
23
18
22
macrogeometry
macrogeometry
5
haptic map
ordinal map
22232425
21
16 18
17
20
19
1
8
3
2
3
4
microgeometry
9
4
10
5
5
Figure 4. Perceptual maps produced by ALSCAL MDS using visual similarity ratings (left)
and haptic similarity ratings (right).
similarity experiment have a shape/texture tradeoff which
is strongly biased towards shape.
Two shape-based category groupings were evident in
the visual map (the three bottom shape groups vs. the two
top shape groups), suggesting the emergence of natural
categories and an intriguing relationship between similarity
judgments and categorization. Although these groupings
could already be seen in the similarity matrix, applying
MDS and plotting the stimuli in perceptual space makes
the grouping much easier to visualize.
In order to test whether these groupings could indeed
be considered “natural categories”, we compared the
similarity map against the category boundaries created by
subjects in the free sorting task (Figure 5). The category
boundaries shown were calculated as follows: for each
adjacency in the stimulus map (e.g., the link between
stimulus 13 and 18), we counted the number of subjects
who identified it as a category boundary; then, for each of
the 5 shape and 5 texture levels, we calculated the average
number of subjects who had identified adjacencies along
that line as belonging to their category boundaries; finally,
shape and texture levels for which this average exceeded
30% were plotted as category boundaries.1 Interestingly,
category boundaries fell between clusters of stimuli in the
space derived from similarity ratings, suggesting that
similarity relationships could be used to predict category
structure (see General Discussion).
The results of the post-experiment debriefing
Figure 5. Category boundaries chosen
by visual free sorting, overlaid on visual
stimulus map. Percentages refer to
average proportion of subjects who
selected this boundary.
questionnaire are shown in Table 1. For each of the
questions we asked (to describe the object’s
appearance/how it felt, to describe how similarity
judgments were made, and to describe object categories),
we calculated the percentage of subjects who: 1)
mentioned either the word shape explicitly or shape-related
words (e.g., “round”, “leg”, “sphere”); 2) mentioned the
word “texture” or texture-related words (e.g., “bumpiness”,
“surface structure”); 3) mentioned other object properties
(e.g., colour, weight, size, material). Both texture and
shape were mentioned by a majority of subjects in the
visual similarity experiment when describing the objects,
explaining how they judged similarity, and describing
object categories. Other properties were not mentioned at
all for similarity and categorization, though other
properties were mentioned for object descriptions. Shape
was mentioned more often than texture when explaining
similarity judgments, a result which correlates well with
the dominance of shape found in the MDS analysis of
similarity ratings. Interestingly, texture was mentioned
almost as often as shape in category descriptions; this
finding agrees with the existence of both shape-based and
texture-based category boundaries revealed by the free
sorting experiment.
3.2. Haptic similarity ratings
Mean haptic similarity ratings for the twenty-five
objects are shown in Figure 2. Clear effects of both texture
and shape variation in the stimuli can be seen. The pattern
of fading diagonals in the matrix is related to shape
variation (e.g., the shape of stimulus 1 is decreasingly
similar to the shapes of stimuli 6, 11, 16, and 21, with
mean similarity ratings of 6.4, 5.5, 4.6, and 3.8,
respectively) whereas the fading box-like pattern is related
to the texture variation (e.g., the texture of stimulus 1 is
decreasingly similar to the textures of stimuli 2, 3, 4, and 5,
with mean ratings of 6.0, 4.8, 3.8 and 3.5, respectively).
The stress plot obtained by running MDS on mean
haptic similarity ratings is shown in Figure 3. Stress drops
sharply below the threshold of 0.2 only once the stimuli are
embedded in a two-dimensional space. Plotting the twodimensional output configuration (Figure 4) enabled us to
interpret these perceptual dimensions as texture (as the first
output dimension) and shape (as the second output
dimension). Thus texture alone was insufficient to explain
the similarity data; instead both texture and shape were
important perceptual dimensions for haptic similarity
judgments. This finding also serves as a demonstration of
subjects’ non-trivial ability to extract these two features
from a high-dimensional haptic measurement space.
1
Although this procedure necessarily results in uni-dimensional
categorization rules, most of the subjects’ category boundaries could be
approximated by uni-dimensional rules up to one or two adjacencies.
Table 1. Responses to debriefing questionnaire (% subjects mentioning a given object property)
Answers after visual similarity ratings
Object
Similarity
Category
description
judgment
description
Answers after haptic similarity ratings
Object
Similarity
Category
description
judgment
description
Mention shape or shaperelated property
90%
90%
90%
70%
100%
100%
Mention texture or texturerelated property
80%
60%
80%
100%
100%
100%
Mention other object
properties (e.g., colour)
40%
0%
0%
30%
10%
10%
The importance of both shape and texture for haptic
judgments can also be seen from the relative shape/texture
tradeoff values for individual subjects, shown in Figure 3.
Although texture was clearly the dominant dimension for
some subjects, the mean tradeoff value for haptic
judgments is close to zero, meaning that both shape and
texture were, on average, important dimensions.
The perceptual map obtained by plotting the stimulus
coordinates in the two-dimensional haptic MDS space is
shown in Figure 4.
The haptic perceptual map is
remarkably regular and bears a strong resemblance to the
ordinal map: both the ordinal configuration of the stimuli is
preserved and the relative dimension weights are the same
(i.e., shape and texture are weighted equally in both cases).
One notable difference is the irregular spacing between
shape and texture levels. This spacing suggests two
texture-based groupings (three leftmost rows and two
rightmost rows) and two shape-based groupings (bottom
three rows and top two rows).
The results of the post-experiment debriefing
questionnaire are shown in Table 1. Both shape and texture
were mentioned by all subjects when asked about
similarity judgments and object categories, while texture
was mentioned slightly more often than shape for object
descriptions. This equal proportion of references to shape
and texture in verbal reports agrees with the mean tradeoff
value derived from the individual subject weights,
confirming that shape and texture both played important
roles in haptic similarity judgments. The fact that other
object properties were seldom mentioned correlates with
the sharp drop in MDS stress for a two-dimensional
solution and indicates that shape and texture were both
sufficient and necessary perceptual dimensions in subjects’
similarity haptic judgments.
4. General discussion
4.1. Shape and texture as critical perceptual
dimensions
In both visual and haptic modalities, subjects were
able to extract the two kinds of parametric variation which
were used to create the stimuli. This is a non-trivial ability
given the high-dimensionality of the visual and haptic
measurement spaces. These two stimulus variations, which
we initially referred to as changes in “macrogeometry” and
“microgeometry” were perceived by the subjects as
changes in “shape” and “texture”.
MDS analysis showed, and verbal report confirmed,
that shape was a necessary and sufficient perceptual
dimension for representing similarity relationships between
the stimuli when they were presented visually. In contrast,
both shape and texture constituted the necessary and
sufficient perceptual dimensions when stimuli were
presented haptically.
In their verbal reports, subjects also made reference to
shape and texture when describing these objects and
categorizing them. Although subjects mentioned other
object properties when describing the objects, they rarely
mentioned any other property when describing similarities
or categories. Thus shape and texture may already have
become so-called diagnostic dimensions for the subjects,
i.e., features which take on perceptual importance due to
experience and task demands [11, 12]. Although one could
argue that diagnosticity was induced by the categorization
task in the visual experiment, there was no such task
involved in the haptic experiment. This raises the
possibility that stimulus dimensions take on category
diagnosticity even in the absence of an explicit
categorization task and that performing a similarity
judgment task may be sufficient to invoke mechanisms of
category diagnosticity.
4.2. Differences in critical perceptual dimensions
for vision and touch
In visual similarity judgments, shape was both a
sufficient and necessary perceptual dimension; texture was
not needed to account for the similarity data. This finding
agrees with the idea advanced by Edelman that shape plays
a crucial role in determining similarity relationships
between objects [17]. This finding is also consistent with
the notion that the extraction of global form is one of the
visual system’s areas of expertise [3]. In a recent
computational study [13], however, we found good
correlation between 1) the similarity matrix derived from
simple pixel-wise differences between images of the
objects and 2) the similarity matrix measured in the present
study, including the dominance of shape over texture. This
result raises the possibility that texture played a lesser role
in similarity judgments because texture-related image
differences occur on a smaller spatial scale and/or have
lower local contrast than shape-related image differences.
Another explanation is that texture changes require more
time to build up perceptual weight in the visual modality
than shape changes do; systematic manipulation of
stimulus presentation times would be required in order to
test this hypothesis.
In haptic similarity judgments, both shape and texture
were important perceptual dimensions. Given that local
material properties are known to be more easily accessible
to the haptic system than global geometric properties [3], it
is not surprising that texture played a more important role
in the haptic similarity judgments than in visual judgments.
However, the finding that shape was an equally important
perceptual dimension for the haptic task was somewhat
surprising. For example, one study had subjects perform
haptic free sorting of 3D objects based on their similarity
and found that material properties such as texture were
more salient than shape [16]. In addition, the fact that
exploration was unimanual should also have biased the
results towards texture, due to the greater difficulty of
integrating shape information relative to bimanual
exploration [18].
Two task parameters may explain the importance of
shape in this experiment: first, subjects may have been
biased towards extracting shape by the contour-following
procedure. Klatzky & Lederman [7] rated the relative
ability of this procedure to extract “exact shape” with a
score of 3, “global shape” with a score of 1, and “texture”
with a score of 1. Had we chosen another procedure, such
as lateral motion (scores of 0 for both shape properties and
2 for texture), texture may have played a more dominant
role in haptic similarity judgments. However, another study
involving the same task found that contour-following was
not associated with a differential emphasis on local versus
global features [14]; the authors suggested that this was
instead because objects were explored for a long period of
time and that global shape properties tend to become more
salient over time.
Exploration time may also have played a role in our
finding that shape was as important as texture for haptic
similarity judgments. The study mentioned above
systematically manipulated exploration time and found that
“local shape” had the same effect as “global shape” on
haptic similarity judgments for a 1s exploration time, but
that the effect of local shape differences decreased as
exploration time was increased up to 16s, up to about a
15% difference in ratings (amongst objects differing in
local shape vs. amongst objects differing in global shape)
[14]. Thus, the 10s exploration time in our haptic
experiment may have resulted in less importance being
accorded to “local shape”, which one might interpret as
texture/microgeometry in our case. However, our subjects
only used the full 10s for the first 2-4 hours of the
experiment; after this point, they reduced their exploration
time, sometimes by up to 50%. Thus, given that
exploration time in our experiment effectively ranged
between 5 and 10s and that the size of the effect reported in
[14] was 5-10% for exploration times of 5-10s, it seems
unlikely that exploration time alone could explain the
lower-than-expected texture weight.
A final possibility is that haptic feature extraction may
have been biased toward shape by the similarity judgment
task itself. If indeed shape plays a critical role in
determining object similarity, the task may have triggered
an additional amount of haptic spatial processing, which
could in fact be carried out within the exploration time
available to subjects. The task effect could also have been
augmented by the fact that the stimuli were novel 3D
objects with relatively complex shape, which may trigger
more spatial processing than familiar objects or objects
with a simpler geometrical structure. Further studies are
needed to disentangle the effects of task, stimulus
complexity, and stimulus familiarity on modality-specific
feature weightings.
4.3. From similarity to categorization
Stimulus clusters in the perceptual map derived from
visual similarity ratings clearly suggested the emergence of
natural categories. Furthermore, these clusters correlated
with the category boundaries chosen by subjects in the free
sorting task (Figure 5). In addition, subjects reported that
the same features (shape and texture) were important both
for making similarity judgments and for assigning category
membership. These results provide a striking example of
the close relationship between similarity and
categorization, a topic of great interest to categorization
researchers (e.g., [15]). It may even be possible to actually
predict natural category boundaries from the clusters in
similarity space, a task we are currently undertaking.
Category boundaries could also be inferred from the
haptic similarity-based maps, though they are not as
obvious as in the visual case. One possibility is that
categorization of such stimuli is a more automatic process
for the visual system, e.g., if categorization is indeed
tightly coupled to spatial processing of global form (which
the visual system can accomplish much more effectively
than the haptic system). A second possibility is that the
similarity ratings scale, which was limited to 7 numbers,
simply does not provide enough information capacity or
resolution to store variance along two perceptual
dimensions and, in addition, variance due to category
boundaries. Further studies are required in order to
elucidate how category structure develops in the haptic
modality.
[4]
4.4. Summary and outlook
[5]
We have shown that the same 3D objects are
represented differently in the human brain depending on
the sensory modality used to perceive them (vision or
touch).
The main discrepancy was the differential
weighting of perceptual dimensions: while shape was the
sole critical perceptual feature for vision, both shape and
texture were critical features for touch. Intrinsic biases in
feature extraction and differences in the task or in the types
of processing triggered by the task may explain the
differences in representations. In future work, we will
investigate the specific role of biases in feature extraction
by computationally extracting 2D and 3D features and
exploring how such features can be combined to simulate
human results.
We found that the features which were critical for
similarity-based representations were also diagnostic for
category membership and demonstrated that similaritybased representations correlated with categorization
behaviour in the visual domain. Future studies will
examine whether categories can indeed be predicted by
similarities and address how category structures differ in
vision and touch.
Finally, the methodology presented here demonstrates
a unique approach for comparing different kinds of shape
rendering. Similarity judgments are a powerful tool to
create perceptual maps of the visualizations and determine
the most relevant perceptual dimensions. Our approach can
help to explain why one visualization may be more
effective than another in a given modality, thus providing
design guidelines for effective multisensory visualizations.
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
Acknowledgements
[16]
We thank Quoc Vuong and Martin Breidt for help
with stimulus design, and anonymous reviewers for their
comments.
[17]
References
[18]
[1]
[2]
[3]
Newell, FN and Ernst, MO and Tjan, BS and Buelthoff,
HH (2001). Viewpoint dependence in visual and haptic
object recognition, Psychol Sci, 2001, 12 , 37-42
Kandel, ER, Schwartz JH, and Jessell, TM (Eds.) (2000).
Principles of Neural Science, 4th edition, New York:
McGraw-Hill Professional Publishing
Klatzky, RL and Lederman, SJ (2003). Touch. In A.F.
Healy & R. W. Proctor (Eds.), Experimental Psychology
(pp. 147-176). Volume 4 in I.B. Weiner (Editor-in-Chief).
Handbook of Psychology, New York: John Wiley & Sons
Loomis, JM, Klatzky, RL, and Lederman, SJ (1991).
Similarity of tactual and visual picture recognition with
limited field of view, Perception, 20, 167-177
Brainard, DH (1997). The Psychophysics Toolbox, Spatial
Vision, 10, 433-436
Pelli, DG (1997). The VideoToolbox software for visual
psychophysics: Transforming numbers into movies, Spatial
Vision, 10, 437-442
Lederman,
SJ
and
Klatzky,
RL
(1993).
Extracting object properties through haptic exploration
Acta Psychol, 84, 29-40
Young FW, Harris, DF (1990). Multidimensional scaling:
procedure ALSCAL. In: Norusis, M (ed.) SPSS Base
System: User’s Guide. Chicago: SPSS, USA, 397–461
Carroll, JD and Chang, JJ (1970). Analysis of individual
differences in multidimensional scaling via an n-way
generalization
of
"Eckart-Young"
decomposition.
Psychometrika, 35, 283-319
Clarke, KR and Warwick, RM (2001). Change in Marine
Communities: An Approach to Statistical Analysis and
Interpretation, 2nd edition, PRIMER-E, Plymouth, UK,
172pp.
Sigala N, Gabbiani, F and Logothetis, NK (2002).
Visual categorization and object representation in monkeys
and humans, J Cogn Neurosci, 14 , 187-198
Schyns, PG & Rodet L (1997). Categorization creates
functional features. Journal of Experimental Psychology:
Learning, Memory and Cognition, 23, 681-696
Cooke T, Wallraven, C, and Buelthoff, HH (2005).
Representation of shape and texture changes in novel, 3D
objects. Proceedings of the 8th Tübingen Perception
Conference, (Eds.) Heinrich H. Bülthoff, Hanspeter A.
Mallot, Rolf D. Ulrich, Felix A. Wichmann, Knirsch
Verlag Kirchentellinsfurt.
Lakatos, S and Marks LE (1999). Haptic form perception:
Relative salience of local and global features, Perception &
Psychophysics, 61, 5, 895-908
Hahn, U and Ramscar, M. (Eds.) (2001). Similarity and
Categorization, Oxford University Press.
Klatzky RL, Lederman S, and Reed, C (1987). There’s
more to touch than meets the eye: the salience of object
attributes for haptics with and without vision, Journal of
Experimental Psychology: General, 116, 4, 356-369.
Edelman, S (1999). Representation and recognition in
vision, MIT Press.
Ballesteros, S and Reales, JM (2004). Visual and haptic
discrimination of symmetry in unfamiliar displays extended
in the z-axis, Perception, 33, 315-327.