Chapter 4
Chapter 4.
Conscious contents and nonconceptual representation ........................................ 4-2
4.1
The role of conscious experience in the study of perception and cognition ................ 4-3
4.2
What subjective experience reveals about psychological processes ............................ 4-4
4.3
The special case of the experience of space................................................................. 4-5
4.3.1
The status of our phenomenal experience of space ............................................. 4-5
4.3.2
The argument for nonconceptual representation of space ................................... 4-6
4.4
The phenomenal experience of mental imagery .......................................................... 4-8
4-1
Chapter 4
Chapter 4. Conscious contents and nonconceptual representation
In the last chapter I raised the issue of nonconceptual representation as a form of
representation that does not involve concepts and therefore does not enter into beliefs and
thoughts about the perceived world. Nonconceptual representations have been widely discussed
in philosophical circles for a number of reasons. One reason is the basic problem that we have
encountered in several places earlier in this book: The need for a way to get information from its
distal causes through proximal effects (e.g., the retinal image) to perceptual beliefs, which are by
definition thoroughly conceptual. The interface is thought to involve a type of informationbearing state that is more complete and finely structured than beliefs, but which nonetheless
qualifies as being a representation. The primary evidence for such a form of representation is the
disparity between appearances and beliefs (which is the hallmark of a perceptual illusion) or
between the concrete, quantitative, panoramic, detailed and fine-grained nature of our conscious
experiences (particularly our visual experiences) and the relative coarseness and abstractness of
our thoughts, beliefs and recollections. When we look at an ambiguous figure, such as a Necker
cube, something changes over time as we watch, and that something is neither the physical
stimulus nor our beliefs about what we see. It is what we refer to as the appearance of the figure
or how we consciously experience it. Since the content of our experience seems to be something
distinct from what we believe about the figure and from the actual projection of the figure on our
retinas it would seem that we need a theoretical vehicle for that content, and such a vehicle
would be some sort of a representation. Similarly when we examine a scene, what we
experience is very different from, on the one hand, what we independently know to be the
information that enters the brain and, on the other hand, from what we think we could capture in
terms of the vocabulary of concepts we are likely to have. In the case of the former discrepancy
the data are very clear that the incoming information is highly incomplete and has a narrow
scope compared with how we experience it (this point will be discussed later in connection with
the special case of the experience of space, and is illustrated in Figure 4-1). The argument from
the richness of experience compared with the relative poverty of our conceptual resources
depends both on how we characterize experience and what we think are the conceptual resources
of the mind. But even without considering the fine points of what are reasonable bounds on our
conceptual apparatus, it seems clear that we are unlikely to have as many distinct concepts for,
say, colors, as there are colors that we can discriminate. Certainly if we consider the number of
color terms in a language we find that they are highly limited [about 11 monoleximic words, see
\Berlin, 1969 #1785]. Yet we can distinguish well over a million different colors [Halsey, 1951
#1784], so we are unlikely to encode each of these as a separate concept or code.1
Notwithstanding such plausible arguments for nonconceptual representation, there are
several questions that need to be considered and several tacit assumptions that need to be
exposed before the hypothesis that there is a nonconceptual representation that corresponds to
conscious contents is taken as established. The most contentious of these is the assumption that
conscious content corresponds to a level of representation in an information-processing or
functional view of the cognitive system. I take up this and other issues in the next few sections.
1
Of course there are many more concepts than there are words. But since each word corresponds to some concept, it
seems reasonable that each concept is at least a potential word in some culture. A million concepts is more than the
total human vocabulary so it seems unreasonable to suppose that every discriminable color could have a
corresponding concept.
4-2
Chapter 4
4.1
The role of conscious experience in the study of perception and cognition
Cognitive science, and particularly vision science, has had a deeply ambivalent relation
with conscious experience. On one hand, the way things appear or what they look like has
always constituted the primary data of the science. When one thing looks bigger in one
condition than in another or when something looks to be moving faster under one condition than
another, or when colors appear different under one lighting condition than another, these are
considered primary data to which theories are expected to respond. On the other hand, the
content of experience has also proven to be one of the most misleading sources of evidence
because it is not neutral with respect to theories. What explanations appear most natural is
determined in part by the way we describe our experiences and conversely, the way we describe
our experiences, even to ourselves, depends on what theories we hold. In fact, the way we
describe the experience often caries with it the implication that the experience itself explains the
data – that the experience of X is explained by alluding to the experience of Y (e.g., the reason it
takes longer to report details from a mental image that is experienced as being small is that the
details are harder to see). Phenomenological evidence has also tempted people to the view that
vision provides a dense manifold of panoramic information. It has suggested that mental images
consist of pictures examined by an inner eye in what Dan Dennett has called a Cartesian Theatre
[Dennett, 1991 #976].
It also has encouraged direct perception theories which claim that we pick up information
about aspects of the world that are prominent in our experience, such as whether the things we
see are suitable for certain purposes – from eating to sitting on (suitability is often referred to as
an “affordance”). It is true that we do not see patterns of light and shadow and patches of color,
we see familiar things such as tables and chairs and people. We also see people on television
screens and in movies. We never see just the front surface of objects, we see entire objects and
we see them as certain kinds of objects, such as our car or the bicycle we used to ride in our
youth. How big something appears or how bright or even what color it is depends on the context
how we see other aspects, such as how far away, whether in direct light and so on. These
adjustments, called constancies, are made prior to our experience, so what we experience is
sometimes thought of as a low-level sensory pattern, but it is in fact more often a high-level
reconstruction of the distal objects. The experience we have when we see objects often involves
emotional memories; we see them as parts of our past. Yet a theory that takes these sorts of
observations as the givens of perception, as the starting point of a theory (as in Gibson’s direct
realism theory), fails to take even the first step towards an explanatory theory of the mechanisms
involved in vision, a theory that might eventually make contact with neuroscience. Science
always needs a way to objectify the problem we want to explain, to make these phenomena
strange. One way to do this is to ask what it would take for a computer to demonstrate some of
the mundane perceptual skills that we exhibit. While this does not guarantee that we will avoid
circular or dead-end approaches, it is one of the ways that we can ensure a detachment from the
temptations of reifying phenomenal experience. But why do we need to be detached from
reifying phenomenal experience? The answer is not a priori, it is just that in the past such
experience has proven to be the most misleading source of evidence and inspiration.
As a source of evidence about how perception works, the appearance of the perceived
world is misleading because it reflects not only the operation of the perceptual system, but also
of our beliefs and expectations and pre-existing folk theories, and it incorporates these to a much
more profound degree than generally believed. The phenomenal experience of a taste reflects, in
part, what we think we are tasting, the experience of our actions reflect what we believe about
the agency of the actions, and so on. Our assumptions about the deliverance of our senses
depends greatly on the content of our subjective experiences and we have every reason to be
4-3
Chapter 4
skeptical of what this reveals about the information that goes on in the process.2 An even more
serious problem with the use of conscious contents as direct indicators of the content of the
outputs of perception is that there is also no room in phenomenology-based theories for the
growing evidence of vision-without-awareness, including change blindness, visuomotor control
without conscious awareness, blindsight, visual agnosia and disorders of visual-motor
coordination and other sources of behavioral and neuroscience data.
The concept of experience in philosophy and psychology as ambiguous. It is used both to
refer to the entire corpus of incoming information – as in “experience shapes our perception” –
and also to the subjective content of this experience – as in “I experience this patch as red or this
line as being shorter than that line”, and so on. But the two are very different. We know this
because in the past 50 years or more experimental and clinical studies have revealed that much of
what is causally efficacious in our sensory input is not available to conscious experience and
much of what is available to our conscious experience is not a cause of our behavior but a
product of our analysis or our theories of mental life.
4.2
What subjective experience reveals about psychological processes
Daniel Wegner etc
At different points in the history of perception research we have oscillated between the
two extremes of accepting phenomenology as the basis for a science of perception and writing it
off as unreliably or even as epiphenomenal. Although perceptual experience cannot be
discounted in the study of perception, neither can one assume that the experience itself is to be
taken at face value as an indicator of the nature of the functional mental states that play a role in
the explanation of how perception works. How are we to reconcile these differences (which
often coexist within individual researchers)?
There is no general solution to this problem. The question of how to interpret a particular
type of observation can only be resolved as we build more successful theories – in particular
when we have at least a sketch of a theory that scales up from individual laboratory experiments
to more general phenomena. The situation we are in is very similar to that which linguists have
been in during the last 60 years. Intuitions of grammatical structure led early linguistics astray
by focusing on surface phenomena. But as generative linguistics became better able to capture a
wide range of generalizations, it found itself relying more, rather than less, on linguistic
intuitions. What changed is that the use of the intuitions was now under the control the evolving
theories. Even such general questions as whether a particular intuitive judgment was relevant to
linguistics became conditioned by the theory itself. Take Chomsky’s famous sentence “Colorless
green ideas sleep furiously” which was introduced to show the distinction between
grammaticality and acceptability judgments. This example engendered considerable debate
because what constitutes grammaticality as opposed to acceptability is not given by intuition but
came from the nascent theory itself.
So my view is that as theories of vision formulate general principles, the theories will
direct us in the interpretation of evidence from conscious experience. It will show us how to
interpret such findings as those of [Wittreich, 1959 #1312]. Wittreich confirmed that when
2
This is not to say that the effective information provided by perception is contaminated by expectations and beliefs
– as supposed by [Whorf, 1958 #790] and his latter-day followers [Gumperz, 1996 #1418]. Rather my claim is that
the contents of one’s conscious experience is contaminated in this way, a very different matter [for more on this, see
\Pylyshyn, 1999 #965]. This suggests that we may need to develop methodologies to take cognizance of this
confounding, the way that signal detection theory provides a methodology for unconfounding sensitivity and
response bias in certain decision-making situations.
4-4
Chapter 4
people walked across the floor of the Ames distorting room they appeared to change in size. But
he also found that this did not happen when the people were well-known to the observer, e.g.,
their spouse, even if they were accompanied in the walk by a stranger (show size did change!).
Even now I think we are in a fairly good position to be incredulous of the theoretical significance
of this finding, given what we know about how reports of conscious experience can be
cognitively penetrable, hypnotism being an extreme example of this. Sometimes we can show
this fairly directly by comparing measures from which the response bias has been
mathematically factored out, as we do when we use the signal detection measure d′ rather than
percent correct. But sometimes we make the decision on the grounds that a theory that takes
certain observations at face value will simply miss the deeper underlying principles.
4.3
The special case of the experience of space
4.3.1 The status of our phenomenal experience of space
Our grasp of space appears to us to be subtle, complex and extremely fine-grained. Our
experience of space is all-pervasive; we experience ourselves as being totally immersed in the
space around us which remains fixed as we move through it or as objects other than ourselves
move through it. Our spatial abilities are remarkable and have resisted a causal explanation
despite the efforts of some of the best minds over the past two centuries. For example, we can
orient ourselves in space rapidly and effortlessly and can perceive spatial layouts based on
extremely partial and ambiguous cues. We can recall spatial relations and recreate spatial
properties in our imagination. Animals low on the phylogenetic scale, who may not even have
concepts, inasmuch as they arguably (though this is by no means obvious) do not how the power
to reason about things that are absent from their immediate perception, exhibit amazing powers
of navigation that prove that they have quantitative representations of the space through which
they travel and that they update these representations continually as they move through the space
or interact with it in various sensory-motor ways. Although perception science is arguably the
most developed of the cognitive sciences there are many areas of vision science where it is far
from clear that we have posed the problems correctly, and the problem of spatial cognition
strikes me as an extremely likely candidate for one of those problems.
The experience of spatial layout is a fundamental datum that presents many constraints
on a theory of early visual representation (some of which I will take up later). But it is also
problematic because it is the experience of how the world is and not of our mental processes.
Because of this it is actually the result of many different mental processes, including the
inferences we draw from our beliefs about the location of objects in world before us. Our
experience is of a stable panoramic layout of spatial locations, some of which are empty while
others are filled with objects, surfaces and features that stand in some spatial relation to one
another. This is the very phenomenology that has leads people to postulate an inner replica of
the perceived world that constitutes the experiential content of our perceived space – a
panoramic display that fills the world around us [Fred Attneave called it "cycloramic" since it
appears to cover 360 degrees of view, \Attneave, 1977 #1500]. If we assume that the content of
experience must somehow arise from a representation that has that content, and that the
representation is constructed from the information we receive through our sensors, then there is a
problem about how such a representation could possibly come about, given the poverty of the
incoming information. The incoming information consists of a small peephole view from the
fovea that jumps about several times a second, during which we are essentially blind, and so on
[the information available to the brain has been described in detail and is a familiar story, see
e.g., \O'Regan, 1992 #733]. So the gap between our visual experience and the available visual
information requires some explanation. While there are many ways to try to fill the gap (some of
4-5
Chapter 4
which, as we will see, appeal to what we call visual indexes or FINSTs) the natural way, given
the form of the experience, is to try to build an internal facsimile of the contents of the
experience and by postulating a process that takes account of the saccades and constructs an
inner picture in synchrony with the eye movements, along the lines depicted in Figure 4-1 below:
Figure 4-1. The intuitive view of the content of our experience of seeing [from
\Pylyshyn, 2003 #1528]
But as we now know, this theory is patently false – there is no inner replica or picture of
any kind in our head, neither literally or in any other non-vacuous sense capable of explaining
how we represent spatial information in perception and thought. What has gone wrong that has
led so many people to succumb to that story? What has gone wrong is that we are using a
particularly natural description of phenomenological experience as the explanandum: we are
trying to explain the content of the experience by positing certain intrinsic properties of a
medium of representation. But we are not entitled to assume that the content of experience
reflects the structure or format or any inherent property of a representation. To do so is to
succumb to the well-known intentional fallacy, the fallacy of attributing properties of what is
being represented to the representation itself (as if our representation of a square object were
itself red and square). Yet so long as we take the content of the perceptual experience as our
primary datum this is where it will lead us. Should we, then, discount the experience and start
afresh from psychophysical data? I will return to this topic in the next chapter when I consider
what a theory of spatial cognition needs to explain.
4.3.2 The argument for nonconceptual representation of space
As we saw in previous chapters, a theory of perception and cognition needs an ultimate
link of some sort with the perceived world in order to ground perception-based mental
representations. Furthermore, this link must ultimately be causal on pain of a regress. Or, more
precisely, the link must not itself be conceptual and must not rely solely on the semantic relation
of satisfaction. In the last chapter I offered a proposal for a particular sort of connection based
on the operation of selective attention – or at least on a particular stage or type of selection that I
called Indexing, which works very much like demonstrative identification. I ended that
discussion with the open question of what happens to the rest of the information in a scene, the
information about objects and properties that are not indexed. I hinted that we may have to live
with the uncomfortable idea that it may be unavailable to the mind, at least at that instant. Such
a conclusion seems particularly implausible in respect to information about space, because it is
here that our experience most strongly insists that we have a grasp on space in some sort of bulk
4-6
Chapter 4
manner that is very different from the punctate index-based account I have been offering. This
intuition is what people have studied under the heading nonconceptual representation of space.
One philosopher who has worked seriously on the problem of characterizing the
nonconceptual experience of space is Christopher Peacocke. In (Peacocke, 1992) he develops a
sophisticated system for characterizing the experience of space in terms of what he calls scenario
content, which he defines as ways in which space can be filled. As I mentioned above, while
characterizing the nonconceptual experience of space is a deep and interesting problem, it is not
clear how cognitive science can build on these ideas, since it is not clear how the detailed
phenomenological experience of space captures the distinctions and the mental structures
required by a causal/functional theory. Perhaps there is a parallel here with the relation between
generative theories of grammar and theories of language learning and parsing. Even while it is
clear that the rules of grammar characterize what a speaker implicitly knows – the underlying
linguistic competence – the rules may not be suited for direct application to parsing or language
learning [for more on this issue as it pertains to language see, \Pylyshyn, 1973 #207]. In fact it is
arguable that the rules themselves (expressed in some generative formalism such as rewrite rules
and transformations) are not explicitly represented at all [Pylyshyn, 1991 #831] . Similarly it is
not clear that the scenarios, to the extent that they correctly characterize the experience of space,
can be taken as constituting a form of (nonconceptual) representation that is functional in
perception.
There are two possible ways of interpreting the audacious claim that even though our
spatial experience is correctly characterized to have something like scenario content, this need
not correspond to the content of a nonconceptual (or any other kind of) representation. (1)
Although a scenario may be an accurate description of one’s experience, it need not be a good
description of a functional mental state. This can be because it mischaracterizes the causally
relevant properties. We see a good deal of this sort of mischaracterization in discussions of
mental imagery. While the descriptions of the phenomenology may be correct, their functional
significance is problematic at best and quite likely simply irrelevant. This way of characterizing
my claim about the role of subjective experience is methodological – there are many sources of
evidence for how a cognitive process works and subjective experience is just one of them and,
like any other source, may have to be discounted in favor of more reliable sources. (2) A second
way of interpreting the claim that a (correct) characterization of experience may not provide the
right information on which to build a causal theory is more radical. It is the suggestion that the
reason that some of the content of experience may not be represented in a way that functions in a
causal account of perception, is that it might be possible for a person to experience sensory
distinctions that are not cognitively functional. And that, in turn, suggests that (at least some)
qualia themselves may not be functional! I am not the first person to make this suggestion
[Block, 2001 #1526]. Other writers have not gone so far as to deny the functional role of
subjective experience, yet have pointed out that the reason that philosophy has stayed faithful to
conscious experience in characterizing perception may not be a good enough reason. That’s the
reason that conscious experience plays the role of grounding justification for beliefs. You are
justified in believing F(x) if you can see that x is F just be looking, where “see” is read as
“consciously experience.” The claim that justification is not a good enough reason to assume that
conscious experience is essential for seeing has been made with characteristic force by Jerry
Fodor in [Fodor, 2004 #1666].
Of course it is also possible that the content of experience – the full fine-detailed
information that we experience, and that might be characterized by some description that
captures the pre-conceptual content (such as Peacock’s scenario content) is available for a very
short time before it either fades or is encoded conceptually. This is consistent with many findings
4-7
Chapter 4
in the experimental literature that provide evidence for what is often called “iconic storage”. For
example, [Sperling, 1960 #1308] found that a great deal of visual information is available for a
fraction of a second after the information input (i.e., the light) has disappeared. This so-called
iconic store does not contain conceptualized information (in fact it is commonly referred to as
pre-categorical store and we cannot search this store for categorized information – e.g., we
cannot pick out the vowels from the brief display) but it is available for any visual process that
itself does not take more than a few hundred milliseconds – including processes that encode
information conceptually.3 While this third option does concede a type of nonconceptual
analogue representation, calling it a representation is misleading insofar as it need not be an
encoding of any sort – not even sub-personal. Moreover it fails the tests for representations I
mentioned earlier. Because it is arguably just a geometrical projection of retinal stimulation (at
least according to one view, championed by Sperling) it cannot misrepresent and its content
(what it represents) does not enter into any generalizations. Consequently it is an informationbearing state in the weak sense, which I have suggested should not be considered a
representation inasmuch as it does not have a semantic character. Some of this may well be a
terminological issue, but it seems to me that we need to distinguish between sensory inertial –
the fact that when activated certain sensors continue their active state for a brief time after the
source of activation ceases to activate the sensor – and representation or encoding. The latter
represents a state of affairs as having certain properties or belonging to a certain category. To use
the philosophical jargon, it represents something as having a certain property so it is a case of
representation as or opaque representation (so called because substituting the representation or
code into a belief context need not preserve the truth of the belief). Even very early perceptual
states (states within early vision) may be true representations in this sense, although they will not
be beliefs, according to the usual understanding of beliefs, because such representations are not
available for general reasoning for two reasons. One reason is that early vision is modular or
encapsulated [to use Fodor’s terminology, \Fodor, 1983 #84] and therefore such representations
may not available outside the early vision module [see \Pylyshyn, 1999 #965]. Another is that it
is likely to be sub-personal and therefore not in the vocabulary of person-level thoughts, despite
meeting conditions of being conceptual. There is no reason why early vision representations
could not be full conceptual representations meeting the criteria suggested earlier (i.e., their
semantic content plays a role in generalizations) and yet fall short of being thoughts. Whether
any particular proposals for such stages in a perceptual process (e.g., Marr’s Primal Sketch or
2.5D sketch) qualify as representations in this sense is an empirical question.4
The notion of the possible nonconceptual content of spatial experience is relevant to the
question of the nature of mental imagery which I discuss in the next section. In the next chapter
I will revisit the question of spatial representation when I offer a suggestion for how
nonconceptual spatial “representation” might arise without any actual internalizing of spatial
properties – i.e., without a representation of spatial properties in the usual sense.
4.4
The phenomenal experience of mental imagery
If we are tempted by the picture of visual perception show in Figure 4-1, then we will be
equally, if not more, tempted by the view that in the absence of input from the eyes, the inner
Some types of information may be available for longer – such as information used for pointing [Tyrrell, 1993
#1773] which seems to be available for about 15 seconds and the effective location of a visual mask, which [Ishai,
1995 #1122] claim may be available for several minutes.
4
As noted in [xxx] in my view one ought to be conservative in admitting an intermediate state as a representation.
To say it is an empirical issue is not to say that one can answer the question of whether it is a representation by
doing an experiment – it is a long-term issue of whether one gains explanatory advantage by treating it as a
representation and referring to what it represents in giving explanations.
4-8
3
Chapter 4
display in that figure can also be filled from memory or from reasoning. Since, according to that
view of visual perception, we have a display surface with the nonconceptual content
corresponding to our experience, it would be logical that we might use it to imagine as well as to
see. This is indeed the received view in much of cognitive psychology [Kosslyn, 1994
#880;Kosslyn, 1980 #144], neuroscience [see the commentaries appearing with my article in
Behavioral and Brain Sciences, \Pylyshyn, 2002 #1302] and even a fair amount of philosophy
[see, for example, \Tye, 1991 #1223, as well as many of the essays reproduced in Block, 1981
#25]. While it is not generally acknowledged that the goal of this exercise is to account for the
experience we have when we entertain mental images, there is little doubt that this is the driving
force. Consider some of the arguments made in favor of what is sometimes referred to as the
“depictive” view of mental images.
4-9