Transparency and Occlusion
Barton L. Anderson
University of New South Wales
One of the great computational challenges in recovering scene structure from images
arises from the fact that some surfaces in a scene are partially obscured by nearer surfaces
or media. Both occluding and transparent surfaces interrupt the projection of more
distant surfaces, and may be considered two ends of a continuum. Occluding surfaces
completely obscure the surfaces that they occlude, whereas transparent surfaces only
partially obscure the surfaces they overlay. The degree to which a transparent surface
obscures an underlying surface depends on its transmittance (i.e., the proportion of light
that it lets through of the underlying layer). Thus, when the transmittance is zero, the
near surface is an opaque occluder; when it is greater than zero, some light of the
underlying layer is transmitted. In order for the visual system to recover scene structure
in contexts in which the transmittance of a near layer falls between zero and one, it must
decompose the image into a “layered” representation that specifies the presence of
multiple surfaces (or a surface and intervening media) along the same line of sight. This
form of decomposition has been termed scission.
In addition to the relationship between transparency and occlusion, the physical
transformations induced by transparent surfaces are intimately related to the physical
transformations that are caused by changes in illumination. This suggests that possibility
that the decomposition that underlies the perception of transparency may also underlie the
separation of illumination from surface reflectance, and hence, the computation of
surface lightness and/or color. Thus, the topic of transparency is intimately related to two
apparently distinct domains: the computation of occlusion relationships; and the
computation of surface lightness. In this paper, I will describe some recent evidence that
reveals the intimate relationship between scission and the perception of surface opacity,
lightness, and depth, and the impact this research has on theoretical frameworks in vision
more broadly.
The computation of transparency
The topic of transparency research emerged as a fundamental area of vision
research with the seminal work of the Italian psychologist Metelli (1970; 1974a,b).
Metelli developed a model of transparency based on the physical setup he used to
generate of transparent images – namely, a disk with a missing sector (episcotister) that
rotated in front of a two-toned background (see Fig. 1). Metelli derived a simple set of
equations that described the relationship between the reflectance of the underlying
background surfaces, the transmittance of the transparent filter (i.e., the size of the
missing sector), and the reflectance of the transparent layer. Metelli argued that
perceived transparency was well predicted by the physical constraints that must be
satisfied to produce a transparent surface, and thus, embraced an “inverse optics”
approach to modeling visual perception. In particular, Metelli used his episcotister
display to derive a physical model for the transmittance (α) and reflectance (t) of the
transparent surface, which was just a weighted sum of the contributions of the light
transmitted from the underlying layer and that reflected by the episcotister:
p = αa + (1-α)t
(1)
q = αb + (1-α)t
(2)
where p is the region containing the transparent layer that overlaps background a; q is the
region containing the transparent layer that overlaps background b, and α is the
transmittance of the transparent layer (i.e., the proportion of the size of the holes in the
episcotister). These equations can be solved to derive separate expressions for the
transmittance and reflectance of the transparent surface:
α = (p-q)/(a-b)
(3)
t = (aq-bp)/(a+q-b-p)
(4)
To make physical sense, α is restricted to be between 0 and 1, which imposes two basic
constraints on the images that are consistent with transparency: the luminance difference
between the regions of transparency (p-q) must have the same sign as the luminance
difference between the regions in plain view (a-b) (to insure that α is positive); and the
magnitude of the luminance difference of the regions of transparency (p-q) must be less
than or equal to the luminance difference of the surface in plain view (a-b) (to insure that
the transmittance falls between 0 and 1). Perhaps the most salient restriction of the
applicability of these equations is that they can only be used to describe transparent filters
containing a uniform reflectance and transmittance. They make no predictions of
displays containing “unbalanced” transparent filters or media, i.e., forms of transparency
that are not uniform in reflectance and/or transmittance. Note that Metelli’s model is a
purely generative model, i.e., his equations described the (simplified) physics of his
episcotister display. Thus, the issue of whether such equations could be used to predict
when, whether, and how transparency was perceived is an issue that required
psychological experimentation.
Nearly three decades of research into transparency perception seemed to provide
compelling evidence that Metelli’s model successfully predicted when transparency was
and was not perceived (Metelli, 1974a,b, 1985; Metelli, Da Pos, & Cavedon, 1985;
Gerbino, Stultiens, Troost, & de Weert, 1990; Kaasrai & Kingdom, 2001). However, we
have recently argued that that this apparent success stems from the particular
methodologies employed to assess his model. Although Metelli derived separate
expressions for the transmittance and reflectance of a transparent surface, until recently,
no experiments assessed whether these expressions accurately captured human perception
of transparent surfaces. Rather, most experiments typically required observers to adjust
(or otherwise judge) the luminance of a test patch in a display until it appeared to form a
uniform transparent filter; they did not measure whether Metelli’s equations could
quantitatively predict the perceived transmittance and reflectance of a transparent filter.
To provide a more direct quantitative test of Metelli’s model, we (Singh &
Anderson, 2002) recently performed a number of experiments to directly assess whether
his equations correctly predicted the perception of transparency. In these experiments,
observers were required to separately match the transmittance and reflectance of a
transparent filter. Observers viewed displays containing a central sinewave grating
surrounded by a higher contrast sinewave grating of the same frequency, orientation and
phase. To enhance the perception of transparency, binocular disparity was added to the
edges of the central patch, giving rise to a clear percept of a homogeneous transparent
filter on lying on top of a high contrast grating. Metelli’s model predicts that the
perceived transmittance of a transparent filter should only depend on the ratio of
luminance differences (i.e., the luminance range) of the regions of transparency to the
regions in plain view (see equation 3). In one of our experiments, the luminance range of
the region in plain view (the high contrast grating) and the luminance difference of the
region of transparency was held constant; only the mean luminance of the region of
transparency was varied. Observers adjusted the luminance range of a matching pattern
to match the perceived transmittance of the transparent filter with varying mean
luminance values. In this experiment, the luminance range of the region of transparency
and the region in plain view are constant, so Metelli’s equations predict that the
transmittance settings in this experiment should be independent of mean luminance. This
was not at all what was observed experimentally. Rather, observer’s matches are very
strongly dependent on the mean luminance of the transparent region (see Fig. 2). In
particular, observers significantly underestimate the transmittance of light filters, and
systematically overestimate the transmittance of dark filters.
The results of this experiment reveal that human observers are not simply
inverting the physics of transparency to compute the properties of transparent surfaces.
The theoretical importance of this finding should not be underestimated, as one of the
main theoretical views of visual processing assumes that the visual system extracts the
properties of the world by performing computations that invert the image formation
process. These findings provide a striking example where human observers have a very
clear sense of a surface property (the transmittance of a transparent layer) that almost
always generates the physically incorrect answer. Why, then, does the visual system err
in this manner? What information is the visual system using to compute these properties
that causes it to give such incorrect responses?
One of the main problems with Metelli’s model is that it assumes that
transmittance is computed on the basis of the ratio of luminance differences between the
regions of transparency and the regions in plain view.1 However, the earliest stages of
cortex do not seem to have access to raw luminance values. Rather, the information about
image structure seems to be largely transformed into a contrast code. Indeed, when the
data from this experiment are plotted in terms of Michelson contrast, it becomes evident
that the visual system uses contrast to scale the transmittance of a transparent surface,
even though this yields the physically incorrect answer. In our sinusoidal displays,
Michelson contrast provides a good measure of perceived contrast, so works well in
accounting for perceived contrast in our displays. However, Michelson contrast fails to
provide an adequate measure of contrast in more complex spatial patterns containing,
e.g., random patches of achromatic surfaces. Despite these shortcomings, recent work
has shown that observer’s transparency judgments can be well described by the perceived
contrast of images (Robillotto, Khang, & Zaidi, 2002; Robillotto & Zaidi, 2004),
1
Strictly speaking, Metelli’s model is formulated on the basis of reflectance differences.
However, Gerbino et al. (1990) showed that Metelli’s equations could be rewritten as difference
in luminance values, and that the form of these equations were identical to equations (1-4).
suggesting that transparency computations are indeed based on some (yet to be
discovered) measure of image contrast.
In sum, there is now clear evidence that the visual system uses the relative
contrast of image regions to compute the opacity of transparent surfaces. In what
follows, I will consider the implications that this discovery has had in understanding
when the decomposition into a layered representation is initiated, and the consequences
that this decomposition can have on not just the properties of the transparent surface, but
the underlying surface as well.
Anchoring perceived transmittance
In addition to the quantitative failures of Metelli’s model, it was also not easily
generalized to a variety of naturally occurring forms of transparency. Metelli’s model
only described conditions of “balanced” transparency, i.e., conditions where the
transmittance and reflectance of the transparent layer is uniform. However, many
naturally occurring forms of transparency – such as that induced by smoke and fog – are
typically unbalanced, particularly in transmittance. Clearly, a more general framework is
needed to understand the wide variety of ways that transparent surfaces and media can
transform image structure. How does the visual system determine whether a scene is in
plain view or viewed through a transparent layer or medium when the properties of the
transparent layer vary continuously?
If image contrast is the currency through which the properties of transparent
surfaces are computed, then any general theory of transparency perception will have to
use image contrast as one of its main ingredients in the computation of transparency.
However, all natural scenes generate variations in image contrast, and the properties of
the underlying surfaces are unknown. So how can the visual system determine whether a
given image arose from a scene in plain view or a higher contrast scene viewed through a
(contrast reducing) transparent layer? The image data is always consistent with both
possibilities, so something is needed to understand how the visual system determines
when transparency is and is not perceived. I have recently proposed that the visual
system employs a transmittance anchoring principle to determine when transparency is
(and is not) inferred (Anderson, 1999, 2003a). The intuitive content of this principle is
that the visual system assumes the fewest surfaces necessary to account for the image
data. More specifically, it states that the visual system treats the highest contrast region
along surfaces and contours as transmittance anchors that are in plain view. All other
contrast values along such contours and/or surfaces are compared to this anchor region,
and decreases in contrast along surfaces are used to infer the presence of transparency.
More specifically, this theory asserts that if there are reductions in contrast along surfaces
or contours that are geometrically continuous, then scission is initiated and transparency
is perceived. The magnitude of the contrast reduction is used to compute the
transmittance of the overlying (transparent) layer (in proportion to this reduction; i.e., the
greater the contrast reduction, the more opaque the transparent surface will appear). In
the limit, where the contrast of the underlying surface goes to zero, the transmittance of
the overlying surface goes to zero, and the near layer becomes an occluder. Note also
that if a change in mean luminance occurs without a corresponding reduction in contrast,
that such transformations are consistent with a change in illumination, and hence, would
be correctly predicted to appear in plain view in both regions of the image. Thus, the
contrast relationships along surfaces and contours could be used to provide information
about both transparent media and changes in illumination.
We have recently shown that this theory correctly predicts the perception of
transparency in both balanced displays, as well as spatially inhomogeneous media
(Anderson, Singh, & Meng, 2006). In addition, we have shown that transmittance
anchoring also has a temporal component: if the contrast of a texture is modulated in
time, the highest contrast region in the spatio-temporal sequence is treated as a region in
plain view, and the spatio-temporal reductions in contrast appear as the intrusion of
transparent media. Thus, the transmittance anchoring principle appears to provide a
foundation on which to predict when transparency is and is not perceived, and the
perceived transmittance of the transparent layer.
Scission and the perception of lightness
The decomposition of an image region into multiple layers can also have a
significant impact on perceived lightness. In the preceding, I have focused on how
variations in image contrast can be used to determine whether transparent surfaces are
present, and if so, how the opacity of the transparent layer is computed. However, when
multiple surfaces are present along the same line of sight, the visual system must also
compute properties of the underlying surface, such as its lightness. Given the close
relationship between the physical transformations induced by transparent surfaces and the
transformations induced by changes in illumination, there is good a priori reasons to
suspect that a process of scission may play a critical role in the perception of surface
lightness. Although a number of authors have suggested the possibility that scission may
play a critical role in lightness perception (Anderson, 1997; Bergström, 1977; Gilchrist,
1977, 1979; Adelson, 1993), such authors have questioned whether an explicit
decomposition actually underlies lightness perception. More recently, we have been
developing a new paradigm that explicitly reveals the role that scission can play in the
perception of surface lightness (Anderson, 1999; 2003a,b; Anderson & Winawer, 2005).
Traditional lightness studies typically employ homogeneous targets to be judged, and
measure the effect different contexts have on their perceived lightness or brightness. In
such paradigms, it can often be difficult to determine whether scission is occurring, and if
so, whether it is playing any causal role in the perceived lightness of a figure. However,
in the paradigm we have developed, if scission occurs, it is phenomenologically very
explicit, and the effects it has on perceived lightness can be directly experienced.
To see the role that scission can play in lightness perception, consider the image
depicted in Fig. 3. This figure appears to contain white chess pieces viewed through dark
smoke, and black chess pieces viewed through white fog. However, the image regions
containing the chess pieces in the top and bottom of the image are actually absolutely
identical; the only difference between the top and bottom images is the overall lightness
of the surrounds. These images were carefully constructed to satisfy the constraints of
transparency. In the top image, all of the boundaries separating the chess pieces from the
surround are lighter inside the chess pieces than outside (so that the boundary separating
the regions is of constant polarity), whereas the opposite polarity holds for the bottom
figure. In addition to the shift in polarity between the top and bottom figure, there are
also significant differences in the way the magnitude of contrast varies along the borders
separating the chess pieces and their surrounds in the two images. Consider, e.g., the
king in the two images. In the top figure, the greatest contrast between the king and the
surround occurs along the bottom right of the piece, whereas the lowest contrast occurs
along the top. Thus, the transmittance anchoring principle states that the bottom right of
this figure should appear in plain view, which is white; whereas the reductions in contrast
that occurs along the boundaries separating the king from its surround should signal the
presence of a transparent medium that varies in opacity (being most opaque where the
contrast of the boundary is lowest, which here, occurs along the top of the king). A
similar analysis holds for the bottom figure, except now the polarity and magnitude
relationships are reversed. In this image, the highest contrast region along the border
separating the king from its surround occurs along its top, and hence, the transmittance
anchoring principle predicts that this portion of the king (which is dark) should appear in
plain view, and lower contrast regions along the contour should appear partially obscured
by transparent media. In the bottom image, the lowest contrast region of the kingsurround border occurs along the lower right of the image, and thus, the opacity of the
transparent layer should be greatest in this region. This is consistent with what observers
report.
In sum, the perception of transparency and lightness in Fig. 3 reveals the close
relationship between the perception of transparency, occlusion, and lightness. The
perception of transparency involves the decomposition of an image into multiple layers,
and the properties of the layers depend critically on exactly how luminance is partitioned
between them. There is a growing body of data suggesting that the contrast relationships
that occur along surfaces and contours play a critical role in determining when scission is
initiated, as well as determining how surface properties such as lightness and opacity are
attributed to the layers that are formed when this decomposition occurs. Such results
reveal severe limitations on inverse optics models of perception, since the computation of
properties such as the transmittance of transparent surfaces are almost always physically
incorrect. Moreover, there is currently no single measure of image contrast that
adequately captures perceived contrast in arbitrary images, which impedes the ability to
predict the precise conditions that lead to scission and the quantitative consequences that
scission should have on perceptual experience. It is therefore of critical importance to
develop a physical measure of contrast that captures the human experience of contrast, so
that it can be used to develop and assess theories of transparency perception. Finally,
although phenomena such as those depicted in Fig. 3 reveal that scission can have a
dramatic effect on perceived lightness, more research is needed to determine if an explicit
decomposition of images into layers is responsible for the perception of lightness (and
color) in all images.
References
Adelson, E.H. Perceptual organization and the judgment of brightness. Science 262,
2042-2044 (1993).
Anderson, B. L. (1997) A theory of illusory lightness and transparency in monocular and
binocular images. Perception 26, 419–453.
Anderson, B. L. (1999) Stereoscopic surface perception. Neuron, 26, 919–928.
Anderson, B. L. (2003a) The role of occlusion in the perception of depth, lightness, and
opacity. Psychological Review, 110, 762–784.
Anderson, B.L. (2003b) The role of perceptual organization in White's illusion.
Perception, 32, 269-284.
Anderson, B.L., Singh, M., & Meng, J. (2006) The perceived opacity of inhomogeneous
surfaces and media. Vision Research, 46, 1982-1995.
Bergström, S.S. (1977) Common and relative components of reflected light as
information about the illumination, colour, and three-dimensional form of objects.
Scandinavian Journal of Psychology, 18. 180-186.
Gerbino, W., Stultiens, C. I., Troost, J. M., & de Weert, C. M. (1990). Transparent layer
constancy. Journal of Experimental Psychology: Human Perception and
Performance, 16, 3–20.
Gilchrist, A.L. (1977) Perceived lightness depends on perceived spatial arrangement.
Science, 195(4274), 185-187.
Gilchrist, A.L. (1979) The perception of surface blacks and whites. Scientific American,
240, 112-123.
Kasrai, R., & Kingdom, F. (2001). Precision, accuracy, and range of perceived
achromatic transparency. Journal of the Optical Society of America A, 18, 1–11.
Metelli, F. (1970). An algebraic development of the theory of perceptual transparency.
Ergonomic, 13, 59–66.
Metelli, F. (1974a). Achromatic color conditions in the perception of transparency. In:
MacLeod, R. B., Pick, H. L. (Eds.), Perception: Essays in Honor of J. J. Gibson.
Cornell University Press, Ithaca, NY.
Metelli, F. (1974b). The perception of transparency. Scientific American, 230, 90–98.
Metelli, F., Da Pos, O., & Cavedon, A. (1985). Balanced and unbalanced, complete and
partial transparency. Perception and Psychophysics, 38, 354-366.
Robillotto, R., Khang, B., & Zaidi, Q. (2002). Sensory and physical determinants of
perceived achromatic transparency. Journal of Vision, 2, 388-403.
Robillotto, R., & Zaidi, Q. (2004). Perceived transparency of neutral density filters across
dissimilar backgrounds. Journal of Vision, 4, 183–195.
Singh, M., and Anderson, B.L. (2002) Perceptual assignment of opacity to translucent
surfaces: The role of image blur. Perception, 31, 531-552.
Figure Captions:
Fig. 1: The episcotister setup used by Metelli to derive his transparency model. A disc
with a missing sector of size α is rotated at high speed over a two toned background. The
proportion of the open to solid regions of the disc determines the transmittance of the
episcotister, and the reflectance of the solid portion of the episcotister (1-α) contributes
additional luminance in regions of transparency.
Fig. 2. Results from Experiment 1 in Singh and Anderson (2002). Observers adjusted a
matching pattern to appear equal in transmittance to a test patch with a fixed luminance
range and mean luminance. Dashed lines in the left figure indicate the predictions of
Metelli’s model; the matches made to four different luminance ranges with varying mean
luminance are given by the filled symbols for a typical observer. The data exhibit strong
and consistent departures from these predictions. When the data are re-plotted in terms
of Michelson contrast (right figure), the data are independent of mean luminance,
demonstrating that observers us perceived contrast to determine the transmittance of
transparent layers.
Fig. 3: A figure demonstrating how the contrast relationships along contours can be used
to determine the perceived transmittance and lightness of surfaces in a complex scene.
The regions within the boundaries of the chess pieces are composed of identical textures
in the top and bottom image, but are decomposed in complementary ways. In the top
figure, the texture is decomposed into dark clouds that obscure light chess pieces, and in
the bottom, the texture appears as light mist that obscured dark chess pieces. Note that
the highest contrast contour segments in both the top and bottom figure appear in plain
view, which occur in different regions for the two images. See text for details.
Figure 1
Figure 2
Figure 3