SSMRecolor: Improving Recoloring Tools with
Situation-Specific Models of Color Differentiation
David R. Flatla and Carl Gutwin
Department of Computer Science, University of Saskatchewan
110 Science Place, Saskatoon, Canada, S7N 5C9
[email protected], [email protected]
ABSTRACT
Color is commonly used to convey information in digital
environments, but colors can be difficult to distinguish for
many users – either because of a congenital color vision
deficiency (CVD), or because of situation-induced CVDs
such as wearing colored glasses or working in sunlight.
Tools intended to improve color differentiability (recoloring
tools) exist, but these all use abstract models of only a few
types of congenital CVD; if the user’s color problems have
a different cause, existing recolorers can perform poorly.
We have developed a recoloring tool (SSMRecolor) based
on the idea of situation-specific modeling – in which we
build a performance-based model of a particular user in
their specific environment, and use that model to drive the
recoloring process. SSMRecolor covers a much wider range
of CVDs, including acquired and situational deficiencies.
We evaluated SSMRecolor and two existing tools in a
controlled study of people’s color-matching performance in
several environmental conditions. The study included
participants with and without congenital CVD. Our results
show both accuracy and response time in color-matching
tasks were significantly better with SSMRecolor. This work
demonstrates the value of a situation-specific approach to
recoloring, and shows that this technique can substantially
improve the usability of color displays for users of all types.
Author Keywords
Accessibility, color, color vision deficiency, recoloring.
ACM Classification Keywords
K.4.2 [Social Issues]: Assistive technologies for persons
with disabilities;
General Terms
Human Factors
INTRODUCTION
Color is commonly used to present information in digital
environments (e.g., charts, web links, spell checking, syntax
highlighting, color-coded maps). For many users, however,
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
CHI’12, May 5–10, 2012, Austin, Texas, USA.
Copyright 2012 ACM 978-1-4503-1015-4/12/05...$10.00.
interpretation of color-coded information is often
problematic because they cannot tell one or more of the
colors apart. There are several reasons for these colordifferentiation problems: people may have a congenital
color vision deficiency (CVD), commonly called ‘color
blindness’, which affects almost five percent of users [4];
people may have an acquired CVD (e.g., cataracts which
reduce color sensitivity); or people may have a situationinduced CVD, such as viewing a screen in bright sunlight,
using a monitor with incorrect settings, or wearing tinted
sunglasses. Up to ten percent of users may have one of
these forms of CVD at any time [19].
All of these types of color vision deficiencies can cause
situations where colors in digital environments are
indistinguishable for the user. The cost of these
interpretation difficulties range from annoyance (e.g., being
unable to distinguish visited from unvisited webpage links)
to severe security and safety issues (e.g., being unable to
see a warning message, or misreading a GPS display).
To assist with this problem, adaptation tools have been
developed that modify the colors in digital presentations to
make them more differentiable (i.e., recoloring tools).
However, current recoloring tools do not work for many
cases of CVD because they only model a particular form of
congenital CVD called dichromatism (in which the user is
missing one of their three types of cone cells). For users
whose CVD arises from a different cause (e.g., acquired
CVD, situation-induced CVD, or other types of congenital
CVD such as anomalous trichromatism), current recoloring
tools do not accurately model the source of the colordifferentiation problem, and cannot perform appropriately.
To address this limitation, we have built a new recoloring
tool (SSMRecolor) based on Situation-Specific Models
(SSMs) of color differentiation. SSMs are performancebased models that can capture the color differentiation
abilities of any individual in any environment, by using an
in-situ calibration procedure. To use SSMRecolor, a user
first carries out a two-minute calibration, which results in
an SSM of their specific color-differentiation abilities in
their current environment [5]. The recolorer uses this model
to determine which colors will be confused, and modifies
these problem colors until the SSM predicts that all colors
are differentiable. The modified colors are then used to
replace the problem colors in the image to produce a
differentiable version of the image.
To evaluate SSMRecolor, we carried out a controlled
experiment that compared our system against a common
Internet-based recoloring tool and a recently-published
recoloring tool that represents the state of the art. We used
two color-matching tasks to test the systems in three
situations that induced an environmental CVD: yellowtinted glasses, a broken monitor with no red capability, and
a darkened monitor. In addition, we tested two groups of
participants – those with congenital CVD and those with
typical color vision. Color matching was used because it is
a typical real-world application of color differentiation.
We found that people’s color-matching performance was
significantly better when using SSMRecolor – matching
accuracy was 90% on average with SSMRecolor, compared
with 70% or less for all other tools. In addition,
SSMRecolor performed consistently in all situations,
whereas the performance of the other tools was widely
variable depending on the type of induced CVD. We found
similar results with response time data – participants found
matching colors significantly faster with SSMRecolor than
with any other tool, and times were consistent across
situations. Last, our results were similar both for
participants with congenital CVD and those without.
Our main contributions are a demonstration that the
situation-specific approach can be implemented in a
recoloring tool, and empirical evidence that the approach
performs better than current tools in non-standard CVD
environments. SSMRecolor requires only a short calibration
step, is applicable to any situation, and works quickly
enough to be used in everyday applications. Situationspecific modeling can substantially increase the coverage of
recoloring tools, for a wide variety of users.
RELATED LITERATURE AND BACKGROUND
Factors that Affect CVD
Many factors can cause color vision deficiencies, including
congenital, acquired, and situation-induced issues. Note that
these types of CVD are not mutually exclusive and people
can have a mixture of more than one type of CVD.
Congenital CVD
Humans see color with three types of light-sensitive cone
cells in the retina. Each cone type is sensitive to a different
portion (long, medium, or short wavelength) of the visible
spectrum. The cones’ spectral sensitivity arises from
photoreceptive proteins that are genetically determined, and
variations in genetics can result in a variation in the spectral
sensitivity of that cone type [22].
Considerable variation exists among non-CVD individuals.
For example, many men have multiple genetic encodings of
their long- and medium-wavelength photoreceptive proteins
leading to variations in male color vision [16]; and there is
evidence that some female carriers of CVD genes may
possess four types of cones instead of three [1]. There is
also considerable variation within the population of those
who have CVD. These variations fall into two categories:
those with cones with a shifted sensitivity and those
missing an entire cone type. There are three types of
sensitivity shifts depending on which cone class is affected:
protanomalous (long wavelength), deuteranomalous
(medium), and tritanomalous (short) [1]. Protanomalous
and deuteranomalous CVDs make up almost 75% of all
congenital CVD [4]. The severity of each of these three
forms of anomalous trichromacy ranges from no difficulties
with day-to-day life to frequent color perception problems.
There are also reported cases of individuals who do not fall
into one of these three categories, and are diagnosed with
extreme anomalous trichromacy [2].
Individuals missing an entire type of cone have dichromatic
CVD: protanopia (long-wavelength cones), deuteranopia
(medium), and tritanopia (short) [1]. Dichromatism
constitutes about 25% of congenital CVD, and manifests in
daily life with frequent difficulties identifying, matching,
and reproducing color [4]. Rarer forms of congenital CVD
include cone monochromatism (two missing cone types,
and rod monochromatism (no cones at all); individuals with
these CVDs only perceive shades of grey [1].
Situation-Induced CVD
Any environmental factor that influences the brightness or
spectral distribution of light entering the eye can induce a
situational CVD. Vision research classifies environmental
brightness into photopic (bright), mesopic (medium), and
scotopic (dark) light levels. In scotopic conditions, only the
rods function, and no color perception occurs. At mesopic
light levels, both the rods and cones contribute to color
perception, thereby changing how we perceive colors (e.g.,
at dusk when blues appear brighter than normal due to the
Purkinje Effect [22]). Color perception works well at
photopic light levels, but excessive light or glare can
overwhelm the capabilities of devices such as LCD panels.
In addition, factors that influence spectral distribution of
light entering the eye can change color perception. This
happens when lighting is colored (e.g., colored spotlights),
or when wearing tinted glasses or contacts. In digital
environments, this factor can occur when display hardware
or graphics software fails or is not properly calibrated.
Acquired (Permanent or Temporary) CVD
Acquired CVD can result from factors that affect the light
entering the eye (e.g., yellowing of the lens with age,
cataracts), as well as neurological damage to the retina or
visual processing centers of the brain [24]. Retinal damage
can result from premature birth, long-term diabetes,
hypertension, macular degeneration, or long-term exposure
to organic solvents like styrene [13]. Neurological damage
can arise from stroke or aneurism, as well as traumatic brain
injury. Acquired CVD can be short-term as well, being
introduced by prescription drugs (e.g., Viagra,
antidepressants) or as a side effect of depression.
Existing Recoloring Tools
The idea of adapting colors on a computer display to match
the color perception abilities of the user began with Meyer
and Greenberg [15], who proposed a CVD simulation
strategy, a computerized color vision test, and the idea of
recoloring for individuals with dichromatism. More
recently, SmartColor [21] allows a designer to specify
visualization properties that constrain the automatic
production of a color scheme that accommodates CVD.
Many recoloring tools have been presented that deal
specifically with images. Methods for transforming color
images to greyscale have been modified to accommodate
individuals with CVD [17,18], and an interactive
accommodation system has been developed [9,10] that
allows the user to explore different recoloring strategies.
Both of these approaches modify all of the colors in the
image, often resulting in dramatic changes. Most recently, a
relatively fast dichromatic recoloring tool aimed at
maintaining the ‘naturalness’ of an image has been
proposed [11] to address this. These systems all perform
recoloring for dichromatic CVD, but do not attempt to
accommodate those with other forms of congenital CVD or
with acquired or situational CVD.
Existing Models of CVD
Recoloring tools rely on models of CVD that provide the
ability to predict which colors on a display will be
confused. Most recoloring tools use simulations of CVD
vision based on early work by Meyer and Greenberg [15] or
Brettel et al. [3,20]. Both approaches produce simulated
protanopic, deuteranopic, and tritanopic views that are
compared to the original image to detect colors that are
differentiable in the original, but not in the simulation. The
image colors are then modified to make them differentiable.
minutes, making the process practical for recoloring tools.
We used this recent SSM as the basis for SSMRecolor.
SSMRECOLOR: AN SSM-BASED RECOLORING TOOL
Here we describe the design and implementation details for
our SSM-based recoloring tool - SSMRecolor. Recoloring
tools use a three-step algorithm to recolor an input image:
1. Reduce the colors in the input image to a ‘key set’ of
colors (usually through quantization).
2. Generate a recolor mapping from the key set of colors to
a replacement set of colors.
3. Replace the original input image colors with the recolor
mapping colors to generate a recolored image.
Steps 1 and 3 have been discussed extensively in previous
work (e.g., [9,17,11]). To find the key set of colors for an
image in SSMRecolor (step 1), a target number of key
colors is provided by the user. Less-frequently-used colors
in the input image are iteratively changed to a perceptuallysimilar color that occurs more frequently, until only the
target number of colors remains. To replace original image
colors with recolor mapping colors (step 3), SSMRecolor
examines each pixel of the original image, switching
original colors for the replacements determined in step 2.
The main contribution in SSMRecolor is in this second step
– generating a recolor mapping. We illustrate the process
with the chart shown at left in Figure 1 (note that all figures
should be viewed in color). A protanopia-simulated view of
this chart is shown at right in Figure 1 to illustrate the
difficulties caused by this type of congenital CVD. Figure 2
contains the key colors for the original chart.
More recently, work has been done to simulate many types
and severities of anomalous trichromacy [14]. Recoloring
tools could make use of these models, but individuals with
CVD rarely know the type or severity of their condition.
Situation-Specific Models of Color Differentiability
Existing models of CVD use a static mathematical
representation of color differentiation which is unable to
adapt to changing circumstances. Basing a model on in-situ
performance can automatically capture the influence of any
factor that affects color perception. As a result, situationspecific models can be constructed for any individual in any
environment, giving these models specificity not available
in the existing mathematical models.
In earlier work, we proposed SSMs for color differentiation
[6] that can accurately represent the color differentiation
abilities of non-CVD and CVD individuals with a variety of
types and severities of congenital CVD, in a variety of
situations. These SSMs were built using an in-situ user
calibration to capture their color differentiation abilities.
The original calibration took over 30 minutes to perform,
but a recent revision [5] reduced the required time to two
Figure 1. Left: a simple Microsoft Excel for Mac 2011 chart
with six categories. Right: this chart as seen by someone with
protanopia. Simulation courtesy www.vischeck.com
Figure 2. The set of key colors (including text and background
colors) for the left chart in Figure 1.
Generating a Recolor Mapping
SSMRecolor builds a mapping from the set of key colors to
a set of replacement colors. This two-step process is
iterative, first identifying a non-differentiable color from
the key color set, then assigning a replacement color for the
identified color. The iteration continues greedily until no
non-differentiable colors are identified (i.e., all the resulting
colors are differentiable). This process is fundamentally
different from previous approaches because any colors that
are differentiable from all other colors are maintained,
resulting in only problem colors being modified. As
mentioned earlier, most existing recoloring tools modify all
of the colors in an image, resulting in unnecessary
introduction of false colors.
example, the problematic blue is replaced by a more vibrant
blue (Figure 3, center).
Color Differentiation Model
An in-situ calibration is performed to generate a situationspecific model of color differentiation for the user [5]. This
calibration captures the effect of any factor (congenital,
acquired, or environmental) on the color differentiation
abilities of a user, and allows predictions to be made about
these abilities. The model provides the Boolean function
areDifferentiable(C1,C2), which predicts whether colors
C1 and C2 are differentiable or not. For illustration
purposes, let us assume that the user has protanopia and has
performed the calibration procedure already.
Replacement Colors
SSMRecolor draws replacement colors from one of two
sets: unconstrained and luminance-maintained. The
unconstrained set is simply a random RGB color generator;
it is unconstrained because nothing restricts the suggested
replacement color. The luminance-maintained set contains
RGB colors of the same CIE LUV luminance as the target.
This set is loaded at runtime from a pre-generated database
by specifying the desired luminance level, and replacement
colors are chosen randomly from this set. For this
illustration, we use the unconstrained set of replacement
colors.
Figure 3. Non-differentiable color networks for a protanopic
individual. Five problem colors (left), two problem colors
(center) and no problem colors (right).
Next, the algorithm again selects the highest degree node
(as there are two nodes with degree of one, the orange color
in the right column bottom row of Figure 3, center is chosen
arbitrarily). Replacement colors are selected until this
node’s edge is eliminated, switching the problematic orange
to a bright green (Figure 3, right). When the color network
contains no more edges, the recolor mapping is built
(Figure 4) and used to generate the recolored image (Figure
5, left). Figure 5 right shows the protanopia-simulated view
for the recolored chart to show that the original color
differentiation problems have now been resolved.
Recolor Mapping Generation Algorithm
To generate the recolor mapping, the ‘areDifferentiable’
function is used to compare key colors. The result of this
comparison is a network where key colors are nodes, and
edges indicate confusion for the user (Figure 3). The color
with the highest degree in the network (i.e., is confused
with the most other colors) is replaced with a color from the
replacement set, and the network is regenerated using the
SSM. This process is repeated until no edges exist in the
color network, at which point the recoloring is complete. A
map from the original key colors to the new set is returned.
It can be seen from the right chart in Figure 1 that some
category color pairs are not easily differentiable for the
protanopic example viewer: ‘screws’ and ‘nuts’, ‘screws’
and ‘rivets’, and ‘washers’ and ‘nails’. Given the key colors
shown in Figure 2 and a color differentiation model for this
individual, the initial non-differentiable color network can
be constructed as shown in Figure 3, left.
Once this network is generated, the highest-degree color is
identified (the blue color in the left column, second row in
Figure 3, left). A replacement color is chosen from the
unconstrained set and the network is regenerated until the
edges connected to this color are eliminated – in our
Figure 4. Recolor mapping from the key colors to the
replacement colors that give no non-differentiable color pairs.
The third from the left and rightmost colors have been replaced.
Figure 5. Left: the recolored version of the original image.
Right: the recolored chart as seen by someone with protanopia.
Simulation courtesy www.vischeck.com
EVALUATION
We carried out a user study to compare the performance of
SSMRecolor with two existing dichromatic recoloring
tools: the Daltonize system available on the WWW
(vischeck.com/daltonize), and three versions of Kuhn’s
recolorer [11]. Kuhn’s tool was chosen because it represents
the state of the art in recoloring, and was the first recoloring
tool to maintain original colors (naturalness) as much as
possible. SSMRecolor also maintains the maximum number
of original colors, making Kuhn’s a reasonable comparator.
We compared these systems in three different CVDinducing situations, with two color tasks, for individuals
with and without congenital CVD.
The study was designed to determine whether the SSM
approach can improve on the state of the art in recoloring –
that is, whether our new recoloring tool allows people both
with and without congenital CVD to better differentiate
colors in a variety of CVD-inducing situations.
Methods
Participants
We recruited 21 volunteers (15 male, mean age 31.0 years)
from a local university – nine with congenital CVD (32.2
years, 8 male), and twelve with no CVD (30.1 years, 7
male). All participants were screened for CVD using the
HRR Pseudoisochromatic Plates [7]; of those with CVD,
five participants had deutan vision, three had protan vision,
and one had unclassified red-green CVD.
CVD-Inducing Situations
A control condition and three types of situationally-induced
CVD were used in the study. We chose situations that were
likely to be commonly encountered in the real world, and
that were representative of different color vision problems.
• Normal (control) view. Colors were presented to the
participant with no added situational CVD.
• Tinted glasses. Participants wore yellow-tinted glasses
while performing the matching task; this had the effect of
altering the hues of the colors on the screen. These
glasses are worn for certain sports, or by people who
experience glare when driving at night.
• Darkened monitor. For this situation, the monitor’s
settings were adjusted to reduce the contrast and
brightness of the colors produced. Brightness was set to
50%, contrast to 0%, and the red, green, and blue primary
gains were set to zero. This condition reduces luminance
differences between colors (common in bright sunlight).
• Broken monitor (no red). This situation simulated a
monitor that cannot display red. All colors displayed on
the screen were programmatically altered to have their
red channel values set to zero.
Recolorers
We compared six recoloring schemes: a control condition
with no recoloring, SSMRecolor, Daltonize, and three
variations of Kuhn’s recolorer.
• SSMRecolor was configured as described above. This
tool uses a situation-specific color differentiation model,
so the participant performed a calibration procedure [5] to
generate the model in each situation. To reduce the effect
of calibration input errors, we performed three
calibrations and used the median values as calibration
values. Three calibrations took 6.84 minutes, on average.
• For the Daltonize tool, the original colors were submitted
as an image to the Daltonize website with the following
settings (guided by the site’s FAQ): red-green stretch
factor was set to 1.3 for all situations, blue-yellow
correction was set to 0.2 for all situations, and luminance
correction was set to 0.0 for the isoluminant situationspecific tasks, and to -1.3 for the other situation-specific
tasks and the Excel task (described below).
• For Kuhn recoloring, the original colors were submitted
as an image to Kuhn’s recoloring tool [12]. As Kuhn’s
tool is a dichromatic recoloring system, it requires the
type of dichromatism (protanopia, deuteranopia, or
tritanopia). We therefore created three versions of the
Kuhn recolorer, one for each type (referred to below as
Kuhn-P, Kuhn-D, and Kuhn-T).
Task and Apparatus
A custom Java application presented a matching task in
which the user had to click on a single color that matched a
given cue color (see Figure 6). Two tasks were developed
that used two different color sets: a situation-specific set of
nine colors, and the first fifteen colors from the Excel 2011
chart color set (Mac version 14.1.3).
Figure 6. Study tasks: nine situation-specific colors (left) and
fifteen Excel colors (right), after recoloring. Participants clicked
on the square that most closely matched the cue color at right.
The study system presented the participant with a 3x3 grid
(situation-specific task) or 3x5 grid (Excel task) of colored
squares. The arrangement of colors in this grid was
randomized, and one of the grid colors was presented as the
cue color on the right of the screen. To reduce the
contextualizing effects of color perception (where adjacent
colors influence the perception of a color), all colors were
presented on a neutral white background with sufficient gap
between color squares to remove contextualizing effects.
The participant’s task was to click on the grid square that
matched the cue color. Each color in every nine-color and
fifteen-color set was chosen once to appear as a cue color.
Participant responses were recorded by the study system.
The study ran in a controlled environment on a Windows 7
PC with a 24-inch 1920x1080 LCD monitor. The darkened
monitor situation was carried out on a second (identical)
computer and monitor, with the display pre-set to the
appropriate values for the condition (see above).
Color Sets for Situation-Specific and Excel Tasks
Each situation-specific task used a different color set,
chosen to highlight difficulties induced by the situation.
The Excel task used the same colors for each situation, and
was chosen to explore aspects of color use in the real world.
Normal situation. Nine isoluminant colors were selected to
focus on the ability of the recolorers to deal with hue and
saturation differences (rather than luminance). Healy
suggests that a maximum of seven isoluminant colors
should be used in a visualization [8]; we added two more to
increase the difficulty of the task for the recoloring tools.
We chose the nine colors from the CIE LUV isoluminant
plane with luminance of 53.3 (obtained using default sRGB
transforms [12]). Eight colors were taken from equallyspaced points around a circle of radius 100 on this plane,
centered at the approximate center of the plane (Figure 7).
These eight plus the central color made up the set of nine.
Figure 9. Red-channel-varying colors used in the broken
monitor condition (left). Brightness-varying achromatic colors
used in the darkened monitor condition (right).
Excel task. The second task used the first fifteen chart
colors from Excel 2011 for Mac (version 14.1.3). Although
fifteen categorical colors in a visualization is an extreme
case, we chose these to introduce enough color
differentiation problems such that the tools would actually
perform some recoloring. These colors were the same for
all situations (see Figure 10).
Figure 7. Colors used in ‘normal’ situation. The isoluminant
circle used to generate colors (left); the resulting colors (right).
Tinted glasses. Here, the same procedure was used, but
colors were chosen from the blue corner of the isoluminant
plane. Blue colors were chosen because yellow glasses filter
out the blue end of the spectrum, so colors that vary in their
‘blueness’ will be difficult to tell apart (Figure 8).
Figure 8. Colors used in ‘tinted glasses’ situation. The isoluminant circle used to generate colors (left); resulting colors (right).
Broken monitor – no red. Nine colors that varied only in red
channel value were chosen by fixing the green and blue
channels at 127 (out of 256), and the red value was set to
one of 25, 51, 76, 102, 127, 153, 178, 204, or 229 (the
colors before removing red are shown in Figure 9, left).
This resulted (after red removal) in nine identical colors,
simulating an extreme color discrimination situation.
Darkened monitor. We chose nine equally-spaced
achromatic colors (i.e., r = g = b). Achromatic colors were
chosen because the settings used in the darkened monitor
situation affect the luminance range of the monitor without
affecting hue and saturation, so colors that vary only in
brightness were used (Figure 9, right).
Figure 10. Fifteen Excel colors used in all situations.
Study Design and Procedure
The study used a 4x6x2x2 mixed factorial design with three
within-participants factors (Situation, Recolorer, and Task),
and one between-participants factor (Congenital CVD). The
levels of these factors were:
• Situation: Normal, Tinted glasses, Dark monitor, Broken
monitor (no red).
• Recolorer: No recoloring, SSMRecolor, Daltonize, KuhnP, Kuhn-D, Kuhn-T.
• Task: Situation-Specific, Excel.
• Congenital CVD: CVD, nonCVD.
Dependent variables were accuracy (the number of trials
with a correct color match) and the time taken to respond.
Participants carried out tasks in each situation type (Latin
square counterbalanced). In each situation, the participant
performed an SSM calibration to capture their color
differentiation abilities for that specific situation.
Participants then carried out color-matching tasks as
described above. With three within-participants factors, and
either nine or fifteen trials per task, there were 4x6x(9+15)
= 576 trials in the study. At the end of the session,
participants filled out a demographics questionnaire. The
entire study (including HRR test) took less than 90 minutes.
Results
Analysis of Match Accuracy
We carried out an omnibus 4x6x2x2 RM-ANOVA to look
for effects of Situation, Recolorer, Task, and CVD on
match accuracy. The first three factors showed main effects
(Situation: F3,57=71.1, p<0.001; Recolorer: F5,95=81.7,
p<0.001; Task: F1,19=230.1, p<0.001). No difference was
found between CVD/non-CVD groups: F1,19=0.16, p=0.69).
example, a monitor with no red produces colors very
similar to what an individual with protanopia
experiences, so the Kuhn-Protan recolorer does well.
The significant effects of Situation and Task were expected,
because of the differences in the types of induced-CVD
situations and in the requirements for the 9-color and 15color tasks. No difference found between CVD and nonCVD participant performance is of interest, and suggests
that recoloring works in a similar fashion for both of these
groups, at least for the colors chosen for this experiment.
• Recolorer x Task (F5,95=24.9, p<0.001). Similarly,
different recolorers performed differently on different
tasks, e.g., Kuhn-D performed poorly in the tinted glasses
situation (Figs. 14 and 15). This recolorer assumes deutan
CVD, so it recolors by transferring red-green variations
to blue-yellow variations. However, the yellow tinted
glasses reduce perception of blue-yellow axis variations,
so this recolorer performed poorly for the nine-color task.
In the fifteen-color task, this transition from red-green to
blue-yellow still occurred, but the colors contain
variations in luminance which provide redundant cues
that allow better color differentiation.
Our main interest is in the differences between recolorers.
Overall results for the different recolorers are shown in
Figure 11; as can be seen in the figure, SSMRecolor
allowed participants to find 20% more correct matches than
the other tools.
• Recolorer x CVD (F5,95=2.83, p<0.05). There was a small
difference in performance between CVD and non-CVD
participants with the different recolorers. CVD
participants performed better overall than non-CVD
people when using the Kuhn-P and Kuhn-D tools. This is
because these tools are optimized for these participants'
congenital CVD, performing a ‘worst case’ recoloring
which significantly aided these participants.
Figure 11. Overall mean accuracy of participants, by recolorer.
Error bars indicate standard error. Asterisks indicate pairwise
significant differences (Bonferroni corrected) vs. SSMRecolor.
We carried out post-hoc pairwise t-tests between the
individual recolorers, using a Bonferroni correction to
maintain alpha of 0.05. These comparisons showed that
accuracy with SSMRecolor was significantly higher than
any of the other tools (all p<0.001). None of the other
comparisons showed any differences – in particular, no
recolorer was significantly better than no recoloring at all.
There were, however, several interactions among our
factors, and so these overall results must be interpreted in
light of these additional analyses. There were four
significant two-way interactions, detailed below and
illustrated in Figures 12-15.
• Situation x Task (F3,57=40.4, p<0.001). The different
situations had markedly different effects on the two tasks,
as seen by comparing Figures 12 and 13 (or 14 and 15 for
CVD participants). For example, tinted glasses had a
much stronger effect on the situation-specific task than
the Excel task. This is because the situation-specific task
was designed to highlight problems induced by the
situation. The Excel colors contain enough color variation
to reduce the effect of the situation.
• Situation x Recolorer (F15,285=28.0, p<0.001). Different
recolorers performed very differently in the different
situations, because the CVD induced by some situations
aligned well with some forms of congenital CVD. For
Figure 12. Nine-color situation-specific task accuracy for
nonCVD participants (in Figs 12-15: error bars show standard
error; asterisks indicate pairwise significant differences
(Bonferroni corrected) vs. SSMRecolor within a situation.)
Figure 13. Fifteen-color Excel task accuracy, nonCVD.
Figures 12-15 show results for different Recolorers in terms
of the different Situations. We also carried out separate
follow-up t-tests in each combination of factors shown in
the charts, to compare SSMRecolor with the other tools; in
the figures, asterisks (*) are placed on individual bars where
there was a significant difference between that tool and
SSMRecolor within the given situation.
Figures 17-18 show response times by recolorer and
situation to show differences indicated by the interactions.
Figure 14. Nine-color situation-specific task accuracy, CVD.
Figure 17. Response time (9-task), by recolorer and situation.
Asterisks indicate pairwise significant differences (Bonferroni
corrected) vs. SSMRecolor within a situation.
Figure 15. Fifteen-color Excel task accuracy, CVD.
Analysis of Response Time
We carried out a second 4x6x2x2 RM-ANOVA to look for
effects of the main factors on response time. Main effects
were found for factors Situation (F3,57=6.88, p<0.001) and
Recolorer (F5,95=17.8, p<0.001), but not for Task
(F1,19=0.12, p=0.74) or CVD (F1,19=1.52, p=0.23). There
were also significant interactions between Situation and
Recolorer (F15,285=4.71, p<0.001), and between Situation
and Task (F3,57=14.1, p<0.001).
Figure 16 shows the mean overall response times for the
different recolorers. Follow-up t-test comparisons between
different recolorers showed that SSMRecolor allowed
significantly faster response time than all of the other tools
(all p<0.001), and that there were no significant differences
between any of the other tools (or the no-recolor case).
Figure 18. Response time (15-task) by recolorer and situation.
DISCUSSION
Our evaluation provided three main results:
1. Accuracy of SSMRecolor is 20% higher than existing
recoloring tools.
2. Selection time for SSMRecolor is almost two seconds
faster than existing recolorers.
3. Increased accuracy and reduced selection time of
SSMRecolor is consistent across a variety of situations,
color sets and users.
Explanation of Main Results
SSMRecolor achieves consistently higher accuracy and
reduced selection time because the situation-specific model
more accurately represents the color differentiation abilities
of the user. By capturing the abilities of the user with a
performance-based in-situ calibration, any factors that
influence color abilities are automatically encoded into the
model. This allows accurate predictions to be made
regarding their ability to differentiate between two colors.
These predictions are the heart of SSMRecolor’s recoloring
algorithm, allowing both the accurate identification of
problem colors as well as the selection of sufficiently
differentiable replacement colors.
Is SSMRecolor Good Enough?
Figure 16. Mean response times (s) by recolorer. Error bars
indicate standard error. Asterisks indicate pairwise significant
differences (Bonferroni corrected) vs. SSMRecolor.
SSMRecolor did not achieve 100% accuracy for every
condition in the study described above. One possible reason
for this is that the situations used in the study were difficult
from a recoloring standpoint (i.e., for some users these
situations did not have a recoloring that would achieve
100% accuracy). If this is the case, then SSMRecolor
should do better in less extreme situations, and indeed that
is what we see (e.g., SSMRecolor achieved near 100%
performance for every ‘normal’ situation). Even in these
less extreme cases, however, perfect accuracy is unlikely –
for example, even in the control case with non-CVD
participants, people still made a few errors due to the
difficulty of the task or selection mistakes. Nevertheless,
there are potential improvements to SSMRecolor that could
increase its accuracy, as discussed below.
Frequency of Calibration
The situation-specific model of color differentiation relies
on an in-situ calibration. This calibration is required when
any of the factors that influence color perception. Although
the calibration is short (~2 min.), it is not reasonable to ask
the user to perform a new calibration whenever the
environment changes. Two ways of reducing calibration
frequency are storing calibrations for future use, and
expanding the applicability of a calibration.
Storing calibrations: A calibration is a set of numbers that
represent key characteristics of color perception, and can
therefore be stored and retrieved to use when similar
environmental conditions arise. By detecting environmental
conditions automatically (e.g., through sensors), an
appropriate calibration could be loaded automatically,
eliminating the need to recalibrate. This can be extended to
a central online repository of all users’ calibrations, further
reducing the frequency of calibration for each user.
• Luminance challenges. Any situation that results in
reduced luminance range and contrast (e.g., using a
mobile device in bright sunlight, using power-saving
reduced-brightness displays) can also benefit from SSMbased recoloring. The study results from the darkened
monitor situation show great promise for assisting users
in these low-contrast situations.
Extending SSMRecolor
Even though the SSM approach achieves higher accuracy
and reduced selection time, there are a number of changes
that can be made to improve it further. These include
preserving the naturalness of the original image, and
extending it to other uses of color.
Preserving Naturalness
Kuhn’s recoloring tool [11] attempts to maximize the
similarity between the original image and the recolored
image. This is achieved by restricting each replacement
color to the luminance of its respective key color, and by
attempting to maintain the visually-perceived difference
between key colors and between replacement colors. The
existing recoloring algorithm for SSMRecolor is quite naïve
by comparison, but can be extended to incorporate the
luminance and visually-perceived difference consistency as
in Kuhn’s method. To do this, the replacement color set can
be divided into a replacement color set for each key color,
such that each set is isoluminant with its respective key
color. Multiple recolor mappings can then be generated,
selecting the recoloring that maximally preserves the
visually-perceived differences in the original image.
Other Uses of Color
Expanding Applicability of each Calibration: Although
each calibration is for a specific user in a specific situation,
the exact influence of environmental changes (e.g., amount
of ambient light) on color differentiation is still poorly
known. As a result, it is unknown how the magnitude of an
environmental change affects our ability to differentiate
colors. Our previous work on SSMs explored this issue [6],
and suggested that SSMs are robust enough to handle small
variations in environmental conditions. This robustness can
be used to reduce the frequency of calibration by allowing
the system to extend the current calibration when conditions
change, rather than requiring a new calibration.
Current situation-specific models only encode basic color
differentiation abilities. Although differentiation is the
central aspect of many color uses, there are additional uses
of color that go beyond differentiation, such as popout
(using color for pre-attentive processing) and highlighting
(using color to draw attention) in information visualizations
[23]. The current version of SSMRecolor makes no effort to
preserve these perceptual properties through a recoloring,
but the SSM approach is extensible to other color uses in
visualization, allowing SSM-based recoloring to preserve
perceptual properties in addition to differentiability.
Generalizing the Results of the Evaluation
SSMs for color differentiation have a tunable parameter
(called the limit offset) that can be used to adjust how
conservative the model is in its predictions of color
differentiability. Increasing the limit offset causes colors
that are actually differentiable to be predicted as not
differentiable. This parameter can be used in SSMRecolor
to adjust the number of original colors that are modified,
and to adjust the differentiability of the replacement set of
colors. By increasing the limit offset, more of the original
colors will be flagged as not differentiable, resulting in their
subsequent recoloring. An increased limit offset will also
cause colors that are more differentiable to be used as
replacements. When the limit offset is small, a minimal
Although our study examined specific types of situationinduced CVD, our findings should extend to real-world
situations that are similar to our experimental conditions.
• Hue variations. Any condition or situation that causes the
perception of hue to change (e.g., cataracts, retinopathy,
colored lighting, tinted sunglasses, uncalibrated hardware)
can benefit from SSM-based recoloring. Our evaluation
shows that SSM recoloring is applicable to a wide range
of situations (including extreme situations), and should
significantly improve the differentiability of colors on
digital displays in these situations.
Adjustable Recoloring
number of colors will be modified; this can help maintain
the naturalness of the original image. With a large limit
offset, many colors will be modified, and the resulting
replacement colors will be more differentiable.
CONCLUSIONS AND FUTURE WORK
We presented a new recoloring tool – SSMRecolor – based
on situation-specific modeling of color differentiation. This
tool was shown to be consistently more accurate and to
provide shorter selection time than existing recoloring tools
across a number of different situations. Our work
demonstrates the value of a situation-specific approach to
recoloring, and shows that the technique can substantially
improve the usability of color displays for users of all types.
In the future, we have two main goals: deployment, and
improved speed. First, SSMRecolor provides assistance for
individuals with both congenital and situationally-induced
CVD, and we believe this success will generalize to
acquired CVD. Because this recoloring tool covers all three
types of CVD, we plan to develop and deploy a real-world
version of this system. This will allow us to further test the
approach with real-world images and color-differentiation
tasks, and will allow broader collection of data.
Second, we will improve the speed of SSMRecolor.
Currently, the tool takes a few seconds to perform color
replacement on a standard PC, which is acceptable for static
assistance (e.g., reading charts), but is not fast enough for
real-time applications (e.g., live video or 3D gaming). Our
goal is to extend the recoloring technique to work in real
time. We plan to use pixel shaders to construct a transparent
‘SSMRecolor overlay’ that can be placed on top of the
interface of any running application, to modify its colors as
they are displayed.
ACKNOWLEDGEMENTS
We would like to thank Giovane Kuhn for providing us
with executable versions of his recoloring tool.
REFERENCES
1. Birch, J. Diagnosis of Defective Colour Vision, 2nd ed.
Oxford University Press, 2001.
6. Flatla, D.R., Gutwin, C. Individual Models of Color
Differentiation to Improve Interpretability of
Information Visualization. Proc. CHI 2010, 2563-2572.
7. Hardy, L.H., Rand, G., Rittler, M.C. H-R-R Polychromatic Plates. J. Opt. Soc. Am. 44, 1954, 509-521.
8. Healy, C. Choosing Effective Colours for Data
Visualization. Proc. IEEE VIS 1996, 263-270.
9. Jefferson, L., Harvey, R. Accommodating Color Blind
Computer Users. Proc. ASSETS 2006, 40-47.
10. Jefferson, L., Harvey, R. An Interface to Support Color
Blind Computer Users. Proc. CHI 2007, 1535-1538.
11. Kuhn, G.R., Oliveira, M., Fernandes, L. An Efficient
Naturalness-Preserving Image-Recoloring Method for
Dichromats. IEEE Trans. Vis. Comp. Graph., 14(6),
2008, 1747-1754, www.inf.ufrgs.br/~oliveira/
pubs_files/ CVD_Mass_Spring/CVD_Mass_Spring.html
12. Lindbloom, B. www.brucelindbloom.com (09/22/2011).
13. Lomax, R., Ridgway, P. Meldrum, M. Does
Occupational Exposure to Organic Solvents Affect
Colour Discrimination? Tox. Rev., 23(2), 2004, 92-121.
14. Machado, G., Oliveira, M., Fernandes, L. A
Physiologically-based Model for Simulation of Color
Vision Deficiency. IEEE Trans. Vis. Comput. Graphics,
15(6), 2009, 1291-1298.
15. Meyer, G., Greenburg, D. Color-Defective Vision and
Computer Graphics Displays. IEEE Comput. Graph.
Appl., 8(5), 1988, 28-40.
16. Neitz, M., Neitz, J. Numbers and Ratios of Visual
Pigment Genes for Normal Red-Green Color Vision.
Science, 267, 1995, 1013-1016.
17. Rasche, K., Geist, R. Westall, J. Detail Preserving
Reproduction of Color Images for Monochromats and
Dichromats. Comput. Graph. Appl., 25(3), 2005, 22-30.
18. Rasche, K., Geist, R., Westall, J. Re-coloring Images for
Gamuts of Lower Dimension. Comput. Graph. Forum,
24(3), 2005, 423-432.
19. Stone, M. A Field Guide to Digital Color. AK Pet. 2003.
2. Birch, J. Extreme Anomalous Trichromatism. In
Mollon, Pokorny, Knoblauch, eds., Normal and
Defective Colour Vision, Oxford Press, 2003, 364-369.
20. Vienot, F., Brettel, H., Ott, L., Ben M’Barek, A.,
Mollon, J. What Do Colour-Blind People See? Nature,
376, 1995, 127-128.
3. Brettel, H., Vienot, F., Mollon, J. Computerized
Simulation of Color Appearance for Dichromats. J. Opt.
Soc. Am. A., 14(10), 1997, 2647-2655.
21. Wakita, K., Shimamura, K. SmartColor: Disambiguation
Framework for the Colorblind. Proc. ASSETS 2005,
158-165.
4. Cole, B. The Handicap of Abnormal Colour Vision.
Clin. and Expir. Optom., 87(4-5), 2004, 258-275.
22. Wandell, B. Foundations of Vision. Sinauer Assoc., 1995.
5. Flatla, D.R., Gutwin, C. Improving Calibration Time
and Accuracy for Situation-Specific Models of Color
Differentiation. Proc. ASSETS 2011, 195-202.
23. Ware, C. Information Visualization: Perception for
Design, 2nd ed. Morgan Kaufmann, 2004.
24. Yates, J., Diamantopoulos, I., Daumann, F. Acquired
(Transient and Permanent) Colour Vision Disorders. In
Operational Colour Vision in the Modern Aviation
Environment, NATO Tech. Rep., 16, 2001, 43-47.