Camera Phone Color Appearance Utility
Finding a Way to Identify Color
Phillip Lachman
Robert Prakash
Elston Tochip
Outline



Motivation
Goal
Methodology






Image Scaling via Edge Detection
Color Identification
Color Selection & Differentiation
Results
Lessons Learned
Future Work
Motivation


Phones becoming the portal able digital platform
for variety of imaging applications i.e. pictures,
video, organizers etc.
Approximately 10 million blind people within the
U.S.


55,200 legally blind children
5.5 million elderly individuals
http://www.afb.org/Section.asp?sectionid=15#num

Color blind people within the U.S.


~ 8% of males
~ 0.4% - 2% females
http://www.otal.umd.edu/UUPractice/color/
Goal

To develop a software application
that will be able to accomplish the
following:
1) Receives a camera quality image
2) Identifies the predominant color(s)
regions within the image
3) Estimates the color name for the
predominant region
4) Audibly transmits the predominant
color to the user
Locating the Target
General Guidelines and Suggestions

Use a White Card

Provides a white color to baseline lighting conditions



Required for computing color of target
Suggested by fellow classmates and Bob Dougherty
Detecting the White Card

Use an Edge Detection Algorithm

Many Image Processing Edge detection methods available


Identify edges by computing changes in gradient around pixels.
Chose Canny Edge Detection algorithm

Fundamentally easy to understand and implement
Edge Detection
Finding the Target: Discrimination

How does the algorithm discriminate the target being
photographed?


Background clutter and scenery complicate the image
Discrimination Solution

The White Card “Scope”


Use the rectangular white card with a square target hole to allow
object color through
Use Edge Detection image processing algorithms to find the white
card

Find the white card, find the target!
Card
Background
Target
Color
Target
Hole
Finding the Target: Discrimination Cont

The White Card Problem

White backgrounds or light color backgrounds cause
edge detection problems in Canny Algorithm
Original Image
After Edge Detection
Where is the card??
Finding the Target: Discrimination Cont

The White Card Problem Cont.

Adding a Black Outline to the card edges and target
hole greatly improve detection!
Original Image
After Edge Detection
There’s the card!!
Finding the Target: Aiming the Camera

How does a blind person AIM the camera to take a
picture of the target?



Photographs may NOT include target
Photographing target too close may not allow enough lighting to
determine color
Aiming Solution- White Card Holder



Use a phone attachment which holds the white card AND attaches to
phone camera
Guarantees white card and target in the camera Field of View
Guarantees camera is not directly on top of the object, providing
ample lighting for color detection
Camera
White
Card
Hinge Assembly to allow
for folding
Mount
Baseboard
Finding the Target: Aiming the Camera

Additional Benefits of Card Holding Device

Fixes Orientation of the card



Chose to have card positioned vertically with edges
parallel to photo edges
Simplifies algorithm detection, increasing speed
Removes excess background scenery

Device maintains a fixed 6-8 inches between camera
and white card

Scene is dominated by white card and maximizes number of
pixels covering the target
Finding the Target: Examples with and
without device
Finding the Target: Edge Detection
Algorithm

Phase 1: Blurring and Sharpening Edges


Preprocess Images to blur and eliminate noisy pixels
Apply a 3x3 Laplacian Matrix Kernel to resulting image



Kernel is an approximation of the second derivative,
highlighting changes in intensity
Matrix = -1 -1 -1
-1
8
-1
-1
-1
-1
Adding results of Gaussian Blurring and Laplace yields
image with cleaner and more distinct lines
Edge Detection: Blurring and
Sharpening
Original Image
After Smoothing and Laplace
Applying Laplace removes
Noise and Smoothes lines
Finding the Target: Edge Detection
Algorithm

Phase 2: Apply Canny Edge Detection
Algorithm

Step 1: Applies Gaussian smoothing in 2 dimensions
to the image via convolution



Size of the mask = 20x20 with a sigma = 5
Step 2: Compute the resulting gradient of the
intensities in the image
Step 3: Threshold the norm of the gradient image to
isolate edge pixels
Edge Detection: Applying Canny Edge
Detection Algorithm
Original Image
After Canny and Threshold
Finding the Target: Edge Detection
Algorithm

Phase 3: Finding the edges in the photo

Step 1: Recursively search, row by row, from the
outer left/right edges of the image towards the
center.

Search for 1 quarter length from right/left side
Target
Hole
Left
Fourth
Card
Finding the Target: Edge Detection
Algorithm

Phase 3 Cont:

Step 2: Bin and compute outer edges based on
which values are closest to the center

Find left/right edges based on bin having AT LEAST 10%
of the total pixels available on the each side
First bin
From Center
Exceeding 10%
First bin
From Center
Exceeding 10%
Right Side

Left Side
Step 3: Compute Top and Bottom Edge

Compute the average row at which both the left and right
edges begin/end

Value gives rough estimate of top/bottom of white card
Finding the Target: Target Hole
Detection Algorithm

Outer Target White Card edges Located


Proceed to Locate INNER Target Hole edges
Phase 4: Identifying Inner Target Hole Edges




Step 1: Crop Canny Thresholded image to dimensions
obtained for outer edge
Step 2: Perform recursive row by row outside to inside
search until a high threshold is found on both sides.
Step 3: Bin and compute left/right edges as before
Step 4: Compute Top and Bottom edges as before
Finding the Target: Output to Color
Detection

Phase 5: Compute overall Target Hole Position in
ORIGINAL image


Sum up inner and outer edge values computed previously
Crop the original image to these dimensions and output to
Color Detection model
Color Detection: Original Idea

Keep it simple:



Use brightest point on white card as
white point.
Normalize R,G and B separately.
Good results, slight reddish tinge.
10
10
20
20
30
30
40
40
50
50
60
60
70
70
80
80
90
90
100
100
20
40
60
80
100
120
140
160
180
200
220
20
40
60
80
100
120
140
160
180
200
220
Color Detection: “Gray World”

Use “Gray world” theory:


Normalize means of RGB to 128
Results slightly better in low lighting
conditions, but less effective under
good lighting.
10
10
20
20
30
30
40
40
50
50
60
60
70
70
80
80
90
90
100
100
20
40
60
80
100
120
140
160
180
200
220
20
40
60
80
100
120
140
160
180
200
220
Color Identification: Original Idea

Keep it simple: Bin using RGB

Problem:
No clear grouping
 Small changes in one value, changes color
dramatically


Solution:
Attempt to identify groups by using max
of R, G and B values.
 Still contained overlaps

Color Identification: CIE- L*a*b

Convert RGB to CIELab

Benefits:


Device independent.
Problems:
Conversion formulas complicated and
processor intensive.
 Light source information is required.


Solution: use HSV
Color Identification: Use HSV

Convert RGB to HSV

The HSV (Hue, Saturation, Value) model
is a simple transformation from RGB.

Hue, the color type (like red, blue, etc)


Saturation, the "vibrancy" of the color:


ranges from 0-360
Ranges from 0-100%
Value, the brightness of the color:

Ranges from 0-100%
Color Identification: RGB to HSV

Equations used for conversion
Color Selection & Differentiation


Currently code identifies 24 colors
based on HSV color system.
Color identification is acceptable,
but starts to fail in low lighting
conditions.
Results
White
Light Green
White
Indian Red
Lessons Learned

Edge Detection: Think about the Big
Picture

User Feasibility is Critical


If a soldier cannot aim a gun, how accurate is
his shot?
Simplicity is Essential


Presetting card orientation led to efficiency and
shortcuts for edge detection
Slanted Orientation requires much more
processing time and development

Original code variants tried to and failed to account
for all orientations
Lessons Learned

Color Detection:



HSV is a compromise between simply
binning on RGB values and conversion to
L*a*b.
Normalization using the white point more
effective than “gray world”.
Minimum level of lighting in required,
since camera is low quality.
References

http://robotics.eecs.berkeley.edu/~mayi/imgproc/cacode.html

http://homepages.inf.ed.ac.uk/rbf/HIPR2/canny.htm

http://www.aquaphoenix.com/lecture/matlab10/page3.html

http://en.wikipedia.org/wiki/Canny

http://www.afb.org

http://www.otal.umd.edu/UUPractice/color/

Class notes on Color and jpeg tutorials
Future Work




Implementing the processing onto an
actual camera phone
Decreasing the processing time to audibly
deliver the color to the user
Increasing color library
Refining overall algorithm to distinguish
more detailed backgrounds.



Patches
Patterns
Color Designs