RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Demosaicing with Improved
Edge Direction Detection
Presented By:
Anthony Karloff
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Overview
•
•
•
•
•
Demosaicing Background
Basics and Challenges
Advanced Methods (State of the Art)
Color Channel Reconstruction
Conclusion
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Demosaicing Background
Background
Basics
Advanced
Methods
Why Image Reconstruction?
•
Incomplete color planes from CCD sensors.
Channel
Reconstruction
Conclusion
Color Filter Array (CFA) on image sensor.
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Basics and Challenges
Background
Basics
Advanced
Methods
Color Plane Interpolation
• Must Interpolate color planes to re-create
image.
Channel
Reconstruction
Conclusion
Red Channel Interpolation
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Basics and Challenges
Background
Basics
Advanced
Methods
Color Plane Interpolation Methods
•
Pixel Averaging
- lose image resolution
R1 G1
P1
G2 B1
Channel
Reconstruction
Conclusion
•
•
Nearest Neighbor
- Poorest Quality
Bilinear/Spline
- Color artifacts at edges
R1 G1
P1
G2 B1
G1 B1 G2
R1 G3 R2
G4 B2 G5
P1
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Basics and Challenges
Background
Basics
Advanced
Methods
Color Artifacts
•
Problem with most simple interpolation
algorithms is the presence of color artifacts.
Channel
Reconstruction
Conclusion
Original Image
Bilinear Interpolation of Bayer Image
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Basics and Challenges
Background
Basics
Advanced
Methods
Color Artifacts
•
Due to interpolation across edges.
Channel
Reconstruction
Conclusion
Artifacts
P1 P2 P3 P4 P5
P6 P7 P8 P9 P10
Bilinear
Interpolation
P1 P2 P3 P4 P5
P6 P7 P8 P9 P10
P11 P12 P13 P14 P15
P11 P12 P13 P14 P15
P16 P17 P18 P19 P20
P16 P17 P18 P19 P20
P21 P22 P23 P24 P25
P21 P22 P23 P24 P25
Dark to Light Edge
over Bayer Pattern
Resulting Edge
after Interpolation
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Advanced Methods
Background
Basics
Advanced
Methods
Channel
Reconstruction
Conclusion
Advanced Techniques for Color
Plane Interpolation
•
Use color plane gradients
•
Group pixels of similar objects
•
Interpolate along edges (not across)
•
Interpolate green color plane first
•
Interpolate image more than one iteration
(refinement)
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Advanced Methods
Background
Basics
Advanced
Methods
Channel
Reconstruction
Conclusion
Using Gradients for Image
Reconstruction
• Better estimation of color plane behavior.
P1 P2 P3
P4 P5 P6
P7 P8 P9
Bayer Pattern for Green
Centered Pixel
1. GRADIENTS
•
Dx ( P5) 
P 4  P6
2
Dy ( P5) 
P 2  P8
2
Dxd ( P5) 
P3  P 7
2 2
Dyd ( P5) 
P1  P9
2 2
Notice that the differences are always from
the same color plane.
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Advanced Methods
Background
Basics
Advanced
Methods
Channel
Reconstruction
Conclusion
Kimmels ‘E’ Function for Pixel
Grouping
• Associates colors of the same object.
P1 P2 P3
P4 P5 P6
P7 P8 P9
Bayer Pattern for Green
Centered Pixel
•
1. GRADIENTS
2. GROUPING
Ie. If P5 and Pi are part of the same
object, E will be close to unity.
Ei( P5) 
1
1  Di ( P5) 2  Di ( Pi) 2
There are eight Ei values for each pixel. One
for each neighbor.
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Advanced Methods
Background
Basics
Advanced
Methods
Using Edge Detection (Wide)
•
Interpolation is best performed in the same
direction as an edge.
Channel
Reconstruction
P1 P2 P3 P4 P5
Conclusion
P6 P7 P8 P9 P10
1. GRADIENTS
Edge detection of radius 3
P11 P12 P13 P14 P15
HG ( P13)  P12G  P14G
P16 P17 P18 P19 P20
VG ( P13)  P8G  P18G
P21 P22 P23 P24 P25
H R ( P13)  P11R  P15R  2  P13R
Bayer Pattern for Red
Centered Pixel
VR ( P13)  P3R  P23R  2  P13R
2. GROUPING
3. EDGE DETECT 3 pxls
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Advanced Methods
Background
Basics
Advanced
Methods
Channel
Reconstruction
Conclusion
Narrow Edge Detection
• Uses narrow edge detection to improve
edges by looking between color planes.
P1 P2 P3 P4 P5
P6 P7 P8 P9 P10
P11 P12 P13 P14 P15
P16 P17 P18 P19 P20
P21 P22 P23 P24 P25
1. GRADIENTS
2. GROUPING
Edge detection of radius 2
Bayer Pattern for Red
Centered Pixel
3. EDGE DETECT 3 pxls
4. EDGE DETECT 2 pxls
HGR ( P13)  P12G  P14G  2P13R
VGR ( P13)  P2G  P8G  2P13R
HGB ( P13)  12  P7 B  P9B  2P8G  P17 B  P19B  2P18B 
VGB ( P13)  12  P7 B  P17 B  2P12G  P9B  P19B  2P14B 
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Advanced Methods
Background
Basics
Advanced
Methods
Channel
Reconstruction
Local Inter-Channel Correlation
• Compare average color differences in a 5x5
region to determine whether the Red or Blue
channel is more closely related to the green.
Conclusion
CGR  G 5 x 5  R 5 x 5
CGB  G 5 x 5  B 5 x 5
P1 P2 P3 P4 P5
P6 P7 P8 P9 P10
P11 P12 P13 P14 P15
1. GRADIENTS
2. GROUPING
3. EDGE DETECT 3 pxls
4. EDGE DETECT 2 pxls
5. COLOR CORRELATION
P16 P17 P18 P19 P20
P21 P22 P23 P24 P25
Bayer Pattern for Red
Centered Pixel
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Advanced Methods
Background
Basics
Advanced
Methods
Channel
Reconstruction
Conclusion
Improved Edge Detector
• Now we can complete the edge detector
3
4
H
H  H R  H G   GR
H GB
V
V  VR  VG   GR
VGB
5
if
CGR  CGB
otherwise
if
CGR  CGB
otherwise
1. GRADIENTS
2. GROUPING
3. EDGE DETECT 3 pxls
4. EDGE DETECT 2 pxls
5. COLOR CORRELATION
6. IMPROVED EDGE DETECTION
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Channel Reconstruction
Background
Basics
Advanced
Methods
Channel
Reconstruction
Conclusion
Channel Reconstruction Overview
• For each pixel we now have: Ei(Pi) H V
• Approximate the red and blue channels
using Bilinear Interpolation.
•
Reconstruct the green channel using edge
detectors and the approximated red and blue.
•
Reconstruct the red and blue channels using
the complete green channel.
1. GRADIENTS
2. GROUPING
3. EDGE DETECT 3 pxls
4. EDGE DETECT 2 pxls
5. COLOR CORRELATION
6. IMPROVED EDGE DETECTION
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Channel Reconstruction
Background
Basics
Advanced
Methods
Channel
Reconstruction
Conclusion
1. GRADIENTS
2. GROUPING
Green Channel Reconstruction
•
For each Green pixel on a red center…


 E P12 ( P12G  P12 R ' )  E P14 ( P14G  P14 R ' ) if
P7 P8 P9

E P12  E P14

P12 P13 P14
 E ( P8G  P8 R ' )  E P18 ( P18G  P18 R ' )
P13G  P13R   P 8
if
E

E
P8
P18

P17 P18 P19
E
(
Pi


Pi
G  PiR ' )
i 8,12,14,18

Bayer Pattern for Red

 EPi

Centered Pixel
i 8 ,12,14,18
H   V
H  V
otherwise
• A similar approach is taken to the finding the
green value at a blue centered pixel
3. EDGE DETECT 3 pxls
4. EDGE DETECT 2 pxls
5. COLOR CORRELATION
6. IMPROVED EDGE DETECTION
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Channel Reconstruction
Background
Basics
Advanced
Methods
Blue and Red Channel Reconstruction
•
Blue and red channels are then completed
using the full green channel.
Channel
Reconstruction
P1 P2 P3 P4 P5
Conclusion
P6 P7 P8 P9 P10
P11 P12 P13 P14 P15
P13B  P13G 
Where…
E 7  K 7  E 9  K 9  E17  K17  E19  K19
E 7  E 9  E17  E19
Ki  PiB  PiG
P16 P17 P18 P19 P20
P21 P22 P23 P24 P25
1. GRADIENTS
Bayer Pattern for Red
Centered Pixel
Similar approach is taken for completing
Red channel.
2. GROUPING
3. EDGE DETECT 3 pxls
4. EDGE DETECT 2 pxls
5. COLOR CORRELATION
6. IMPROVED EDGE DETECTION
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
Conclusion
Background
Basics
Conclusion
Advanced
Methods
•
Highly computational and hence slow.
Channel
Reconstruction
•
•
Not suitable for real-time applications.
•
Improved Edge Quality.
Conclusion
Drastically reduces color artifacts.
Thank You
RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR
References
Background
Basics
Advanced
Methods
[1] D. Darian Muresan, S. Luke, and T. W. Parks, “Reconstruction of Color Images From CCD Arrays,” Cornell
University, Ithaca NY. 1485,
Channel
Reconstruction
[2] R. Kimmel, “Demosaicing: Image Reconstruction from Color CCD Samples” IEEE Transl. J. Image
Processing, vol. 8, Sept. 1999.
Conclusion
[3] Xiaomeng Wang, Weisi Lin, Ping Xue, “Demosaicing with Improved Edge Direction Detection” IEEE Transl.
J. Image Processing, 2005.