ColorBars: Increasing Data Rate of LED-to-Camera
Communication using Color Shift Keying
Pengfei Hu, Parth H. Pathak, Xiaotao Feng, Hao Fu and Prasant Mohapatra
ACM CoNEXT 2015
Outline
➢
Visible Light Communication (VLC) background
○ LED-to-camera communication, applications
➢
Limitations of existing approaches
○ Data rate and reliability
➢
Our approach - Color shift keying
➢
Challenges and ColorBars system design
○ Color flicker, inter-frame loss and receiver diversity
➢
Performance evaluation
○ RGB LEDs, Android and iPhone
VLC - Background
➢
Utilize 430 - 790 THz spectrum for communication
Transmitter - LEDs
Receiver - photodiode,
image sensor
LED-to-Camera Communication
➢
Transmitter identity and information association
○ Users can identify an LED transmitter and receive information specifically
transmitted by that LED
○ Difficult with most types of RF communication and NFC
➢
Applications
○ Cameras on smartphones and wearables like smart-glasses
○
In retail stores, user can visualize LED on merchandise and receive more
information about it
○
Location-specific broadcast - LED lights in an office can broadcast floor map
and navigation information
Rolling Shutter
➢
CMOS image sensors
○ Matrix of photodiodes
○ Commonly used in mobile devices
➢
Rolling shutter phenomenon
○ Scan one column of pixels at a time
○ Subsequent scanning enables communication
➢
Example: ON-OFF Keying
Limitations of Existing Schemes
➢
Current LED-to-camera communication techniques have many limitations
○ Low data rate
○ Low reliability due to inter-frame gap
○ Human perceivable flickering
➢
Two existing approaches
ON-OFF Keying (OOK)
-
Ambient light
interference
ON-OFF flickering
Frequency Shift Keying (FSK)
-
Longer symbol
duration
< 12 bytes/sec. [1,2]
Our Approach
➢
Color Shift Keying (CSK)
○ Use different colors as data
symbols
○ Especially suitable for camera
receivers
➢
How?
○ Use RGB tri-LED instead of white
LED (Blue LED with yellow
phosphor coating)
○ Pulse Width Modulation for
mixing R, G and B colors
CSK Constellation Design
➢
Originally proposed in IEEE 802.15.7 standard
○ Constellation based on CIE 1931 color space
8 CSK
16 CSK
Challenges - CSK for LED-to-Camera Communication
➢
Color flicker
○ LED’s primary purpose is illumination
○ Guaranteeing white light even when using CSK
○ Avoiding human perceivable color or color change
Challenges - CSK for LED-to-Camera Communication
➢
Color flicker
○ LED’s primary purpose is illumination
○ Guaranteeing white light even when using CSK
○ Avoiding human perceivable color or color change
➢
Inter-frame data loss
○ Inter-frame gap of camera receivers
○ Unidirectional communication
○ Reliable data delivery
Challenges - CSK for LED-to-Camera Communication
➢
Color flicker
○ LED’s primary purpose is illumination
○ Guaranteeing white light even when using CSK
○ Avoiding human perceivable color or color change
➢
Inter-frame data loss
○ Inter-frame gap of camera receivers
○ Unidirectional communication
○ Reliable data delivery
➢
Receiver diversity
○ Differences in cameras based on color filters, types and arrangements
○ Same transmitted color captured differently by different cameras
○ Reduce demodulation errors
Challenges - CSK for LED-to-Camera Communication
➢
Color flicker
○ LED’s primary purpose is illumination
○ Guaranteeing white light even when using CSK
○ Avoiding human perceivable color or color change
➢
Inter-frame data loss
○ Inter-frame gap of camera receivers
○ Unidirectional communication
○ Reliable data delivery
➢
Receiver diversity
○ Differences in cameras based on color filters, types and arrangements
○ Same transmitted color captured differently by different cameras
○ Reduce demodulation errors
Avoiding Color Flicker
➢
Temporal summation
○ Human eye “accumulates” incoming photons for a period of time until it
saturates
○ The time period is called “critical duration”
➢
Luminance and color perception
○ The perceived color is the average of the received colors during the critical
duration due to the temporal summation (Bloch’s law of vision)
Avoiding Color Flicker
➢
Difficult to ensure equal proportion of R, G and B within critical duration
➢
Dedicated illumination symbols
○ Inserted periodically
○ Amount depends on symbol frequency, also subjective
Higher symbol freq. =>
more demodulation errors
10 volunteers
1000 Sym./sec.
3000 Sym./sec.
Challenges - CSK for LED-to-Camera Communication
➢
Color flicker
○ LED’s primary purpose is illumination
○ Guaranteeing white light even when using CSK
○ Avoiding human perceivable color or color change
➢
Inter-frame data loss
○ Inter-frame gap of camera receivers
○ Unidirectional communication
○ Reliable data delivery
➢
Receiver diversity
○ Differences in cameras based on color filters, types and arrangements
○ Same transmitted color captured differently by different cameras
○ Reduce demodulation errors
Inter-frame Data Loss
➢
Camera receiver
○ Data transmitted during inter-frame gap is lost
○ Unidirectional communication from LED to camera
➢
Reed-Solomon error correction coding
○ RS(n,k) can correct t bit errors where 2t = n - k, n is the size of the codeword
and k is number of data bits
Size of a symbol
(in bits)
illumination ratio
Number of symbols
received between two
consecutive frames
Number of symbols lost
between two consecutive
frames
Packetization
➢
Data, parity and illumination symbols are encapsulated in a packet
○ Packet header includes
- Flag indicating data or calibration packet
- Size of the packet
➢
Size of a packet is chosen to be total size of a frame and inter-frame gap
Challenges - CSK for LED-to-Camera Communication
➢
Color flicker
○ LED’s primary purpose is illumination
○ Guaranteeing white light even when using CSK
○ Avoiding human perceivable color or color change
➢
Inter-frame data loss
○ Inter-frame gap of camera receivers
○ Unidirectional communication
○ Reliable data delivery
➢
Receiver diversity
○ Differences in cameras based on color filters, types and arrangements
○ Same transmitted color captured differently by different cameras
○ Reduce demodulation errors
Receiver Diversity
CIELab color space
Camera diversity
➢
Camera exposure time and ISO also affect the captured color
➢
Calibration packets
○ Transmitted periodically
○ Includes all symbols of current CSK modulation
○ Receiver uses the symbol colors for matching during demodulation
Demodulation
➢
Demodulation uses CIELab color space
○ Color is converted from RGB to {a, b}
➢
Brightness is non-uniformly distributed in
the captured frame
○ Dimension L is removed
CIELab color space
➢
Colors of calibration packets used for
matching (Euclidean distance in a,b-plane)
Performance Evaluation - Experiment Setup
➢
Transmitter
○ BeagleBone Black (1 GHz processor, 512
MB memory)
○ Off-the-shelf RGB tri-LED
○ Maximum frequency of color change approx. 4500 Hz
○ PWM controller for color generation
➢
Receiver
○ Nexus 5 (Android) and iPhone 5S (iOS)
○ Nexus 5 - Android app for decoding, demodulation and error correction
○ iPhone 5S - captured video processed offline
Inter-frame Loss Ratio
➢
Cameras on Nexus 5 and iPhone5S
○ Frame rate - 30 frames per second
Transmission Symbol
Rate (Hz)
Received Symbol Rate (Hz)
Nexus 5
iPhone 5S
1000
772.84
640.55
2000
1506.11
1263.56
3000
2352.65
1887.73
4000
3060.67
2431.01
Avg. Inter-frame Loss
Ratio
0.23
0.37
Symbol Error Rate (SER)
➢
Fraction of symbols incorrectly demodulated
○ Low SER for 4 and 8 CSK even at high symbol frequency
○ High inter-symbol interference at 16 and 32 CSK
Nexus 5
iPhone 5S
Throughput
➢
Number of received symbols without any error correction
○ Excluding the illumination symbols
➢
Lower throughput of iPhone 5S due to its larger inter-frame gap
Nexus 5
iPhone 5S
Goodput
➢
RS decoding and error correction
➢
Lower goodput of iPhone 5S is due to higher overhead of parity bits (larger interframe gap)
Nexus 5
5.2 Kbps
iPhone 5S
2.5 Kbps
Conclusions and Future Work
➢
CSK is an attractive scheme for LED-to-camera communication
○ Higher achievable data rate than OOK and FSK
➢
Design of ColorBars
○ Flicker free operation, reliable communication
➢
Open challenges and future work
○
Larger LED - camera distance
- programmable LED with higher lumens
○
Optimizing CSK constellation based on other factors
- Camera type, ambient light
Thank you!
Questions/comments?