ICCP
ICCP
#5
#5
ICCP 2014 Submission #5. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
000
001
002
003
004
005
006
Phase Messaging Method for Time-of-flight Cameras
007
008
009
010
011
Paper ID 5
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
Anonymous ICCP submission
1. Introduction
As cameras become ubiquitious, communication between displays and cameras has become an active research
area. Visible-Light Communication (VLC) has been developed where standard cameras detect patterns intentionally
placed within images. These patterns can be fixed as in QRcodes [14], dynamic as in Visual MIMO, [4, 27] or LEDto-LED applications [22], and coded using OFDM as in
Pixnet [17, 21]. The main theme in these methods, is communication by modulating light intensity from an observed
059
060
061
062
063
064
065
066
067
068
069
Abstract
Ubiquitous light emitting devices and low-cost commercial digital cameras facilitate optical wireless communication system such as visual MIMO where handheld
cameras communicate with electronic displays. While
intensity-based optical communications are more prevalent
in camera-display messaging, we present a novel method
that uses light phase for messaging and time-of-flight (ToF)
cameras for receivers. With intensity-based methods, light
signals can be degraded by reflections and ambient illumination. By comparison, communication using ToF cameras
is more robust against challenging lighting conditions. Additionally, the concept of phase messaging can be combined
with intensity messaging for a significant data rate advantage. In this work, we design and construct a phase messaging array (PMA), which is the first of its kind, to communicate to a ToF depth camera by manipulating the phase
of the depth camera’s infrared light signal. The array enables message variation spatially using a plane of infrared
light emitting diodes and temporally by varying the induced
phase shift. In this manner, the phase messaging array acts
as the transmitter by electronically controlling the light signal phase. The ToF camera acts as the receiver by observing and recording a time-varying depth. We show a complete implementation of a 3 × 3 prototype array with custom
hardware and demonstrating average bit accuracy as high
as 97.8%. The prototype data rate with this approach is 1
Kbps that can be extended to approximately 10 Mbps.
054
055
056
057
058
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
Figure 1. An example application of “smart signage” that communicates to cameras in the environment with infrared LED’s
that send invisible messages with light phase for communication
with time-of-flight depth cameras. These cameras are increasingly
more commonplace and can be expected in mobile robotics and
devices. The LEDs are depicted large for clarity but would be unnoticeable in a real application.
image. Recently, high quality depth cameras using timeof-flight (ToF) have become readily available at low-cost,
creating an entirely new opportunity for communications.
Creating a depth-pattern is a more challenging optical task
when compared to intensity patterns. However, since depth
is measured with ToF cameras using the phase of reflected
light, manipulating the phase of the reflected light signal effectively changes the measured depth, resulting in a phasebased message.
The applications for depth cameras are numerous since
most intelligent sensing applications can benefit from scene
geometry information. We develop a novel method for com-
088
089
090
091
092
093
094
095
096
097
098
099
100
101
102
103
104
105
106
107
ICCP
ICCP
#5
#5
ICCP 2014 Submission #5. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
108
120
121
122
123
124
125
ToF Camera
ToF Camera
Distance d
Distance d
(a)
ToF Camera
ToF Camera
Customized Phase
Customized
MessagingPhase
Array
Messaging Array
d1
d1
d2
d2
dn
dn
(b)
126
127
128
129
130
131
Figure 2. Phase Messaging Array (PMA) method. (a) A customized PMA with n LEDs is physically at distance d away from
a ToF camera. (b) By modulating phases for LEDs on PMA, LEDs
“seem” at individual distances away in depth images from the ToF
camera. Messages are represented by these measured distances.
132
133
134
135
136
137
municating with depth cameras using the phase of infrared
(IR) light. The transmitter arrays can be used as beacons,
tags or communication channels that interact with RGBD cameras for real world applications. One can envision
“smart signage” with invisible light messaging to cameras
in the environment (see Figure 1); vehicle id tags transferring information to road cameras or cameras on other vehicles; robots that gather task information like access keys
stored in fixed beacons; product tags with information (shelf
life, expiration date and price) that are read during scene
depth-imaging so that the product can also be localized.
While visible light communication has become a popular
research area, ambient illumination and reflections in real
world environments can degrade the signal quality. Messaging with light phase instead of intensity has distinct advantages. First, time-of-flight cameras are designed to measure
the phase of the source light signal (typically 15-30 MHz)
by correlating the source and the reflected signal. This design means that depth cameras are tuned to their source light
signal and are relatively unaffected by environment lighting. Therefore, reflections and ambient lighting do not degrade the quality of the signal. This is a major problem
for operations of intensity-based systems. Additionally, the
time-of-flight camera measures time delays on the order of
nanoseconds, and this short time interval makes the method
particularly useful for mobile scenarios where the camera
and messenger are in motion (e.g. handheld camera, automotive applications). The time interval of the light path
is orders of magnitude smaller than that of the system motion, so that motion blur is not an issue. On the contrary,
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
intensity-based messaging is vulnerable to degradation by
motion blur. Furthermore, the use of IR light enables messaging without visibility to human observers.
In this work, we design and construct a phase messaging array (PMA), which is the first of its kind, to communicate to a depth camera by manipulating the phase of the
depth camera’s own light signal (see Figure 2). The light
signal is replicated and the phase is shifted by our custom
circuitry. Replicating the camera’s own light signal is a key
design component of the system. The time-of-flight camera
is generally unresponsive to light signals that are not exactly
matched in frequency to their source signal. Sensing and
replicating the original camera signal with a message encoded in the phase shift ensures independence of the transmitter (PMA) and receiver (camera) in the communications
channel. A spatial distribution of IR light emitting diodes
allows messaging units to be packed in the x-y plane for
an increased data rate. The array driving circuitry measures
the depth camera’s source signal and effectively delays that
signal according to the desired message. In this manner, the
depth camera observes a time-varying depth that is electronically controllable. We show a prototype implementation of
a 3×3 array with 4 phase symbols and messaging accuracies
around 97.8%. The data rate of this 3 × 3 array approaches
1 Kbps which can be readily extended to a data rate more
than 10 Mbps (see our discussion in Section 4.2).
……
115
116
117
118
119
Customized Phase
Customized
MessagingPhase
Array
Messaging Array
……
109
110
111
112
113
114
2. Related Work
Visible-Light Communications (VLC) is a general term
for methods that use visible light instead of radio frequencies for communication. A theme in VLC in recent years
has been to extend existing infrastructure to create new
communication channels. Li-Fi [9] uses light-bulbs that
not only illuminate the room, but can also transmit temporally dynamic data by modulating light intensity. The data
rate of Li-Fi with a single color LED reaches 1.67 Gbps;
however, specialized equipment is needed for both the receiver and transmitter. Increasing the field-of-view in visible light commuincations has been addressed in [25] with
the use of hemispherical lenses at the receivers. However,
the use of ordinary cameras provides the benefit of an inherently wide field-of-view. Visual MIMO [3, 4] is a method
that uses any light-emitting spatial array such as an LCD
display as the transmitter and uses ordinary cameras as the
receivers. This method is a type of Camera-Display Communication (CDC) since the framework enables cameras
and electronic displays to communicate. Other CDC methods include QR-codes [1, 14], Bokode [16] and Pixnet [21].
QR-codes are comprised of black-white block patterns, and
are quite common in recent years applied to existing ads
or signage. Unlike Visual MIMO, the QR-code is passive
so the information is static and the pattern is visible to the
human observer. Similar to Visual MIMO, QR-codes are
designed for use with general purpose cameras, allowing
2
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
ICCP
ICCP
#5
#5
ICCP 2014 Submission #5. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
broad real-world applicaton. Bokode and Pixnet also display message codes with light-emitting devices as transmitters. The difference is that Bokode exploits the bokoh effect
of ordinary camera lenses for long-distance static tag reading, while Pixnet generalizes the popular OFDM transmission algorithms [2, 15] to address the unique characteristics
of the LCD-camera link. Visual MIMO [27, 28] proposes
a photometric modeling for hidden imagery and can realize
a dual use the electronic display: to display an image for
human observation while simultaneously transmitting hidden bits for decoding by a camera. The heirarchy of this
work in optical communications in real-world environments
progresses from barcodes [18] that are one-dimensional and
static, to QR-codes that are two-dimensional and static, to
the various CDC methods that use controllable intensity
patterns and are two dimensional and time-varying. Our
method adds to this framework with phase messaging that
uses controllable phase patterns for 2D time-varying messaging. We continue the theme of using ordinary cameras
for messaging since low-cost unmodified ToF cameras are
used for the receiver.
270
271
272
273
274
4
s(t)
r(t)
3.5
τ
3
2.5
Signal Strength
216
2
275
276
277
278
279
280
1.5
1
0.5
0
x1
x0
x2
x3
−0.5
−1
0
1
2
3
4
5
6
Time(t)
Figure 3. Four bucket method. Get four equidistant samples of
the received signal r(t) within one cycle. Their cross-correlations
with the sent signal s(t) are c(x0 ), c(x1 ), c(x2 ), c(x3 ). They can
be used to solve the depth, which is proportional to the delay τ .
IR LEDs, since they are readily available at low cost. 1
Denote the modulation frequency as fm . The sent signal
s(t) and the received signal r(t) with delay τ are given by
Ordinarily, ToF cameras are used to directly estimate 3D
scene structure complementing traditional multiple-view reconstruction methods [10,11]. Applications using ToF cameras include 3D scene reconstruction [6, 23], reflectance
capture using ultrafast imaging [20], frequency analysis to
address multipath interference [5, 13, 26] and human pose
detection/tracking [7, 8]. These depth cameras are increasingly common in various application domains such as intelligent vehicles, robotic environment mapping and surveillance. Our novel framework deploys ToF depth cameras
as receivers for optical communications. Compared with
intensity-based messaging methods, this phase-based messaging is more robust against background illumination as
shown in our experimental results. Also high measurement
precision in phases (for example, Softkinetic ToF cameras
have 32768 phase levels) can create a larger coding space
for an increased data rate.
s(t) =
r(t) =
a cos (2πfm t) ,
A cos (2πfm (t − τ )) + B
(1)
(2)
where A and B are the unknown scale and offset of the
reflected signal and these parameters are estimated at the
camera. For s(t) and r(t), the cross-correlation is
Z T2
r(t)s(t + x) dt
c(x) = lim
T →∞
− T2
aA
cos(2πfm x + 2πfm τ ) + B.
(3)
2
There are three unknown variables in Eqn 3: amplitude A,
offset B and delay τ . These variables can be solved by
“the four-bucket method” [12, 19]. The basic principle is
to obtain 4 equidistant samples at locations (x0 , x1 , x2 , x3 )
within a modulation cycle as in Figure 3. The crosscorrelations of these four outputs with the source signal
s(t) is denoted by c(x0 ), c(x1 ), c(x2 ), c(x3 ). The three unknown variables can be solved based on these correlations
in closed form. The amplitude A can be solved as
q
2
2
[c(x3 ) − c(x1 )] + [c(x0 ) − c(x2 )]
A=
.
(4)
2
The offset B is mainly attributed to background illumination and can be solved as
=
3. Methods
3.1. Introduction to Time-of-flight Camera
We summarize the basic operation of ToF cameras as
presented in [12, 19]. While most intensity-based digital
camera are passive, ToF cameras rely on integrated light
sources. During imaging, photons emitted from this light
source travel to objects in the scene. After surface reflection, the returning light is sensed by the ToF camera. The
light travels a distance equal to twice the object depth relative to the camera. There are two main types of ToF cameras: (1) pulse-modulation (PM) (2) continuous-wave (CW)
For this work, we use a continuous wave ToF camera using
B=
1 Creative
c(x0 ) + c(x1 ) + c(x2 ) + c(x3 )
.
4
(5)
Interactive Gesture Camera Developer Kit from Intel costs
$149. DepthSence311 from SoftKinetic costs $299. (December 2013)
3
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
ICCP
ICCP
#5
#5
ICCP 2014 Submission #5. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
324
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
ToF Camera
Analog circuit@15MHz
FIFO
A0
ToF
Camera
LED
Multiplexer
Am-1
p0
pn-1
…...
331
332
333
334
335
Message
Stream
Shift Register
325
326
327
328
329
330
Data Format
Converter
…...
Microprocessor
LEDs of the
ToF Camera
Photo
Detector
sc(t)
sac(t)
Amplifier
Phase
Modulation
Figure 4. Basic Operating Principle of the Phase Messaging Array (PMA). For clear illustration of the method, only one transmitter
LED is shown. Data flow is labeled with red arrows and control flow is in green. As a communications channel, the LED’s on the PMA
are the message transmitters and the ToF camera is the receiver. The PMA senses the signal from the ToF LED light source using a
photo detector. This copy of the light signal is denoted sc (t). After amplification, sac (t) is used to create multiple phase-delay versions
p0 , · · · , pn−1 . Delays range in (0, 2π] and are achievable in our design. These phase delays are symbols sent to multiplexers as output
candidates. The control flow is the bit version of the message stream. At the multiplexer, the combination [Am−1 , · · · , A0 ] selects the
corresponding phase-delayed candidate signals to sequentially drive the transmitter LED in the next cascade.
The delay τ is the time of flight for light sent from camera
LED, reflected from object’s surface and received by the
ToF camera and can be solved as
c(x3 ) − c(x1 )
τ = arctan
.
(6)
c(x0 ) − c(x2 )
camera using a photodiode to convert the light signal into
the electrical signal sc (t). This copied light signal is amplified to sac (t) and delayed to multiple phase-delay versions
p0 , · · · , pn−1 . Delays range in (0, 2π] and are controllable
in our design. In this manner, we create a dictionary of
phase symbols and that are sent to the multiplexers as output candidates. The signals sc (t), sac (t) and pi (t) comprise
the data flow component of the P M A circuit. The control flow is the bit version of the message stream. At the
multiplexer, the combination [Am−1 , · · · , A0 ] selects corresponding phase symbol in a sequential way to drive the
transmitter LED in the next cascade2 .
There is a simple intuitive method of achieving the phase
delay that fails in practice: applying a signal generator or
crystal to produce signals at the same modulation frequency
fm of the ToF camera. For the “four bucket method”,
it is critical to make sure the sent signal s(t) and the received signal r(t) have the same modulation frequency fm .
Though labeled with the fixed frequency value fm , a signal generator or crystal always produces outputs with some
variance around fm . This frequency error converts into a
delayed phase error and makes the measured depth values
random. We have shown this experimentally in Figure 5(a)
with the following procedure: A signal generator produces
a sinusoid waveform at 15MHz. This source drives an IR
LED facing to a receiver ToF camera working at the same
modulation frequency fm = 15MHz. As expected, the
The value of τ is proportional to the distance d between the
ToF camera and the captured object as τ = 2d
c , where c is
light speed. Then the delayed phase between s(t) and r(t)
can be denoted as φ = 2πfm τ = 2πfm 2d
c . Since φ ranges
in (0, 2π], the measured distance d reaches its maximum
dmax when φ = 2π. For a ToF camera working at fm =
15MHz, dmax =10m.
3.2. Phase Messaging Array Method
Controllable Phase Delay The design of our proposed
PMA method is illustrated in Figure 4. For simplicity, the
figure shows the basic operating principle with a single LED
on the phase messaging array. As a communications channel, the PMA is a message transmitter and the ToF camera
is the message receiver. The goal is to create phase delays
at the PMA circuit that mimic depth variations in order to
message to the ToF depth camera. Delaying light directly
would typically require variation of the refractive index of
the medium, and controllable delays in this manner are not
practical. However, as an electical signal, we can readily introduce delays using an appropriately designed cascade of
logic gates. This approach is a key innovation in the design. We sense the signal from the source LED of the ToF
2A
m−1 is the Most Significant Bit (MSB), while A0 is the Least Significant Bit (LSB).
4
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
ICCP
ICCP
#5
#5
ICCP 2014 Submission #5. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
350
300
300
250
250
200
150
sac(t)
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
Inv1
Inv2
100
100
50
50
0
0
C2
Inva
Invb
p0
0
0.5
1
1.5
2
2.5
3
360⁰
C3
p1
180⁰
Invc
p2
p3
(a) Phase modulation scheme
0
0.2
0.4
0.6
0.8
1
1.2
1.4
0 ⁰
(b) Four phase images
1.6
Time(second)
Time(second)
(a) Source: signal generator
Inv3
C1
Inv4
150
350
350
(b) Source: copied signal
300
Phase histogram
300
444
445
446
447
448
449
450
451
452
453
454
455
Inv0
200
Phases(deg)
439
440
441
442
443
350
Phases(deg)
433
434
435
436
437
438
Phases(deg)
432
250
200
150
100
0
200
400
600
800
1000
Time(s)
(c) Plot of phases
(c) Signal amplification
200
150
100
50
50
0
250
1200
1400
0
0
50
100
150
200
250
300
350
Phases(deg)
(d) Histograms of phases
Figure 6. Phase modulation. (a) In the phase modulation scheme,
“deterioration prevention” and “phase delay control” are two basic
considerations. (b) Four phase images of an LED displayed with
delayed signals p0 , p1 , p2 and p3 . Phase values are mapped to
colors for visualization. (c) Plot of measured phases for an LED
displayed with four delayed signals in turn. (d) Histogram of measured phases in (c). Red dotted lines are hard thresholds for phase
symbol assignment.
Figure 5. Comparison of two sources. (a) Measured phases of
an IR LED driven by a 15MHz output from a signal generator are
random, because of the frequency noise of the signal generator.
(b) Measured phases of an IR LED by our proposed PMA method
are stable. (c) Amplification of sc (t) in blue to sac (t) in red.
measured phases of the LED from the depth image are random.
Our solution is to sense and copy the source signal from
LED of the receiver ToF camera. An example is illustrated
in Figure 5(c): the copied signal sc (t) in blue has an amplitude of 400mV with unit voltage 200mV; the amplified
signal sac (t) in red has an amplitude of 5V with unit voltage 2V. This amplified signal sac (t) drives an LED facing
the same ToF camera. Compared with Figure 5(a), the measured phases of the LED in Figure 5(b), are stable enough
for messaging.
from the receiver ToF camera. Though the location of the
LED is fixed, ToF camera measures it at four possible distances, corresponding to four phases p0 , p1 , p2 and p3 . For
visualization, these four candidate phases are mapped to individual colors. These four delayed signals are displayed
sequentially on an IR LED controlled by a timer as in Figure 6(c). The histograms of all measured phases are shown
in Figure 6(d). The histograms show the presence of outliers from the phase symbols due to transition states. We
currently remove outliers by using an indicator LED to temporally identify transition times. Omitting transition times,
the measured phase has a standard deviation σp = 1.2 from
an individual LED. The phases can be separated by hard
thresholds indicated by the red dotted lines.
Phase Modulation For messaging, the mimicing signal
of an LED from the receiver ToF camera is amplified into
sac (t). In our proposed scheme in Figure 6(a), there are four
phase-delayed candidates: p0 , p1 , p2 and p3 . Delays range
in (0, 2π] and are controlled by capacitors C1 , C2 and C3 .
Each phase candidate is connected to the same channel
of all multiplexers and each multiplexer drives one transmitter LED. Since each multiplexer works as a “switch”,
the configuration is a heavy load to output inverters Ia , Ib ,
Ic and Id , and may degrade the output signals severely. To
prevent signal deterioration between cascades, the output
signal pi for each cascade and the one sent to the next cascade are not the same. Additionally, inverter I0 is used to
increase driving ability of the input signal.
Figure 6(b) illustrates four phase images of an IR LED
Messaging and Signal Selection Each of four delayed
phases p0 , p1 , p2 and p3 (in red) are connected to the same
channel of all multiplexers. Every multiplexer drives one
transmitter LED facing towards the receiver ToF camera.
The bit streams from the control flow (in green) determines
which candidate to select.
One multiplexer case is shown in Figure 7(a). By the
truth table in Figure 7(b) [24], signal A1 A0 = 102 selects
the signal with phase p2 . In the decoding scheme, messages
are extracted by converting the measured phase value into
the corresponding binary code using the truth table in Figure 7(b).
5
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
ICCP
ICCP
#5
#5
ICCP 2014 Submission #5. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
output
A0
Multiplexer
A1
p0
p1
p2
p3
A1
A0
output
0
0
1
1
0
1
0
1
p0
p1
p2
p3
ure 7(b) and labels p2 p1 p0 p3 p2 p1 p0 p3 p1 are decoded to the
binary form “100100111001001101”.
360⁰
(b) Truthtable
0 ⁰
Figure 7. Signal selection for messaging. (a) A 4×1 multiplexer
with four channels (in red) and two bits A1 A0 (in green) for selection. (b) The truthtable of the multiplexer in (a).
(a) Snapshot of 9 LEDs
4. Experiments
In our proposed scheme, the receiver is a CW ToF camera SoftKinect DS311. Light from LEDs of this camera
is IR with wavelength of 850nm and modulation frequency
fm =15MHz. The depth sensor has a resolution of 160×120.
The PMA transmitter is illustrated in Figure 8. There are
five basic components: (1) photo detector and signal amplifier; (2) phase modulator; (3) multiplexers; (4) an array
of 9 LEDs; (5) shift register to convert the serial control
flow into the parallel one for all multiplexers. The control
flow is converted into a binary stream and buffered at a microprocessor MSP430G in this experiment. For the PMA
transmitter circuit, it is critical to select component chips
with delay far less than one period, i.e. 66ns.
593
323.5
172.4
p2
p3
p0
232.9
276.8
327.1
p1
p2
p3
182.3
236.7
242.3
p0
p1
p1
(c) Phase label assignment
Figure 9. Phase symbol assignment for one phase image. (a)
A phase image with 9 LEDs. For visualization, phase values are
mapped to colors. (b) Mean phase values of 9 LEDs in degree. (c)
Phase symbol assignment with p0 -p3 by the threshold method.
Of the nine LEDs in the prototype, one LED is used as
an indicator to signal the frame transition. The other eight
LEDs provide 16 message bits per captured frame. We send
8000 random bits using the PMA and extract messages at
the ToF camera. The average accuracy of five independent
experiments is 97.8% with a standard deviation of 1.3%.
The camera frame rate for experiments is 24 fps (framesper-second). In practice, the transmission rate at the PMA
is modified to allow for repetitive frames to assist the synchronization and the measured number of repetitive frames
has a mean of 6.3, resulting in an effective frame rate of 4
fps. With 16 bits per frame (2 bits per LED), the effective
message rate is approximately 64 bps. By addressing synchronization without using repetitive frames, the data rate
for this method can reach 960 bps by setting 60fps, i.e. the
maximum frame rate of SoftKinect DS311. An accuracy
of 97.8% corresponds to an uncoded bit error rate of 2.2%.
Appropriate error correction codes such as Reed-Solomon
for burst error and convolutional codes for random errors
can be used to improve performance of accuracy, at the
tradeoff of bit rate. The appropriate channel characterization and coding is reserved for future work.
575
576
577
578
579
580
587
588
589
590
591
592
272.6
(b) Phase values in degree
570
571
572
573
574
581
582
583
584
585
586
599
600
601
602
603
604
180⁰
(a) Multiplexer
Figure 8. Phase messaging array (PMA) circuit for experiments.
We test the system in an indoor environment by sending
random bits from PMA and measure the accuracy (or recognition rate) at the receiver side (i.e. ToF camera). Here,
recognition rate is given by the number of correctly recognized bits divided by the number of transmitted bits. One
snapshot of the captured phase image is illustrated in Figure 9(a). For message extraction, we get average phase values for each LED as in Figure 9(b) and assign phase labels by the threshold method. Refer to the truthtable in Fig-
4.1. Robustness
We compare the influence of ambient background illumination for 1) phase-based messaging with a ToF camera (SoftKinetic DS311) as the receiver and 2) intensitybased messaging using typical cameras (smart phone Samsung SIII and consumer digital camera Sony DSC-RX100)
as receivers. The transmitter for phase-messaging is our
6
594
595
596
597
598
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
ICCP
#5
ICCP
#5
ICCP 2014 Submission #5. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
array, and a 60 frames-per-second camera, the expected data
rate would be 600000 × 2 × 8 = 9, 600, 000 ≈ 10 Mbps .
PMA displaying four phases sequentially; while the transmitter for the intensity-based messaging is a visible light
LED switching between on and off and controlled by a simple electronic timer. Plots of captured intensity and phases
are shown in Figure 10. For the normal indoor condition,
both ToF camera and intensity-based cameras have switching states as expected. We then apply a bright ambient illumination (from an ordinary fluorescent office light) that
is manually switched on and off. With this strong timevarying background illumination, the captured values from
intensity-based camera is significantly degraded. We repeat
sending 8000 random bits under varying background illumination for five times. The average bit accuracy is 95.8%
with a standard deviation of 2.1%, for the phase-based messaging. The intensity-based messaging gives an accuracy
of 100% with no background illumination, but the accuracy dips to approximately 70% with random blinking background illumination. As expected, the phase-based messaging method is much more robust against ambient or background illumination than intensity-based messaging.
5. Conclusion
In this work, we propose and develop a communication
methodology with a low-cost continuous-wave ToF camera
and low-cost custom circuitry. The novel method, called
Phase Messaging Array (PMA), is the first of its kind that
allows messaging to the ToF camera using electronically
controllable phase. The message bits can be encoded temporally via phase delays and spatially in the x-y coordinate
frame. We demonstrate high accuracy messaging with the
PMA operating independently from the ToF camera, suggesting applications where PMA’s in scenes can act as messaging tags, beacons or landmarks for auxilliary communication to mobile systems employing depth cameras. A key
demonstrated advantage of phase-messaging is the performance under various lighting conditions allowing operation
in outdoor and bright light environments. Messaging accuracies for all cases are above 95%.
4.2. Data Rate Discussion
References
The system we present in this paper is a complete implementation of the proof of concept 3 × 3 phase-messagingarray with nine transmitter LEDs. In the current implementation, each LED can emit a time varying signal taking on
values from a set of four phase delays, i.e. a 2 bit binary
message stream. The basic concept can be easily extended
to increase the number of LEDs packed in the spatial plane
and to increase the number of phase symbols. Of the nine
LEDs in the prototype, eight messaging LEDs provide 16
message bits per captured frame. The ToF camera works at
the framerate of 24 fps so that the prototype can transmit up
to 384 bps with synchronization, and 960 bps at 60 fps. An
extension of the current approach can be made by increasing
the number of phase symbols to span the 360-degree range.
With the current level of phase noise (σp = 1.2) approximately (360/2.4) = 150 symbols or approximately 7 bits can
be implemented and still be separated enough for discrimination. By also increasing the spatial density of the LEDs
to a 100 × 100 array, and operating at 60 fps, the extended
data rate would be 4.2 Mbps. Further modification of the
hardware of the PMA to reduce the phase variance so that
256 symbols can be used would enable full utilization of the
8-bit depth camera range and give a data rate of 4.8 Mbps.
A key interesting point is that this communications channel
is achievable while the depth camera continues its normal
operation as a depth camera. No additional hardware on the
ToF camera is necessary for its role as the message receiver.
As future work, phase-based messaging can be combined
with varying the amplitude of the light signal to achieve a
combination of both intensity-based and phase-based messaging (in environments with low ambient light). For example, with 8-bits of intensity variation, a 100 × 100 pixel
[1] ISO/IEC 18004: 2006 Information technology Automatic
identification and data capture techniques QR code 2005 bar
code symbology specification, 2006.
[2] J. Armstrong. Ofdm for optical communications. J. Lightwave Technol., 27(3):189–204, Feb 2009.
[3] A. Ashok, M. Gruteser, N. Mandayam, T. Kwon, W. Yuan,
M. Varga, and K. Dana. Rate adaptation in visual mimo.
Proceedings of IEEE Conference on Sensor, Mesh and Ad
Hoc Communications and Networks, pages 583–591, 2011.
[4] A. Ashok, M. Gruteser, N. Mandayam, J. Silva, M. Varga,
and K. Dana. Challenge: mobile optical networks through
visual mimo. Proceedings of the sixteenth annual international conference on Mobile computing and networking (MobiCom), pages 105–112, 2010.
[5] A. Bhandari, A. Kadambi, R. Whyte, L. Streeter, C. Barsi,
A. Dorrington, and R. Raskar. Multifrequency time of flight
in the context of transient renderings. In ACM SIGGRAPH,
pages 46:1–46:1, 2013.
[6] Y. Cui, S. Schuon, D. Chan, S. Thrun, and C. Theobalt.
3D shape scanning with a time-of-flight camera. In IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR), pages 1173–1180, 2010.
[7] M. V. den Bergh and L. V. Gool. Combining RGB and ToF
cameras for real-time 3D hand gesture interaction. In IEEE
Workshop on Applications of Computer Vision (WACV),
pages 66–72, 2011.
[8] V. Ganapathi, C. Plagemann, D. Koller, and S. Thrun. Real
time motion capture using a single time-of-flight camera. In
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 755–762, 2010.
[9] H. Haas. High-speed wireless networking using visible light.
SPIE Optoelectronics and Communications, 2013.
[10] M. Hansard, S. Lee, O. Choi, and etc. Time of Flight Cameras: Principles, Methods, and Applications. Springer, 2013.
7
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
ICCP
ICCP
#5
#5
ICCP 2014 Submission #5. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
756
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
Samsung SIII
350
810
811
812
813
814
RX100
200
80
180
70
160
250
60
Intenisity
140
Intenisity
Phases(deg)
300
120
100
80
200
50
40
60
30
40
150
0
50
100
150
200
250
300
20
350
0
50
100
150
Time(s)
200
250
300
350
400
20
450
0
100
200
Time(s)
350
300
400
500
600
Time(s)
200
80
180
250
60
Intenisity
Intenisity
140
120
100
80
200
50
40
60
30
40
150
0
50
100
150
200
250
300
20
0
50
100
150
Time(s)
Phase
200
250
300
Time(s)
Intensity
350
400
450
20
0
100
200
300
400
500
600
Time(s)
Intensity
Figure 10. Influence of ambient background illumination. We compare the influence of time-varying ambient light for phase-based
messaging with a ToF camera (SoftKinetic DS311) and intensity-based messaging (using a smart phone camera Samsung SIII and a
consumer digital camera Sony DSC-RX100). We can observe that phase-based messaging is more robust against time-varying ambient
illumination than intensity-based cameras.
[11] R. I. Hartley and A. Zisserman. Multiple View Geometry in
Computer Vision. Cambridge University, 2004.
[12] R. P. Horaud. Tutorial on 3D Sensors, Apr 2013.
[13] A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi,
A. Dorrington, and R. Raskar. Coded time of flight cameras: Sparse deconvolution to address multipath interference
and recover time profiles. ACM Trans. Graph., 32(6):167:1–
167:10, Nov. 2013.
[14] K. Kamijo, N. Kamijo, and G. Zhang. Invisible barcode with
optimized error correction. In IEEE International Conference on Image Processing, pages 2036 –2039, oct. 2008.
[15] X. Li, J. Vucic, V. Jungnickel, and J. Armstrong. On the capacity of intensity-modulated direct-detection systems and
the information rate of aco-ofdm for indoor optical wireless applications. IEEE Transactions on Communications,
60(3):pp. 799–809, 2012.
[16] A. Mohan, G. Woo, S. Hiura, Q. Smithwick, and R. Raskar.
Bokode: Imperceptible visual tags for camera based interaction from a distance. ACM Trans. Graph., 28(3), July 2009.
[17] M. R. H. Mondal and J. Armstrong. Impact of linear misalignment on a spatial OFDM based pixelated system. In
Asia-Pacific Conference on Communications, 2012.
[18] A. Q. Morton. Packaging history: The emergence of the
Uniform Product Code (UPC) in the United States. History
and Technology, 11(1):101111, 1994.
[19] C. D. Mutto, P. Zanuttigh, and G. M. Cortelazzo. Time-ofFlight Cameras and Microsoft Kinect(TM). Springer Publishing Company, Incorporated, 2012.
[20] N. Naik, S. Zhao, A. Velten, R. Raskar, and K. Bala. Single view reflectance capture using multiplexed scattering and
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
8
815
816
817
818
819
820
821
822
823
824
825
70
160
300
Phases(deg)
768
769
770
771
772
773
No Blinking Amb. Light
763
764
765
766
767
ToF Camera
With Blinking Amb. Light
757
758
759
760
761
762
time-of-flight imaging. In Proceedings of SIGGRAPH Asia
Conference, pages 171:1–171:10, 2011.
S. D. Perli, N. Ahmed, and D. Katabi. Pixnet:designing
interference-free wireless links using lcd-camera pairs. ACM
Int. Conf. on Mobile Computing and Networking, 2010.
S. Schmid, G. Corbellini, S. Mangold, and T. R. Gross. Ledto-led visible light communication networks. In Proceedings
of the fourteenth ACM international symposium on Mobile
ad hoc networking and computing (MobiHoc), 2013.
A. Velten, T. Willwacher, O. Gupta, A.Veeraraghavan,
M.G.Bawendi, and R. Raskar. Recovering three-dimensional
shape around a corner using ultrafast time-of-flight imaging.
Nature Communication, 3:745+, 2012.
J. Wakerly. Digital Design: Principles and Practices (4th
Edition). Prentice-Hall, Inc., 2005.
T. Q. Wang, Y. A. Sekercioglu, and J. Armstrong. Analysis of an optical wireless receiver using a hemispherical lens
with application in mimo visible light communications. J.
Lightwave Technol., 31(11):1744–1754, Jun 2013.
D. Wu, M. O’Toole, A. Velten, A. Agrawal, and R. Raskar.
Decomposing global light transport using time of flight
imaging. In IEEE CVPR, pages 366–373, 2012.
W. Yuan, K. Dana, A. Ashok, M. Varga, M. Gruteser,
and N. Mandayam. Photographic steganography for visual
mimo: A computer vision approach. IEEE Workshop on the
Applications of Computer Vision, pages 345–352, 2012.
W. Yuan, K. Dana, A. Ashok, M. Varga, M. Gruteser, and
N. Mandayam. Spatially varying radiometric calibration for
camera-display messaging. IEEE Global Conference on Signal and Information Processing (GlobalSIP), 2013.
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863