Analysis of compressed
depth and image streaming
on unreliable networks
Pietro Zanuttigh, Andrea Zanella, Guido M. Cortelazzo
The problem




Remote browsing of 3D scenes over wireless
channels
3D representations usually require huge amount
of data
Real time browsing
Wireless links are typically unreliable
 Robustness
to packet loss issues
3D Scenes
2 Basic type of data
Texture
Geometry
can be
Geometry
represented by a set
of depth maps
View warping


Depth information allows to associate each
sample in each view to a point in the 3D space
If the camera positions are known it is possible
to compute the position of the point in another
view
Why using depth maps?


Both texture and geometry info are compressed
by using JPEG2000 coding
Same coding/decoding scheme for both texture
& geometry
 Gain

JPEG2000 is a wavelet coding scheme that
yields multilayer representation
 Gain

in simplicity
in modularity
JPEG2000 is standardized
 Gain
in interoperability
Motivations


Today 3D system transmits info over TCP connections
TCP guarantees reliable transport at the cost of
unpredictable time delay


UDP does not introduce extra delay but may experience
packet losses


Might impair visual quality
A possible tradeoff might consist in using UDP together
with protection schemes


Might impair navigation fluidity
Data packets have very different relevance: Unequal Error
Protection (UEP)
A proper design of such schemes require good
knowledge of the effect of packet losses on the
reconstructed view…
Aim of this work

This work aims at shading light on these
aspects, answering the following
questions:
 How
performance degrades with increasing
loss probability?
 Which packets are more important?
 Is it better to protect geometry or texture info?
Remote visualization of 3D scenes
QuickTime™ and a
decompressor
are needed to see this picture.




Client-Server remote visualization system
Scene represented as a set of views and depth
maps scalably compressed in JPEG2000
Interactive browsing at client side
JPIP transmission protocol
Architecture of the system
Server



Client
The server holds the 3D scene description as a
set of images and depth maps scalably
compressed in JPEG2000
It decides which elements of the compressed
streams are the most suitable to be transmitted
The client renders the required view exploiting
the data received from the server
Simulation scenario (1)
QuickTime™ and a
decompressor
are needed to see this picture.



The server transmits 1 view (texture) and 1 depth
map (geometry)
Depth information is used to warp the view to
novel viewpoints
2 Test models
 ‘Goku’ (synthetic model)
 ‘Soccer Player’ (reconstructed from real world)
Simulation Scenario (2)
Target scenario: wireless link, UDP
protocol with no retransmissions
 Lossy channel

 Random
packet loss (1%, 5% and 10%)
 Loss of a consecutive packet batch

Comparison of the rendered views with
and without packet loss
Lossless reconstruction
example


Original texture size ~
40 KByte
JPIP frame size:
variable from ~0 byte
to ~1 Kbyte
 Black
area and highest
resolution info
transmitted in frames
with very small size
Loss of texture information (1/2)

QuickTime™ and a
decompressor
are needed to see this picture.


MSE due to the loss of
texture packets vs packet
loss rate & angle between
available and required
view (soccer player)
Distortion increases with
the amount of lost
packets (expected)
Distortion independent of
the selected viewpoint
Loss of texture information (2/2)
QuickTime™ and a
decompressor
are needed to see this picture.
Plot shows MSE due to the loss of a batch of texture
packets as a function of the position of the lost packets
batch and of the angle between the available and
required view



Dependent on lost packet position
JPEG2000/JPIP transmit compressed data
packets in order of relevance, losing earlier
packets is worse
Unequal Error Protection could be exploited
Loss of depth information (1/3)



Causes samples in the rendered views to be
misplaced
Critical on edges
Big impact on MSE
Loss of depth information (2/3)
QuickTime™ and a
decompressor
are needed to see this picture.
QuickTime™ and a
decompressor
are needed to see this picture.
Distortion increases with the amount of lost
packets and depends on the position of
the lost packet (as in the texture case)
Loss of depth information (3/3)
QuickTime™ and a
decompressor
are needed to see this picture.
0
30
MSE due to packet loss increases with the angle
between available and required view
(key difference with texture)
Depth and texture comparison
Plot shows MSE due to the loss of depth and
texture packets as a function
of the amount of lost packets and of the angle
between the available and required view
QuickTime™ and a
decompressor
are needed to see this picture.


Depth information more important, but probably overestimated by
MSE metric
Depth impact depend on the angle:

For small angles texture & depth errors have similar impact
 For larger angles depth become much more important
Conclusions
Very different relevance of different
packets (JPEG2000 transmits them in
order of relevance)
 Depth loss impact depends on the
viewpoint, texture one does not

Final considerations (2)
Fast Browsing
Focus on detail
Prefer response time
(use UDP + light Unequal
Error Protection)
Prefer Reliability
(use TCP or UDP + heavy
Forward Error Correction
also depending on the path
rate and delay)
Focus on depth information
to rapidly warp between
viewpoints
Focus on texture information
to appreciate visual details
Final considerations (3)

Adding redundancy might be detrimental in case
packet losses are due to contention instead of
wireless link errors
 Cross
Layer Optimization (CLO) techniques shall be
used on the wireless link to shield end-to-end
applications from wireless unreliability

Other transport protocols, such as Stream
Control Transmission Protocol (SCTP), might be
considered
Future work
Analysis of more complex simulation
scenarios with multiple views and depth
maps
 Design of ad-hoc hybrid TCP-UDP (SCTP)
protocols
 UEP techniques for 3D models

We’d like to dedicate this work to
Federico Maguolo
He was supposed to join us on this project
but his tragic death has prevent us from all the
excellent ideas and contributions
he would for sure have given to this work.