Measuring Quality of
Experience for Successful
IPTV Deployments
Dr. Stefan Winkler
Outline
• Challenges
– Digital Video Quality Issues
– Traditional Measurements (QoS) vs.
Quality of Experience (QoE)
• Possible Solutions
– QoE Measurement Approaches
– End-to-end QoE Management
• Conclusions
Digital Video Challenges
Demanding traffic profiles

High bandwidth streams

High traffic volumes

Live, VOD
High end-user expectations

Defined with decades of history

Grow rapidly with HD

Low tolerance for poor quality
Service quality
degradations
Difficult diagnosis,
troubleshooting
Rising
management and
OPEX costs
Higher customer
churn
Network effects

Video impacted heavily with
minor network impairments

Multi-vendor network
complicates diagnosis /
troubleshooting
New architectures

Sensitive video processing
devices create possibility for
various impairment sources

Ad-insertion, middleware
What Drives End-Users
Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.com
Service Providers View
Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.com
Service Providers’ View
Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.com
7
Sources of Video Issues
Consider all elements for true end-to-end solution
Compression Artifacts
Original
MPEG-2
H.264
PSNR vs. QoE
Same amount of distortion (PSNR) – different perceived quality
Understand & model human vision system
QoS vs. QoE
QoS
QoE
• Quality of Service
–
–
–
–
Network-centric
Delay, packet loss, jitter
Transmission quality
Content agnostic
• Quality of Experience
–
–
–
–
Content impairments
Blockiness, Jerkiness, …
End-user quality
Application driven
QoS vs. QoE
Same network impairments
Packet Loss: 1%
Delay: 10ms
Jitter: 50us
Bandwidth: 500kbps
Different perceived quality!
MDI vs. QoE
• Media Delivery Index (MDI)
• MDI consists of two metrics:
– Delay Factor (DF)
– Media Loss Rate (MLR)
• MDI limitations:
–
–
–
–
MDI assumes constant bit rate (CBR) traffic
MDI does not consider video payload or content
MDI values are not intuitive
MDI doesn’t correlate with video quality
MDI vs. QoE
5
MOS
V-FACTOR
4
3
2
1
0
Jitter
1 to 50ms
3000
Media Loss
2500
Media Loss
2000
1500
1000
500
0
Packet Drop
1 in 500 periodic
Packet Drop
1 in 500 poisson
Packet Drop
1 in 500 uniform
Packet Drop
1 in 500 and Jitter
Duplicate Packets
1 in 10
Duplicate Packets
1 in 500
Reordering Packets
1 in 500
QoS/QoE Cycle
Alignment gap
End-user
Service
provider
Desired
QoE
Targeted
QoS
Value gap
Execution gap
Perceived
QoE
Delivered
QoS
Perception gap
Adapted from ITU-T Rec. G.1000 and COM12–C185–E
Outline
• Challenges
– Digital Video Quality Issues
– Traditional Measurements (QoS) vs.
Quality of Experience (QoE)
• Possible Solutions
– QoE Measurement Approaches
– End-to-end QoE Management
• Conclusions
Full-Reference Approach
Sender
Video
Receiver
Compression/
Transmission
System
Video
Full Ref. Quality
Full reference information Measurement
• Comparison of individual video frames
• Offline analysis (capture is required) – lab applications
• High detail and accuracy
• Alignment procedure
No-Reference Approach
Sender
Video
Receiver
Compression/
Transmission
System
Video
No-Ref. Quality
Measurement
• Non-intrusive, in-service measurement
• Real-time monitoring applications
• No alignment required
Reduced-Reference Approach
Sender
Video
Feature
Extraction
Receiver
Compression/
Transmission
System
Reduced Ref.
Measurement
Video
Feature
Extraction
• Monitoring applications
• Correlation of content and network impairments
• Encrypted environments
Content & Network Metrics
(Correlation Engine)
"Vision is the most highly developed of the human senses, so people are even more sensitive to flaws
in video images than, say, the sound of a telephone conversation.”
Ken Wirt, Cisco Vice President Consumer Marketing, Jan 2008
20
Vision Modeling
Sensitivity
• Contrast perception
– Visibility of different patterns
– Frequency dependencies
• Masking effects
Temporal frequency [Hz]
• Color perception
Visibility
threshold
Target contrast
– Interaction of content and impairments
– Texture, edges, luminance
Masking
curve
– Spatial and temporal masking
Spatial frequency [cpd]
Threshold
without
masker
Masker contrast
End-to-end QoE
Deep Content Analysis (bitstream)
Deep Content Analysis
(pixel by pixel)
Source content and
encoder / transcoder
validation
Content Impairments:
• Blockiness, blur
• Jerkiness
• Freeze/black frame
• Noise, Color
Detect content impairments
Deep inspection to associate content
to timestamps (eg: TS1 = I-Frame)
Network (header or stream) Analysis
Content Stream Analysis:
• PES inspection
• PCR jitter etc.
Detect QoS issues
Content analysis where possible (unencrypted)
Inspection of QoS to associate timestamps to
impairments (eg: TS1 = Packet Loss)
TS1 = I-Frame
Q-Advisor
Correlation
Engine
TS1 = Packet Loss
Packet Loss -> I-Frame
Human Vision
System Model
Video
Quality
Reports
Network Impairments:
• Loss
• Delay
• Jitter
• Bandwidth
IPTV QoE Management
1. Understand the Service
2. Understand the Problem
 Is there an issue?
 Does it matter?
 What does the customer see?
 What is the exact cause?
Issue
1.0
5
Imperceptible
4
Perceptible
3
Slightly Annoying
2
Annoying
1
Very Annoying
Possible Causes
Blockiness
Encoder
Transcoder
Network Loss
Blur
Camera (focus)
Encoder
Transcoder
Freeze Frame, Jerkiness
Encoder (dropped
frames)
Network loss
Bad
synchronization
Black Screen, Blue
Screen
No Video Signal
(source)
Ads not inserted
Major network
loss
Color
Encoder
Camera
Transcoder
Video Noise (analog
noise)
Camera
STB
Noise (digital)
Encoder
Transcoder
Audio
Microphone
Encoder (bad
mono stereo
encoding
Encoder (lip
sync)
STB (bad
filtering)
STB
3. Understand the Solution
 What is the impairment source?
25
Conclusions
• QoE is application-driven
– Measure both network and content impairments
• QoE is user-oriented
– Measure how end-user perceives service issues
• End-to-end quality measurement
– Cover different impairment sources
– Identify problem causes
Contact Info
Stefan Winkler
[email protected]
Company:
qoe.symmetricom.com
Further Reading:
stefan.winkler.net/book.html