Spatio-Temporal Quincunx
Sub-Sampling
. . and how we get there
David Lyon
Overview






Sampling in Television and Film
The problems of aliasing
Filtering requirements
Conversion between differing formats
Problems that can occur
How we can mitigate some of the problems and maintain
or improve quality
Sampling Theory
 Harry Nyquist – 1889 to 1976
“The number of independent pulses that can be put through a
telegraph channel per unit time is limited to twice the bandwidth
of the channel”
Sampling Theory
 Harry Nyquist – 1889 to 1976
“The number of independent pulses that can be put through a
telegraph channel per unit time is limited to twice the bandwidth
of the channel”
 Later Nyquist-Shannon
“Exact reconstruction of a continuous-time baseband signal from
its samples is possible if the signal is bandlimited and the
sampling frequency is greater than twice the signal bandwidth”
Sampling Theory
Amplitude
Fs
Frequency
Sampling Theory
Amplitude
Fs
Frequency
 Audio:
20kHz bandwidth, Fs = 44.1kHz, 48kHz
Sampling Theory
Amplitude
Fs
Frequency
 Audio:
20kHz bandwidth, Fs = 44.1kHz, 48kHz
 Video:
5.75MHz bandwidth, Fs = 13.5MHz
30MHz bandwidth, Fs = 74.25MHz
Aliasing
Amplitude
Nyquist
Frequency
Fs
Frequency
Aliasing
Amplitude
Nyquist
Frequency
Fs
Frequency
 Frequencies above Fs/2 are “reflected” into the lower
portion of the spectrum and become entangled with the
low-frequency signals
Aliasing
Amplitude
Nyquist
Frequency
Fs
Frequency
 Frequencies above Fs/2 are “reflected” into the lower
portion of the spectrum and become entangled with the
low-frequency signals
 These signals CANNOT be removed afterwards
Aliasing
Amplitude
Nyquist
Frequency
Fs
Frequency
 Frequencies above Fs/2 are “reflected” into the lower
portion of the spectrum and become entangled with the
low-frequency signals
 These signals CANNOT be removed afterwards
 Filtering BEFORE sampling is needed
Image Sampling
Temporal –
frames
Vertical lines
Horizontal pixels
Image Sampling
 Horizontal resolution
Sampling rate of 720, 1280, 1920 or 2048 samples/picture width
•
Resulting resolution of 360, 640, 960 or 1024 cycles/pw
Image Sampling
 Horizontal resolution
Sampling rate of 720, 1280, 1920 or 2048 samples/picture width
•
Resulting resolution of 360, 640, 960 or 1024 cycles/pw
 Vertical resolution
Sampling rate of 480, 576, 720, 1080 samples/picture height
•
Resulting resolution of 240, 288, 360 or 540 cycles/ph
Image Sampling
 Horizontal resolution
Sampling rate of 720, 1280, 1920 or 2048 samples/picture width
•
Resulting resolution of 360, 640, 960 or 1024 cycles/pw
 Vertical resolution
Sampling rate of 480, 576, 720, 1080 samples/picture height
•
Resulting resolution of 240, 288, 360 or 540 cycles/ph
 Temporal resolution
Sampling rate of 24, 25, 30, 50, 60 . . . samples/second
•
Resulting resolution of 12, 15, 25, 30 cycles/sec
Re-sampling
 Image size changes are common
Re-sampling
 Image size changes are common
Simple example of interpolating a 1080 picture to 480:
Input resolution is 540 cycles/ph
Output resolution is 240 cycles/ph (division by 2.25)
Amplitude
•
•
1080
Vertical
Frequency
Filter
Amplitude
Potential Alias
480
Vertical
Frequency
Re-sampling
 Interpolation is only one part of the problem
Filtering is needed to control the signal spectrum and avoid the
introduction of aliases
Simple interpolators are generally poor filters
Re-sampling
 Interpolation is only one part of the problem
Filtering is needed to control the signal spectrum and avoid the
introduction of aliases
Simple interpolators are generally poor filters
 Alias terms are “folded” about the Nyquist point
Inverted in frequency, inverted “movement”
Highly noticeable to the human eye, which references its own
internal 3D model
Re-sampling
 Interpolation is only one part of the problem
Filtering is needed to control the signal spectrum and avoid the
introduction of aliases
Simple interpolators are generally poor filters
 Alias terms are “folded” about the Nyquist point
Inverted in frequency, inverted “movement”
Highly noticeable to the human eye, which references its own
internal 3D model
 Alias terms left in the image will be shifted again in any
subsequent operations
Potentially cumulative problems
3D Sampling
Temporal –
frames
Restricted by
practical
limitations
Vertical lines
Horizontal pixels
Linked by
aspect ratio
and pixel
shape
Spatio-Temporal Sampling
Spatial
Frequency
No of Lines
Temporal
– frames
Potential
alias
Potential
alias
Frame Rate
Spectrum
Spatial
- lines
Temporal
Frequency
Spatio-Temporal Sampling
Spatial
Frequency
No of Lines
Temporal
– frames
Potential
alias
Potential
alias
Frame Rate
Spectrum
Spatial
- lines
 Filtering:
Spatial – optical LPF and lens MTF
Temporal
Frequency
Spatio-Temporal Sampling
Spatial
Frequency
No of Lines
Temporal
– frames
Potential
alias
Potential
alias
Frame Rate
Spectrum
Spatial
- lines
 Filtering:
Spatial – optical LPF and lens MTF
Temporal – integration time of sensor system
Temporal
Frequency
Spatio-Temporal Sub-Sampling
Spatial
Frequency
No of Lines
Temporal
– frames
Potential
alias
Potential
alias
Frame Rate
Spectrum
Spatial
- lines
 Where is the filter?
Temporal
Frequency
Up-conversion
Horizontal
Spatial
Frequency
No of Lines
Vertical
Temporal
?
Spectrum
Frame Rate
Temporal
Frequency
Up-conversion
Horizontal
Spatial
Frequency
No of Lines
Vertical
Temporal
?
Spectrum
 Adaptive filtering
Frame Rate
Temporal
Frequency
Up-conversion
Spatial
Frequency
Horizontal
No of Lines
Vertical
Temporal
?
Spectrum
 Adaptive filtering
 Motion compensation
Frame Rate
Temporal
Frequency
Format Interchange
Spatial
Frequency
Film
1080p
500c/ph
720p
250c/ph
480i
1080i
1080p (24)
0c/ph
0c/s
15c/s
30c/s
Temporal
Frequency
Format Interchange
Spatial
Frequency
Film
1080p
500c/ph
720p
250c/ph
480i
1080i
1080p (24)
0c/ph
0c/s
15c/s
30c/s
Temporal
Frequency
 Conversion
between formats
requires care
Format Interchange
Spatial
Frequency
Film
1080p
500c/ph
720p
250c/ph
480i
1080i
1080p (24)
0c/ph
0c/s
15c/s
30c/s
Temporal
Frequency
 Conversion
between formats
requires care
 Mixing formats
such as film and
video is to be
avoided
Format Interchange
Spatial
Frequency
Film
1080p
500c/ph
720p
250c/ph
480i
1080i
1080p (24)
0c/ph
0c/s
15c/s
30c/s
Temporal
Frequency
 Conversion
between formats
requires care
 Mixing formats
such as film and
video is to be
avoided
 1080p downconversion might
raise new
challenges
Over-sampling
 Commonly applied to audio – eg 96kHz down to 48kHz
Amplitude
Allows the use of a high performance digital filter:
96
Frequency
Amplitude
Filter
48
Frequency
Over-sampling
 Commonly applied to audio – eg 96kHz down to 48kHz
Allows the use of a high performance digital filter:
Over-sampling
 Commonly applied to audio – eg 96kHz down to 48kHz
Allows the use of a high performance digital filter:
 1080p allows similar gains for outputs of 720p and 1080i
Good temporal filtering must introduce delay
Over-sampling
 Commonly applied to audio – eg 96kHz down to 48kHz
Allows the use of a high performance digital filter:
 1080p allows similar gains for outputs of 720p and 1080i
Good temporal filtering must introduce delay
 Film sampling at >1080 lines/ph also allows controlled
down-sampling
Conclusion
 Spatio-temporal quincunx sub-sampling (aka interlace) is
likely to be with us for some time
Conclusion
 Spatio-temporal quincunx sub-sampling (aka interlace) is
likely to be with us for some time
 Modern cameras and processing can stress the format
unless care is taken
Conclusion
 Spatio-temporal quincunx sub-sampling (aka interlace) is
likely to be with us for some time
 Modern cameras and processing can stress the format
unless care is taken
Imprinted alias is difficult (or impossible) to remove
Camera integration is an important filter for interlace
Conclusion
 Spatio-temporal quincunx sub-sampling (aka interlace) is
likely to be with us for some time
 Modern cameras and processing can stress the format
unless care is taken
Imprinted alias is difficult (or impossible) to remove
Camera integration is an important filter for interlace
 Poor anti-alias filtering leads to additional compression
concatenation artefacts
Conclusion
 Spatio-temporal quincunx sub-sampling (aka interlace) is
likely to be with us for some time
 Modern cameras and processing can stress the format
unless care is taken
Imprinted alias is difficult (or impossible) to remove
Camera integration is an important filter for interlace
 Poor anti-alias filtering leads to additional compression
concatenation artefacts
 1080p down-conversion could make the stress worse