Information-Theoretic Modeling
Lecture 12: Summary and conclusion
Jyrki Kivinen
Department of Computer Science, University of Helsinki
Autumn 2012
Jyrki Kivinen
Information-Theoretic Modeling
Summary
Roughly speaking, information-theoretic modelling on this course
has meant making statements of the form
“The complexity of data D is n bits.”
and making conclusions based on that.
Jyrki Kivinen
Information-Theoretic Modeling
Summary
Roughly speaking, information-theoretic modelling on this course
has meant making statements of the form
“The complexity of data D is n bits.”
and making conclusions based on that.
Examples:
Jyrki Kivinen
Information-Theoretic Modeling
Summary
Roughly speaking, information-theoretic modelling on this course
has meant making statements of the form
“The complexity of data D is n bits.”
and making conclusions based on that.
Examples:
Source coding: assume data comes from i.i.d. source, use
entropy as complexity measure
Jyrki Kivinen
Information-Theoretic Modeling
Summary
Roughly speaking, information-theoretic modelling on this course
has meant making statements of the form
“The complexity of data D is n bits.”
and making conclusions based on that.
Examples:
Source coding: assume data comes from i.i.d. source, use
entropy as complexity measure
Huffman coding: shortest symbol code for known frequency
distribution
Jyrki Kivinen
Information-Theoretic Modeling
Summary
Roughly speaking, information-theoretic modelling on this course
has meant making statements of the form
“The complexity of data D is n bits.”
and making conclusions based on that.
Examples:
Source coding: assume data comes from i.i.d. source, use
entropy as complexity measure
Huffman coding: shortest symbol code for known frequency
distribution
MDL: complexity = information + noise
Jyrki Kivinen
Information-Theoretic Modeling
Summary
Roughly speaking, information-theoretic modelling on this course
has meant making statements of the form
“The complexity of data D is n bits.”
and making conclusions based on that.
Examples:
Source coding: assume data comes from i.i.d. source, use
entropy as complexity measure
Huffman coding: shortest symbol code for known frequency
distribution
MDL: complexity = information + noise
We’ll next briefly summarize the course contents by theme.
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Statistical information theory
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Statistical information theory
Key ideas:
basic concepts: entropy, conditional entropy,
mutual information etc.
basic results: data processing inequality etc.
noiseless source coding and the source coding
theorem
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Statistical information theory
Key ideas:
basic concepts: entropy, conditional entropy,
mutual information etc.
basic results: data processing inequality etc.
noiseless source coding and the source coding
theorem
More technical:
asymptotic equipartition property, proof of
source coding theorem
noisy channel coding and the channel coding
theorem
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Statistical information theory
Key ideas:
basic concepts: entropy, conditional entropy,
mutual information etc.
basic results: data processing inequality etc.
noiseless source coding and the source coding
theorem
More technical:
asymptotic equipartition property, proof of
source coding theorem
noisy channel coding and the channel coding
theorem
Further topics:
theory and practice of error correcting codes
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Data compression
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Data compression
Key ideas:
decodable and prefix codes, Kraft(-McMillan)
inequality
symbol codes, block codes
Shannon code length, Shannon-Fano code
more advanced algorithms: Huffman, arithmetic
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Data compression
Key ideas:
More technical:
decodable and prefix codes, Kraft(-McMillan)
inequality
symbol codes, block codes
Shannon code length, Shannon-Fano code
more advanced algorithms: Huffman, arithmetic
proof of Kraft inequality
analysis of Huffman and arithmetic coding
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Data compression
Key ideas:
decodable and prefix codes, Kraft(-McMillan)
inequality
symbol codes, block codes
Shannon code length, Shannon-Fano code
more advanced algorithms: Huffman, arithmetic
More technical:
proof of Kraft inequality
analysis of Huffman and arithmetic coding
Further topics:
practical implementation of arithmetic coding
other types of codes (Lempel-Ziv etc.)
course Data Compression Techniques
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Minimum Description Length
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Minimum Description Length
Key ideas:
models and codes, model classes, universal
models/codes
two-part, mixture and NML codes
code lengths and the MDL principle
(k/2) log2 n
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Minimum Description Length
Key ideas:
More technical:
models and codes, model classes, universal
models/codes
two-part, mixture and NML codes
code lengths and the MDL principle
(k/2) log2 n
calculating two-part, mixture and NML code
lengths in some application scenarios
example applications: subset selection, denoising
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Minimum Description Length
Key ideas:
models and codes, model classes, universal
models/codes
two-part, mixture and NML codes
code lengths and the MDL principle
(k/2) log2 n
More technical:
calculating two-part, mixture and NML code
lengths in some application scenarios
example applications: subset selection, denoising
Further topics:
efficient NML computation on more complex
model classes
various application areas
Fisher information: where does (k/2) log2 n
really come from?
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Kolmogorov complexity
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Kolmogorov complexity
Key ideas:
motivation and basic definition
universal and prefix machines
invariance theorem, uncomputability
basic upper bounds and examples
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Kolmogorov complexity
Key ideas:
motivation and basic definition
universal and prefix machines
invariance theorem, uncomputability
basic upper bounds and examples
More technical:
proof of uncomputability
conditional Kolmogorov complexity
Jyrki Kivinen
Information-Theoretic Modeling
Course contents
Kolmogorov complexity
Key ideas:
motivation and basic definition
universal and prefix machines
invariance theorem, uncomputability
basic upper bounds and examples
More technical:
proof of uncomputability
conditional Kolmogorov complexity
Further topics:
universal prediction
there’s a lot of theory about Kolmogorov
complexity
applications in logic and theory of computing
Jyrki Kivinen
Information-Theoretic Modeling