Overview

Information is what remains after one
abstracts from the material aspect of the
physical reality ...

How to do it?
1
The Library of Babel

The universe (which others call the Library) is composed of an indefinite
and perhaps infinite number of hexagonal galleries, with vast air shafts
between, surrounded by very low railings

Jorge Luis Borges (1899-1986)
What is an „A“ ?

What makes something similar to
something else (specifically what makes,
for example, an uppercase letter 'A'
recognisable as such)

Metamagical Themas, Douglas
Hoffstader, Basic Books, 1985
2

First law of thermodynamics: conservation of energy


Second law: entropy


The change in the internal energy of a closed thermodynamic
system is equal to the sum of the amount of heat energy
supplied to the system and the work done on the system
The total entropy of any isolated thermodynamic system tends
to increase over time, approaching a maximum value
Entropy is a measure of disorder of the
configuration of states of the atoms or other
particles, which make up the system
3



Any physical system that is made up of many, many
tiny parts will have microscopic details to its physical
behavior that are not easy to observe (Matt McIrvin)
There are various microscopic states the system can
have, each of which is defined by the state of motion of
every one of its atoms, for instance
But all we can measure easily are its macroscopic
properties like density or pressure
Secend Law of Thermodynamics
The Second Law of Thermodynamics
can be nicely stated as follows
 A physical system will, if isolated (that is,
if energy cannot get in or out), tend
toward the available macroscopic state
in which the number of possible
microscopic states is the largest

4

Suppose that the "macrostate" is the total
of the dice
There are six ways to get a total of 7 from
the "microstates" of the two dice
 Only one way to get a total of 2 or 12
 7 is more likely


In statistical thermodynamics, Boltzmann's equation is
a probability equation relating the entropy S of an ideal
gas to the quantity W, which is the number of
microstates corresponding to a given macrostate

k is Boltzmann's constant equal to 1.38062 x
10-23 joule/kelvin and
5
Entropy - Information
Experiments starts at t0 and ends at t1
 At t0 we have no information about the
results of the experiment
 At t1 we have all information, so the
Entropy of the experiment is 0
 From t1 to t0 we have wone information

Relationship to log2
"

!
Differs by a constant, 1.4427
"

1
pi log pi = "# pi log 2 pi
#
log2 i
i
1
pi log10 pi = "# pi log 2 pi
#
log10 2 i
i
Differs by a constant, 3.3219
!
6
Information
Information is the uncertainty which
declines through the appearance of a
character
 The information content is defined by the
probability that this character appears
 Information is the gain of knowledge
 Information can be transmitted





Noiseless communications:
The decoder at the receiving end receives exactly the
characterssent by the encoder
The transmitted characters are typically not in the
original message's alphabet.
For example, in Morse Code appropriately spaced
short and long electrical pulses, light flashes, or sounds
are used to transmit the message
7
Information
Ii = log 2 (ui ) = log 2 (1/ pi ) = "log 2 ( pi )
!
Entropy in Information since

Entropy measured in bits
I = H(F) = "# pi log 2 pi
i
!
8
Only two characters


In terms of decimal expansion, the measurement has
determined a certain fixed numbers of the expansion of
m
Greater precision of the second measurement with x2
and y2
• x1 < x2 < m < y2 < y1

The gain of information after the second measurement can be
measured in the known number of digits of the decimal
expansion of m
9

The gain of information by decimal digits denoted by I10
can be expressed by following formula
# y1 " x1 &
I10 = log10 %
( = log10 (y1 " x1 ) " log10 (y 2 " x 2 )
$ y2 " x2 '
• y1-x1 range of error of the first observation
• y2-x2 range of error of the first observation
• log10(y2-x2) approximately number of digits in decimal expansion
!
We have seen that a non-constant
function I cannot be invariant of the affine
measurement group unless it involves at
least three variables
 Relative nature of information measure:

A single measurement does not
provide information
10
Trade-off between the accuracy of
measurement and the range of variability
of the measuring instrument
 Uncertainty of the observed value and
the uncertainty of the frequency range to
perform the measurement are related
 If one is small, the other has to be large



A Fourier transform of f is a function of frequency v
Let be Δv the frequency range

It can be proved that
ΔtΔv ≥ 1/4 π

If Δt is small, f corresponds to a small interval of the
whole function


Lower frequencies are not represented
For higher frequency range, a big interval of the function

Good frequency representation, bad time resolution, Δt is big
11


I(Δt,Δt´)=log2(Δt,Δt´)=-log2(ΔN/ ΔN´)
I(Δt,Δt´)+ I(ΔN/ ΔN´)=0
• Information about one variable is gained at the expense of an
equal loss of information about the other variable
• Principle of conservation of information



Each physical change of state, such as the firing of a
neuron is accompanied by the release of certain
amount of energy ∆E
The corresponding change of information ∆I is given by
∆E= c∆I




c is a constant which is proportional to the temperature T
c=k T log 2
k is Boltzmann´s constant
I measured in bits
12

Our structure, a tree is organized in levels
Suppose each node belonging to a given
level has the same number of children
 The levels are labeled from 0 to L, level 0 is
the root, level L are the leafs


How many nodes should each level contain in order
that the average waiting time required is as small as
possible?


The average witing/search time through such a tree will be
minimized if the number of nodes N(k) which belongs to the
level k is related to the number N=N(L)
N(k)=Nk/L
for k=1,...,L
• Example L=2, N(1)=N1/2
13

With N=N(L) follows



tavarage=L/2*N(1)=L/2*N1/L
Value of L which minimize tavarage
tavarage=L/2*N1/L=L/2*exp(1/L*log(N))
• dt/dL=1/2(1-log(N)/L)*exp(1/L*log(N))=0
• L=log(N)≈0.69*log2(N)
• tmin=log(N)/2*exp(log(N)/(log N))
• tmin=log(N)/2*exp(1)
• tmin≈0.94*log2(N)
Information Gain

Length of arc of a circle of radius r subtended by an
angle Θ is S(Θ)=r*Θ if Θ is measured in radians
• S(α)= α*r
• S(β)= β*r

I=log2(S(α)/ S(β))=log2(α/ β) bits
14
Let be α =2*π the a priori knowledge (one
“measurement”)
 Let be β a measurement


I(β) =log2(α/ β) =log2(2*π/ β) bits

We need only one measurement!
Example




As we move in small steps from P to 0 along P0 we see that
each angle whose corresponding information is 1 bit
The information gain is passing from one straight line to the
next, it is I(π )-I(π )=log2(π /π)=0, since the angle remains
unchanged
When the right angle at vertex 0 is reached, there is a positive
gain of information log2(π /(π/2))=1 bit
At the next step, passing from right angle to the straight angle
there is an information loss log2((π/2)/π)=-1 bit
15