Probability & Random Variables
Probability & Random Variables
Prof. Jae Young Choi, Ph.D.
Pattern Recognition Lab., Jungwon University
Department of Biomedical Engineering
Email: [email protected]
URL: http://bprlab.tistory.com/notice/4
RANDOM VARIABLES
• ▣ RANDOM VARIABLES
– D/ Random Variable X(s) : (랜덤 변수, 불규칙 변수, rv)
– 정의 구역이 표본 공간 S 인 실변수 함수로 실험에서 정의된
S 의 원소를 s라면 s∈S, rv는 X(s)로 정의 ~ X(s)
– rv는 대문자 X, Y, Z로 표시하며, 그 특정값은 소문자 x, y, z로 표시
• Ex) Tossing a coin {H, T} and a die {1, 2, 3, 4, 5, 6}
– A/ 동전 던지기에서
⑴ H가 나타나면: 주사위의 숫자를 그대로 사용 ~ X
⑵ T가 나타나면: 주사위의 숫자에 2배하여 음수(-)로 만듦 ~ -2X
Review of Probability
Random Variables
 A random variable, x, is a real-valued function defined on the
events of the sample space, S. In words, for each event in S, there
is a real number that is the corresponding value of the random
variable.
 Viewed yet another way, a random variable maps each event in S
onto the real line. That is it. A simple, straightforward definition.
Event s
X
Mapping
R.V X(s)
Review of Probability
Random Variables
Review of Probability
Random Variables (Con’t)
Example: Consider again the experiment of drawing a single card
from a standard deck of 52 cards. Suppose that we define the
following events. A: a heart; B: a spade; C: a club; and D: a diamond,
so that S = {A, B, C, D}.
A random variable is easily defined by letting x = 1 represent event A,
x = 2 represent event B, and so on.
Sample space
A
B
C
R.V space
D
1
4
2
3
Review of Probability
Random Variables (Con’t)
 Note the important fact in the examples
just given that the probability of the events
have not changed
 All a random variable does is map events
onto the real line.
RANDOM VARIABLES
RANDOM VARIABLES
• ▣ RANDOM VARIABLES
• Ex) Tossing a coin {H, T} and a die {1, 2, 3, 4, 5, 6}
– Sample Space S
S= {(H,1), (H,2), (H,3), (H,4), (H,5), (H,6), (T,1), (T,2), (T,3), (T,4),
(T,5), (T,6)}
↓ ↓
↓
↓
↓ ↓
↓ ↓ ↓ ↓ ↓ ↓
X(s)= 1
2
3
4
5
6
-2 -4 -6 -8 -10 -12
Mapping (Function)
RANDOM VARIABLES
RANDOM VARIABLES
• Ex) Continuous rv. ~ Clock Face
– 시계판을 임의로 돌렸을 때 초침이 가리키는 실수를 s라면
⑴ rv X = X(s) = s2
w/ 0 ＜ s ≤ 12
⑵ Sample Space S Mapping SX = {0 ＜ x ≤ 144}
• ▶ Random Variable X = X(s)
– s∈S → X(s) = Numerical Assignment to X based on s in S
RANDOM VARIABLES
• Conditions for a Function to Be a Random Variable
RANDOM VARIABLES
• ▣ RANDOM VARIABLES (rv)
– ▶ Conditions for a Function to be a rv
• Not a multivalued function, i.e. ∃ a unique value X(s), ∀s ∈ S
• X가 rv가 되기 위한 조건 2가지
① 집합 {X≤x}가 임의의 실수 x에 대한 사건이어야 함
P{X≤x} = sum of probabilities, ∀s ∈ S satisfying X(s)≤x
② P{X=∞} = 0 and P{X=-∞}=0
– ▶Discrete rv and Continuous rv
• Discrete rv : Discrete Sample Space에서 정의된 rv
ex) 주사위 던지기에서 X(s) = s,
w/ s∈{1≤s≤6, s는 정수}
• Continuous rv : Continuous Sample Space에서 정의된 rv
ex) 시계판을 임의로 돌려서 나타나는 실수 X(s) = s2, {0 ＜ x ≤144}
• Mixed rv : 위의 2가지 rv들을 혼용하여 사용할 경우
RANDOM VARIABLES
RANDOM VARIABLES
RANDOM VARIABLES
Example of Discrete Random Variable
RANDOM VARIABLES
Example of Discrete Random Variable (cont’d)
RANDOM VARIABLES
 Thus far we have been concerned with random variables whose
values are discrete. To handle continuous random variables we
need some additional tools. In the discrete case, the probabilities
of events are numbers between 0 and 1.
 When dealing with continuous quantities (which are not
denumerable) we can no longer talk about the "probability of an
event" because that probability is zero. This is not as unfamiliar as
it may seem.
 For example, given a continuous function we know that the area of
the function between two limits a and b is the integral from a to b
of the function. However, the area at a point is zero because the
integral from,say, a to a is zero. We are dealing with the same
concept in the case of continuous random variables.
RANDOM VARIABLES
Probability of Continuous Random Variable
RANDOM VARIABLES
• D/ Unit-Step Function : u(x)
1
u ( x)  
0
u (x)
if x  0
1.0
if x  0
• D/ Unit-Impulse Function : d(x)
1
d ( x)  
0
d (x)
if x  0
1.0
if x  0
• Relationship : u(x) vs. d(x)
du ( x )
d ( x) 
,
dx
u ( x) 

x

d ( s )ds
RANDOM VARIABLES
• ♣ DISTRIBUTION FUNCTION
RANDOM VARIABLES
♣ CUMULATIVE DISTRIBUTION FUNCTION
RANDOM VARIABLES
• ♣ CUMULATIVE DISTRIBUTION
FUNCTION, FX(x)
FX ( x)Function
 PX (Cumulative
x, x  Distribution

• D/ Distribution
Function, CDF)
:
  x |   x  , x is a real number 
n
n
i 1
i 1
FX ( x)   PX  xi u ( x  xi )   P( xi ) u ( x  xi )
• Discrete CDF :
• Properties of CDF, FX(x) :
– (1) 경계값 :
FX(-∞) = 0,
FX(+∞) = 1,
– (2) 단조비감소 :
FX(x1) ≤ FX(x2), if x1 ＜ x2
– (3) 구간확률 :
P{x1 ＜ x ≤ x2} = FX(x2) - FX(x1)
0 ≤ FX(x) ≤ 1
– (4) 우방극한값만 존재하고 연속이다 : Continuous from the right side
FX(x+) = FX(x)
D/ x+ = lime0 (x + e) w/ e ＞
RANDOM VARIABLES
• ♣ CUMULATIVE DISTRIBUTION
FX ( x)  P X  x
FUNCTIONS
• D/ Distribution Function :
• Axiomatic Skeletons
–
–
–
–
• T/
A1/
A2/
A3/
A4/
0 ≤ FX(x) ≤ 1 (-∞ ＜ x ＜ ∞)
FX(-∞) = 0, FX(∞) = 1
FX(x) is non-decreasing as x increases.
P(x1 ＜ X ≤ x2) = FX(x2) - FX(x1)
P(X ＞ x) = 1 - FX(x)
Proof) From A2/ FX(∞) = 1 and from A4/, let x1 = x, x2 = ∞
→ P(x ＜ X ＜ ∞) = FX(∞) - FX(x)
∴ P(X ＞ x) = 1 - FX(x)
RANDOM VARIABLES
• ♣ CUMULATIVE DISTRIBUTION
FUNCTIONS
RANDOM VARIABLES
• ♣ CUMULATIVE DISTRIBUTION
FUNCTIONS
RANDOM VARIABLES
• ♣ CUMULATIVE DISTRIBUTION
FUNCTIONS
RANDOM VARIABLES
• ♣ DENSITY FUNCTION, fX(x)
• D/ Density Function (Probability Density Function, PDF) :
f X ( x) 
dFX ( x)
,
dx
x  
n
n
i 1
i 1
• Discrete PDF : f X ( x)   PX  xi d ( x  xi )   P( xi ) d ( x  xi )
• Properties of PDF, fX(x) :
– (1) 경계값 :
fX(x) > 0,

– (2) 전체적분은 1이다 :

– (3) CDF와의 관계 :
FX ( x)  

f X ( x)dx  1
x

– (4) 구간확률 :
∀x
f X ( s)ds
P{x1  x  x2 }  
x2
x1
f X ( x)dx
RANDOM VARIABLES
• ♣ DENSITY FUNCTION
• D/ Density Function : f ( x)  dFX ( x)  lim FX ( x  e )  FX ( x)
X
e 0
dx
e
f X ( x)dx  Px  X  x  dx
RANDOM VARIABLES
• ♣ DENSITY FUNCTION
• D/ Density Function : f ( x)  dFX ( x)  lim FX ( x  e )  FX ( x)
X
e 0
dx
e
f X ( x)dx  Px  X  x  dx
• Axiomatic Skeletons
– A1/
fX(x) ≥ 0

(-∞ ＜ x ＜ ∞)
– A2/

– A3/
FX ( x)  
– A4/
P{x1  x  x2 }  

f X ( x)dx  1
x

f X ( s)ds
x2
x1
f X ( x)dx
RANDOM VARIABLES
• ♣ DENSITY FUNCTION
RANDOM VARIABLES
• ♣ GAUSSIAN RV
– D/ Gaussian rv :
• X is called a“Gaussian rv”if its PDF is represented by a normal
density function (정규밀도함수),
f X ( x) 
 x   X 2 
  N  X ,  X2
exp  
2
2 X 
2 X2

1

• CDF of a“Gaussian rv”is represented by
a normal distribution (정규분포)”
 s   X 2 
x
1
ds
FX ( x) 
exp  
2
2 
2 X 
2 X


RANDOM VARIABLES
♣ GAUSSIAN RV
RANDOM VARIABLES
• ♣ GAUSSIAN RV
– D/ F (x), Normalized Gaussian CDF ~ 평균 0, 분산 1
F ( x) 
1
2
 s2 
 exp   2 ds  N 0, 1  FX ( x) | X 0, X 1
x
• Properties
– F (x)에서 x  (x-X)/X 를 대입하면 일반 정규밀도함수를 구할 수 있
다
 x  X 

FX ( x)  F 
 X 
– x 의 음수 영역에 대해서는
또는
F ( x)  1  F ( x)
x  0
F ( x)  F ( x)  1
x  0
RANDOM VARIABLES
• ♣ GAUSSIAN RV
– D/ F (x), Normalized Gaussian CDF ~ 평균 0, 분산 1
RANDOM VARIABLES
• ♣ GAUSSIAN RV-Example
RANDOM VARIABLES
• ♣ GAUSSIAN RV-Example
RANDOM VARIABLES
• ♣ GAUSSIAN RV-Example
RANDOM VARIABLES
• ♣ GAUSSIAN RV-Q Approximation
RANDOM VARIABLES
• ♣ GAUSSIAN RV
– D/ Q-Function :
– T/ Q( x)  Q( x)  1,
Q( x ) 
1
2


x
 s2 
exp   ds
 2
Q( x)  F ( x)  1
– D/ Error Function : erf ( x) 
2


x
0


exp  s 2 ds
Built-in function in MATLAB
– T/
Q(x) 
1
 x 
1

erf



2
 2 
RANDOM VARIABLES
♣ Other Distribution and Density Examples
RANDOM VARIABLES
♣ Other Distribution and Density Examples
RANDOM VARIABLES
♣ Other Distribution and Density Examples
RANDOM VARIABLES
♣ Other Distribution and Density Examples
RANDOM VARIABLES
♣ Other Distribution and Density Examples
RANDOM VARIABLES
♣ Other Distribution and Density Examples
RANDOM VARIABLES
♣ Other Distribution and Density Examples
RANDOM VARIABLES
♣ Other Distribution and Density Examples
RANDOM VARIABLES
♣ Other Distribution and Density Examples
RANDOM VARIABLES
♣ Other Distribution and Density Examples
RANDOM VARIABLES
♣ Other Distribution and Density Examples
Rayleigh
Density
Rayleigh
Distribution
CONDITIONAL DISTRIBUTION &
DENSITY
• ♠ CONDITIONAL DISTRIBUTION
• Recall, Conditional Probability of A given B:
P( A  B)
P( A | B) 
w/ P( B)  0
P( B)
• D/ Conditional Distribution :
Let an event A = {x| X ≤ x} for rv. X, then P(A | B ) = P{X ≤ x |
B}
= FX(x | B ) is defined as“conditional distribution function”of X.
FX ( x | B)  PX  x | B) 
P({ X  x}  B)
P( B)
w/ P( B)  0
{ X  x}  B
w/
is the joint event of all outcomes s.t.
X(s) ≤ x and s ∈ B, and P(B) ≠ 0
CONDITIONAL DISTRIBUTION &
DENSITY
• ♠ CONDITIONAL DISTRIBUTION
• Properties of FX(x | B ) :
(1) 경계조건 :
FX ( | B)  0, FX ( | B)  1
0  FX ( x | B)  1
(2) 단조비감소성 : FX ( x1 | B)  FX ( x2 | B)
(3) 구간확률 :
if
x1  x2
P{x1  X  x2 | B}  FX ( x2 | B)  FX ( x1 | B)
(4) 우방극한값만 존재 :
FX ( x  | B)  FX ( x | B)
CONDITIONAL DISTRIBUTION &
DENSITY
• ♠ CONDITIONAL DENSITY
• D/ Conditional Density :
“conditional density function”of rv. X is the derivative of
the conditional distribution function, s.t.
f X ( x | B) 
dFX ( x | B )
dx
• NOTE :
If rv. X is discrete or mixed, FX(x|B ) contains
step discontinuities, then fX(x|B ) then has
impulse functions at discontinuities.
CONDITIONAL DISTRIBUTION &
DENSITY
• ♠ CONDITIONAL DENSITY
• Properties of fX(x | B ) :
(1) 경계조건 :
(2) 총밀도적분 :
f X ( x | B)  0



f X ( x | B)dx  1
(3) CDF vs. PDF : F ( x | B)  x f ( s | B)ds
X
 X

(4) 구간확률 :
P{x1  X  x2 | B}  
x2
x1
f X ( x | B)dx
CONDITIONAL DISTRIBUTION &
DENSITY
♠ EXAMPLE
CONDITIONAL DISTRIBUTION &
DENSITY
♠ EXAMPLE
CONDITIONAL DISTRIBUTION &
DENSITY
♠ EXAMPLE
CONDITIONAL DISTRIBUTION &
DENSITY
• ♠ METHODS OF DEFINING CONDITIONING
EVENTS
• 조건 event B를 정하는 방법은 여러 가지가 있다. 예를 들어,
B = {X ≤ b} 라면
 ☞ Conditional Distribution Function :
P{ X  x} FX ( x)
F X ( x | X  b) 
• (i) x ＜ b 일 때 :
F X ( x | X  b) 
• (ii) x ≥ b 일 때 :
FX ( x) / FX (b)
 F X ( x | X  b)  
1

P{ X  b}

P{ X  b}
1
P{ X  b}
xb
xb
Therefore,
dF ( x | X  b)  f X ( x) / FX (b)

0
dx

X
 ☞ Conditional
:
 f X( x |Density
X  b)  Function
FX (b)
xb
xb
ONAL DISTRIBUTION & DENSITY
♠ METHODS OF DEFINING CONDITIONING
From our assumptions that the conditioning event has nonzero probability, we have 0  F ( x)  1
so EVENTS
that the conditional distribution function is never smaller than the ordinary distribution
X
function
FX ( x | X  b)  FX ( x)
f X ( x | X  b)  f X ( x ) x  b
RANDOM VARIABLES
• ◆ DISCRETE vs. CONTINUOUS DENSITY
& DISTR.
Discrete Density & Distr.
Continuous Density & Distr.
RANDOM VARIABLES
• ♣ DISTRIBUTION & DENSITY EXAMPLES
• ▶ Binomial rv. ~ X
⑴ Binomial Density Function :
n
f X ( x)     p k (1  p) n k d ( x  k )
k 0  k 
n
for n = 1, 2, …, and 0＜p＜1
⑵ Binomial Distribution Function : F ( x)  n  n  p k (1  p) n k u ( x  k )
 
X
k 0
k 
NOTE : Bernoulli (independent & repeated) trial experiment ~ S = {success,
failure}
• ▶ Poisson rv. ~ X
⑴ Poisson Density Function :
e bb k
f X ( x)  
d (x  k)
k
!
k 0

k constant b＞0,
for 
abreal
⑵ Poisson Distribution Function :
e b
u( x  k )
k
!
k 0
FX ( x)  
b = λT (λ~avg. rate, T~주기)
Ex) Counting-type 프로세스 응용: 제조 산업의 불량율, 전화오는 횟수, 음극의 전자 방출
율
cf. If n→∞ and p→0, then np ≒ b (constant) & Binomial → Poisson
RANDOM VARIABLES
• ♣ DISTRIBUTION & DENSITY EXAMPLES
Binomial
Poisson
RANDOM VARIABLES
• ♣ DISTRIBUTION & DENSITY EXAMPLES
• ▶ Uniform rv. ~ X
⑴ Uniform Density Fn. :
 1

f X ( x)   b  a

 0
a xb
elsewhere
for -∞＜a＜∞, b＞a
 0
x a
⑵ Uniform Distr. Fn. :
FX ( x)  
b  a
 1
Ex) PC의 Random Number Generator
xa
a xb
xb
• ▶ Exponential rv. ~ X
 1 ( x  a ) / b
 e
⑴ Exponential Density Fn. : f X ( x)   b

0

xa
xa
for -∞＜a＜∞, b＞0
⑵ Exponential Distr. Fn. :
1  e  ( x a ) / b
FX ( x)  
0

Ex) 어떤 aircraft로부터 반사된 radar신호 세기의 변화
xa
xa
RANDOM VARIABLES
• ♣ DISTRIBUTION & DENSITY EXAMPLES
Uniform
Exponential
RANDOM VARIABLES
• ♣ DISTRIBUTION & DENSITY EXAMPLES
• ▶ Rayleigh rv. ~ X
Rayleigh
⑴ Rayleigh Density Fn. :
2
2
 ( x  a )e  ( x  a ) / b
f X ( x)   b

0

xa
xa
for -∞＜a＜∞, b＞0
⑵ Rayleigh Distr. Fn. :

1  e  ( x  a )
FX ( x)  

0

2
/b
xa
xa
Ex) Band Pass Filter (BPF) 통과한 신호 잡음의 진폭,
여러 가지 계측 시스템의 잡음
• ☞ NOTE: Refer to MATLAB Examples of Densities & Distributions
RANDOM VARIABLES
• ♣ DISTRIBUTION & DENSITY EXAMPLES
• Histograms
Uniform
Rayleigh
RANDOM VARIABLES
• ♡ MATLAB EXAMPLES
• Histograms
% gaushist.m histogram of gausian r.v.
n=1000;
x=2*randn(1,n)+5*ones(1,n);
% generate vector of samples
plot(x);
% draw 1st figure
xlabel('Index'); ylabel('Amplitude'); grid;
pause;
[m,z]=hist(x);
% calculate counts in bins and bin coordinates
w=max(z)/10;
% calculate bin width
mm=m/(1000*w);
% find probability in each bin
v=linspace(min(x),max(x));
% generate 100 values over range of rv X
y=(1/(2*sqrt(2*pi)))*exp(-((v-5*ones(size(v))).^2)/8); % gaussian PDF
bar(z,mm,'w');
% plot histogram
hold on;
% retain histogram plot
plot(v,y);
% superimpose plot of gaussian PDF
xlabel('Random Variable Value'); ylabel('Probability Density');
hold off;
% release hold of plot