### 7-Learning Objective Functions

```Learning Objective Functions
Machine Learning CSx824/ECEx242
Bert Huang
Virginia Tech
Outline
•
Optimization and machine learning
•
Types of optimization problems
•
Unconstrained, constrained
•
Lagrange multipliers for constrained optimization
•
Convex, non-convex, discrete
Optimization and Machine Learning
ŵ
argmin
w 2Rd
2
>
w w+
n
X
>
log(1 + exp( yi w xi ))
i=1
regularizer
loss
parameter space
hypothesis space
objective function
Types of Optimization Problems
•
Unconstrained, constrained
•
Lagrange multipliers for constrained optimization
•
Convex, non-convex, discrete
Unconstrained Optimization
-∞
∞
Constrained Optimization
-∞
-10
100
∞
Penalty Functions
∞
-∞
-10
∞
100
∞
Lagrange Multipliers
argmin f (x) subject to g (x) = 0
x
g (x) = I (x 2 [ 10, 100])
argmin max f (x) + ↵g (x)
x
-∞
-10
100
↵
Lagrangian
∞
x2 + (1 - x) y
argmin max f (x) + ↵g (x)
x
↵
Convexity
f (↵x1 + (1
↵)x2 )  ↵f (x1 ) + (1
↵)f (x2 )
↵ 2 [0, 1]
x1 , x2 2 R
d
Convex Functions
Logistic Regression Neg. Log Likelihood
Bernoulli Negative Log Likelihood
Non-Convex Optimization
x
h
y
f (x) = (wh (wx x))
Local Optimization
Bounding Methods
Bounding Methods
Discrete Optimization
•
Feature selection, decision tree learning
•
Often intractable
•
Relaxation to continuous optimization
Categorizing Machine Learning Techniques
Convex
Constrained
Nonconvex
Unconstrained
multilayered perceptron
expectation maximization
support vector machines
logistic regression
naive Bayes