Learning Objective Functions Machine Learning CSx824/ECEx242 Bert Huang Virginia Tech Outline • Optimization and machine learning • Types of optimization problems • Unconstrained, constrained • Lagrange multipliers for constrained optimization • Convex, non-convex, discrete Optimization and Machine Learning ŵ argmin w 2Rd 2 > w w+ n X > log(1 + exp( yi w xi )) i=1 regularizer loss parameter space hypothesis space objective function Types of Optimization Problems • Unconstrained, constrained • Lagrange multipliers for constrained optimization • Convex, non-convex, discrete Unconstrained Optimization -∞ ∞ Constrained Optimization -∞ -10 100 ∞ Penalty Functions ∞ -∞ -10 ∞ 100 ∞ Lagrange Multipliers argmin f (x) subject to g (x) = 0 x g (x) = I (x 2 [ 10, 100]) argmin max f (x) + ↵g (x) x -∞ -10 100 ↵ Lagrangian ∞ Saddle Point Optimization x2 + (1 - x) y argmin max f (x) + ↵g (x) x ↵ Convexity f (↵x1 + (1 ↵)x2 )  ↵f (x1 ) + (1 ↵)f (x2 ) ↵ 2 [0, 1] x1 , x2 2 R d Convex Functions Logistic Regression Neg. Log Likelihood Bernoulli Negative Log Likelihood Non-Convex Optimization x h y f (x) = (wh (wx x)) Local Optimization Bounding Methods Bounding Methods Discrete Optimization • Feature selection, decision tree learning • Often intractable • Relaxation to continuous optimization Categorizing Machine Learning Techniques Convex Constrained Nonconvex Unconstrained multilayered perceptron expectation maximization support vector machines logistic regression naive Bayes Saddle Point support vector machines expectation maximization perceptron support vector machines Discrete decision tree learning