CS188 Discussion Week 14: Reinforcement Learning
By Nuttapong Chentanez
Want to compute
Markov Decision Process (MDP)
-Consists of set of states s  S
-Transition model model, T(s, a, s’) = P(s’ | s, a)
- Probability that executing action a in state s leads to s’
-Reward function R(s, a, s’),
- Can also write Renter(s’) for reward for entering into s’
- Or Rin(s) for being in s
-A start state (or a distribution)
-May be terminal states
Bellman’s Equation
Idea: Optimal rewards = maximize over first action and then follow
optimal policy
How?
Simple Monte-Carlo (Sampling)
*(s) =
Value Iteration
Theorem: Will converge to a unique optimal values. Policy may
converge long before values do.
Policy Iteration
Policy Evaluation: Calculate utilities for a fixed policy
Policy Improvement: Update policy based on resulting converged utility
Repeat until policy does not change
In practice, no need to compute “exact” utility of a policy just rough
estimate is enough.
Combine dynamic programming with Monte-Carlo
-One step of sampling and use current state value
-Aka. Temporal Difference Learning (TD)
Reinforcement Learning
Still have MDP

Still have an MDP:

A set of states s  S

A set of actions (per state) A

A model T(s,a,s’)

A reward function R(s,a,s’)

Still looking for a policy (s)

However, the agent don’t know T or R

Must actually try actions at states out to learn
Reinforcement Learning in Animals
Studied experimentally in psychology for > 60 years
Rewards: food, pain, hunger etc.
Example: Bees learn near-optimum foraging plan in artificial flowers
field
Dolphin training
Model-Based Learning
-Can try to learn T and R first then

Simplest case:

Count outcomes for each s,a

Normalize to give estimate of T(s,a,s’)

Discover R(s,a,s’) the first time we experience (s,a,s’)
Passive Learning
-Given a policy (s), try to learn V(s), don’t know T, R
TD Features
-On-line, Incremental, Bootstrapping(use info that we learn so far)
-Model free, Converge as long as  decreases over time eg (1/k), <1
Problems with TD
-TD is for learning value of state under a given policy
-If we want to learn optimal policy, won’t work
-Idea, learn state-action pair (Q-values) instead
Q-Functions
-Utility of starting at state s, taking action a, then follow  thereafter
-Q-value of optimum policy, utility of starting at state s, taking action a,
then follow optimum policy afterward
Another solution: Exploration Function
Idea: instead of exploring a fixed amount, can explore area where the
value is not yet established
Eg. f(u,n) = u + k/n, k is a constant
Q-Learning Algorithm, applying TD idead to learn Q *
Practical Q-Learning
-In realistic situation, too many states to visit, may even be infinite
-Need to learn from small training data
-Be able to generalize to new, similar states
-Fundamental problem in machine learning
Function Approximation
-Inefficient/Infeasible to learn each state-action pair q-value
-Suppose we approximate Q(st,at) with a function f with parameters 
-Can do gradient-descent update
Learn Q*(s,a) values
-Receive a sample (s,a,s’,r)
-Consider your old estimate:
-Consider your new sample estimate:
-Modify the old estimate towards the new sample:
-Equivalently, average samples over time:
Converge as long as  decreases over time eg (1/k), <1
Exploration vs. Exploitation
-“If always take best current action, will never explore other action and
never know there is a better action”
-Explore initially then exploit later
-Many scheme for balancing exploration vs. exploitation

Simplest: random actions (-greedy)

Every time step, flip a coin

With probability , act randomly

With probability 1-, act according to current policy
(best q value for instance)
Will explore space but keep doing random action even after the learning
is “done”
A solution: Lower  over time
Project time
- vt is the reward received
- Idea: Gradient indicate the direction of changes in  that most
increase Q
Want Q to looks more like vt, so modify each parameter
depending on whether
increasing or decreasing a parameter will make Q more like vt
- In this simple form, does not always work, but the idea is on the right
track
- Feature selection is difficult in general