Towards robotic assistants in nursing homes:
challenges and results
Joelle Pineau
Michael Montemerlo
Martha Pollack *
Nicholas Roy
Sebastian Thrun
Carnegie Mellon University
*University of Michigan
Introducing Pearl –
A mobile robotic assistant for elderly people and nurses
ROLE:
cameras
Providing info
(TV, weather)
LCD mouth
microphone
& speakers
touchscreen
handle bars
sonars
carrying tray
laser
mobile base
The Nursebot Project
Reminding to eat,
drink, take meds
Monitoring Rx
adherence & safety
Calling for help
in emergencies
Supporting
communication
Remote
health services
Providing
physical assistance
Management
support of ADLs
Linking caregiver
and resources
Moving things
around
Joelle Pineau
The Nursebot project in its early days
The Nursebot Project
Joelle Pineau
Architecture
High-level controller
Navigation
The Nursebot Project
Cognitive support
Communication
Joelle Pineau
Architecture
High-level controller
Navigation
Cognitive support
Communication
• Localization and map building
(Burgard et al., 1999)
• People detection and tracking
(Montemerlo et al., 2002)
The Nursebot Project
Joelle Pineau
Architecture
High-level controller
Navigation
Cognitive support
Communication
• Autominder system
(Pollack et al., 2002)
The Nursebot Project
Joelle Pineau
Architecture
High-level controller
Navigation
Cognitive support
Communication
• Speech recognition: Sphinx system
(Ravishankar, 1996)
• Speech synthesis: Festival system
(Black et al., 1999)
The Nursebot Project
Joelle Pineau
The role of the top-level controller
High-level controller
Navigation
Cognitive support
Communication
ACTION SELECTION - based on the trade-off between:
- goals from different modules;
- goals with varying costs / rewards;
- reducing uncertainty versus accomplishing goals.
The Nursebot Project
Joelle Pineau
Speech recognition with Sphinx
The Nursebot Project
Joelle Pineau
Robot control under uncertainty
Speech=“today”
Belief State
P(st=weather-today)=0.5
P(st=appointment-today )=0.5
State
weather-today
USER
Action={ say-weather,
update-appointment,
clarify-query}
The Nursebot Project
Joelle Pineau
Robot control using Partially Observable Markov
Decision Processes (POMDPs)
Belief state
P(s2
)
Observations
Costs / Rewards
P(s1)
State
USER + ENVIRONMENT + WORLD
Actions
Problem: Which action allows the robot to maximize its reward?
The Nursebot Project
Joelle Pineau
Methods to solve POMDPs
Objective: Find a policy, (b), which maximizes reward.
POMDP
New methods?
Performance
AMDP
FIB
QMDP
MDP
O(S2A) O(S2AO) O(S2AB)
UMDP
O(S2AT)
T
O(S2AO )
Complexity
The Nursebot Project
Joelle Pineau
New approach: A hierarchy of POMDPs
Idea: Exploit domain knowledge to divide one POMDP into many smaller ones.
Motivation: Complexity of POMDP solving grows exponentially with # of actions.
Assumption: We are given POMDP M = {S,A,,b,T,O,R} and hierarchy H
subtask
Act
abstract action
Navigate
ExamineHealth
VerifyPulse
VerifyMeds
Move
ClarifyGoal
primitive action
North South East West
The Nursebot Project
Joelle Pineau
PolCA: Planning with a hierarchy of POMDPs
Step 1: Select the action set
Navigate
AMove = {N,S,E,W}
Move
North
South
East
ClarifyGoal
West
ACTIONS
North
South
East
West
ClarifyGoal
VerifyPulse
VerifyMeds
The Nursebot Project
Joelle Pineau
PolCA: Planning with a hierarchy of POMDPs
Step 1: Select the action set
Step 2: Minimize the state set
Navigate
AMove = {N,S,E,W}
SMove = {X,Y}
Move
North
ACTIONS
North
South
East
West
ClarifyGoal
VerifyPulse
VerifyMeds
The Nursebot Project
South
East
ClarifyGoal
West
STATE FEATURES
X-position
Y-position
X-goal
Y-goal
HealthStatus
Joelle Pineau
PolCA: Planning with a hierarchy of POMDPs
Step 1: Select the action set
Step 2: Minimize the state set
SMove = {X,Y}
Step 3: Choose parameters
Move
North
ACTIONS
North
South
East
West
ClarifyGoal
VerifyPulse
VerifyMeds
The Nursebot Project
Navigate
AMove = {N,S,E,W}
STATE FEATURES
X-position
Y-position
X-goal
Y-goal
HealthStatus
South
East
ClarifyGoal
West
PARAMETERS
{bh,Th,Oh,Rh}
Joelle Pineau
PolCA: Planning with a hierarchy of POMDPs
Step 1: Select the action set
Step 2: Minimize the state set
Navigate
AMove = {N,S,E,W}
SMove = {X,Y}
Step 3: Choose parameters
Move
ClarifyGoal
Step 4: Plan task h
North
ACTIONS
North
South
East
West
ClarifyGoal
VerifyPulse
VerifyMeds
The Nursebot Project
STATE FEATURES
X-position
Y-position
X-goal
Y-goal
HealthStatus
South
East
West
PARAMETERS
PLAN
{bh,Th,Oh,Rh}
h
Joelle Pineau
First study in simulation
• 20-questions domain:
Agent:
User:
Agent:
User:
Agent:
User:
Agent:
Objective:
“Is it an animal?”
“No.”
“Is it a vegetable?”
“Yes.”
“Is it green?”
“No.”
“...?”
Actions
Observations
Plan a sequence of questions allowing the
agent to identify the chosen object.
Small complication…. the user can change objects at any
time (Pr = 0.1), without telling the agent.
The Nursebot Project
Joelle Pineau
The hierarchy
Begin
Vegetables
…
Minerals
Animals
Vegetable? Mineral? Animal?
…
Mammal? Herbivore? RABBIT TURTLE
The Nursebot Project
…?
Joelle Pineau
Methods to solve POMDPs
POMDP
PolCA
Performance
AMDP
FIB
QMDP
MDP
UMDP
Complexity
The Nursebot Project
Joelle Pineau
Results
Domain: |S|=12, |A|=20, |O|=3
0
POMDP
PolCA-D1
-20
FIB
QMDP
-40
R -60
-80
-100
-120
0.01
0.1
1
10
100
1000
10000
100000
1000000
Time (secs)
The Nursebot Project
Joelle Pineau
A new decomposition
Begin
Vegetables
…
The Nursebot Project
Minerals
…
Animals
Vegetable? Mineral? Animal?
…
Joelle Pineau
A new decomposition
Begin
Vegetables
Minerals
Animals
…
Fruits
…
The Nursebot Project
Plants
Vegetable? Mineral? Animal?
…
Fruit?
…
Joelle Pineau
Results
Domain: |S|=12, |A|=20, |O|=3
0
POMDP
PolCA-D1
PolCA-D2
FIB
QMDP
-20
-40
R
-60
-80
-100
-120
0.01
0.1
1
10
100
1000
10000
100000
1000000
Time (secs)
The Nursebot Project
Joelle Pineau
Methods to solve POMDPs
POMDP
PolCA
Performance
AMDP
FIB
QMDP
MDP
UMDP
Complexity
The Nursebot Project
Joelle Pineau
PolCA in the Nursebot domain
• Goal: A robot is deployed in a nursing home, where it provides
reminders to elderly users and accompanies them to
appointments.
• Domain: |S|=512, |A|=20, |O|=19
• Hierarchy:
The Nursebot Project
Joelle Pineau
Sample scenario
The Nursebot Project
Joelle Pineau
Results for dialogue system
POMDP policy
MDP policy
0.18
0.1
The Nursebot Project
0.1
Joelle Pineau
Summary
• We have developed a first prototype robot able to serve as a
mobile nursing assistant for elderly people.
• The top-level controller uses a hierarchical variant of POMDPs
to select actions.
• This allows it to acquire necessary information and successfully
complete assigned tasks.
• Probabilistic techniques have been found to be very useful to
flexibly model and track individuals.
The Nursebot Project
Joelle Pineau
The Nursebot team
CMU - Robotics:
Greg Armstrong
Michael Montemerlo
Joelle Pineau
Nicholas Roy
Jamie Schulte
Sebastian Thrun
CMU - HCI/Design:
Francine Gemperle
Jennifer Goetz
Sarah Kiesler
Aaron Powers
U. of Pittsburgh - Nursing:
Jacqueline Dunbar-Jacobs
Sandra Engberg
Judith Matthews
U. of Pittsburgh - CS:
Don Chiarulli
Colleen McCarthy
U. of Freiburg - CS:
Maren Bennewitz
Wolfram Burgard
Dirk Schulz
U. of Michigan - CS:
Laura Brown
Dirk Colbry
Cheryl Orosz
Bart Peintner
Martha Pollack
Sailesh Ramakrishnan
Standard Robotics:
Greg Baltus
For more details: www.cs.cmu.edu/~nursebot
The Nursebot Project
Joelle Pineau
Our vision of robotic healthcare
Management
support of
ADLs
Reminding
to eat, drink,
& take meds
Linking
the caregiver
to resources
Providing
physical
assistance
Moving things
around
Providing
information
(TV, weather)
Monitoring
Rx adherence
& safety
Calling for help
in emergencies
Supporting
inter-personal
communication
Enabling use
of remote
health services
The Nursebot Project
Joelle Pineau
Profile of an aging population
 450,000 nurses to recruit before 2008 (USA)
The Nursebot Project
Joelle Pineau
Localization and map building
The Nursebot Project
Joelle Pineau
People tracking
The Nursebot Project
Joelle Pineau
Analysis of movements
The Nursebot Project
Joelle Pineau
Autominder System
The Nursebot Project
Joelle Pineau
The family of Markov models
State ambiguity?
Choice of
action?
no
yes
The Nursebot Project
no
yes
Markov Chain
Hidden Markov Model
(HMM)
Markov Decision
Process (MDP)
Partially Observable
Markov Decision
Process (POMDP)
Joelle Pineau
The POMDP model
The POMDP is a septuple { S, A, , b, T, O, R }
rt-1
State:
...
Action:
Belief:
...
st-1
ot-1
bt-1
at-1
rt
st
ot
bt
...
??
...
Problem: Which action allows the robot to maximize its reward?
The Nursebot Project
Joelle Pineau
Execution with a hierarchy of POMDPs
At each time step, traverse the hierarchy from top to bottom.
For each subtask, consult the policy at h(bt)
If at is an internal node > move to that subtask.
Act
Navigate
ExamineHealth
VerifyPulse
VerifyMeds
Move
ClarifyGoal
North South East West
The Nursebot Project
Joelle Pineau
How to choose parameters?
• Consider for example TNavigate(s,a,s’)
• Case #1: a is a primitive action.
TNavigate(s,ClarifyGoal,s’)  T(s,ClarifyGoal,s’)
S’
0.7
S’
Navigate
North
ClarifyGoal
S’
0.3
The Nursebot Project
Move
S
South
S’
North
South
ClarifyGoal
East
West
Joelle Pineau
How to choose parameters?
• Consider for example TNavigate(s,a,s’)
• Case #1: a is a primitive action.
TNavigate(s,ClarifyGoal,s’)  T(s,ClarifyGoal,s’)
• Case #2: a is an abstract action.
TNavigate(s,Move,s’)  T(s, Move(s) ,s’)
S’
0.9
North
ClarifyGoal
S’
S’
Navigate
0.2
Move
S
South
0.1
S’
North
South
ClarifyGoal
East
West
0.8
The Nursebot Project
Joelle Pineau
The details
States:
1 per object (e.g. tomato, cucumber, rabbit, turtle, tulip, ….)
Observations:
“yes”, “no”, “”
Actions:
1 guess per objet
+ many questions
Rewards: question
correct guess
incorrect guess
= -1
= +5
= -20
Begin
Vegetables
…
Minerals
Animals
Vegetable? Mineral? Animal?
…
Mammal? Herbivore? RABBIT TURTLE
The Nursebot Project
…?
Joelle Pineau