Funder: DARPA CSSP
Imbuing Human-Robot
Teams with Intention
Recognition
Dr. Gita Sukthankar
[email protected]
Students: Ken Laviers, Bennie Lewis
Intelligent Agents Lab
Software
Agents
Intention
(Plan,
Activity, Goal)
Recognition
Humans
(Biological Agents)
Robots
(Mechanical Agents)
Research Problems

Improving plan, activity, and intention recognition





Determining when to act autonomously


Transfer-of-control
Identifying what to do


Fast
Sufficiently accurate
Acquiring training data
Making it sample-efficient
This can be the hardest problem!
Not annoying the human users!


Mutual predictability
Human must be able to infer the intentions of the agents/robots
Domains




Improved game and simulation AI
Intelligent training and tutoring systems
Human-robot interfaces
Elder-care home monitoring systems
Example: Adaptive Opponents
Exploit adversarial models to improve team decision-making
Online Play Recognition




Create tree of team spatiotemporal traces
Combine output from multiple
classifiers to reliably
recognize plays
Modify policy of key players
to improve play of entire team
Adapt in real-time to the
strategy employed by the
human player
Learning to Adapt
Play Recognizer

Divide and conquer the
problem into several
learning modules


?



Play recognizer
Successor state estimator
Reward estimator
Individual modules are
inaccurate but combine to
learn an effective play
adaptation.
Use Monte Carlo search
to rapidly evaluate large
number of play
adaptations
Results
Adaptive agents improves the yardage gained in a play
and reduce the number of interceptions
over the standard game AI.
8
Human-Agent-Robot Teams

Robots need humans to use their past experience and
common-sense knowledge:



Humans need help with repetitive tasks:



To process ambiguous sensor data
To solve complicated planning problems (e.g., figuring out the
grasp points on objects)
Monitoring multiple information streams (video or audio)
Toggling between multiple robots
Agents can facilitate HRI by:


Monitoring the humans to identify operator distraction
Remembering and propagating commands intelligently across
teams of robots
User Interface
(view from an overhead camera)
User Interface
(gamepad control is popular with our student test subjects)
RSARSim

(video)
Learn Models of User Distraction





Agent
Learn model of user distraction
by inserting artificial visual
distractions into simulation
Identify which of the three
robots the user is paying
attention to
Features based on robot
motion trajectories
Use EM to fit parameters to
HMM model
Perform transfer-of controlwhen distraction levels go over
a certain level
Learn Models of User Distraction
5 class
2 class
HMM
37%
88%
SVM
28&
67%
Bayes
28%
67%
DT
28%
67%



Identification of user distraction
level more accurate than
models that don’t remember
past state
Two state classification
accuracy shows our decision
threshold (control vs. nocontrol)
Statistically significant
improvements (p<.05) on time
required to find the total
number of victims in urban
rescue scenario
Multi-Robot Manipulation




Sensors on robot are insufficient
for good grasp planning
Toggling rapidly between robots
is complicated for users
Idea: leverage commands given
by user to one robot to
propagate (and translate) for
second robot)
User study evaluating command
paradigm:



Follow Me: 2nd robot joins the 1st
robot
Mirror Me: 2nd robot copies the 1st
robot
Scenario involves moving piles
of objects to a goal location,
some of which require two
robots to move
Human-Agent-Robot Teams




User study on 20 users had very promising results
Introducing these two new primitives results in
reductions in both time required to complete the task and
in reducing the number of object drops in most of the
scenarios
Favorable responses on the post-test questionnaire
Current work:
 Incorporating a learning by demonstration mode to
allow users to learn the primitives rather than having
them preprogrammed by the developer
Conclusions
Robots
(Mechanical Agents)
Software
Agents
Humans
(Biological Agents)
Agents are well-positioned to serve as an enabler of mutual
predictability through a combination of intention recognition
and communication monitoring.
Multi-Robot Manipulation