Jacob
an instruction agent in VR
Marc Evers ([email protected])
6 March 2000
1
Contents
context & project description
approach & technology
current state & next steps
architecture of the system
design details
conclusions
2
Context
pilot project of VR Valley Twente
objective: establishing a regional knowledge
centre on VR
organisations: UT (INF, WB), HE, HSA, Carp,
MST, ...
6 pilot projects
http://www.vr-valley.com/
3
Jacob Project
Jacob as assistant
tasks: instruction, navigation, etc.
Jacob shows intelligent behaviour
multimodal interaction
interaction between multiple agents
manipulation of objects in the virtual
world
4
Approach
use existing knowledge and theories
Intelligent Tutoring Systems, Computer
Graphics, AI, …
software engineering focus
how to build such a system in a
maintainable, adaptable, …, way
use OO techniques & software architecture
knowledge
5
Technology
runs on any PC or Workstation with web
browser+VRML browser+3D graphics card
programming languages
Java 1.1; VRML 97
standards
H-Anim for Jacob’s body
under consideration: Java 3D, JavaSpeech
6
Current state
Jacob teaches construction task
Towers of Hanoi
manipulation of objects in virtual world
by Jacob and by user
Jacob has explicit model of the task
simulation of physical processes
gravity, collisions
simple animation
7
Next steps
natural language dialogue
‘intelligence’ for Jacob
multi user Jacob (DeepMatrix)
integration in VMC, in 2 steps:
first, Jacob will play isolated role in VMC
second, Jacob interacts with VMC
8
Basic architecture
‘task’ layer (?)
abstract 3D World: agents, cubes, tables, ...
abstraction of 3D platform
3D platform: VRML/EAI; Java 3D; ...
9
World View
10
Class VWorld
simulation of physical aspects
gravity, collisions
other physical constraints
management of VE objects
agents & objects
keeps track of the state of VE objects
observations
11
Class VEObject
defines general behaviour of VE objects
bounding box, position, size
gravity, (b)locking
relations like isOnTopOf
12
Subclasses of VEObject
Cube
detects picking, dropping, dragging by user
Avatar
knows position and orientation of user
Jacob
Jacob’s manifestation in the world
lookAt, moveObject, walktTo, shakeHead
13
‘Physics’ protocol
: DraggingSensor
NodeProxy
: VWorld
c : Cube
: GravityThread
: World
Listener
1: drop performed
2: agentMovesObject(c)
3: enforceWorldConstraints( )
4: startGravity( )
5: start( )
6: setPosition( )
7: update( )
8: worldStateChanged()
14
‘Physics’ protocol
: Gravity
Thread
3: enforceWorldConstraints( )
7: update( )
5: start( )
6: setPosition( )
2: agentMovesObject(c)
c : Cube
:
VWorld
4: startGravity( )
1: drop performed
: DraggingSensor
NodeProxy
8: worldStateChanged()
: World
Listener
15
Collision avoidance protocol
c : Cube
: VWorld
: VEObject
b1 : Bounding
Box
b2 : Bounding
Box
1: setPosition(v)
2: avoidCollision(c,v)
3: avoidCollision(c,v)
4: canCollide( )
5: getBoundingBox( )
6: getBoundingBox( )
7: avoidCollision(b2, v)
8: get...
16
Collision avoidance protocol
1: setPosition(v)
c : Cube
4: canCollide( )
5: getBoundingBox( )
2: avoidCollision(c,v)
6: getBoundingBox( )
3: avoidCollision(c,v)
:
VWorld
:
VEObject
7: avoidCollision(b2, v)
b2 :
BoundingBox
b1 :
BoundingBox
8: get...
17
Instruction View
18
Class Task
knows how to create the ‘scene’
has knowledge on performing the task
Schema: hierarchical structure
Composite design pattern
action, explanation, pre/postcondition,
invariant
schema interface is generic
SchemaIterator determines current task state
19
Class InstructionModel
InstructionRule
rules that fire other rules or start instruction
methods based on conditions
InstructionMethod
executed until goal or state is reached
example:
while user does nothing: wait, give hint, wait,
show move
20
Task update protocol
: VWorld
t : Task
: Schema
Iterator
: Schema
: Instruction
Model
1: worldStateChanged( )
2: updateCurrent( )
3: precondition(t)
4: postcondition(t)
5: invariant(t)
6: getSubSchemas( )
7: taskStateChanged( )
8: currentSchema( )
21
Task update protocol
1: worldStateChanged( )
:
VWorld
2: updateCurrent( )
t : Task
8: currentSchema( )
7: taskStateChanged( )
: Schema
Iterator
3: precondition(t)
4: postcondition(t)
5: invariant(t)
6: getSubSchemas( )
: Instruction
Model
:
Schema
22
Concluding remarks
current results can be generalised
Jacob based on existing knowledge and
theories
instruction agent concept is very general
instruction, assistance, design guidance…
demo & documentation
http://www.cs.utwente.nl/~evers/jacob/
23
Jacob Screenshot
24