VR - software, the missing piece?
DISTRIBUTION/
COMMUNICATION
HARDWARE
Software
SIMULATION
SOUND
INTERACTION
GRAPHICS
REALTIME ASPECTS
2005-02-09
1
©Anders Backman, Dept. Computing Science,
VR05 - Software
HUMAN ISSUES
SOFTWARE
APPLICATIONS
2
Software
2005-02-09
Software
int main(int argc, char **argv) {
// Init
// Loop forever and do:
// Read devices
VRdevice_read();
// Update objects
VRupdate_objects();
// Render the Scene
VRrender_scene();
}
n
VR/Game engine
n
n
Its a complex system
Hardware:
n
Visual display
n
Image generator
n
Tracking system(s)
n
n
n
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
Sound system(s)
Interaction devices
n
Computing environment
n
n
n
n
n
n
n
Network
n
n
n
n
5
n
Scripting
AI
Physics
Collision detection
©Anders Backman, Dept. Computing Science, VR05 - Software
n
Network
Interaction
LOD
Culling
Rasterizing
2005-02-09
So we have a bunch of technological
issues to solve
n
Rendering
n
Heterogenous, TCP, UDP, LAN, Modem,...
Software
Communication
n
Gloves etc..
Operating system, single multi CPU, shared/distributed memory...
2005-02-09
Simulation
n
Headspeaker, 2-4-8 speakers
4
Software:
n
One graphics card, Multi-pipe onyx, Cluster of PC:s
Camera based, magnetic, ultrasound, ...
n
n
Software
n
Computer screen, CAVE, Powerwall, ...
n
n
3
©Anders Backman, Dept. Computing Science, VR05 - Software
©Anders Backman, Dept. Computing Science, VR05 - Software
6
How to render as many polygons as fast as
possible, how to simulate ragdolls, etc...
But dont forget the final result, if everyone
have to focus on how to store the triangles
in the most efficient way to avoid stalling
the graphics pipe? Then who is going to
focus on the content?
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
1
Software
n
Software
We must also have a production ”pipeline”
n
n
n
Character
Modelling/Animation
To be able to let the modelers focus on
modelling, sound-designers work on sound,
artists work on texturing/design, etc.. We
need to develop tools that aids them in the
process of creating the final result.
Content is king.
Now what is a production pipeline?
Simulation
modelling
AI design
Environment
Scriping/events
Modelling/texturing
Sound environment
7
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
Software
n
2005-02-09
n
VR - More than just graphics
Software
n
n
"Homegrown tools have the advantage that they can be
customized for the needs of your shop, you have the code to
fix any bugs or add any enhancements.
The disadvantage of in-house tools is that they are far more
expensive to build and maintain, often costing more than offthe-shelf tools. In many cases it is utterly impossible to build
your own tools because of the scope of the application, for
example 3D modeling and animation packages or bitmap
editors."
It is an integration of technologies
n
Software libraries from many
vendors/projects could possible be involved
n
n
Hardware
n
n
I've seen it take people four days of messing around with the
PlayStation 2's blend modes just to get a texture with alpha
to show up correctly.
9
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
VE system architecture
n
Virtual
Environment
Generation
system
head position
hand
position
(Grapics
software and
workstation)
2005-02-09
Component based
n
As we are using lots of bricks to complete the house,
we should strive to use well defined components to
be able to
n
Immersive
Visualization
Facility
Debug
n
n
Isolate the error
Specialized debug tools for individual components
Audio
synthesizer
n
Replace
Audio
localizer
n
Upgrade
n
Optimize
n
n
Haptic
system
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
Components
Display
electronics
Head- and
handtracking
electronics
Design is critical
Reuse
The use of design patterns can help getting a
common language between implementors.
10
n
graphics
11
©Anders Backman, Dept. Computing Science, VR05 - Software
Why software?
Writing it your own
n
8
n
©Anders Backman, Dept. Computing Science, VR05 - Software
12
To switch from one physics engine to another
Hide API changes to the rest of the system
..
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
2
Software
n
Software - Modeling
There are tools for lots of different areas:
n
n
n
n
Authoring (events, …)
n
n
Rendering
n
Distribution/Communication
n
Sound
n
n
n
n
n
n
Communicates with the external VR hardware (trackers…)
Graphics, usually a Scene Graph API on top of OpenGL/DirectX
Handling the consistency of a distributed DB.
Rhino3D
n
Alias
Maya
LightWave
AC3D
Blender
n
n
n
n
Simulation
n
n
n
Simulation code for physical /artificial entities.
13
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
Software
Tools comes in different levels
There are a lot of 3D file-formats out there. Problems arise when trying
to convert between them. Materials, textures disappear, ...
n
VRML 1 .WRL
n
VRML 2 .WRL
n
n
n
For prototyping a higher-level tool is often
desirable
Sometime low-level API give you the control
you need over details
Different tools are suitable for different
purposes (special purpose tools)
n
n
n
n
AutoDesk .3DS
n
n
n
n
Software - Fileformats
n
COLLADA (www.collada.org)
n
n
n
n
2005-02-09
n
n
n
Exporters already exist for: 3DSMax, Maya, SoftImage XSI
n
n
n
Mesh, textures, animation, skinning, shaders, ...
n
Rigidbody info: mass, meshes, material, ...
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
The more intelligence the file format has, the harder it is to
convert to something else
n
n
n
17
©Anders Backman, Dept. Computing Science, VR05 - Software
There are a lot of converters out there too:
Simulation information
n
3D studio mesh format.
Geometry (mesh+textures) + animations.
Software - Converters
XML based
Will contain rendering specification
n
Contains no textures or materials
Wavefront .OBJ
WorldToolKit Neutral File Format NFF
Inventor .IV
Every scenegraph API usually have its own native fileformat.
16
Is an Industry initative to create a portable file format
for 3D content.
Alias, Discreet, Softimage, 3DLabs
n
n
Defacto standard for Vis-sim business.
Hiearcheal format
Supports links to other .flt files
AutoDesk .DXF
n
©Anders Backman, Dept. Computing Science, VR05 - Software
Second standard, contains a lot: interpolation, sound, key framing, ...
No support for named objects
n
n
2005-02-09
First standard, no events and such, only geometry
MultiGen’s OpenFlight .FLT
n
n
15
©Anders Backman, Dept. Computing Science, VR05 - Software
Software - File-formats
Low-level: API:s
n High-Level: GUI, drag-n-drop, Word for VR!!
n
…
2005-02-09
n
n
Great at NURBS, although low control over polygons
14
n
n
Great for modelling, easy to use
No real control over polygons.
Easy to write plugins to get full control
No support for hiearcheal fileformat except VRML
n
n
Handling of spatial sound connected to objects
Great for real-time modeling
Full control over polygons!
Hiearcheal file format.
3D Studio
Handling the actions in the application.
Device support
n
n
Converting file-formats
n
n
n
n
Translating
n
Multigen
Creating the 3D models
n
n
n
Modeling
18
What should a pure renderer do with physics information such as
Mass?
PolyTrans from Nugraf
Crossroads (freeware)
ivToflt, wrlToflt, … (SGI only)
Some work, some don’t. (In some cases)
60% of work with a VR project can directly be related to
fileconversion problems.
Make sure you have a workflow ready.
A well developed process of the whole chain of data, from
generating it to rendering/simulating it.
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
3
Software etc...
n
Software - Graphics
n
n
n
n
n
n
n
www.windsoft.com
Small cool software for
making lightmaps
Evaluation versions
available.
Reads AC3D!
Exports .x (which osg can
read)
Has DLL interface for
someone to write
importers/exporters
19
2005-02-09
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
Plib (www.plib.org)
OpenSG (www.opensg.org)
n
n
n
n
§ OpenScenegraph (www.openscenegraph.org)
§ A Core scenegraph
§ Handles the scene only, no windowing, user interaction, simulation etc...
This is supposed to be handled by additional toolkits and libraries.
§ Well object orientated, using Designpatterns.
§ Open source, get too peek at the source, AND to modify it.
§ Readers for most common formats, the community adds that.
§ OpenFlight, 3DS, AC3D, VRML1, LWO, OBJ, GEO
§ Portable
§ Unix, Windows, MacOs X, Solaris, HP, (Playstation2)
§ Impostors
§ First implementation in a scenegraph.
§ Occlusion culling
§ -Under heavy development
n
n
n
n
n
§ Things changes (interfaces, functionality)
§ Reaching 1.0, wich will mean a more stable core.
§ - Documentation
§ Code examples the best way to get into it.
§ Documentation is on the way.
22
Swedish company
Mature
Readers for most common filetypes
Strictly OO
Not OpenSource, but free for academic use.
2005-02-09
n
I-COLLIDE
n
SWIFT++
n
OPCODE
n
n
n
n
Performer (www.sgi.com)
n
n
n
n
n
n
n
n
n
23
n
Optimized for SGI (but now exists for Linux and Windows)
A lot of functionality
Multiprocessing
System stress and load management
Asynchronous database paging
Morphing
Clip Mapping for handling of really large textures.
Fixed-frame-rate capability
DVR (Dynamic Video Resolution) (On SGI Onyx)
©Anders Backman, Dept. Computing Science, VR05 - Software
Not really easy to 2005-02-09
work with.
©Anders Backman, Dept. Computing Science, VR05 - Software
Software - Collision Detection
German research project
Core scenegraph
Multithreaded core design
Gizmo3D (www.tooltech.com)
©Anders Backman, Dept. Computing Science, VR05 - Software
Software - Graphics
Support for 6DOF, Stereo, Collision detection, sound..
SGI/NT/Linux
- Slow
+ Java = Portable, stable?
Software - Graphics
One of the first portable graphic API:s, now also OpenSource
A mature multiplatform API.
2005-02-09
Aimed towards CAD industry
Tessellation, optimization, Occlusion culling, multiprocessor
support (SGI).
NT, SGI (SUN, HP)
Almost shutdown from SGI.
Not an API but still powerful
Portable
Performance?
Outgrowing it self?
X3D
©Anders Backman,
Dept. Computing Science, VR05 - Software
Mostly used as 2005-02-09
a fileformat. Not as a script
language.
See OpenGL except for portability!!
20
VRML
n
21
OpenInventor
Java3D (www.sun.com)
n
n
Direct3D
n
n
©Anders Backman, Dept. Computing Science, VR05 - Software
Industry standard for accessing the rendering hardware in an
portable way.
View models interactively in 3D space
Control color and lightning
manipulate pixels...
Portable
Full control
In most cases to low level to start from scratch developing a whole
immersive VR application
n
n
Optimizer/Cosmo3D (www.sgi.com)
n
OpenGL,
n
Software - Graphics
n
Graphic APIs.
n
LightMapMaker
n
n
n
Models are preprocessed into convex polyhedra.
C++
AABB trees
Arbitary meshes (convex/non-convex), polygon soups
SOLID
n
n
24
Works only on convex polyhedra
Version 3.5 commercial
Open source version FreeSOLID
(http://sourceforge.net/projects/freesolid/)
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
4
Software
n
Software
A VR toolkit should contain most of the
following parts
n
Real time graphics
n
n
Have a stable path of importing models
n
n
n
IDE/API: Multigen, SmartScene, VEGA Prime
VEGA is one of the most competent VR development toolkits
available. Lots of different modules for:
n
n
n
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
n
n
n
University of Iowa
Open source
Developed to make code once, compile and run anywhere for VR.
Incorporates hardware support with display support.
GUI:s for setting up the environment.
You dont have to decide the interaction hardware while writing the
code. This is configurated later on.
n
n
n
n
n
n
Write on a PC running windows, using mouse for interaction
Later run on a multipipe onyx with trackersystem
Support for cluster rendering
Sound support
Does not include a scenegraph, depends on either OpenGL,
Performer, OSG, OpenSG
Sounds wonderful, wheres the catch?
n
n
n
n
Large
No rendering, authoring,...
Monumental software
Many requirements of tools/versions for getting it compiled.
28
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
Virtools
Commercial
n
Support for different projection systems
26
n
Software
n
Exchanging minimal level of data over a network so several users can cooperate
in a VE.
Dead reckoning algorithms
Interest groups
VR Juggler
Colosseum3D (VRlab)
n CAVElib, Commercial
n Maverick Open Source, AIG, UK
n DIVE, SICS, Sweden.
n Simple Virtual Environment (SVE), Georgia Tech,
USA.
n MR Toolkit, University of Alberta, Canada
n DIVERSE, Open Source (built on Performer)
n VR Juggler, Open Source, Iowa State University
n VHD++
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
n
Support for trackers, gloves, ...
Stereo Viewing
n
n
29
Networking
n
Research
n
Inverse kinematics, rigid-body-dynamics, ...
n
n
Listing of VR software
n
Controlling 3D sound sources and listeners, loading of different sound files.
Sensors
n
©Anders Backman, Dept. Computing Science, VR05 - Software
Where actions are programmed without need for compiling, etc. Trigger for
actions.
n
n
Software
n
n
Bounding box, geometry
2005-02-09
27
Simulation
Interpolators, movie-clips, ...
25
n
n
n
Primitives: box, polygons, spheres, cones, ...
Collision detection
n
n
Audio
Animation
n
n
n
n
Model libraries
n
n
Scripting language
Importing models, filters
n
n
Level-of-detail, culling, shaders...
n
Marine simulation - Dynamic ocean simulation
Radar simulation
Special effects - Explosions, smoke, fire, missile trail…
Terrain generation
Sounds a lot like the US Defence to me?
Expensive
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
n
n
n
n
Commercial 3D engine
VR/Games
Visual Programming
Many plugins
n
n
n
30
Physics (Havok)
AI (AI inplant)
Browser plugins
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
5
VRED (www.vred.org)
n
n
n
n
Based on
OpenSG
Python
scripting
ODE (open
dynamics
engine)
integration
Free
Game engines
n
n
n
n
n
Numerical Design´s Gamebryo
Monolith´s Jupiter
Renderware
Unreal3
Important issues:
n
n
n
n
n
n
n
31
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
Software
n
n
n
n
n
n
n
Usually the simulation has been pushed away for the
graphics.
This is undergoing a change. For example the games
requires more and more precise simulation
underneath. AI, physics, …
n
EON Reality’s EON (now using OpenSceneGraph)
Superscape VR
n
Muse Technologies Inc.
n
Gone out of business
n
n
Dvise
n
n
Gone out of business.
n
n
n
Not developed anymore.
Boston Dynamics Inc.
DI guys human simulation
Magma from ReachIn Technologies
33
2005-02-09
n
©Anders Backman, Dept. Computing Science, VR05 - Software
Software
In an Immersive Visualization system where people is trying to
find pockets of oil from measurements of the earth’s soil,
maybe an authentic simulation of the friction is not of primarily
interest.
For a vehicle simulator, where the vehicle’s interaction with the
terrain is important, it is obvious that some sort of physical
simulation is needed.
34
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
Software
n
Whereas a simulation driven simulation is based on the entities in the simulation.
Rather then building the graphical scene and than add dynamic behavior, there
is a simulation which contain all the information needed to simulate and generate
the visible parts of the object.
Physical
n
Visual
n
A graphic driven simulation will prioritize the
graphics. The attributes of the graphic entity
is given to the simulation
n
n
Graphic representation
(scene graph)
Simulation representation
A
B
n
Surface properties, geometry
Visual/physical
properties
E
Visual/physical
properties
Simulation code for E
In each timeframe check for
contact with D (ground)
2005-02-09
Mass, geometry, friction coefficient
Simulation code for C
If User enter C then
Do Action C
C
D
35
This is getting more and more important.
It always depends on what you are looking for.
Application development Interfaces
n
©Anders Backman, Dept. Computing Science, VR05 - Software
Simulation driven or graphic driven?
API: WorldToolKit
IDE: WorldUp
World2World
n
n
2005-02-09
Integrated Development Environment
n
n
32
Software
Sense8
n
Workflow
Debugging
Portability
Features
Stability
Design
Documentation
©Anders Backman, Dept. Computing Science, VR05 - Software
36
Visual/physical
properties
2005-02-09
Generates
Graphics
©Anders Backman, Dept. Computing Science, VR05 - Software
6
Software
n
Vortex
n
Havok
n
n
n
Physics SDK
Physics SDK
Plugins for 3D Studio
n
Renderware Physics
n
Meeqon Physics SDK
ODE – Open source
AI Inplant
n
n
n
n
37
Former Mathengine
AI simulation
2005-02-09
©Anders Backman, Dept. Computing Science, VR05 - Software
7