Component-based Intelligent Visualization
H. Hagen1, H. Barthel1, A. Ebert1 and M. Bender2
1
Intelligent Visualization and Simulation Systems
DFKI GmbH Kaiserslautern, Germany
2
Department of Computer Sciences and Microsystem Technology
University of Applied Sciences Kaiserslautern, Germany
Abstract:
In this paper we propose a visualization system architecture combining
component technology with multi agent technology. Component technology is
used as a development platform on which visualization modules as well as
agents are implemented as reusable software components. The agents in our
approach are used to automatically satisfy individual user demands and to
dynamically adapt to changing system loads and different hardware
configurations
Key words:
visualization system, intelligent visualization architecture, component
technology, multi agent technology
1.
INTRODUCTION
Visualization applications are used to analyze data and retrieve
information coded in the data by utilizing many complex data processing and
visualization algorithms, demanding a substantial knowledge about such
algorithms from the developer. In order to reduce the development effort,
visualization systems like AVS from Advanced Visual Systems, IRIS
Explorer from SGI or Data Explorer from IBM have been developed in the
last 20 years. These systems offer huge object oriented class libraries for a
simplified development of new modules and a visual prototyping
environment with which the data flow of applications can be interactively
specified by the user.
These systems are very powerful yet, but they are huge and monolithic
because they were developed over several years and extensions to such
1
2
H. Hagen, H. Barthel, A. Ebert and M. Bender
systems were done at the expense of the quality and clarity of the overall
architectural design. The modules are strongly coupled with the respective
system, that means the system demands special functionality from every
module before that module can be used in the system. Consequently modules
developed for one system are not usable in other systems without major
changes in the source code because other systems require other functionality.
Moreover they lack flexibility regarding the visualization process by
either providing a high rendering quality with limited interaction
possibilities, or real-time visualization with reduced rendering quality.
Furthermore, the user has to manually adjust the balance between the desired
frame rate and the resulting rendering quality by modifying the related
control parameters or by discovering and (ex)changing the causative system
components.
Orthoslice
Probe
IsoSurface
Figure 1. Example visualization using SGI IRIS Explorer.
In our opinion, an appropriate visualization system architecture should
support the development of new visualization modules without being fixed
to the environment in which they are used, and the architecture should
include an intelligent control unit capable of automatically supervising and
tuning all system components during runtime.
In order to reach this objective, in our approach we are combining multi
agent technology with component technology. Component technology with
its dynamic linking properties offers an ideal platform to achieve the
uncoupling of visualization modules and visualization system, and intelligent
agents (implemented as components themselves) are able to take over the
H. Hagen, H. Barthel, A. Ebert and M. Bender
3
automatic customization of parameters affecting the rendering quality and
the rendering performance.
According to the proposed architecture we present a prototype
implementation consisting of
– a visualization framework composed of reusable software components
which can be used as the building blocks in composing applications,
– the component-based visual prototyping system CVP with which new
applications can be plugged together interactively by using the
components of the visualization framework, and
– agents which are automatically controlling the texture resolution and the
LOD selection based on the overall rendering performance.
The organization of this paper is as follows: We first give a short
introduction on component technology (section 2), then we will introduce
our visualization framework and the CVP-System (sections 3 and 4), and in
section 5 we describe how agents are used in our approach to control the
rendering pipeline. We conclude our paper with suggestions for future work
(section 6).
2.
COMPONENT TECHNOLOGY
2.1
Introduction
The complexity of present-day software development cannot be
satisfactorily reduced by the concept of object-orientation because of the
constraints imposed by particular programming languages and operating
systems. The next frontier in software technology is the increasingly
important area of component-based development, in which a component
typically has all the properties of an ordinary object, but in addition can be
deployed in different operating environments without change or
recompilation. The size of components can range from small utility objects
to large systems or business objects. Application construction thus becomes
more an issue of component assembly rather than of program development.
There are many different definitions of the term component. Most
commonly used is a definition stating that a component is an encapsulated
piece of software with a standardized, contractually specified interface
giving access to a corresponding functionality (service) that piece of
software wants to offer. The definition of interfaces as well as its
administration is regulated by a component system (repository), which also
provides a communication infrastructure based on these interfaces. An
4
H. Hagen, H. Barthel, A. Ebert and M. Bender
important point here is that the component system offers a repository in
which interfaces are stored and which is accessible during runtime. So, for
example, after the registration of the components interfaces in the repository,
if a client component wants to make use of a service a server component
offers, the client asks the component system for the interface of the serving
component and then accesses the corresponding functionality (see figure 2).
Client
Client
Register
Interface
Access Service
Query
Interface
Server
Server
Register
Interface
Component System
Componentsystem
Figure 2. Dynamic linking.
So, with component technology the strict separation between
implementation and provided functionality supports the development of
independent and reusable software components, which leads to the following
advantages:
– dynamic linking: Components can be replaced, added to or deleted from
an application at runtime.
– faster application development: The effort necessary to build/assemble
new applications can be reduced in the case suitable components are
available.
– small costs: If an application can be assembled out of already existing
components, smaller costs arise due to shorter development times.
– reliability: Components are permanently extended and improved leading
to a higher reliability of the respective application.
– flexible system architecture: Easy exchange and customization of
components leads to an improved extendibility and maintainability of
applications.
To benefit from these advantages in our approach we are using a
component system as the base development platform on which the
5
H. Hagen, H. Barthel, A. Ebert and M. Bender
visualization modules as well as the agents are implemented as reusable
software components (see figure 3). Consequently the modules and the
communication between these modules only depend on the component
system and can be used wherever this component system is available,
independent from the underlying hardware and the operating system.
Components
Componen
Componen
Agents
Agent
Agent
Visual
Visual
Visual
Prototypin
Prototyping
Prototypin
Utilities
Utilitie
Utilitie
Component
System
Componentsyst
Componentsyst
COM/DCOM
• COM/DCO
• COM/DCO
CORBA
• CORB
• CORB
Java Bean
Beans
•• Java
• Java
Bean
Operating
System
OperatinSyste
OperatinSyste
Figure 3. Component system as a base development platform.
The three most important component systems which we can choose from
are COM/DCOM from Microsoft, CORBA from the OMG Object
Management Group, and Java Beans from Sun. We have decided to use
Sun’s Java Beans component technology because it is directly integrated into
the Java language and requires no extra effort to transfer a Java class to a
Java Beans component. With the use of so called bridges both DCOM and
CORBA components can be used from within a Java Beans component.
2.2
Java Beans
The Java Beans component architecture defines that a Java Bean is a
reusable software component that can be manipulated visually in a builder
tool. These builder tools may operate entirely visually, allowing the direct
plugging together of Java Beans, or they may enable users to write Java
classes that interact with and control a set of beans.
Here the functionality a bean provides may vary: a bean may be a simple
GUI element such as a button or a slider, a sophisticated database viewer or
a marching cubes component generating an isosurface on a structured point
set. But common to all Java Beans is
6
H. Hagen, H. Barthel, A. Ebert and M. Bender
– the support for “introspection” so that a builder tool can analyze how a
bean works,
– the support for “customization” so that when using an application builder
a user can customize the appearance and behavior of a bean,
– the support for “events” as a simple communication metaphor than can
be used to connect up beans,
– the support for “properties”, both for customization and for programmatic
use, and
– the support for persistence, so that a bean can be customized in an
application builder and then have its customized state saved away and
reloaded later.
From a developers point of view it is important to stress that a bean is not
required to inherit from any particular base class or interface and that it is
also entirely usable by human programmers because all the key APIs have
been designed to work well both for human programmers and for builder
tools. These APIs describe the three most important features of a Java Bean:
– the set of properties it exposes,
– the set of methods it allows other components to call, and
– the set of events it fires.
Properties are named attributes associated with a bean that can be read or
written by calling appropriate methods on the bean. Thus for example, a
bean might have a “foreground” property that represents its foreground
color. This property might be read by calling the method
“Color getForeground()”
and the property might be updated by calling the method
“void setForeground(Color c)”.
The methods a Java Bean exports are just normal Java methods which
can be called from other components or from a scripting environment. By
default all of a beans public methods will be exported, but a bean can choose
to export only a subset of its public methods.
Events provide a way for one component to notify other components that
something interesting has happened. Under the event model of Java an event
listener object can be registered with an event source. When the event source
detects that something interesting happens it will call an appropriate method
on the event listener object.
H. Hagen, H. Barthel, A. Ebert and M. Bender
3.
7
THE VISUALIZATION FRAMEWORK
During the visualization process data is transformed into images in order
to efficiently and accurately represent the information that is coded in the
data. According to the visualization pipeline this transformation is done in a
dataflow oriented manner by converting the data from its original form into
graphic primitives and displaying these primitives by using a graphics
library. Consequently we distinguish between source components generating
data, filter components consuming and producing data, and mapper
components mapping the data to graphic primitives and rendering these
primitives using a graphics library like OpenGL.
Figure 4. Visualization examples.
For the prototype implementation presented in this paper we use
Kitware’s visualization toolkit VTK which offers a modern object-oriented
C++ class library composed of a huge set of classes for the dataflow oriented
processing and visualization of data. This class library is accessible from
within Java by utilizing the Java Native Interface API (JNI) allowing Java
code that runs inside a Java Virtual Machine* (VM) to interoperate with
applications and libraries written in other programming languages, such as
C, C++, and the assembly language. Because JNI imposes only a little
overhead we can take advantage of the native performance the VTK class
library offers.
8
H. Hagen, H. Barthel, A. Ebert and M. Bender
Figure 5. The visual prototyping environment of IBM Data Explorer.
The visualization framework of our prototype implementation actually
contains over 170 visualization modules implemented as Java Beans
components. Here the flow of data is realized by events encapsulating the
data flowing through the visualization pipeline. To be more precisely we
actually map the following data set types onto the respective event types:
–
–
–
–
structured points
unstructured points
structured grid
polygonal data
⇒ StucturedPointsEvent,
⇒ UnstructuredPointsEvent,
⇒ UnstructuredGridEvent, and
⇒ PolyDataEvent.
So, for example a component reading a structured points set acts as a
StructuredPointsEvent source, and a marching cubes component computing
an isosurface of a structured points data set and producing triangles acts as a
StucturedPointsEvent listener and as a PolyDataEvent source. Consequently
it is possible to implement the components of the framework independent
from each other, requiring only the knowledge of incoming and outgoing
events which are based on data set types which are unusual to change. With
the CVP-System introduced in the next section we can add and replace
components even at runtime by utilizing the dynamic linking properties
component technology offers. Figure 4 shows some example visualizations
done with the visualization components of the prototype system.
4.
THE CVP-SYSTEM
Most of the commercial visualization systems available today offer a
visual prototyping environment in which the data flow of applications can be
specified interactively by the user. Here, according to the visualization
H. Hagen, H. Barthel, A. Ebert and M. Bender
9
pipeline, an application is deemed to be a network of modules connected by
links over which the data flows from source objects via filter objects to
mapper objects. So, with the use of a visual prototyping environment the
user can rapidly realize new visualization applications without the need for
editing source code. Figure 5 shows an example for such an environment
(IBM Data Explorer).
Even though these builder tools have been proved very useful, a module
to be used in such an environment is requested to implement a special
functionality. But because every system has its own functional requirements,
a module normally cannot be used in other systems without major changes in
the source code.
Figure 6. The CVP-System.
To overcome this drawback we have implemented the component based
visual prototyping system CVP taking advantage of the dynamic linking
properties component technology offers. More precisely, in the CVP-System
a Java Beans component can be used without requiring the implementation
of mandatory functionality. And because a bean is not required to inherit
from any particular base class or interface, every bean can be used to
interactively plug together new applications. Moreover the system on
demand creates a graphical user interface allowing the customization of the
bean so that the user can interactively adjust the components properties.
Figure 6 shows an example CVP-Session realizing a scalar warping
application.
10
5.
H. Hagen, H. Barthel, A. Ebert and M. Bender
AGENT-BASED CONTROL
The way in which the visualization is done still lacks flexibility regarding
the visualization process by either providing a high rendering quality with
limited interaction possibilities, or real-time visualization with reduced
rendering quality. That’s because the user manually has to adjust the balance
between the desired frame rate and the desired rendering quality by
modifying the related control parameters or by discovering and (ex)changing
the causative system components. Furthermore because of the continually
changing conditions and the vast variety of parameters the monitoring and
tuning of visualization applications is a complex challenge, even for experts
with a widespread experience in visualization.
READER
MARCHING
CUBE
TEXTURE
COORDS
DATA GENERATION
LEVEL OF
DETAIL
DATA CONVERSION
REACTIVE AGENT
KNOWLEDGE BASE
RENDERER
DATA VISUALIZATION
ENVIRONMENT
KNOWLEDGE BASE
CONTROLLING AGENT
USER DEMANDS
ENVIRONMENT
LEGEND
REACTIVE AGENT
DELIBERATIVE AGENT
CONTROL LOOP
Figure 7. Agent-controlled visualization pipeline.
In order to improve this unsatisfying situation, we are extending our
component based visualization system architecture by incorporating multi
agent technology. A multi agent system is a software system build by a
number of agents communicating with each other in order to solve more
complex problems. Here an agent is a piece of software operating
autonomously according to its explicitly given goals and reacting on changes
in its environment.
H. Hagen, H. Barthel, A. Ebert and M. Bender
11
Figure 8. Snapshot of the example application.
Our approach make use of two types of agents: reactive agents and
deliberative agents. A reactive agent only has a limited knowledge about the
context it is living in and therefore only performs a simple task, whereby a
deliberative agent has knowledge about a greater context and therefore has
the ability to handle more complex problems, especially by using and
controlling reactive agents.
Regarding the visualization process a module or a group of modules
always performs one fixed task; this means that the terms of regulation (e.g.
frame rate and rendering quality) can be described by straightforward static
rules. Thus, it is reasonable to assign a reactive agent to either a module or a
group of modules and to control the modules’ parameters on demand. Since
a reactive agent only needs the knowledge in the module specific context it
acts similar to a control loop.
In contrast, a deliberative agent has the ability to handle complex
problems, but in general it is not able to process such jobs in real time on its
own because of the applied sophisticated calculation methods. The
supervision of the reactive agents, as well as the analysis of the information
12
H. Hagen, H. Barthel, A. Ebert and M. Bender
generated by them, is done by such a deliberative agent, which we call a
controlling agent. Based on the overall knowledge about the involved
visualization components and with respect to the user specific demands it
automatically responds to changes in the environment (e.g. the system load)
by modifying the desired values of the subordinated reactive agents.
Figure 7 demonstrates how reactive and deliberative agents may be used
to automatically control the flow of data through the visualization pipeline.
To achieve a user defined rendering quality and performance, reactive agents
are assigned to each component of the visualization pipeline and are
managed by the controlling agent. Exemplarily we will briefly describe the
interplay of the data conversion and visualization modules and the
controlling manager.
The reactive agent of the data conversion module controls the level of
detail component by altering its parameters affecting the level of detail
algorithm, the reduction level and the accuracy demand. The output of the
data conversion module is the effective accuracy level and the number of
generated triangles. The reactive agent of the data visualization module
controls the render component by varying the representation model (wire
frame, solid), the shading model, the rendering algorithm, the resolution and
the lighting and texturing parameters. The output of the data visualization
module is the current frame rate or the render time respectively. Both
modules receive their permitted parameter ranges from the controlling agent,
which in its turn is biased by user demands and the system environment.
Figure 8 shows a snapshot of the visual prototyping system CVP while
assembling a medical volume data visualization application. Here a CT data
set consisting of 93 image slices through the human head is imported, an
isosurface is generated using a marching cubes algorithm, and different LOD
resolutions of the geometry and the texture image are computed.
Without the use of intelligent agents the user manually has to adjust the
parameters in order to achieve a reasonable rendering speed, but because of
the multitude of the parameters it is very difficult to get both a high
rendering quality and high interaction rates.
6.
CONCLUSION AND FUTURE WORK
In this paper we proposed as visualization system architecture combining
multi agent technology with component technology. Component technology
is used as an ideal development platform on which the visualization modules
as well as the agents are implemented as reusable software components.
Consequently the modules and the communication between these modules
only depend on the component system and can be used wherever this
H. Hagen, H. Barthel, A. Ebert and M. Bender
13
component system is available, independent from the underlying hardware
and operating system. And with the use of reactive and deliberative agents
the adjustment of parameters affecting the rendering quality and the
rendering performance can be achieved automatically according to explicit
user demands or changes in the system environment.
Our future plans include the utilization of a knowledge base in which
knowledge is collected about the available visualization components, how
these components are to be tuned by reactive agents, and how reactive agents
are to be monitored by deliberative agents.
REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
Advanced Visual Systems: “ActiveX Visualization Components”, Technology
Whitepaper, Advanced Visual Systems Inc., 1999.
Barthel: „Komponentenbasiertes visuelles Prototyping von
Visualisierungsanwendungen“, Dissertation, University of Kaiserslautern, 2000.
Burnett, Goldberg, Lewis,: “Visual Object-Oriented Programming”, Khoral Research
Inc., 1994.
Ebert, Divivier, Bender, Barthel: “A Visualization System using Multi Agent
Technologies”, Visualization 2000, Salt Lake City, 2000.
ECOOP: “European Conference on Object-Oriented Programming”, ECOOP
Proceedings, 1996.
Franklin, Graesser: “Is it an Agent, or just a Program? A Taxonomy for Autonomous
Agents”, Third International Workshop on Agent Theories, Architectures, and
Languages, 1996.
Griffel: „Componentware: Konzepte und Techniken eines Softwareparadigmas“,
dpunkt-Verlag, 1998.
Hagen, Müller, Nielson: Scientific Visualization – Overviews, Methodologies,
Techniques, IEEE, 1997.
Harmon: “Components Development Strategies”, Cutter Information Corp. On
Managing and Developing Component-Based Systems, July 1998.
Heuer-Hasenpatt, Hollunder, Kittlaus, Schumacher: „Bausteinorientierte
Anwendungsentwicklung“, SIZ GmbH, 1998.
Jacobsen: “Software Reuse – Architecture, Process and Organization for Business
Success”, ACM Press, Addison Wesley, 1997.
Nwana: “Software Agents: An Overview”, Journal, Knowledge Engineering Review,
11(3), 1996.
Ranze, Müller: „Über den Einsatz von Agenten in Umweltanwendungen“, 11.
Symposium “Informatik im Umweltschutz“, 1997.