Introduction to
Search for
Knowledge Project
(SfK)
Thursday 20th July
Sara Collins
Presented by David Fergusson
Ischia, Italy 9th - 21st July 2006
1
SfK = simulation of typical
e-Science Research
• Collaboration between scientists (your group)
• Exploring large amounts of Data to find particular patterns of interest
– Astronomy
– Particle physics
– Biomedicine
– Geophysics
….
• Using results of other researchers’ work
Ischia, Italy 9th - 21st July 2006
2
The Pillars of Wisdom
background
Pillar –
Overrides background
Rectangular
Constant height
Aligned with x-y axis
Plaque –
Overrides background
Rectangular
Constant height
> Or < pillar height
Aligned with x-y axis
Total of 11 Pillars
Wise Words
Ischia, Italy 9th - 21st July 2006
Word or phrase –
Overrides background
Rectangular
Constant height
> Or < plaque height
Aligned with x-y axis
3
Hints
Given
• Boundary of the Surface
• hints,(Xi, Yi), obtained by “previous research teams”
Ischia, Italy 9th - 21st July 2006
4
Search space
10000
-10000
10000
-10000
Ischia, Italy 9th - 21st July 2006
5
Real search space
characteristics
• In real life
– Noise is much larger and pervasive (ie. Top of pillars)
• Ratio of signal to noise is artificially large
– Search spaces are larger
– Don’t normally know the complete parameters of your
search space
• Eg. Boundaries & alignment
• So, here, we do not need statistics to analyse
the patterns
• Do not need to run complex models
Ischia, Italy 9th - 21st July 2006
6
Integrating practical
characteristics
Technology
name
Condor
Characteristic
API
Non-web services
scripting
OMII
plain web services
java
GT4
WSRF
java
gLite
Data &
computation
resources
scripting
Ischia, Italy 9th - 21st July 2006
7
Integrating practical
characteristics 2
• Scripting technologies - rely on parameter
modification and re-submission
– May require re-joining the queue
• Java technologies - rely on programmatic
parameter sweeps
– May add functionality
Ischia, Italy 9th - 21st July 2006
8
Framework
Expected to
• Write program(s) / script(s)
– To run explorations across Surface,
– to find
• Pillars, then plaques
• Do visualisation on Plaques to read Wisdom
Words
• Recognise the pattern
• Making full use of capabilities
Ischia, Italy 9th - 21st July 2006
9
Client side architecture
Ischia, Italy 9th - 21st July 2006
10
Server side architecture
Ischia, Italy 9th - 21st July 2006
11
Condor Integrating Practical
• You will be provided with a Java application
– You can concentrate on the search for the pillars
– You need to run it with Condor to file the pillars: you have access to all
computers in this room, since they all run Condor
– If you did the advanced Condor practical, this should be straightforward
• Possible Solutions
– A basic solution will involve no programming, only running the application
– An advanced solution might use a small program to write a set of Condor jobs (or
a DAG) and simplify life for you.
Worker Computers
(your computers)
Your computer submits and runs
jobs: it is a shared resource Ischia, Italy
9th - 21st July 2006
Condor Matchmaker
12
Introduction to plain Web Services
seqGen::generate
• Complete the web service client for
Generate a vector of numbers [start,step,limit]
ExampleSecureService::returnIssuerDN based on
ExampleSecureService::returnSubjectDN client. xprodGen::generateX
Generate
• Increase the verbosity of the DynamicInvoker
class.an array (x,y) based on two vectors
surfaceGen::probe
Pillar Exercises
Take a set of (x.y) points and
• Complete
Clientthe clients to invoke the seqGen, return the height at these points (x,y,h)
Services
xprodGen
and probe methods.
Machine
(dualxeon)
• Extend
a Java method to perform a simple workflow filestore::createFile
(You!)
Write surface data to unique filename on server
that invokes the remote web services.
• Use the provided PillarFinder, PillarReader and PlotWS::makePlot3D_xyz
Usespace
GnuPlot to generate a graphical image
Regular clients to search & visualise the pillar
Services
(ws-01)
OMII Integrating Practical
Services
(ws-02)
OGSA-DAI
Hints database
Services
(ws-32)
Services
(ws-31)
Ischia, Italy 9th - 21st July 2006
13
Integrating Exercise Layout
User Interface
ISCHIA
1.
RB/BDII
SE
SampleGen
PillarFinder
PillarReader
2.
CE
SE
LFC
Catalog
GILDA
AMGA
Catalog
Ischia, Italy 9th - 21st July 2006
14
GT4 Integrated Practical
Visualising results
Ischia, Italy 9th - 21st July 2006
16
Manage your work load
• Start with Condor
• Work from technologies where you have
someone who has completed an advanced
practical
• Try to work with one ‘scripting’ and one ‘java’
example
• We will review progress at 11:00 & 16:00 and
supply more help if required.
Ischia, Italy 9th - 21st July 2006
17
4 “Help desks”
• One per technologies
• To simplify support - visit technology desk
Ischia, Italy 9th - 21st July 2006
18
Analyse technologies
• Don’t just look for the “fastest”
• If there are differences, try to understand why
– Eg. Latency, architectural difference
• Try to think why the architectural decisions were
taken
– For instance Service Oriented Architectures
– It’s unlikely that differences are due to one technology
being “wrong”
Ischia, Italy 9th - 21st July 2006
19
Analyse your methods
• Continuously review your strategy and
tactics
• Review where you allocate team
resources (people)
• Finally review how you might improve your
approach next time.
Ischia, Italy 9th - 21st July 2006
20