Event Infrastructure
Alasdair Allan
University of Exeter
The eSTAR/TALONS Interconnect
Alasdair Allan
University of Exeter
Robert R. White
Los Alamos National Laboratories
1
The eSTAR concept
• It’s an ad-hoc peer-to-peer network of heterogeneous
resources utilising a collaborative agent paradigm for
decision making
• Treat telescopes and databases in a similar manner,
both being made available on an observational grid
• Disguise complexity, users hate complexity
VOEvent Workshop II, Tucson
2
eSTAR in a nutshell
VOEvent Workshop II, Tucson
3
What is an agent anyway?
• An agent is “just software” not magic
Loosely, an agent is a computational entity which:
• Acts on behalf of another entity in an autonomous
fashion
• Performs its actions with some level of proactivity
and/or responsiveness
• Exhibits some level of the key attributes of learning,
co-operation and mobility
VOEvent Workshop II, Tucson
4
Multi-agent systems
• A multi-agent system is one that consists of a number
of agents, which interact with one-another.
• In the most general case, agents will be acting on
behalf of users with different goals and motivations.
• To successfully interact, they will require the ability to
cooperate, coordinate, and negotiate with each other,
much as people do…
VOEvent Workshop II, Tucson
5
Multi-agent systems for eSTAR
• We’ve built the first agent based astronomical system,
and it was clearly the correct choice of architecture.
• What we’ve built in eSTAR is a collaborative agent
system which schedules telescopes.
• Complicated telescope schedules are built by humans
using a multi-pass approach.
• Traveling salesman problem…
VOEvent Workshop II, Tucson
6
Peer-to-Peer
• Agents operate in a peer-to-peer manner and can
make use of these interconnections between people
and data.
• Carrying out intelligent resource discovery could
mean that your agent looks to your collaborators
agent for data and expertise before it looks to
“central” sources.
VOEvent Workshop II, Tucson
7
The world is flat…
• The world is small and flat, but it is
none the less still very complex.
• Architectures which take account
of this are intuitive, and will map
well into the real world.
© Terry Pratchett
VOEvent Workshop II, Tucson
8
The eSTAR network
Embedded
Agent
The VO
© Nik Szymanek
User Agents
Gateway
Service
VOEvent Workshop II, Tucson
Embedded
Agents
• GRB rapid follow-up programme
• Search for exo-planets
• Testbed for Robonet-1.0 open time
9
The eSTAR network
Embedded
Agent
The VO
© Nik Szymanek
User Agents
Gateway
Service
VOEvent Workshop II, Tucson
Broker
Embedded
Agents
Alert
Agent
GCN
10
What we’ve done so far?
• Convenience layer and parser (Perl & C++)
– building common cases (including GCN)
– different parser methodologies
• Prototype event network live and on sky
• Limited (hard-wired logic) brokering
• Archiving and persistent storage
• RSS test feed(s)
– programmatic, or human readable?
VOEvent Workshop II, Tucson
11
Where are we going?
• Event brokering (aggregators)
– time critical
– non-time critical (data mining?)
• Event archiving and persistent storage
– REST interface
– persistence of data products
• Search service
– REST interface
• Event Feeds
– pull, or push?
VOEvent Workshop II, Tucson
12
Event feeds
There are two ways to feed events to consumers,
• Pull model (feeds)
– polling
– e.g. RSS feed
• Push model (forwarding)
– via REST, SOAP or vanilla TCP
VOEvent Workshop II, Tucson
13
Pull model
• Advantage
– under client control
• Disadvantage
– as slow as the polling interval
A pull model, e.g. RSS, is never going to work for rapid
response cases like GRB or transient follow-up.
VOEvent Workshop II, Tucson
14
Push model
• Advantage
– theoretically faster, depending on architecture
• Disadvantage
– administrative nightmare for publisher
A push model, e.g. GCN, is the only way to do rapid
response work. However it isn’t really optimal, so we’ll
probably be stuck using both approaches. Shouldn’t
optimise our standard for distribution via RSS model.
VOEvent Workshop II, Tucson
15
Aggregators
Why should we aggregate event feeds?
• Consolidation
– Do we need to support multiple publishers?
• Removal of duplicate events
• Added semantic content
• Trust issues
VOEvent Workshop II, Tucson
16
Storage
Why should we permanently store event messages?
• The data itself will be replicated elsewhere in the VO.
• Why do we care about the original message?
• If we want to search, we have to store.
• However, to me, the best use case for persistent
storage is actually event feeds…
VOEvent Workshop II, Tucson
17
Atom feed
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title xmlns="http://www.w3.org/2005/Atom">eSTAR/TALONS GCN Feed</title>
<author xmlns="http://www.w3.org/2005/Atom">
<name xmlns="http://purl.org/atom/ns#">eSTAR Project</name>
<email xmlns="http://purl.org/atom/ns#">[email protected]</email>
</author>
<link xmlns="http://purl.org/atom/ns#" type="application/atom+xml" rel="self” href="http://www.estar.org.uk/atom.xml"/>
<entry xmlns="http://purl.org/atom/ns#" xmlns:default="http://www.w3.org/1999/xhtml">
<title xmlns="http://purl.org/atom/ns#">ivo://gcn.gsfc/hete/396943a</title>
<content xmlns="http://purl.org/atom/ns#" mode="xml">
<div xmlns="http://www.w3.org/1999/xhtml">
<![CDATA[<?xml version = '1.0' encoding = 'UTF-8'?>
<VOEvent version="1.0" id="ivo://gcn.gsfc/hete/396943a”>
.
.
.
</VOEvent>
]]>
</div>
</content>
<author xmlns="http://purl.org/atom/ns#">
<name xmlns="http://purl.org/atom/ns#">GCN</name>
<email xmlns="http://purl.org/atom/ns#">[email protected]</email>
</author>
</entry>
.
.
.
</feed>
VOEvent Workshop II, Tucson
18
RSS feed
<?xml version="1.0" ?>
<rss version="2.0">
<channel>
<title>VOEvent GCN Notices</title>
<language>en</language>
<description>RSS feed for GCN notices</description>
<link>http://voeventnet.caltech.edu/GCN.html</link>
<item>
<title>MILAGRO_Source trigger 886-1</title>
<description>
<p>Possible GRB The event time is 2005-12-01T16:24:15 UT. Location RA 232.1248 Dec 62.9797 (J2000)</p>
</description>
<pubDate>Fri, 02 Dec 2005 16:09:52 PST</pubDate>
<link>http://voeventnet.caltech.edu/Notices/ivo_GCN_MILAGRO_58_886-1_2005-12-01T16:24:15.xml</link>
<author>Scott Barthelmy GCN [email protected]</author>
<comments>http://gcn.gsfc.nasa.gov/gcn3_archive.html</comments>
</item>
.
.
.
</channel>
</rss>
VOEvent Workshop II, Tucson
19
<enclosure>
• Its an optional sub-element of the RSS <item> tag
that allows the feed publisher to include a link to a
file.
• It has three required attributes. The url=“ ” attribute
says where the enclosure is located, length=“ ” says
how big it is in bytes, and type=“ ” says what its type
is, a standard MIME type. e.g,
<enclosure url=“http://estar.org.uk/event.pl?id=886-1” length=“2440” type=“application/xml+voevent” />
• This is how podcasting works..
VOEvent Workshop II, Tucson
20
RSS feed
<?xml version="1.0" ?>
<rss version="2.0">
<channel>
<title>VOEvent GCN Notices</title>
<language>en</language>
<description>RSS feed for GCN notices</description>
<link>http://voeventnet.caltech.edu/GCN.html</link>
<item>
<title>MILAGRO_Source trigger 886-1</title>
<description>
<p>Possible GRB The event time is 2005-12-01T16:24:15 UT. Location RA 232.1248 Dec 62.9797 (J2000)</p>
</description>
<pubDate>Fri, 02 Dec 2005 16:09:52 PST</pubDate>
<link>http://voeventnet.caltech.edu/Notices/ivo_GCN_MILAGRO_58_886-1_2005-12-01T16:24:15.xml</link>
<author>Scott Barthelmy GCN [email protected]</author>
<comments>http://gcn.gsfc.nasa.gov/gcn3_archive.html</comments>
</item>
.
.
.
</channel>
</rss>
VOEvent Workshop II, Tucson
21
RSS feed
<?xml version="1.0" ?>
<rss version="2.0">
<channel>
<title>VOEvent GCN Notices</title>
<language>en</language>
<description>RSS feed for GCN notices</description>
<link>http://voeventnet.caltech.edu/GCN.html</link>
<item>
<title>MILAGRO_Source trigger 886-1</title>
<description>
<p>Possible GRB The event time is 2005-12-01T16:24:15 UT. Location RA 232.1248 Dec 62.9797 (J2000)</p>
</description>
<pubDate>Fri, 02 Dec 2005 16:09:52 PST</pubDate>
<link>http://voeventnet.caltech.edu/Notices/ivo_GCN_MILAGRO_58_886-1_2005-12-01T16:24:15.xml</link>
<author>Scott Barthelmy GCN [email protected]</author>
<comments>http://gcn.gsfc.nasa.gov/gcn3_archive.html</comments>
<enclosure url=“http://estar.org.uk/event.pl?id=886-1” length=“2440” type=“application/xml+voevent” />
</item>
.
.
.
</channel>
</rss>
VOEvent Workshop II, Tucson
22
Unanswered questions
We have our document standard, so at this stage there
should be only two issues outstanding,
• Message passing protocol(s)
– How the documents are passed
• Transport standard(s?)
– How the documents are carried
VOEvent Workshop II, Tucson
23
Protocol work
The recent work on interoperability between eSTAR and
TALONS has show the importance of,
• ACK messages
• IAMALIVE messages
eSTAR
Broker
VOEvent Workshop II, Tucson
Gateway
Service
24
Should we mandate transport?
• Why? Nobody thought about RFC 1149 when IP
datagram packets were standardised…
• See http://www.blug.linux.no/rfc1149/
VOEvent Workshop II, Tucson
25
Authentication
• Transport layer issue?
• Format issue?
VOEvent Workshop II, Tucson
26
Authorisation
• Transport layer issue?
VOEvent Workshop II, Tucson
27
Things to think about…
• Standards never survive widespread contact with
users intact. At least no good standard does…
• If people twist and pull the standard to do something
differently than we designed it to do, that’s our fault.
• The users are always right. No, seriously…
VOEvent Workshop II, Tucson