The SEALS Platform for
Scholarly Research
Asun Gómez-Pérez
[email protected]
Ontology Engineering Group
Universidad Politécnica de Madrid
http://www.seals-project.eu/
Acknowledge to: Miguel Esteban Gutiérrez,
Raul García-Castro, Nandana Mihindukulasooriya
and the SEALS consortium
14.07.2017
1
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Ed Hovy Requirements
• A kind of framework that
graphically depicts the
whole scholarly
environment/workflow - on
this, a set of markers, like
Post Its notes, that each
indicate a technology
available for use at the
relevant point - also on this,
another set of Post Its notes
that each indicate some
kind of bottleneck or gap
Asun Gómez-Pérez
Requirements
• A kind of system that helps
me to reproduce
experiments that appear in
research papers and
compare my technology
with other technologies
14.07.2017
2
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Scenario
Scholarly workflows
Tool1
Tool2
Tool3
Tool1



Tool4
Tool5
How do I know which
tools are appropriate?:
Reproduce Experiments
(unbiased, repeatable,
…)
Tool1
Tool4
Tool5
Tool1
Tool2
Tool6
Tool6
Tools
Evaluations
Tool6
Tool7
Results
Test data
My tool
My results
14.07.2017
3
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Heterogeneity in the tools to be compared
• Data Sets
– Levels of complexity
– Scalability
– Synthetic, hand-crafted
and real world data
• Tools
– Execution environment
– Functionalities
– Criteria to evaluate:
scalability, interoperability,
etc.
14.07.2017
4
• Metrics
– Precision/recall,
information losses of
data, etc.
• Evaluations
– Fully automatic
– The user is in the loop
– Hybrid approach
– Extrinsic and intrisic
• Starting point
– From scratch
– Extend and adapt
preliminar approaches
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Dimensions
Goals
selection
Interpretation
selection
Metrics
selection
Test data
selection
Evaluation
workflows
implementation
Evaluation
design
Assumptions
selection
Criteria
selection
Tools
selection
Tool features
selection
Evaluation
services
implementation
Execution
requirements
selection
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
SEALS platform for technology evaluation
•
SEALS Platform Features
• Open (everybody can use it)
• Scalable (to users, data size)
• Extensible (to more tests, different technology,
more measures)
• Sustainable (beyond SEALS)
• Independent (unbiased)
• Repeatable (evaluations can be reproduced)
6
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
The SEALS platform
Results Comparisons
Evaluations
Tools
Results
Semantic Technology
Recommendation
Framework
Test Data
7
7
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
SEALS Logical Architecture
Virtualization:
two execution
environments
Evaluation Organisers
Technology
Providers
SEALS Portal
Runtime
Evaluation
Service
Technology
Adopters
SEALS
Service Manager
Software agents
SEALS Repositories
Test Data
Repository
Service
Tools
Repository
Service
Results
Repository
Service
Evaluation
Descriptions
Repository Service
14.07.2017
8
8
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Evaluation Execution Process
at 10,000ft
Infrastructure
request
Tool
deployment
Execution
request
analysis
Tool
undeployment
Execution
environment
preparation
Evaluation
description
execution
Infrastructure
release
Execution
environment
clean-up
ER
Test data
stage-in
Activity
Execution
Results
storage
Test data
stage-out
10.07.2012
9
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Evaluation services
Tools
Use your dataset to
evaluate others tool
Define your own
experiment
Evaluations
Test data
My tool
Evaluations
My results
Tools
My test data
Tools
Test data
My evaluation
My results
Exploit results
Reproduce
experiments
Results
My results
My tool
My test data
14.07.2017
10
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Achievements
Specification of the
Platform Architecture
SEALS entities (dataset, tools,
evaluations and results) are
described using Ontologies
Implementation and Evaluation
of Repositories
Test Data
Repositor
y Service
Tools
Repositor
y Service
Results
Repositor
y Service
Evaluation
Descriptions
Repository
Service
http://www.seals-project.eu/ontologies/
Workflows
Implemented in BPEL
Semantic Technology
Recommendation Framework
Results Comparisons
11
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 14.07.2017
Conference. 19-20th March 2013. Amsterdam
SEALS support to scholarly workflows
Scholarly Portal
Tool1
Tool3
Tool1
Tool2
Tool1
Tool4
Tool5
Tool6
Tool2
Tool1
Tool4
Tool5
Tool6
Tool7
Tool6



SEALS
Service Manager
Runtime
Evaluation
Service
SEALS Repositories
Scholarly
Workflow
Repository
Test Data
Repository
Tools
Repository
Results
Repository
Evaluation
Descriptions
Repository
12
12
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Thank you for your attention!
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam