MMG: from proof-of-concept to
production services at scale (part II)
Lars Ailo Bongo (ELIXIR-NO, WP6)
WP4 F2F, 19-20 June 2017,
Prague, Czech
ELIXIR-EXCELERATE is funded by the European
Commission within the Research Infrastructures programme
of Horizon 2020, grant agreement number 676559.
www.elixir-europe.org/excelerate
Outline
MMG updates
Short term needs (= EOSC pilot)
Long term needs
Demo
Discussion about business models
2
MMG on Elixir Compute Platform
Done:
•
•
•
•
•
META-pipe on cPouta, CESNET and EGI Federated Clouds
META-pipe integrated with ELIXIR AAI
EMG on Embassy cloud and GCP
Marine metagenomics portal
Metagenomics workshops using cPouta
In progress:
•
•
•
•
META-pipe on AWS
Open META-pipe web app for ELIXIR users
EMG in CWL
Documentation of best practices
TODO:
•
•
•
•
3
Test META-pipe and EMG at scale
ELIXIR compute cloud business model
Deploy META-pipe and EMG production service on cloud
Incorporate other MMG pipelines such as BioMaS
MMG EOSC Pilot
1. MMG, Compute Platform, EGI Elixir CC
2. Aims:
1.
2.
3.
4.
Evaluate the performance of META-pipe and EMG at scale using
EOSC resources.
Cost-optimize the analyses on EOSC.
Evaluate the use of elasticity in EOSC for execution of job queues.
Develop a full-service delivery model and potential business model
between the stakeholders and entities involved.
3. Not funded
4. Next step?
https://docs.google.com/document/d/124x5ygyE5xIUVHJOq94TwoqLxHgABxGhmra
wEmXdN5w/edit#
4
MMG needs (next two years)
Cloud
• Business model / resources needed to analyze end-user data
• Resources needed to generate reference databases
• Pipeline optimizations
Storage and data transfer
•
•
•
•
5
Object store
Cache for pipeline reference databases and VMs
Optimize META-pipe deployment (reference DBs >> user data)
Optimize EMG data transfer to/from cloud
MMG needs (next two years)
AAI
• Integrated with object store
Service registry/ help desk
• Help end-users to get resources
6
MMG needs (after Excelerate)
Cloud
• Sustainable business model
• Maintain cloud setup tools
• Easy deployment of new pipelines, databases, and services
Storage and data transfer
• Access data in ENA on behalf of users (as in EBI)
AAI
• Established service
Service-registry / help-desk
• ?
7
Business model alternatives
1. Users gets resources
1.
2.
3.
4.
Project or national funding/ resource
Pan-European resources (EOSC?)
Direct payment (industry)
Elixir pool of funding
2. Service provider gets resources
1.
2.
3.
Elixir node(s)
Elixir funded
Pan-European resources
3. Resource provider gets funding
1.
2.
8
Elixir nodes
Elixir funded
Business model alternatives
1. User gets resources
1.
2.
Many small allocations
Compute platform problem
2. Service provider gets resources
1.
2.
Few big allocations
MMG problem
3. Resource provider gets funding
1.
2.
9
Few big grants
Resource provider problem
Summary
How to get resources at scale?
How to establish compute platform business model?
What happens to MMG use case and compute platform after
Excelerate?
How can others take advantage of MMG experiences?
10
Demo
Alternative 1
11