When data management meets
cosmology and astrophysics: some
lessons learned from the Petasky project
Farouk Toumani
LIMOS, CNRS, Blaise Pascal University
Clermont-Ferrand, France
●
Expérience de cosmologie de
4ème génération :
●
Télescope de 8,4 m
●
Cerro Pachon (Chili)
●
Astronomie très grand champ :
caméra 9,6□
●
●
Tout le ciel visible en 6
bandes optiques (20000 □)
Poses de 15 s, 1 visite / 3
jours
Journées de l’interdisciplinarité,
10-11 Décembre, 2014, Paris, France
10 ans, 60 Pbytes de données
●
15/03/12
Emmanuel Gangler – Réunion LIMOS
Petasky
http://com.isima.fr/Petasky
Mastodons(program!of!the!Interdisciplinary!Mission!of!CNRS!
•
INS2I
LIMOS (UMR CNRS 6158, Clermont-Ferrand)
✦
LIRIS (UMR CNRS 5205, Lyon)
✦
LABRI (UMR CNRS 5800, Bordeaux)
✦
LIF (UMR CNRS 7279, Marseille)
✦
LIRMM (UMR CNRS 5506, Montpellier)
IN2P3
✦
LPC (UMR CNRS 6533, Clermont-Ferrand)
✦
APC (UMR CNRS 7164, Paris)
✦
LAL (UMR CNRS 8607, Paris)
✦
Centre de Calcul de l’IN2P3/CNRS (CC-IN2P3)
INSU
✦
LAM (UMR CNRS 7326, Marseille)
✦
•
•
Petasky: scientific challenges
• Management of scientific data in the fields of cosmology and astrophysics
➡
Large amount of data
➡
Complex data (e.g., images, uncertainty, multi-scales...)
➡
Heterogeneous formats
➡ Various
and complex processing (images analysis, reconstruction of
trajectories, ad-hoc queries and processings, …)
• Scientific challenges
➡ Scalability
➡ Data
integration
➡ Data
analysis
➡ Visualisation
• Application context :
LSST project
Science in an exponential world
The availability of very large amounts of data and the ability to efficiently
process them is changing the way we do science
• Science paradigms1
1. Empirical description of natural phenomena
2. Theoretical science: models and generalization
3. Computational science: simulation of complexe phenomena to validate
theories
4. Data Intensive science : collecting and analyzing large amount of data
1Jim
Gray, eScience Talk at NRC-CSTB meeting Mountain View CA, 11 January 2007.
From Astronomy to astroinformatics
• Modern digital detectors, CCDs,
• Early use of scientific computing, numeric simulations, ..
➡
Antikythera mechanism, between 150 to 100 BC
➡
Supernovae Cosmology Project, 1986
➡
1024x1024 CCD camera, 2 megabytes every five minutes
International Virtual Observatory Alliance (IVOA)
-
Web of astronomical data
➡
Sloan Digital Sky Survey (SDSS)
➡
GAIA, launched in 12/2013 and started the scientific observations in
7/2014
• A culture of sharing data
➡
Data with non-commercial value (more open than healthcare or
biomedical science field)
From Astronomy to astroinformatics
• Modern digital detectors, CCDs,
• Early use of scientific computing, numeric simulations, ..
➡
Antikythera mechanism, between 150 to 100 BC
➡
Supernovae Cosmology Project, 1986
➡
1024x1024 CCD camera, 2 megabytes every five minutes
International Virtual Observatory Alliance (IVOA)
-
« … a mechanical computer used for
calculating lunar, solar and stellar
calendars »
Web of astronomical data
➡
Sloan Digital Sky Survey (SDSS)
➡
GAIA, launched in 12/2013 and started the scientific observations in
7/2014
• A culture of sharing data
➡
Data with non-commercial value (more open than healthcare or
biomedical science field)
From Astronomy to astroinformatics
• Modern digital detectors, CCDs,
• Early use of scientific computing, numeric simulations, ..
➡
Antikythera mechanism, between 150 to 100 BC
➡
Supernovae Cosmology Project, 1986
➡
1024x1024 CCD camera, 2 megabytes every five minutes
International Virtual Observatory Alliance (IVOA)
-
« … a mechanical computer used for
calculating lunar, solar and stellar
calendars »
Web of astronomical data
➡
Sloan Digital Sky Survey (SDSS)
➡
GAIA, launched in 12/2013 and started the scientific observations in
7/2014
• A culture of sharing data
➡
Data with non-commercial value (more open than healthcare or
biomedical science field)
From Astronomy to astroinformatics
• Modern digital detectors, CCDs,
• Early use of scientific computing, numeric simulations, ..
➡
Antikythera mechanism, between 150 to 100 BC
➡
Supernovae Cosmology Project, 1986
-
1024x1024 CCD camera, 2 megabytes every five minutes
« … a mechanical computer used for
calculating lunar, solar and stellar
calendars »
➡
International Virtual Observatory Alliance (IVOA)
SDSS
- Web of astronomical data
➡
Sloan Digital Sky Survey (SDSS)
➡
GAIA, launched in 12/2013 and started the scientific
observations
working in
in 2000
• Started
7/2014
• In 2010, a total archive of 140 TB
•
2.5 m Telescope, 54 CCD imager
• A culture of sharing data
➡
Data with non-commercial value (more open than healthcare or
biomedical science field)
From Astronomy to astroinformatics
• Modern digital detectors, CCDs,
• Early use of scientific computing, numeric simulations, ..
➡
Antikythera mechanism, between 150 to 100 BC
➡
Supernovae Cosmology Project, 1986
-
1024x1024 CCD camera, 2 megabytes every five minutes
« … a mechanical computer used for
calculating lunar, solar and stellar
calendars »
➡
International Virtual Observatory Alliance (IVOA)
SDSS
- Web of astronomical data
➡
Sloan Digital Sky Survey (SDSS)
➡
GAIA, launched in 12/2013 and started the scientific
observations
working in
in 2000
• Started
7/2014
• In 2010, a total archive of 140 TB
•
2.5 m Telescope, 54 CCD imager
• A culture of sharing data
➡
Data with non-commercial value (more open than healthcare or
biomedical science field)
From Astronomy to astroinformatics
• Modern digital detectors, CCDs,
• Early use of scientific computing, numeric simulations, ..
➡
Antikythera mechanism, between 150 to 100 BC
➡
Supernovae Cosmology Project, 1986
-
1024x1024 CCD camera, 2 megabytes every five minutes
« … a mechanical computer used for
calculating lunar, solar and stellar
calendars »
➡
International Virtual Observatory Alliance (IVOA)
SDSS
- Web of astronomical data
➡
Sloan Digital Sky Survey (SDSS)
➡
GAIA, launched in 12/2013 and started the scientific
observations
working in
in 2000
• Started
7/2014
• In 2010, a total archive of 140 TB
•
2.5 m Telescope, 54 CCD imager
• A culture of sharing data
➡
Data with non-commercial value (more open than healthcare or
biomedical science field)
From Astronomy to astroinformatics
• Modern digital detectors, CCDs,
• Early use of scientific computing, numeric simulations, ..
➡
Antikythera mechanism, between 150 to 100 BC
➡
Supernovae Cosmology Project, 1986
-
1024x1024 CCD camera, 2 megabytes every five minutes
« … a mechanical computer used for
calculating lunar, solar and stellar
calendars »
➡
International Virtual Observatory Alliance (IVOA)
SDSS
- Web of astronomical data
➡
Sloan Digital Sky Survey (SDSS)
➡
GAIA, launched in 12/2013 and started the scientific
observations
working in
in 2000
• Started
7/2014
• In 2010, a total archive of 140 TB
•
2.5 m Telescope, 54 CCD imager
• A culture of sharing data
➡
Data with non-commercial value (more open than healthcare or
biomedical science field)
How much bytes…
10005
1015
péta
10004
1012
téra
10003
109
giga
10002
106
méga
10001
103
kilo
How much bytes…
10005
1015
péta
10004
1012
téra
A single text character
giga
A typewritten page
A high-resolution photograph
méga
The complete works of Shakespeare
1 byte
2 kilobyte s
2 megabytes
5 megabytes
A minute of high-fidelity sound
kilo
A pickup truck filled with books
The contents of a DVD
A collection of the works of Beethoven
50,000 trees made into paper and printed
The print collections of the U.S. Library of
Congress
All
U.S. academic research libraries
10 megabytes
1 gigabyte GB )
17 gigabyte s
20 gigabytes
1 terabyte ( TB )
10 terabytes
2 petabytes
10003
109
10002
106
10001
103
All hard disk capacity developed in 1995
http://searchstorage.techtarget.com/definition/How-many-bytes-for
20 petabytes
Sizes of the astronomical datasets
PB
TB
GB
MB
KB
1980
1990
2000
2010
Sizes of the astronomical datasets
PB
TB
GB
MB
KB
1980
1990
2000
2010
Sizes of the astronomical datasets
PB
TB
GB
MB
KB
1980
1990
2000
2010
Sizes of the astronomical datasets
PB
TB
GB
MB
KB
1980
1990
2000
2010
Sizes of the astronomical datasets
PB
TB
GB
MB
KB
1980
1990
2000
2010
Sizes of the astronomical datasets
PB
TB
GB
MB
KB
1980
1990
2000
2010
E-science evolution
E-science evolution
E-science evolution
E-science evolution
Homo FTP-GREPus
E-science evolution
Homo FTP-GREPus
E-science evolution
Homo FTP-GREPus
In 20041
FTP/GREP 1GB in a minute
FTP/GREP 1TB in 2 days
FTP/GREP 1PB in 3 years
1Where
The Rubber Meets the Sky Giving Access to Science Data, Jim Gray and Alex Szalay
E-science evolution
Homo FTP-GREPus
In 20041
FTP/GREP 1GB in a minute
FTP/GREP 1TB in 2 days
FTP/GREP 1PB in 3 years
1Where
The Rubber Meets the Sky Giving Access to Science Data, Jim Gray and Alex Szalay
E-science evolution
Homo FTP-GREPus
In 20041
FTP/GREP 1GB in a minute
FTP/GREP 1TB in 2 days
FTP/GREP 1PB in 3 years
Homo Numericus
1Where
The Rubber Meets the Sky Giving Access to Science Data, Jim Gray and Alex Szalay
E-science evolution
Homo FTP-GREPus
In 20041
FTP/GREP 1GB in a minute
FTP/GREP 1TB in 2 days
FTP/GREP 1PB in 3 years
Homo Numericus
1Where
The Rubber Meets the Sky Giving Access to Science Data, Jim Gray and Alex Szalay
E-science evolution
Homo FTP-GREPus
In 20041
FTP/GREP 1GB in a minute
FTP/GREP 1TB in 2 days
FTP/GREP 1PB in 3 years
Homo Numericus
1Where
The Rubber Meets the Sky Giving Access to Science Data, Jim Gray and Alex Szalay
Grid computing
Cloud computing
Virtualization
MapReduce
New hardware
NoSQL
…
Data-driven discovery in Astrophysics
Telescopes
Observatories
Data-driven discovery in Astrophysics
Telescopes
Observatories
Digitized
data
Data-driven discovery in Astrophysics
Telescopes
Observatories
Digitized
data
Processing
pipeline
Data-driven discovery in Astrophysics
Telescopes
Observatories
Digitized
data
Processing
pipeline
Information/
knowledge
Data-driven discovery in Astrophysics
Composants
d’un
Composants
SGBD
d’un
Composants
SGBD
d’un SGBD
Telescopes
Observatories
User/Application
User/Application
Composants
d’un
SGBD
DBA Composants
DBA
d’un SGBD
Digitized
Transaction commands
Queries,
updates
Transaction commands
Queries,
updates
data
Query
Transaction
Metadata, compiler
manager
statistics
Transaction commands
Query
Transaction
DDL
Transaction
Execution
Logging
and
manager
engine
recovery
Index, file
and record
requests
Logging and
Index/file/record
Log
recovery
manager
pages
Data,
BufferBuffer
manager
Data,
metadata,
indexes
Storage
Storage
Buffer
manager
Storage
Log
pages
Page
commands
DDL commands
Transaction
DDL
DDL
Query
Execution
Logging
Concurrency
and
compiler
Metadata,
compiler
engine
recoverycontrolstatistics
Processing
Query
plan pipeline
Index/file/record
Execution Concurrency
Log
control
manager
engine
Lock
tableData,
pages
Data,
Index, metadata,
file
and
record
Page
indexes
requests
commands
metadata,
ge indexes
ands
DBA
DDL commands
Metadata, compiler
Metadata,
manager
compiler
manager
compiler
DBA
User/Application
statistics
statistics
Metadata
Transaction commands
Query
DDL commands
Queries,
plan
updates
Query
plan
Metadata,
statistics
ex, file
record
quests
Transaction commands
DDL commands
Transaction
Logging
Concurrency
and
manager
recoverycontrol
Metadata
Metadata
DDL
compiler
DBA
DDL
Concurrency
compiler
control
Metadata
Logging and Log
recoveryLock
table
pages
Concurrency
control
Lock
table
metadata,
indexes
Information/
BufferBuffer
Index/file/record
knowledge
manager
manager
Lock tableData,
Buffer
metadata,
indexes
Storage
Buffer
Storage
manager
manager
Storage
Buffer
Dedicated
Storage
manager
archives
Storage
Metadata
DDL commands
Log
pages
Lock table
Data-driven discovery in Astrophysics
Composants
d’un
Composants
SGBD
d’un
Composants
SGBD
d’un SGBD
Telescopes
Observatories
User/Application
User/Application
Composants
d’un
SGBD
DBA Composants
DBA
d’un SGBD
Digitized
Transaction commands
Queries,
updates
Transaction commands
Queries,
updates
data
Query
Transaction
Metadata, compiler
manager
statistics
Transaction commands
Query
Transaction
DDL
Transaction
Execution
Logging
and
manager
engine
recovery
Index, file
and record
requests
Logging and
Index/file/record
Log
recovery
manager
pages
Data,
BufferBuffer
manager
Data,
metadata,
indexes
Storage
Storage
Buffer
manager
Storage
Log
pages
Page
commands
DDL commands
Transaction
DDL
DDL
Query
Execution
Logging
Concurrency
and
compiler
Metadata,
compiler
engine
recoverycontrolstatistics
Processing
Query
plan pipeline
Index/file/record
Execution Concurrency
Log
control
manager
engine
Lock
tableData,
pages
Data,
Index, metadata,
file
and
record
Page
indexes
requests
commands
metadata,
ge indexes
ands
DBA
DDL commands
Metadata, compiler
Metadata,
manager
compiler
manager
compiler
DBA
User/Application
statistics
statistics
Metadata
Transaction commands
Query
DDL commands
Queries,
plan
updates
Query
plan
Metadata,
statistics
ex, file
record
quests
Transaction commands
DDL commands
Transaction
Logging
Concurrency
and
manager
recoverycontrol
Metadata
Metadata
DDL
compiler
DBA
DDL
Concurrency
compiler
control
Metadata
Logging and Log
recoveryLock
table
pages
Concurrency
control
Lock
table
metadata,
indexes
Information/
BufferBuffer
Index/file/record
knowledge
manager
manager
Lock tableData,
Metadata
DDL commands
Public
Buffer
Log
pages
Lock table
metadata,
indexes
Storage
Buffer
Storage
manager
manager
Storage
Buffer
Dedicated
Storage
manager
archives
Storage
Education
Science
Data-driven discovery in Astrophysics
Composants
d’un
Composants
SGBD
d’un
Composants
SGBD
d’un SGBD
Telescopes
Observatories
User/Application
User/Application
Composants
d’un
SGBD
DBA Composants
DBA
d’un SGBD
Digitized
Transaction commands
Queries,
updates
Transaction commands
Queries,
updates
data
Query
Transaction
Metadata, compiler
manager
statistics
Transaction commands
Query
Transaction
DDL
Transaction
Execution
Logging
and
manager
engine
recovery
Index, file
and record
requests
Logging and
Index/file/record
Log
recovery
manager
pages
Data,
BufferBuffer
manager
Data,
metadata,
indexes
Storage
Storage
Buffer
manager
Storage
Log
pages
Page
commands
DDL commands
Transaction
DDL
DDL
Query
Execution
Logging
Concurrency
and
compiler
Metadata,
compiler
engine
recoverycontrolstatistics
Processing
Query
plan pipeline
Index/file/record
Execution Concurrency
Log
control
manager
engine
Lock
tableData,
pages
Data,
Index, metadata,
file
and
record
Page
indexes
requests
commands
metadata,
ge indexes
ands
DBA
DDL commands
Metadata, compiler
Metadata,
manager
compiler
manager
compiler
DBA
User/Application
statistics
statistics
Metadata
Transaction commands
Query
DDL commands
Queries,
plan
updates
Query
plan
Metadata,
statistics
ex, file
record
quests
Transaction commands
DDL commands
Transaction
Logging
Concurrency
and
manager
recoverycontrol
Metadata
Metadata
DDL
compiler
DBA
DDL
Concurrency
compiler
control
Metadata
Logging and Log
recoveryLock
table
pages
Concurrency
control
Lock
table
metadata,
indexes
Information/
BufferBuffer
Index/file/record
knowledge
manager
manager
Lock tableData,
Metadata
DDL commands
Public
Buffer
Log
pages
Lock table
metadata,
indexes
Storage
Buffer
Storage
manager
manager
Storage
Buffer
Dedicated
Storage
manager
archives
Storage
Education
Science
Astrometric pipeline
Outputs object positions
Astrometric pipeline
Outputs object positions
Spectroscopic pipeline
redshifts, classification of
objects, …
●
Expérience de cosmologie de
4ème génération :
The LSST project
Télescope de 8,4 m
Large Synoptic Survey telescope
Cerro Pachon (Chili)
●
●
Astronomie très grand champ :
□
caméra 9,6
Telescope
of 8.4 m
●
A new window over the sky:
●
●
●
15/03/12
Tout le ciel visible en 6
bandes optiques (20000 □)
Poses de 15 s, 1 visite / 3
jours
10 ans, 60 Pbytes de données
Emmanuel Gangler – Réunion LIMOS
The LSST
Organisation projet :
PetaSky
Non-proft corporation
● US : 33 partenaires ; 670 M$
● Chili : site
● France : IN2P3 (~15 M€)
Telescope
Caméra
Data Management
Outreach
« The data volumes […] of LSST are so large that the limitation on our ability to do
science isn't the ability to collect the data, it's the ability to understand […] the
data »
Andrew Conolly (U. Washington)
“How do you turn petabytes of data into scientific knowledge?”
Kirk Borne (George Mason U.)
Data management challenges in LSST
“How much the (LSST) project will tell us about our solar system, the dark energy
problem and more, will depend on how well we can process the information the
telescope and its camera send back to us - an estimated sum of around ten
petabytes of data per year.”
(Mari Silbey, Space: the big data frontier, http://www.smartplanet.com/blog/thinking-tech/space-the-big-data-frontier/12180)
Data management challenges in LSST
“How much the (LSST) project will tell us about our solar system, the dark energy
problem and more, will depend on how well we can process the information the
telescope and its camera send back to us - an estimated sum of around ten
petabytes of data per year.”
(Mari Silbey, Space: the big data frontier, http://www.smartplanet.com/blog/thinking-tech/space-the-big-data-frontier/12180)
“Plans for sharing the data from LSST with the public are as ambitious as the
telescope itself”
Anyone with a computer will be able to fly through the Universe, zooming past objects a hundred
million times fainter than can be observed with the unaided eye. The LSST project will provide
analysis tools to enable both students and the public to participate in the process of scientific
discovery.
LSST data scales
Tens of thousands of billions of photometric observations over tens of
billions of objects
LSST data scales
Tens of thousands of billions of photometric observations over tens of
billions of objects
•
1-10 Millions events per night, 3 billions of sources
•
16 TB each 8 hours at a rate of 540 MB/second
Images
•
•
-
12,8 GB/39 per second (data rate of 330 MB/s)
-
100 PB final archives of images
Transients
-
•
•
1-3 PB, Relation with 100 attributes/5000 billions of tuples
Objects catalog :
-
Relation with 500 attributes, 40 Billions of tuples
-
100-200 TB
Estimation for the end of the project
• 400 000 Billions of tuples (different versions of data in addition to replication),
• Raw data: 60 PB
• Catalog database: 15 PB
• Total volume after processing : several hundred of PB
LSST data scales
Tens of thousands of billions of photometric observations over tens of
billions of objects
•
1-10 Millions events per night, 3 billions of sources
•
16 TB each 8 hours at a rate of 540 MB/second
Images
•
•
-
12,8 GB/39 per second (data rate of 330 MB/s)
-
100 PB final archives of images
Transients
-
•
•
Highest data rate in current
astronomical surveys: 4.3 MB/s Sloan Digital Sky Survey (SDSS)
1-3 PB, Relation with 100 attributes/5000 billions of tuples
Objects catalog :
-
Relation with 500 attributes, 40 Billions of tuples
-
100-200 TB
Estimation for the end of the project
• 400 000 Billions of tuples (different versions of data in addition to replication),
• Raw data: 60 PB
• Catalog database: 15 PB
• Total volume after processing : several hundred of PB
LSST data scales
LSST year 1
LSST year 10
Raw data
6 PB
60 PB
Archive
19 PB
270 PB
Disk (DAC)
16 PB
90 PB
DB
(baseline)
0,5 PB
5 PB
Moore
equivalent
2014
1.2 TB
12 TB
LSST data scales
LSST year 1 Table&
LSST year 10
Raw data
Archive
Disk (DAC)
6 PB Object'
60 PB 109'TB'
Moore
equivalent
2014
#tuples&
#a/ributes&
38'B'
470'
19 PB Moving'Object'
270 PB 5'GB'
6'M'
100'
16 PB Source'
5'T'
125'
32'T'
7'
0,5 PB Difference'Image'
5 PB 71'TB'
Source'
200'B'
65'
1.2 TB CCD'Exposure'
12 TB 0.6'TB'
17'B'
45'
90 PB 3.6'PB'
Forced'Source'
DB
(baseline)
Size&
1.1'PB'
LSST data scales
A change of scale from TB to PB
Data volumes & rates are unprecedented
in astronomy
Relative Etendue (= A )
320
280
20000
Estimated Nightly Data Volume
Etendue (m2 deg2)
240
15000
200
160
GB 10000
120
5000
80
All facilities assumed operating100% in one survey
0
40
0
LSST
PS4
PS1
Subaru
CFHT
SDSS
MMT
Steven M. Kahn
SLUO Meeting Presentation
June 7, 2007
DES
4m
VST
Raw
Catalog
LSST Pan-STARRS 4
VISTA
IR
4
SDSS
LSST will make tens of trillions
photometric observations of tens of
billions of objects
Queries per difficulty level
Supported queries
• Retrieve any type of information about a single object (identified by a given objectId),
including full time series.
SELECT * FROM Object JOIN Source USING (objectId) WHERE objectId =
293848594;
Few seconds
• Retrieve any type of information about a group of objects in a small area of sky, including
neighborhood-type queries.
SELECT * FROM Object WHERE qserv_areaSpec_circle(1.0, 35.0, 5.0/60)
≃1 hour
• Analysing light curves across large area.
SELECT O.objectId, myFunction(S.taiMidPoint, S.psfFlux) FROM Object AS O JOIN
Source AS S USING (objectId) WHERE O.varProb > 0.75 GROUP BY O.objectId;
≃1 day (24h)
• Analysing light curves of faint objects across large area.
SELECT O.objectId, myFunction(V.taiMidPoint, FS.flux) FROM Object AS O JOIN
ForcedSource AS FS ON (O.objectId = FS.objectId) JOIN Visit AS V ON (FS.visitId =
V.visitId);
≃1 week
! !
Queries per difficulty level
Expensive/impossible queries
• Expensive queries
– Find objects far away from other objects (for a large number of objects).
Question: what is the largest distance we should plan to support for distance-based
queries involving (a) small number of objects, (b) all objects on the sky?
– Sliding window queries: Find all 5 arcmin x 5 arcmin regions with an object density
higher than rho
• Impossible queries
– Large size results
• Select all pairs of stars within 1 arc min of each other in the Milky Way region.
–Expensive or hidden computation (e.g., Join)
• Near neighbor query on the Source or ForcedSource table
• Joining large tables between different LSST data releases
• Time series analysis of every object
• Cross-match with very large external catalog (e.g. LSST with SKA)
• Any non-spatial join on the entire catalog (Object, Source, ForcedSource)
• Join of Source with ForcedSource
Examples of User Defined Functions
•
•
•
q3c_ang2ipix(ra, dec) -- returns the ipix value at ra and dec
•
q3c_ellipse_join(ra1, dec1, ra2, dec2, major, ratio, pa) -- like q3c_join, except (ra1, dec1) have to
be within an ellipse with major axis major, the axis ratio ratio and the position angle pa (from north
through east)
•
q3c_radial_query(ra, dec, center_ra, center_dec, radius) -- returns true if ra, dec is within radius
degrees of center_ra, center_dec. This is the main function for cone searches. function should be
used if when the index on q3c_ang2ipix(ra,dec) is created)
•
q3c_ellipse_query(ra, dec, center_ra, center_dec, maj_ax, axis_ratio, PA ) -- returns true if ra,
dec is within the ellipse from center_ra, center_dec. The ellipse is specified by major axis, axis ratio
and positional angle. function should be used if when the index on q3c_ang2ipix(ra,dec) is created)
•
•
•
q3c_poly_query(ra, dec, poly) -- returns true if ra, dec is within the postgresql polygon poly.
•
q3c_ipixcenter(ra, dec, bits) -- the function returning the ipix value of the pixel center of certain
depth covering the specified (ra,dec)
q3c_dist(ra1, dec1, ra2, dec2) -- returns the distance in degrees between (ra1,dec1) and (ra2,dec2)
q3c_join(ra1, dec1, ra2, dec2, radius) -- returns true if (ra1, dec1) is within radius spherical
distance of (ra2, dec2). It should be used when the index on q3c_ang2ipix(ra2,dec2) is created.
q3c_ipix2ang(ipix) -- returns a 2-array of (ra,dec) corresponding to ipix.
q3c_pixarea(ipix, bits) -- returns the area corresponding to ipix at level bits (1 is smallest, 30 is the
cube face) in steradians.
Main thesis of this talk
How to form a new generation of scientists capable to
exploit the new technologies to pursue science goals at an
unprecedent scale?
G.Longo 1
1Talk, workshop
« New challenges in astro- and environmental informatics in the Big Data era », May, 2014, Szombathely, Hungary
Main thesis of this talk
How to form a new generation of scientists capable to
exploit the new technologies to pursue science goals at an
unprecedent scale?
G.Longo 1
It should not be up to the scientists but to the
technology (data management system) to
overcome the computing barriers between them
and the data
1Talk, workshop
« New challenges in astro- and environmental informatics in the Big Data era », May, 2014, Szombathely, Hungary
Main thesis of this talk
How to form a new generation of scientists capable to
exploit the new technologies to pursue science goals at an
unprecedent scale?
G.Longo 1
It should not be up to the scientists but to the
technology (dataAnalysts
management
system)
to
instead of Hackers
overcome the computing barriers between them
and the data
1Talk, workshop
« New challenges in astro- and environmental informatics in the Big Data era », May, 2014, Szombathely, Hungary
Main thesis of this talk
How to form a new generation of scientists capable to
exploit the new technologies to pursue science goals at an
unprecedent scale?
G.Longo 1
It should not be up to the scientists but to the
technology (dataAnalysts
management
system)
to
instead of Hackers
overcome the computing barriers between them
and the data
1Talk, workshop
« New challenges in astro- and environmental informatics in the Big Data era », May, 2014, Szombathely, Hungary
Main thesis of this talk
This
approach
has
been
very
successful
in
How to form a new generation of scientists capable to
business
domain
but intogeneral
not sogoals
successful
exploit
the new
technologies
pursue science
at an
in the scientific domain
unprecedent scale?
G.Longo 1
It should not be up to the scientists but to the
technology (dataAnalysts
management
system)
to
instead of Hackers
overcome the computing barriers between them
and the data
1Talk, workshop
« New challenges in astro- and environmental informatics in the Big Data era », May, 2014, Szombathely, Hungary
What makes DB technology successful in
business domain?
• Abstractions
✓
Relation instead of files, blocks, tablespaces,
segments, extents, access path
✓
Relational algebra instead of algorithms
• Declarative query language
✓
Express what you want not how to get it
• Optimization
✓
Rather naive techniques but enough for the
business world
Example: query compilation
Select Name
From Employé E, Département D, Projet.P
Where E.empno=D.manager and E.empno=P.mgr
Example: query compilation
Exemple (cont.)
Select Name
From Employé E, Département D, Projet.P
Plan
d’exécution
Where E.empno=D.manager and E.empno=P.mgr
Index Nested loop
(A.x=C.x)
Merge-join
(A.x=B.x
Sort
Table-scan
A
Table-scan
Employé
Index scanProjet
C
Index-scan
Sort
Table-scan
B
Table-scan
Département
Petasky: data management challenge
Techniques to build an efficient and easy to use data access system
at a reasonable cost
•
•
•
Specialized Hardware
Programming
Ad-hoc optimization
•
•
•
Commodity machines
Querying
Generic system
Space of solutions and associated
challenges
! Clearly beyond the capacities of centralized systems
•
•
Distributed and parallel systems
✓
Data distribution
✓
Computation distribution
✓
Failure resilience
Storage model
✓
row store vs. column store
✓
(sophisticated) Indexes
•
Benefit from modern hardware
•
Complexity theory and cost models
✓
Standards measures: I/O, data transfer, ..
✓
Cost of coordination
Explored approaches
• Big data approaches
✓
✓
Distributed and parallel systems
➡
MapReduce-like approaches (shared nothing architecture)
➡
Parallel DBMS (shared all thing architecture)
➡
Spatial partitioning (QSERV for LSST)
Column store DBMSs
➡
•
Vertica, MonetDB, ...
Data integration to the rescue
➡
Declarative approach
●
Agrégation des résultats
●
Le temps de calcul
assage à grande échelle (au CC IN2P3) :
œuds physiques, 120 GB disque, 16 GB RAM
es, 3000 chunks → 50 GB / nœud
lélisation du déploiement : espace disque = taille DB x2
23/1/14
Emmanuel Gangler – Workshop Mastodons
des surcoûts (tables en cache)
Limitations
•
ités de performance :
Limited queries
✓
- distance ~1 arcmin
→ 300 noeuds
-
non spatial joins
munications, multi-threading
Non-partitionable
aggregates
réponse
~proportionnel
-
de
✓
User
defined functions
mbre de
nœuds
✓
Load balancing
✓
Ad-hoc query rewriting
13/40
Mediation-based
approach
(cont.)
Data integration at the rescue
User query
Answers
Semantic
Mappings
. . .
View on
Source 1
Answers
View on
Source 2
View on
Source 3
View on
Source N
Query
Wrapper 1
Answers
Wrapper 2
Wrapper 3
Wrapper N
Query
Documents
Source 1
Source 2
Source 3
Source N
uted on clusters of several nodes and both their inputs and outputs are files
ibuted file system (e.g.,Mapreduce
Hadoop Distributed
File System (HDFS)).
approaches
a new programming model intended to facilitate the
1 development
MapReduce of
Execution
Overview
scalable parallel computations on large
server clusters
Mapreduce approaches
Hadoop!
Hadoop++!
HAIL!
HadoopDB!
Hive!
Query!language
procedural
Procedural
Procedural
DeclaraGve
DeclaraGve
Simple!indexes
Not!supported
supported
Supported
supported
Supported
MulG!indexes
Not!supported
Not!supported
supported
Supported
Supported
Complex!indexes Not!supported
Not!supported
Not!supported
supported
Supported
Storage!
System
HDFS
HAIL
Classical!DBMS
HDFS
Manual
Manual
AutomaGc
Manual
[Dean!et!al,!04]
Index!choice
HDFS
O
[Di3rich!et!al,!10]
[Di%rich!et!al,!12]
[Abouzeid!et!al,!09]
[Thusoo!et!al,!10]
Data Mapreduce
loading (1/2)
approaches
Minutes"
2000"
1500"
Hadoop!
Hadoop++!
HAIL!
HadoopDB!
Hive!
procedural
Procedural
Procedural
DeclaraGve
DeclaraGve
supported
Supported
Not!supported
supported
Supported
Difference"(Hive">>"HadoopDB):"300"%"for"
25"machines"and"200"%"for"50"machines"
•  Hive:"
Supported
[Di3rich!et!al,!10]
[Dean!et!al,!04]
1000"
500"
Query!language
0"
250"GB"
500"GB"
Simple!indexes
1"TB"
25"machine"
Not!supported
Hive"(HDFS)"
global"hash"
250"GB"
supported
local"hash"
[Di%rich!et!al,!12]
500"GB"
[Abouzeid!et!al,!09]
1"TB"
50"machines"
Supported
tuning"
loading.4me.for.1.TB9.50.machines.
MulG!indexes
Not!supported
• 
33%.
39%.
Complex!indexes Not!supported
15%.
Storage!
System
HDFS"
global"hash"
Index!choice
13%.
Not!supported
• Not!supported
2X"of"Ime"for"2X"of"data"volume""
•  25"!"50"machines:"same"Ime"
•  HadoopDB:"
HDFS
local"hash"
tuning"
supported
Supported
HDFS
HAIL
Classical!DBMS
HDFS
Manual
Manual
AutomaGc
Manual
•  2X"of"data:"90"%K120"%""supplementary""
•  25"!50"machines:"25"%""gain""
O
[Thusoo!et!al,!10]
Data Mapreduce
loading (1/2)
approaches
Data loading (2/2)
1500"
Hadoop!
Hadoop++!
HAIL!
6%$
HadoopDB!
Hive!
procedural
Procedural
Procedural
DeclaraGve
DeclaraGve
supported
Supported
[Di3rich!et!al,!10]
[Dean!et!al,!04]
30%$
1000"
500"
Query!language
250"GB"
500"GB"
1"TB"
25"machine"
Not!supported
Simple!indexes
Load%&me%
250"GB"
supported
indexing%
Hive"(HDFS)"
global"hash"
local"hash"
Indexing$for$1$TB$3$50$machines$$
Hive$
loading.4me.for.1.TB9.50.machines.
MulG!indexes
Not!supported
• 
200%
33%.
150%
39%.
15%.
Storage!
System
global"hash"
Index!choice
100%
13%.
50%
0%
[Abouzeid!et!al,!09]
94%$
Load%&me%
[Thusoo!et!al,!10]
indexing%
Indexing$for$1$TB$3$50$machines$
HadoopDB$
Not!supported
supported
Supported
Difference"(Hive">>"HadoopDB):"300"%"for"
•  Hive:%
25"machines"and"200"%"for"50"machines"
objec&d%
O
ra%
Not!supported
• Not!supported
2X"of"Ime"for"2X"of"data"volume""
•  25"!"50"machines:"same"Ime"
decl%
Supported
•  25%!%50%machines:%gain%of%15%%
supported
Supported
HAIL
Classical!DBMS
HDFS
Manual
AutomaGc
Manual
•  2X"of"data:"90"%K120"%""supplementary""
500%GB% •  25"!50"machines:"25"%""gain""
1TB%
2%TB%
250GB%
Sourceid%
tuning"
HDFS
tuning"
1"TB"
50"machines"
Supported
•  HadoopDB:"
HDFS
local"hash"
500"GB"
•  Hive:"
Complex!indexes Not!supported
HDFS"
[Di%rich!et!al,!12]
70%$
0"
Index%size%(GB)%
Minutes"
2000"
scienceccdexposure%
Manual
Data Mapreduce
loading (1/2)
approaches
Data loading (2/2)
1500"
Hadoop!
Hadoop++!
HAIL!
6%$
HadoopDB!
Hive!
procedural
Procedural
Procedural
DeclaraGve
DeclaraGve
supported
Supported
Query processing
Hive vs. HadoopDB (2/6)
[Di3rich!et!al,!10]
[Dean!et!al,!04]
30%$
1000"
500"
Query!language
500"GB"
1"TB"
2000"
250"GB"
1800"
25"machine"
Not!supported
1600"
Load%&me%
supported
indexing%
Time"(seconds)"
1400"
Hive"(HDFS)"
global"hash"
local"hash"
Indexing$for$1$TB$3$50$machines$$
1200"
Hive$
1000"
loading.4me.for.1.TB9.50.machines.
MulG!indexes 800"Not!supported
• 
200% 600"
33%.
400"
39%.
150% 200"
[Thusoo!et!al,!10]
[Abouzeid!et!al,!09]
global"hash"
Storage!
System
Index!choice
100%
Q1"
50%
tuning"
HDFS
160"
Load%&me%
140"
indexing%
Indexing$for$1$TB$3$50$machines$
120"
HadoopDB$
100"
0"
Q2"
Q3"
Q4"
HDFS
objec&d%
O
ra%
Supported
Hive"
HadoopDB" Not!supported
Hive"
Supported
Hive" supported
HadoopDB"
Hive"
HadoopDB"
Hive"
HadoopDB"
• Not!supported
2X"of"Ime"for"2X"of"data"volume""
500"GB"
1"TB"
250"GB"
500"GB"
1"TB"
•  25"!"50"machines:"same"Ime"
Q1"
Q2"
decl%
scienceccdexposure%
Manual
Q3"(RA)"
Q4"
HAIL
Classical!DBMS
HDFS
Manual
AutomaGc
Manual
•  2X"of"data:"90"%K120"%""supplementary""
500%GB% •  25"!50"machines:"25"%""gain""
1TB%
2%TB%
250GB%
Sourceid%
94%$
•  25%!%50%machines:%gain%of%15%%
20"
•  HadoopDB:"
local"hash"
0%
tuning"
HadoopDB"
250"GB"
13%.
200"
50"machines"
Supported180"
•  Hive:"
0"
1"TB"
Not!supported
supported80"
Supported
Difference"(Hive">>"HadoopDB):"300"%"for"
•  Hive:%60"
25"machines"and"200"%"for"50"machines"
40"
HadoopDB"
Hive"
Complex!indexes Not!supported
15%.
500"GB"
Time"(seconds)"
250"GB"
Simple!indexes
HDFS"
[Di%rich!et!al,!12]
70%$
0"
Index%size%(GB)%
Minutes"
2000"
Physical storage
Row store vs. column store
Row$Oriented$Database$
• High cost of I/O
• Analytical queries are expensive
• Problem with null values
Column$Oriented$Database$
• Cost of join
Hybrid storage
T
T1
T2
Hybrid Database
Hybrid storage
T
Column1(
ORACLE(
MonetDB(
Hybrid(
Q1(
00:05:00,00(
00:00:46.00(
00:06:00,00(
Q2(
00:04:42,00(
00:00:00.12(
00:02:28,00(
Q3(
00:04:15,00(
00:00:00.03((
00:00:01,00(
Q4(
00:04:04,00(
Q5(
00:00:02,19(
Q6(
T1
T2 00:00:00.90(
00:03:40,00(
00:00:42.00(
00:05:06,00(
00:00:02,50(
00:00:03,00(
00:39:00,00(
Q7(
00:00:18,14(
00:00:6.3(
00:00:18,00(
Q8(
00:07:38,00(
00:00:39.2(
01:40:00,00(
Q9(
00:01:39,00(
00:00:45.00(
Q10(
00:04:03,00(
(00:08:13,00(
Hybrid Database
00:00:09,00(
Learned lessons and research directions
No one fits all
•
•
•
MapReduce-based algorithms can be useful to implement physical operators
Hybrid system: row/column store
Need for more research on
✓
Abstraction adequate to the scientific domain
-
Array data model (SCiDB)?
✓
Support of user defined functions
✓
Optimization techniques embedded in the data management system
✓
Scalability of information integration framework
Scientists
Data mining/learning/statistics
Foundational approach to build scientific data
management system
Data management
Visualisation
Working cross disciplines
… working cross cultures
Scientists
Data mining/learning/statistics
Foundational approach to build scientific data
management system
Data management
Visualisation
Merci