FITS TAP
by
Jai Won Kim, Univ of Heidelberg, MPE
([email protected])
Gerard Lemson, MPA
([email protected])
Matthias Egger, MPA
([email protected])
IVOA Meeting – Nara, Japan 2010
Dec 7, 2010
Introduction
•
FITS: Astronomy’s common data file format
– metadata + data in a single file.
•
For publishing astronomy science data on-line in IVOA standard
– Metadata is very important.
– IVOA standards require specific metadata.
•
in this talk
Challenges building IVOA standard services.
•
GOM.FITS
– Relational database model for homogeneous, flexible and efficient storage &
retrieval of metadata independent of FITS file types .
•
Mapping FITS keywords to IVOA standards metadata by using SQL.
•
Publish and access via TAP service
IVOA Meeting – Nara, Japan 2010
2
Dec xx, 2010
Access to FITS files
1 FITS file
N homogenous FITS files
M catalogs of N FITS files
IVOA Meeting – Nara, Japan 2010
3
Dec xx, 2010
Implementing IVOA Services
Images:
SIAP
Spectra:
SSAP
Observations:
ObsTAP
.....
IVOA Meeting – Nara, Japan 2010
4
Dec xx, 2010
Challenges building (IVOA) services for
multiple archives, diverse FITS files
• FITS files are files
– hard to query efficiently.
• FITS Headers are very diverse.
– mission dependent: ROSAT, SDSS, XMM, .....
– not necessarily known which FITS headers will be useful in advance.
• Different (IVOA standard) services need different set of metadata.
– need to identify proper keywords for each service
• FITS Headers are not complete.
– some of the IVOA service required metadata might be stored in FITS
data unit (eg., image metadata stored as a data in a binary table).
– not in FITS files at all.
IVOA Meeting – Nara, Japan 2010
5
Dec xx, 2010
Mapping FITS Keywords to
SIA metadata for ROSAT images
SIMPLE =
BITPIX =
NAXIS =
NAXIS1 =
NAXIS2 =
EXTEND =
CONTENT =
ORIGIN =
DATE
=
TELESCOP=
INSTRUME=
OBS_MODE=
IRAFNAME=
MJDREFI =
MJDREFF =
ZERODATE=
ZEROTIME=
RDF_VERS=
FILTER =
OBJECT =
RA_NOM =
DEC_NOM =
ROLL_NOM=
EQUINOX =
OBS_ID =
ROR_NUM =
OBSERVER=
DATE-OBS=
TIME-OBS=
DATE_END=
TIME_END=
MJD-OBS =
T
16
2
512
512
T
/ file does conform to FITS standard
/ number of bits per data pixel
/ number of data axes
/ length of data axis 1
/ length of data axis 2
/ FITS dataset may contain extensions
'IMAGE '
/ PSPC TOTAL-BAND IMAGE
'USRSDC '
/ origin of processed data
'06/12/96'
/ FITS creation date (DD/MM/YY)
'ROSAT '
/ mission name
'PSPCB '
/ instrument name
'POINTING'
/ obs mode: POINTING,SLEW, OR SCAN
'rp700072n00_im1.imh' / IRAF file name
48043 / MJD integer SC clock start
8.79745370370074E-01 / MJD fraction SC clock start
'01/06/90'
/ UT date of SC start (DD/MM/YY)
'21:06:50'
/ UT time of SC start (HH:MM:SS)
'3.6
'
/ Rationalized Data Format release version number
'NONE
'
/ filter id: NONE OR BORON
'3C220.3 '
/ name of object
1.448500E+02 / nominal RA (deg)
8.326000E+01 / nominal DEC (deg)
-7.495722E+01 / nominal ROLL (deg CCW North)
2.000000E+03 / equinox
'US700072P.N1'
/ observation ID
700072 / ROR number
'LAWRENCE, CHARLES,R.' / PI name
'27/02/91'
/ UT date of obs start (DD/MM/YY)
'13:12:22.000'
/ UT time of obs start (HH:MM:SS)
'27/02/91'
/ UT date of obs end (DD/MM/YY)
'20:40:57.000'
/ UT time of obs end (HH:MM:SS)
4.831455E+04 / MJD of seq start
FITS standard keywords
Direct mapping b/w SIA metadata & FITS keywords
Mapping required computation
SIA metadata not available in FITS files
IVOA Meeting – Nara, Japan 2010
6
Dec xx, 2010
Common approach
Map keywords to
IVOA metadata
SIAP
Map keywords to
columns
SSAP
Map columns to
IVOA metadata
ObsTAP
....
IVOA Meeting – Nara, Japan 2010
.....
7
Dec xx, 2010
GOM.FITS Solution
SIAP
gom.fits
SSAP
ObsTAP
....
IVOA Meeting – Nara, Japan 2010
.....
8
Dec xx, 2010
GOM.FITS Data Model
FITS file
Primary HDU
SIMPLE =
BITPIX =
NAXIS =
NAXIS1 =
NAXIS2 =
EXTEND =
COMMENT
COMMENT
COMMENT
COMMENT
CONTENT =
ORIGIN =
DATE
=
TELESCOP=
INSTRUME=
OBS_MODE=
T / file does conform to FITS standard
16 / number of bits per data pixel
2 / number of data axes
512 / length of data axis 1
512 / length of data axis 2
T / FITS dataset may contain extensions
FITS (Flexible Image Transport System) format defined in Astronomy and
Astrophysics Supplement Series v44/p363, v44/p371, v73/p359, v73/p365.
Contact the NASA Science Office of Standards and Technology for the
FITS Definition document #100 and other FITS information.
'IMAGE '
/ PSPC TOTAL-BAND IMAGE
'USRSDC '
/ origin of processed data
'06/12/96'
/ FITS creation date (DD/MM/YY)
'ROSAT '
/ mission name
'PSPCB '
/ instrument name
'POINTING'
/ obs mode: POINTING,SLEW, OR SCAN
Extension HDU
XTENSION
= 'BINTABLE
'
BITPIX =
NAXIS =
NAXIS
1 =
NAXIS
2 =
PCOUNT =
GCOUNT =
TFIELDS=
TTYPE
1 = 'SRC_NR '
TFORM
1 = '1I
'
TUNIT
1 = 'NONE '
TTYPE
2 = 'LEV1_X '
TFORM
2 = '1E
'
TUNIT
2 = 'pixel '
IVOA Meeting – Nara, Japan 2010
/
8/
2/
32 /
56 /
0/
1/
9/
/
/
/
/
/
/
binary table extension
8-bit bytes
2-dimensional binary table
width of table in bytes
number of rows in table
size of special data area
one data group
(required keyword
)
number of fields in each row
DETECT source number
data format of the field
: 2-byte INTEGER
physical unit of field
Image X position
(15 arcsec pixels
)
data format of the field
: 4-byte REAL
physical unit of field
9
Dec xx, 2010
Mapping using SQL from
FITS headers to IVOA metadata
• Pivotting
– turn a table using unique values of one column to create multiple columns in
another.
Pivot Table SQL
TABLE_A
select
,
,
,
from
group
TABLE_B
a.ID as ID
MAX(Case When a.KEYWORD='INSTRUME' Then a.VALUE Else null End) as INSTRUME
MAX(Case When a.KEYWORD='RA_NOM' Then a.VALUE Else null End) as RA_NOM
MAX(Case When a.KEYWORD='DEC_NOM' Then a.VALUE Else null End) as DEC_NOM
TABLE_A a
by a.ID
IVOA Meeting – Nara, Japan 2010
10
Dec xx, 2010
Mapping using SQL from
FITS headers to ObsTap for ROSAT
ROSAT HRI
RCW103 Pointing
11
Dec xx, 2010
images credit: Max-Planck-Institut für extraterrestrische Physik (MPE)
IVOA Meeting – Nara, Japan 2010
Making region using SQL from
FITS headers to ObsTap for ROSAT
ROSAT HRI
RCW103 Pointing
IVOA Meeting – Nara, Japan 2010
images credit: MPE
12
Dec xx, 2010
Mapping using SQL: calculations and
custom metadata in tabular form.
SSA metadata for zCOSMOS
Insert into ivoa.SSA ( ... )
(select 'http://www.g-vo.org/zCOSMOS/’ + zf.accessURL as AccessRef
, zf.instrume as Instrument
, z.redshift as DerivedRedshift
, '580.0' as SpectralResolutionPower
, 1E-10*(zf.crval + (1 - zf.crpix)*zf.cdelt) as MinWavelength
, 1E-10*(zf.crval + (zf.naxis1 - zf.crpix)*zf.cdelt) as MaxWavelength
...
from zcosmos.ZCOSMOS_VIMOS_BRIGHT_DR2_TABLE z
,(select h.objectid as hduId
, f.objectid as fitsFileId
, MAX(f.accessURL) as accessURL
, MAX(CASE WHEN hc.keyword='INSTRUME' Then hc.value Else null End) as instrume
, MAX(CASE WHEN hc.keyword='NAXIS1' Then cast(hc.value as int) Else null End) as naxis1
, MAX(CASE WHEN hc.keyword='CDELT1' Then cast(hc.value as real) Else null End) as cdelt
, MAX(CASE WHEN hc.keyword='CRVAL1' Then cast(hc.value as real) Else null End) as crval
, MAX(CASE WHEN hc.keyword='CRPIX1' Then cast(hc.value as real) Else null End) as crpix
...
from fits.catalogue c, fits.fitsFile f
, fits.hdu h, fits.headercard hc
where c.shortName = 'ZCOSMOS_BRIGHT_DR2'
and c.objectId = f.catalogueId
and f.objectId = h.fitsFileId
schemas
and h.objectId = hc.hduId
- fits
group By h.objectId, f.objectId
- zcosmos
) zf
- ivoa
where zf.accessURL= z1.fileName)
IVOA Meeting – Nara, Japan 2010
13
Dec xx, 2010
Plugins - Compute spectral bounds for
CDFS X-Ray follow-up spectra
IVOA Meeting – Nara, Japan 2010
14
Dec xx, 2010
IVOA Meeting – Nara, Japan 2010
15
Dec xx, 2010
Conclusion
• Use standard technology: RDB and SQL
– No proprietary language
– Flexible query or mapping
• Extensible
– Easy to add other metadata not in FITS headers as tables in database
– Able to extract metadata stored in FITS data units or from more
complex calculations using Java plugins
• Easy search on multiple archives
– Not restricted to IVOA standards
• Future developments
–
–
–
–
–
Updating database by executing plug-ins after uploading FITS files.
Improving user interface
Supporting custom table upload in tool
Writing WCS-Region plugin
…
IVOA Meeting – Nara, Japan 2010
16
Dec xx, 2010
Acknowledgement
• Thanks to
– Tom McGlynn for nom.tam.fits and feedback.
– Pat Dowler et.al. for OpenCADC.
IVOA Meeting – Nara, Japan 2010
17
Dec xx, 2010