PDS Imaging Node
PDS4 Migration
Lisa Gaddis (USGS)
Sue LaVoie (JPL)
November 30, 2012
PDS Management Council Meeting
UCLA
PDSMC November 29-30, 2012
1
PDS4 Migration Strategy
• Migrate only past missions & newly awarded missions (InSight)
– Justification: for active missions, community analysis is ongoing with existing
software & tools
– Prioritize based on need for class development/follow-on missions and popularity
– Retain PDS3 labels & metadata within PDS4 structure
• Migrate mission datasets requiring no (or simple) data manipulation
– Retain copy of any data requiring manipulation
• Migrate metadata only of mission datasets requiring data manipulation
– Justification: complex data migration is too risky
– Retain PDS3 data format & label
– Create PDS4 labels and metadata to enable complex data product searches
across all holdings
– Decisions/Action required by MC & Data Modelers
• Policy allowing waiving of data migration for complex legacy data sets
• Dictionary terms/keywords enabling description of legacy formats
• Modify IMG systems to support both PDS3 & PDS4 data sets & formats
PDSMC November 29-30, 2012
2
PDS4 Migration Approach
• Process
–
–
–
–
–
–
–
–
–
–
–
–
Identify data set for migration
Create/update Discipline Classes, provide input to PDS-wide Classes
Map PDS3 to PDS4 labels and ancillary files/data
Create schema
Modify existing software tools & scripts to automatically create labels, collection
and bundles
Generate Sample Bundle(s)
Verify
Iterate
Informal IMG internal review & testing
Load Bundle into IMG Registry
Identify next data set
…..
– Repeat iterative process until Data Model and Classes are “final”, refining all along
the way. Then perform the full data set migration on priority data sets
PDSMC November 29-30, 2012
3
PDS4 Migration - Top Priority Data Sets
PDSMC November 29-30, 2012
4
PDS4 Migration – Waive Data Format Change
PDSMC November 29-30, 2012
5
Migration Progress
• Clementine 750nm Basemap (Chris)
– Sample Bundle validated, reviewed by EN, updated and redelivered for ingestion
• Folders in bundle: data (6 map projected single band images & ISIS detached labels),
document (.txt, .doc, .htm, .pdf versions of “volinfo” suite), miscellaneous (delivery &
transfer manifests), xml_schema
– Cartography Class - Incorporating Federal Geographic Data Committee (FDGC)
model. Still needs fine tuning
– Tools used: oXygen XML Editor, IMG Generate Tool, EN Validate Tool, ad hoc
scripts
• Mars Pathfinder (MPF) IMP EDR (Elizabeth)
– Sample Bundle validated and delivered to EN for review and feedback
• Folders in bundle: data (200 files), browse (browse for each data file), context (instrument,
instrumet host, investigation, resources, etc.), document (EDR SIS), geometry (gazetteer),
xml_schema
– Imaging Instrument Parameter Class – in good shape for this data set &
Clementine data set
– Tools used: oXygen XML Editor, IMG Generate Tool, EN Validate Tool, ad hoc
scripts
PDSMC November 29-30, 2012
6
Migration – Next Up
• Clementine UVVIS 5-band Mosaic (Chris)
– Multispectral data set
– Preliminary Bundle layout defined
• Mars Pathfinder (MPF) RVRCAM (Elizabeth)
– Tables and Engineering data sets
• Phoenix OM, RAC, SSI EDR (Amy)
– Precursor for InSight (launch 2016; Geo is lead node)
• Landed mission products and metadata
• Phoenix, MER, MSL heritage
• Work closely with Geo throughout
– Class work: Camera Model, Landed Geometry, Mars InSitu, likely more
PDSMC November 29-30, 2012
7
System & Tools Status
• Accomplishments
– Installed all PDS2010 software on IMG operational machine, including Registry,
Registry UI, Harvest, and Validate
• Used Validate tool on test bundles prior to delivery to EN for review
– Continued development of IMG Generate Tool (label generation) to support
ongoing data migration activities
– Created internal IMG Wiki to track PDS4 migration details, progress, lessons
learned, open items, versions of dictionaries and schemas used, etc.
• Plans
– Harvest test & completed Bundles into IMG Registry
– Work with Engineering Node to communicate with central Registry
PDSMC November 29-30, 2012
8
Back-up
PDSMC November 29-30, 2012
9
Imaging Node: PDS4 Data Migration
# Data Sets & Class Work
19
• IMG manages a lot of
complex data with a
wide variety of formats
–
Discipline Class Work
No Discipline Class Work
–
–
Data Sets Requiring Class Work
1
12
Compression
Instrument Parameters
21
29
Surface Geometry
Camera Model
Cartography
Mosaic
14
11
Calibra on
Spectral Qube
15
52
203 data sets, ~13 M
products
Many of these are not
PDS4-compliant
Data Manipula on
No Data Manipula on
151
• 90% of IMG data sets
require
184
9
# Data Sets & Data Manipula on
15
Anaglyph
–
Assessment & revision
of classes of keywords
• Discipline Classes
• PDS-wide Classes
Manipulation of data to
achieve compliance
• 25% of IMG data
sets
• But, we assume that
migration of all data
sets is not required
Data Sets for Each Data
Manipula on Issue Type
1
9
simple compression
9
complex compression
prefix& suf fi x
interleaved
3
3
spectral qube
undefin
e
d
10
PDSMC November 29-30, 2012
10
IMG Data Migration: Data Manipulation
• Data Manipulation
Data Sets for Each Data
Manipula on Issue Type
1
9
simple compression
9
complex compression
prefix& suf fi x
interleaved
3
3
spectral qube
undefin
e
d
10
– 25% of IMG data sets
• 52 of 203 data sets
– Potentially very time consuming and
costly
– Process:
•
•
•
•
•
Engage experts
Software development & testing
V&V
Peer Review?
Release
– Issues:
• Retain PDS3 version – forever?
PDSMC November 29-30, 2012
11
IMG Data Migration: Classes
• Class development will require
Data Sets Requiring Class Work
9
1
12
Compression
Instrument Parameters
21
29
Surface Geometry
Camera Model
Cartography
–
–
• Classes needed for IMG - prelim analysis
(not all included on chart)
– Calibration – PDS-wide
– Compression – PDS-wide
– Geometry
• Camera Model – IMG
• Surface Geometry – PDS-wide
• SPICE Geometry – PDS-wide
Mosaic
14
11
Calibra on
Spectral Qube
15
15
Anaglyph
IMG for Discipline-specific classes
Cross-node participation for many others
–
–
–
–
–
–
–
–
–
Cartography – IMG
Imaging Instrument Parameter – IMG
Telemetry Parameters – IMG
Command Parameters – IMG
Processing History – IMG
Imaging Instrument Compression – IMG
Mosaic – IMG
Anaglyph (?) - IMG
Spectral Qube (?) - IMG
PDSMC November 29-30, 2012
12
Past
Missions
PDSMC November 29-30, 2012
13
Active
Missions
PDSMC November 29-30, 2012
14