Extending OPeNDAP to
Offer Remapping Services
EGU General Assembly 2013
Session ESSI2.5 —
Service Architecture challenges for
multi-disciplinary systems and Future Internet
Wednesday, 10-April, at 14:00
Dave Fulker, President, OPeNDAP, Inc.
James Gallagher,Vice President, OPeNDAP, Inc.
OPeNDAP Is
✦ A data-access method/protocol (DAP)
•
•
Architecturally service oriented
Realized/deployed in multiple languages
✴ Client libraries embedded in NetCDF, MATLAB...
✴ Servers that run under Apache...
•
A NASA “community standard”
✦ A not-for-profit corporation
•
Headquartered in Rhode Island, USA
2
Current Paradigm
✦ DAP-compatible servers offer clients easy
access to (remote) data, with side benefits:
•
•
Greater interoperability
✴ In essence, middleware, with a powerful data model
Fewer needs for (wholesale) data transfers
✦ But, actions are limited mainly to subsetting
•
•
No summarization or other statistics
No remapping or other transforms
3
E.g., DAP(2) Offers
Index-Based
Subsetting
Like NetCDF—but as a
Web service—users may
4
• Skip indices
• Limit index ranges
• Lower dimensionality
OPeNDAP’s Data-Access Protocol
DAP2 (1993) & DAP4 (July)
✦ A request is a URL (with constraints)
that yields a response, which may be
•
•
Metadata only (a textual description)
•
Used directly or via NetCDF library
Variables, organized in groups, structures, &
arrays of (binary) integers, reals...
✦ Requests w constraints produce subsets
•
But, DAP4 sets the stage for “queries”
to invoke other functions
5
Proposition
✦ Future data-access services should offer
highly flexible, data-proximate operations:
•
•
•
Deliver subsets defined in complex ways
Enable actions on derived dataset values
Simplify complex uses (multi-source integration,
e.g.) via user-specified remapping operations
✦ Such services must be useful for both
•
•
End-users (typically within client apps)
Brokering services (in EarthCube, e.g.)
6
DAP4 Sets the Stage
✦ DAP2 internally supports limited functions
& function composition, but...
✦ DAP4 treats functions and/or a functional
language as extensions, w allowances for
•
•
Calculated metadata responses, e.g., units
Use of POST to submit large or complex
requests, e.g., convolution fields
✦ Thus: DAP4 is extensible enough to
allow domain-specific languages, e.g.
7
Relevant OPeNDAP Successes* with
Extended Server Functions
✦ Multi-instrument inter-calibration at NASA
✦ Cloning the function syntax from the
GrADS Data Server (GDS)
•
Enables use by netCDF-based DAP clients
(at ECMWF, e.g.) without code changes
✦ UGRID (irregular mesh) subsetting
*
All experimental, these have been funded
by NASA & NOAA
8
Irregular-Grid
Subsetting
• Subsets are defined by
polygonal regions
• Subsetting yields another
(smaller) irregular grid
• Subsets retain geometric/
topological relationships
(i.e., no regridding!)
From Bill Howe, U WA
9
A Few Remapping* Ideas
*broadly defined
✦ Binning: return a space-time distribution
(as counts on a user-spec’d raster) of values
satisfying some (complex) criterion
✦ Masking: accept queries with rasters as
space-time constraints
✦ Perhaps a (limited?) functional language for
user-defined capabilities
✦ Provision of OGC’s WPS, etc.
10
These Ideas Will Require
✦ Reference to higher-level data models
•
Semantics of function approximation over
partitioned (space-time) domains
✴ Sampled at discrete points
✴ Piecewise continuous (step functions or linear, e.g.)
✴ Linear combinations of basis functions
✴ Conservation principles (mass, energy, etc.)
✦ Changes in the data access paradigm
•
Remapping takes time and resources
✴ Authorization, etc.
✴ Results caching, etc.
11
Consider Studies that Involve
Multi-Source Data
✦ Pertinent data often are
•
Very remote—residing at multiple locations,
far from the investigator
•
Very large—moving them (whole) is difficult
✦ Users (perhaps via brokers) would benefit
from data-access services offering
•
Remapping operations (broadly defined),
performed close to the data
12
Questions/Comments?
(on My Proposition, Reprised:)
✦ Future data-access services should offer
highly flexible, data-proximate operations:
•
•
•
Deliver subsets defined in complex ways
Enable actions on derived dataset values
Simplify complex uses (multi-source integration,
e.g.) via user-specified remapping operations
✦ Such services must be useful for both
•
•
End-users (typically within client apps)
Brokering services (in EarthCube, e.g.)
13