Building a User-Driven GEOSS:
Methods to Capture, Analyze, and Prioritize User Needs
GEO Task US-09-01a and
Earth Observations Identification
Lawrence Friedl, USA-NASA
User Interface Committee Member
US-09-01a Task Lead
ISRSE-34 Symposium • Sydney, Australia
10.April .2011
Earth Observations Needs
Overview of Presentation Elements
This presentation discusses a document-based approach to
identify observation needs, using teams of experts to:
a) review analytic methods to assess needs in documents, and
b) concur on priority-setting criteria to rank the needs.
Based on results of GEO Task US-09-01a: Identifying Critical Earth
Observation Priorities.
I.
Approach and Meta-analysis
II.
Results
III.
Findings
IV.
Advantages and Limitations
Discussion
2
Earth Observations Needs
Section I.
Approach and Meta-analysis
3
Earth Observations Needs
Task US-09-01a
GEO Task US-09-01a
Objective:
Establish and conduct a process for identifying critical
Earth observation priorities common to many GEO
societal benefit areas, involving scientific and technical
experts, taking account of socio-economic factors, and
building on the results of existing systems’ requirements
development processes.
GEO Societal Benefit Areas (SBA):
Agriculture
Disasters
Biodiversity
Ecosystems
Climate
Energy
Health
Water
Weather
4
Earth Observations Needs
Task US-09-01a
GEO Task US-09-01a
Objective:
Establish and conduct a process for identifying critical
Earth observation priorities common to many GEO
societal benefit areas, involving scientific and technical
experts, taking account of socio-economic factors, and
building on the results of existing systems’ requirements
development processes.
GEO Societal Benefit Areas (SBA):
Agriculture
Disasters
Biodiversity
Ecosystems
Climate
Energy
Health
Water
Weather
5
Earth Observations Needs
Task US-09-01a
US-09-01a Process: Nine Steps
The process lists the steps serially, yet some are done in parallel.
Step 1:
Step 2:
Step 3:
Step 4:
UIC Members identify Advisory Groups and Analysts for each SBA
Determine scope of topics for the current priority-setting activity
Identify existing documents regarding observation priorities for the SBA
Develop analytic methods and priority-setting criteria
Step 5:
Step 6:
Step 7:
Step 8:
Step 9:
Review and analyze documents for priority Earth observations needs
Combine the information & develop a preliminary report on the priorities
Gather feedback on the preliminary report
Perform any additional analysis
Complete the final report on Earth observations for the SBA
6
Earth Observations Needs
Task US-09-01a
General Approach
Document-based analysis. Examined
observation needs expressed in publiclyavailable documents from past ~10 years.
Analysis included over 1700 documents.
An “Advisory Group” and an “Analyst”
worked to identify documents, analyze
them, and prioritize observations within
each SBA. AGs involved 6-23 people from
developed & developing countries that
represent experts in an SBA. (Invited
CoPs, IGOS, others to be on AGs).
167 AG members & 43 countries in total.
Developed individual SBA reports, which
specified the observation priorities for that
SBA.
Based on the SBA reports, the
Task Team conducted a metaanalysis across the individual SBA
reports to determine observations
priorities common to many SBAs.
The team combined and prioritized
parameters from the SBA lists and
prepared an over-arching report to
identify “Earth observation
priorities common to many SBAs.”
The report includes findings,
lessons learned, and
recommendations.
7
Earth Observations Needs
Task US-09-01a
Cross-SBA Analytic Methodologies
Method 1: Tally of All Priorities
Frequency analysis is a simple tally of the SBAs that require a given observation.
(Total of 146 observations were included in this prioritization.)
Methods 2&3: Weighted Sums of All Priorities
Weighted frequency analysis is a weighted sum of the number of SBAs that
require a given observation, taking into account the high/medium/low importance
assigned by SBA Analysts. Different weighting schemes in the two methods
(Same 146 parameters as Method 1.)
Method 4: Top 15 Priorities by SBA
This key parameters method is based on each SBA Analyst preparing a list of
the “top 15” for that SBA. (Total of 99 observations were included.)
Final Set: Ensemble approach across the methods.
Calculated mean rank and the range of ranks for all 146 parameters across
methods. Natural breaks at top 19 and top 36 parameters.
8
Earth Observations Needs
Section II.
Results
9
Earth Observations Needs
Task US-09-01a
SBA-specific Observation Priorities
Each SBA identified a set of observation priorities as well as the 15 “mostcritical” observations. Overall, there were 146 observations identified as priority
needs from combining the SBA lists. The Task Team used an ensemble of 4
prioritization methods to integrate and prioritize for the Cross-SBA Analysis.
Agriculture SBA
Disasters SBA
Energy SBA
10
Earth Observations Needs
Task US-09-01a
8 observations
(5% of 146 total)
are common to 6
or more SBAs
29 observations
(20%) are
common to 4 or
more SBAs
100 observations
(68%) are
common to 2 or
more SBAs
11
GEO Task US-09-01a
Highest Ranked Observations
(#1-20) and Associated SBAs
All of the 20 Highest-ranked
observations are common to
4 or more SBAs
All of 30 Highest-ranked obs. are
common to 3 or more SBAs
3 Highest-Ranked:
Precipitation, Soil Moisture,
Surface Air Temperature
Note: Some observations may be relevant to an SBA
even though not included in the SBA’s set of priorities.
* Biodiversity SBA Team did not produce
a set of Earth observations priorities.
12
Earth Observations Needs
Section III.
Findings
13
Earth Observations Needs
Task US-09-01a
Task Report – Findings
•
•
•
•
•
•
•
•
Precipitation Reigns the Cross-SBA Analysis
Methods Showed Agreement at Highest-Rankings
Priorities of a Single SBA May Not Be on the Cross-SBA List
Task’s Approach Produced Users’ Needs in Users’ Terminology
Articulation of Observation Needs in Documents Varied
Regional Needs Incorporated but Not Featured
Availability of Documents by Region Varied
Insufficient Information across Documents on Parameter
Characteristics
14
Earth Observations Needs
Task US-09-01a
Task Report – Findings
• Priorities of a Single SBA May Not Be on the Cross-SBA List
This approach, by design, focused on the commonality of priority observations to many
SBAs. Thus, some observations of critical importance to a particular SBA do not appear
in the Cross-SBA list of priority observations.
• Task’s Approach Produced Users’ Needs in Users’ Terminology
Users didn’t always use terms that the Earth obs. community may be familiar with.
Many of the needed observation parameters were expressed as phenomena of interest
rather than technical specifications of the parameter. Overall, the demand-side, userbased approach of the task produced a rich array of observations and revealed a need
for follow-on user engagement to refine parameter characteristics.
• Insufficient Information across Documents on Parameter Characteristics
Some of the documents included quantitative information on the required characteristics
(e.g., accuracy, latency) of critical observation parameters, while other documents
lacked such information. At times it was available. However, the required observation
parameter characteristics vary widely according to the user and application.
15
Earth Observations Needs
Task US-09-01a
Task Report – Recommendations
Follow-on
Activities
Process
Improvements
• Gather information and engage users on specific characteristics of the
priority Earth observations, especially Precipitation.
• Conduct an assessment of the current and planned availability of the
priority Earth observations.
• GEO and/or Regional Caucuses could consider pursuing similar
assessments at regional levels.
• Consider additional analytic methods to gathering users’ needs and
pursue an ensemble of approaches.
• Prescribe the prioritization methods, SBA sub-areas, and other aspects
of the SBA analyses.
• Pursue broader incorporation of documents in many languages.
• Continue the use of ad hoc Advisory Groups, with refinements.
• Strongly consider a single organization to manage the individual SBA
analyses.
• Articulate an SBA’s community of users to support systematic collection
of users’ needs.
16
Earth Observations Needs
Task US-09-01a
Task Report – Recommendations
• GEO and/or Regional Caucuses could consider pursuing similar
assessments at regional levels.
User needs vary by geographic region, and user needs unique to a single region were
not likely to appear among highly-ranked observations of the Cross-SBA analysis
• Pursue broader incorporation of documents in many languages.
Documents describing user needs, especially regional and national needs, exist in
languages other than English, yet such documents were not discovered or were
underrepresented. The Task Team recommends that future endeavors plan and provide
sufficient resources for the identification of documents in many languages and for
necessary translations
• Articulate an SBA’s community of users to support collection of users’
needs.
Users and end-users are broad terms. Future Task Teams should develop or refine a set
of User Types for each SBA. The Task Teams should ensure that the Analysts employ the
User Types as guidance in collecting information and representing needs, utilizing them
to conduct a gap analysis of their activities to address any bias or gaps.
17
Earth Observations Needs
Section IV.
Advantages and Limitations
18
Earth Observations Needs
Task US-09-01a
Task Lead’s Comments
Task’s efforts and results represent significant contributions and first
steps within GEO to articulate Earth observation priorities. GEO has
documented in a transparent way how Earth observation needs have
been identified, involving numerous organizations and experts.
The value of the Task’s results and cross-SBA report is at least
twofold:
• Provide a baseline and entry point for further engagement with
end users on their needs.
• Confirm any expected priority observations as backed up by an
analysis of the literature.
19
Earth Observations Needs
Task US-09-01a
Lessons Learned
• Task Approach Achieved Desired Diversity in Prioritization
Methods
• Variety in Analysts’ Approaches Introduced Complexities
• Approach to Sponsorship of Analysts Impacted Process
• Approach to Selection of SBA Sub-Areas Introduced Challenges
• Advisory Groups Played Valuable Yet Variable Roles
Use of existing documents provides a level of objectivity
The ranking of an observation in the Cross-SBA list does not imply
objective importance of that observation as much as commonality
in need.
20
Earth Observations Needs
Task US-09-01a
A Document-Based Approach
Process Comments
• Analytic methods to extract observations and needs
- Extraction and inference
• Prioritization methods and priority-setting criteria
- Opportunity to tailor to SBA-specific issues (e.g., Daily Adjusted
Life Years, World Energy Outlook)
- For individual analyses, need to be internally consistent
- For meta-analyses, need commonality and comparability across
types (e.g., ordered list, tiers, grouped by some factor)
Note: The US-09-01a approach, by design and broadly-stated, was to be a
natural experiment involving a variety of methods; methods were tied to the
respective SBA communities as represented by the AGs.
21
Earth Observations Needs
Task US-09-01a
A Document-Based Approach
Assumptions
• Documents were produced in an unbiased way
• Documents express needs for existing and new
observations
22
Earth Observations Needs
Task US-09-01a
A Document-Based Approach
Advantages
• Independent and objective (based on what
organizations have already stated publicly)
• Reflects collective needs of organizations across
multiple people
• Shows documented, expressed needs
• Based on a ‘democratic’ principle in which all were
eligible to be considered
• Generate observation needs in the terms and lexicon
of the respective user communities
23
Earth Observations Needs
Task US-09-01a
A Document-Based Approach
Limitations
• Difficult to capture very-recent needs (the time aspect of
the approach features long-term, sustained expressions
of need);
• Advisory Group member engagement and potential bias
• Need for representative set of documents for all regions
and SBAs
• Level of specificity and technical information on needs
may vary across documents; inference of needs
introduces possible error
24
Earth Observations Needs
Task US-09-01a
A Document-Based Approach
p. 51
25
Earth Observations Needs
Task US-09-01a
Recommendations
• Document-based analysis provides an objective basis to an identification of
Earth observation needs
– should be part of a ensemble of analytic & prioritization methods
– should be employed to identify users’ terminology for their needs
• A process involving a document-based analysis across themes should
prescribe the analytic/prioritization method(s) and deliverables as well as
allow for SBA-specific variants
• Allocate significant time, resources, etc. to include range of documents
across regions, languages
• Articulate what a ‘representative’ sample comprises
– should use and internal gap analysis to ensure appropriate breadth
• Allocate sufficient time for review of preliminary results by people outside of
advisory groups; this review may stimulate the identification of a host of
new documents)
26
Earth Observations Needs
Task US-09-01a
Contacts & Acknowledgements
Lawrence Friedl, US-09-01a Task Lead
Amy Jo SWANSON, Task Coordinator
Erica ZELL, Lead Analyst
Adam CARPENTER, Asst. Lead Analyst
Special recognition to the Analysts and
Advisory Group members for the
respective SBAs.
Final Cross-SBA Report and Individual SBA Reports
are available at the US-09-01a Task Website:
http://sbageotask.larc.nasa.gov/
27
Earth Observations Needs
Discussion
28
Earth Observations Needs
Back-up Materials
29
Earth Observations Needs
Task US-09-01a
Advisory Groups for SBAs
• Advisory Groups ranged from 6-23 members.
• Sought broad international and regional representatives.
• Involved GEO CoPs, former IGOS Themes, GEO Countries
and GEO Participating Organizations.
30
Earth Observations Needs
Task US-09-01a
This chart
presents the
30 highestranked Earth
observations,
shown
according to
score in the
Cross-SBA
analysis. The
range in ranks
is also shown.
Ranks are
‘inverted’ so
highest score
is 146.
31
Earth Observations Needs
Task US-09-01a
General structure:
An “Advisory Group” and an “Analyst” work together to identify
documents, analyze them, and prioritize observations within each SBA.
Advisory Group (1 per SBA)
Functions:
- Will help to identify documents
- Comment on analytic methods
and priority-setting criteria
- Review the analysts’ findings,
priorities, and reports.
Analyst (1 per SBA)
Functions:
- Will read and analyze the documents
- Develop an analytic method and
priority-setting criteria
- Conduct the meta-analysis to identify
common priorities within a SBA.
Involved 6-23 people from developed
and developing countries that
represent experts in an SBA.
The Analyst was the primary coordinator
and organizer of the activity to meet the
schedules and deadlines.
Involved Communities of Practice,
former IGOS Themes, GEO Countries
and Participating Organizations.
Interacted with and utilized the Advisory
Group to vet prioritization methods and
review results and reports.
32
Earth Observations Needs
Task US-09-01a
Advisory Groups
33
Earth Observations Needs
Task US-09-01a
A Document-Based Approach
34
Earth Observations Needs
Task US-09-01a
Final Report
Published October 2010.
Delivered to GEO Plenary VII as
annex to UIC Report and
distributed at Plenary.
Final Cross-SBA Report and
Individual SBA Reports are
available at the
US-09-01a Task Website:
http://sbageotask.larc.nasa.gov/
35