Venus Exploration Analysis Group (VEXAG) Replies to NASA Headquarters’ Request for Information for Lessons Learned from Recent Planetary Science Division Announcements of Opportunity for Discovery, New Frontiers and JUICE 1) Allocation of additional pages for each Flight Element 2) Reference to AG Goals as a complement to the Decadal Report 3) Level of detail required 4) New Technology Demonstration VEXAG Competed Missions and ROSES Topical Analysis Group Prepared by Gordon Chin (NASA GSFC) Submitted by Lori Glaze (NASA GSFC) October 2013 L. Glaze VEXAG Comments
Title: VEXAG 1. Allocation of additional pages for each Flight Element
Name of submitter and contact information (institutional affiliation, email address)
Dr. Lori S. Glaze, on behalf of the Venus Exploration Analysis Group (VEXAG)
NASA Goddard Space Flight Center
[email protected]
Statement of the issue addressed
Additional proposal pages are needed to adequately explain the flight hardware proposed
for multi-element missions.
Significance to a wide range of proposers
Many proposed Discovery missions in recent years - including in-situ missions to Venus,
Mars, and Titan - have been comprised of several flight elements. It has proven quite
difficult to adequately describe each element in the given page allocation, given the
multiplicity of systems and functions of each flight element. Such flight elements include
carrier spacecraft, orbiter spacecraft, entry vehicle, probe, lander, rover, airplane, and
balloon/gondola. We note that with 25 pages typically allocated to Technical Description,
Management, and Cost (TMC), covering diverse topics such as management, cost, and
risk issues, the needed descriptions of the flight hardware for multi-element missions –
including all necessary mass/power tables, systems descriptions, and block diagrams – is
problematical. Consequently, in the competitive AO environment, such multi-element
missions are placed at a disadvantage since the level of detail for each element
necessarily has to be less than for single-element missions due to the strict, one-size-fitsall, page count. A multi-element mission can receive many TMC weaknesses because of
the reduced level of detail that was allowed by page count.
Impact of not resolving the issue
Proposers of multi-element missions will continue to be placed at a disadvantage in their
ability to adequately present their mission hardware in the limited page count of the AO,
potentially resulting in non-selection of the best Discovery proposals for Step 2.
Suggested pathway(s) to resolve the issue
As was done in the last Discovery AO, where it was recognized that each instrument
should have a page allocation sufficient to adequately describe the instrument, we suggest
that in the next AO an allocation of pages be given to each proposed flight element to
1
L. Glaze VEXAG Comments
allow each element to be adequately discussed. A viable working number appears to be 4
pages per flight element.
Impacts of the suggested solution(s)
The additional pages sought will allow proposers to provide the depth of information
sought by review panels, thus significantly improving the review process and the
likelihood that truly exceptional missions that meet or exceed Step-1 TMC criteria will be
selected.
2
L. Glaze VEXAG Comments
Title: VEXAG 2. Reference to AG Goals as a complement to the Decadal Report
Name of submitter and contact information (institutional affiliation, email address)
Dr. Lori S. Glaze, on behalf of the Venus Exploration Analysis Group (VEXAG)
NASA Goddard Space Flight Center
[email protected]
Statement of the issue addressed
To aid reviewers in their assessment of competed mission proposals, an AO usually refers
to compliance with the most recent Decadal Survey as the baseline for traceability of
proposed mission scientific objectives. However, the Decadal Survey typically covers a
very broad array of objectives that encompass a wide range of investigations, with little
information on the priorities among and details of such investigations. Moreover, these
investigations span all degrees of difficulty, from those that could reasonably be
accomplished with small missions to those that could only succeed via flagship. Thus,
given the lack of detail on goals, objectives, and investigations provided in the Decadal
Survey, reviewers of Discovery proposals not intimately involved with the current state
of exploration for a target under review may be left confused. A possible outcome is that
missions may be chosen that poorly satisfy the priorities of the relevant
Assessment/Analysis Groups, as espoused in their Goals, Objectives, and Investigations
document.
Significance to a wide range of proposers
Finding a means to provide reviewers with the latest “state of the field” against which to
assess the science merit of proposals helps assure the best use of NASA funds for
planetary exploration.
Impact of not resolving the issue
Evaluating proposals without using the Goals/Objectives/Investigations documents
generated by the Assessment/Analysis (“AGs”) for individual scientific communities
may not lead to selection of the scientifically best missions.
Suggested pathway(s) to resolve the issue
When available, it is recommended that the published Goals, Objectives and
Investigations prepared by the various NASA Assessment/Analysis Groups be taken into
consideration in the evaluation process of AO responses as a complement to the Decadal
3
L. Glaze VEXAG Comments
Survey. These Assessment/Analysis Group reports may also be used to clarify the
relative priorities within a set of competing proposals that is aimed at the same target of
investigation.
Impacts of the suggested solution(s)
The published Goals, Objectives and Investigations formulated by the NASA community
AGs have undergone extensive review and vetting within the individual AGs and have
also been promulgated and reviewed within the broader science community. These
objectives are generally aligned with the Decadal Survey, but are typically much more
detailed, thus helping to clarify the fields of interest that may be inherent in a complex
target of study. The AG recommendations will greatly aid reviewers in their assessment
of the relative importance of proposals that may be submitted for the same target of
investigation, providing increased fidelity in the selection process.
4
L. Glaze VEXAG Comments
Title: VEXAG 3. Level of detail required
Name of submitter and contact information (institutional affiliation, email address)
Dr. Lori S. Glaze, on behalf of the Venus Exploration Analysis Group (VEXAG)
NASA Goddard Space Flight Center
[email protected]
Statement of the issue addressed
Step-1 proposals are sometimes given major weaknesses for perceived inadequate
systems descriptions that pertain to details that were not required to be addressed by the
Step-1 AO and that normally are addressed during Step 2 or later. The NASA review
process should monitor reviewers to ensure that such issues do not inappropriately skew
proposal evaluations and categorizations. This is particularly important when specialized
technical reviewers who are not familiar with the NASA AO proposal review process are
brought in to address a specialized system (e.g., aircraft, balloon, Thermal Protection
Systems).
Significance to a wide range of proposers
All Step-1 proposal evaluations would benefit from limiting reviewer
expectations for submitted technical details. Currently any proposal might be subject to
reviewers expecting details in a Step 1 proposal that would normally be provided during
Phase A or later.
Impact of not resolving the issue
Proposers deserve an even playing field in the review process for their proposals
as established by the Step-1 AO. If this issue is not resolved, and review panels seek
detail beyond what is required by the Step-1 AO, proposals involving specialized systems
in particular are liable to continue to be unfairly judged by specialized technical
reviewers who are less familiar with the proposed technology or who do not realize the
limits of technical maturity that is required for Step 1 proposals.
Suggested pathway(s) to resolve the issue
Reviewers should be advised of the level of detail required for a Step 1 proposal
in the Step-1 AO. The submitted technical material for a proposal should then be
evaluated only to the extent defined in the Step-1 AO. As Panel Chairs and Program
Scientists monitor the review process as it transpires, they should pay particular attention
5
L. Glaze VEXAG Comments
to ensure that unfair criticisms beyond the expected bounds of Step-1 evaluation do not
evolve into Major Weaknesses.
Impacts of the suggested solution(s)
The careful monitoring of specialized reviewers in the proposal review process will
significantly improve the likelihood that truly exceptional missions that meet or exceed
Step-1 TMC criteria will be selected.
6
L. Glaze VEXAG Comments
Title: VEXAG 4. New Technology Demonstration
Name of submitter and contact information (institutio nal affiliation, e mail address)
Dr. Lori S. Glaze, on behalf of the Venus Exploration Analysis Group (VEXAG)
NASA Goddard Space Flight Center
[email protected]
Statement of the issue addressed
The recent Discovery AO offered the possibility for new technology, such as the
Advanced Sterling Radioisotope Generator (ASRG), for flight demonstration. As a result
two proposals were selected for Step 2 studies that proposed using ASRGs. In the end,
the ASRG may have been offered prematurely. However, the end result was that other
proposals that were more technologically robust and had lower risk ratings were
overlooked in favor of using these proposals that used a more ambitious technological
approach that NASA was looking to demonstrate. It appears therefore that this element of
the AO did not offer a level-playing field between proposals or targets of investigation
that did not require new technology to carry out their missions and high science, but also
high risk, technology demonstration missions.
Significance to a wide range of proposers
The NASA desire to implement new technology should not drive the selection of
missions. Missions have been crafted in a careful fashion to balance science objectives,
technological risks, and minimization of cost. Implementation of new technology, if
required, should be only one element of this complex selection process, not a dominant
one.
Impact of not resolving the issue
Teams that respond to a Discovery AO devote a great deal of effort to mission
formulation. If proposals that offer implementation of new technology are given an
advantage over those that offer lower risk through using established technology, the end
result may be to overlook equally, or even more valuable, science and targets of
investigation that do not require use of new technology and crowd out many viable
mission concepts.
Suggested pathway(s) to resolve the issue
This recommendation is that the proposed use of new technology in a mission does not
7
L. Glaze VEXAG Comments
offer an unfair selection advantage to those proposals that do not use new technology.
Major technical risks due to the proposed use of new technology should be treated as
other major technical risks in the categorization process. If NASA desires a flight
demonstration of new technology, then it should do so in a program that is committed to
it solely, not in a competition with other missions that may not need it.
Impacts of the suggested solution(s)
This solution encourages NASA to demonstrate new technology in an environment that
allows higher risk and has a clear objective of showing that the technology can work. By
including this demonstration in a competed environment, the objectives of science and
technology demonstration are so intertwined that the selection criteria is skewed to favor
a much more risky approach, thereby crowding out missions that may be more robust and
carry lower risk, but offer equally compelling science.
8