Accreditation
Authorities
Danny Thomas
(256) 922-0802
[email protected]
What is M&S Accreditation?
•• Problem
Problem Definition
Definition
•• Problem
Problem Parameters
Parameters
•• Problem
Problem Criticality
Criticality
•• M&S
M&S Data
Data Needed
Needed
•• M&S
M&S Credibility
Credibility Needed
Needed
The provider is:
Responsible
for
releasing,
certifying, assessing, verifying,
validating,
the
model
or
simulation and/or quantifying the
uncertainty of the data given.
The decision maker is:
Responsible for approving or
accrediting the simulation that is
used. Or more simply put, just
believing the data provided.
Requirements
The official endorsement that a
model or simulation or federation
of models and simulations and
its associated data is acceptable
for use for a specific purpose.
Problem Needing
M&S Use
Top-Down
M&S CREDIBILITY
Thresholds Required
ACCREDITATION
CRITERIA
V&V
How Good
Is It?
Bottom-Up
Bottom-Up
M&S
M&S Criteria
Criteria
Assessment
Assessment Data
Data
Assessment
Accreditation is:
Is It Good
Enough?
&A
Collected
Collected and
and Quantified
Quantified
Criterion
Criterion Evidence
Evidence from
from
M&S
M&S Artifacts
Artifacts
M&S Artifacts
•• M&S
M&S Reqts
Reqts
•• M&S
M&S Designs
Designs
•• M&S
M&S Code
Code
•• M&S
M&S Data
Data
•• M&S
M&S Use
Use
Accreditation
Accreditation Is
Is The
The End
End User’s
User’s Determination
Determination An
An M&S
M&S Is
Is Acceptable
Acceptable For
For Use
Use
Accreditation Authorites
Page 2
What is M&S Accreditation?
Accreditation is:
The formal approval for use for a specific purpose of an M&S,
given that:
• The M&S and its associated data have been objectively and
quantitatively evaluated by a cognizant experts and
• Evidence exists supporting the ability of the M&S to produce real-worldrepresentative data within the operational ranges of, and within the
accuracy required by, the specific purpose.
There is an implication that someone or some group has the
authority to approve use of model or simulation. That entity is the
“Accreditation Authority”
Accreditation
Accreditation Is
Is The
The End
End User’s
User’s Determination
Determination An
An M&S
M&S Is
Is Acceptable
Acceptable For
For Use
Use
Accreditation Authorites
Page 3
Definitions of VV&A Roles & Responsibilities
VV&A Roles
M&S
Results
Users
A management agent or board responsible for specifying the acceptance criteria and has authority to
accredit an M&S application for intended use in accordance to prescribed acceptance criteria.
How Good Does
it Need to Be?
Subject Matter
Expert (SME)
A recognized NASA leading practitioner in a given field of study and or discipline appropriately related
to the intended use of the M&S, will provide peer review.
Accreditation
Agent
The individual, group, or organization that will define accreditation plan and conduct of accreditation
assessment based on V&V of M&S application to ensure evidence of credibility is compliant with
specified accreditation criteria.
User / Analyst
The individual, group, or organization that will define the requirements for, select and execute the
M&S, and use the data generated from the M&S, to conduct specific analysis. User will provide
assistance with specification of acceptability criteria.
USER’s
VIEW
M&S
Users
Accreditation
Authority
Role Descriptions
Developer
The individual, group, or organization responsible for developing the model and or simulation.
The developer will conduct V&V activities to comply with specified accreditation and V&V plans.
V&V Agent
The individual, group, or organization will define, plan and conduct verification and validation activities
to generated and document the evidence of M&S credibility in compliance with acceptability criteria.
M&S
Builders
VV&A
Services
Accreditation Authorites
VV&A
Broker
An assigned VV&A professional practitioner responsible for the coordination and oversight of
prescribed accreditation process and VV&A activities for each of the CxP M&S Accreditation
Candidates.
VV&A
Lead
Responsible for management oversight of VV&A efforts and acting initiator of accreditation process for
all designated M&S Accreditation Candidates. The lead assigns brokers to M&S Accreditation
Candidate, monitor and report status of all VV&A activities.
DEVELOPER’s
VIEW
M&S
How Good
Is it ?
Page 4
M&S VV&A Accreditation Authority
Accreditation
Authority
Organizational Authority
‰ Reports to program/project management
‰ Must have the authority to prescribe & approve specific M&S use
‰ Must have the authority to commit resources for specific M&S VV&A
Qualifications
‰ Must have the technical cognizance to assess the proper use of specific M&S applications
‰ Must be able to weigh cost, schedule, and risk against alternatives
M&S VV&A Responsibilities
‰ Specifies (and documents) the programmatic priorities for specific M&S use
- based upon technical need, schedule, and any associated risk factors
‰ Approves resource expenditures and schedules for specific M&S VV&A
‰ Specifies (and documents) the technical requirements for specific M&S use
- specifies the specific minimum levels reqd for each VV&A accreditation criterion
‰ Reviews and Approves Accreditation Plans for specific M&S use
‰ Makes Go/No-Go decisions for each of Three VV&A Phases
‰ Makes accreditation decision based upon results of VV&A assessment results, and other inputs
- Approves and signs accreditation memos and memorandum of records for M&S
‰ Reports to project management on any special conditions of specific M&S use
- caveats, limitations, assumptions, constraints of specific M&S
Accreditation Authorites
Page 5
Accreditation Authority Examples
♦ Meteorite Engineering Model (MEM) Example
• Environmental M&S
• Wide Applicability
• Authority is a uniquely qualified and recognized SME
♦ SpaceNet Example
• Industry/Academic Developer
• Multiple discipline SME panel
• Authority is analysis lead
♦ Campaign Manifest Analysis Tool
• NASA Developer
• Special purpose application
• Authority is sponsor
♦ Fully Unstructured Navier-stokes 3-dimensional (FUN3D)
• Aero CDF Code
• Development community distinct from Users Groups
• Authority receiving panel / supplying panel?
♦ Ares Critical Math Model (CMM) process
• NASA and contractor developers
• Accreditation Authority is MSFC Chief Engineer
Accreditation Authorites
Page 6
Define Accreditation Criteria
♦ Related to credibility but not necessarily
synonymous
•Includes programmatic considerations – like use cost
•Addresses suitability
•Considers actual conditions of use
♦ Based on DoD, DoE and Industry standards
♦ Selected from the an encyclopedic list based on the
objectives of the Accreditation Authority and the
circumstances of the simulation effort
Accreditation Authorites
Page 7
The Three Dimensions
Rigorous
4
Production
3
Developmental
2
Research
1
Top-Level
Credibility
Assessment
Credibility Criteria
2
3
4 1
Accreditation Authorites
2
DOMAIN A
1
4
3
3
2
2
DOMAIN B
1
DOMAIN C
1
Sub criteria C4
Sub criteria C5
Sub criteria C6
1
Sub criteria C3
0
Sub criteria C1
0
Sub criteria C2
3
Sub criteria B5
0
Sub criteria B6
1
Sub criteria B3
0
Sub criteria B4
0
Sub criteria B2
3
Sub criteria B1
1
Sub criteria A3
2
Sub criteria A1
3
Credibility Levels
)
Sub Criteria Credibility
R
3
Used Right ? (UR)
4
1
0
0
0
0
0
0
Gaps
Meets or exceeds requirement
Fails requirement by one unit
Credibility requirement-level assignments by accreditation authority, to accredit a specific M&S for a specific use
Measured value of criteria achievement for specified application
Fails requirement by two or more units
72 Lower Level
Accreditation Criteria
U
Built Well? (BW)
(U
?
3
se
d
1
2
4
ig
ht
1
4
Built Well ? (BW)
4
0
2
Match
Intended
Use ?
(IU)
Sub criteria A2
3
Credibility Levels
Domain Credibility
4
R
Match Intended Use? (IU)
Multi-Dimensional M&S Credibility Space
Page 8
Fundamental Touch Points for
M&S Accreditation vs. M&S Results-Credibility
Analysis xyz
data flow
defined?
OUT
IN
Data
user
data
Data
provider
?
?
Analysis for xyz
data
data
OUT
IN
Data
user
Data
provider
? ?
?
“USED
RIGHT ?”
data
?
Before Use
Before Use
M&S
Support
Plan
Produce
Analysis xyz
Report
M&S
Results
Credibility
M&S is Fit for Intended Use
M&S is Built Well
“FIT TO
PURPOSE ?”
“BUILT
WELL ?”
M&S
needs
‰ BUILT WELL
‰ FIT TO PURPOSE
and
‰ USED RIGHT
[ AFTER USE]
Conduct Analysis and do
Pre and Post Data-Processing
After Use
M&S Use for xyz
[ Known M&S Results-Credibility]
Analysis xyz
needed-results
defined?
Analysis xyz
output data
needs defined?
Credibility Assessment
Use M&S to Generate
Needed Analysis Data
Analysis xyz
Results
After Use
Analysis xyz
input data
needs defined?
Are the M&S “needs” for Analysis xyz defined?
Analysis
Problem xyz
Integrated
Analysis
SIAP
Plan
After Use
“ The Needs” for
M&S Application
M&S
Accreditation for
Specific Use
M&S
status
M&S
Development,
Management
Accreditation Authorites
?
VV&A
[ BEFORE USE]
Technical Reviews
Page 9