Big Picture for Autonomy
Research in DoD
Approved for Public Release 15-1707
Soft and Secure Systems and Software Symposium
Dr. Robert Grabowski
Jun 9, 2015
For internal MITRE use
© 2015 The MITRE Corporation. All rights reserved.
|2|
Robotic Experience






9 Years US Navy nuclear power program
PhD CMU, small heterogeneous teams
15 years of robotics research
2005 DARPA Grand Challenge finalist
Novel perception
Test and evaluation, metrics
CGN-35
USS Truxtun
Physics
Millibots
Meteor
© 2015 The MITRE Corporation. All rights reserved.
Centaurs
|3|
Agenda






Tenets of autonomy research
Autonomy survey evolution
Definitions and spaces
Survey of autonomy in DoD
Autonomy in motion
Acceptance gap
© 2015 The MITRE Corporation. All rights reserved.
|4|
Tenets of Autonomy Research
 Focus on uniqueness
– Where autonomy is unique and how this uniqueness impacts the tester
– Why current approaches insufficient, what new approaches are needed
 Recognize perspectives
– Understand how each looks at autonomy different, what drives them
– Develop an appropriate language to communicate across perspectives
– Pivot from development of autonomy to test of autonomy
 Drive with models
– Provide relationships between perspectives
– Assure completeness / gap analysis (don’t leave holes)
– Mechanism for decomposition (towards a plan)
 Make it actionable
– Use insights to drive investment, new technologies, methodologies
– Build in capability to measure progress
© 2015 The MITRE Corporation. All rights reserved.
|5|
Autonomy
H i s t o r y, d e f i n i t i o n s a n d
fights
© 2015 The MITRE Corporation. All rights reserved.
|6|
Evolution of thinking
 Early investigation focused on
platforms, mostly provided portfolio
and catalog
© 2015 The MITRE Corporation. All rights reserved.
2010
|7|
Evolution of thinking
 Autonomy becomes S&T priority
2010
2010 SecDef
RDE Focus
© 2015 The MITRE Corporation. All rights reserved.
|8|
Evolution of thinking
 Autonomy becomes S&T priority
 First systematic focus on autonomy
 Introduction of MUM-T
 Introduction of issue of trust
2010
2010 SecDef
RDE Focus
© 2015 The MITRE Corporation. All rights reserved.
2011 DoD
Integrated Roadmap
|9|
Evolution of thinking
 Autonomy becomes S&T priority
 First systematic focus on autonomy
 Introduction of MUM-T
 Introduction of issue of trust
 Shift from platforms to decisions
 Shift towards science of autonomy
2010
2010 SecDef
RDE Focus
2011 DoD
Integrated Roadmap
2012
DSB Role Autonomy
© 2015 The MITRE Corporation. All rights reserved.
| 10 |
Evolution of thinking
 Autonomy becomes S&T priority
 First systematic focus on autonomy
 Introduction of MUM-T
 Introduction of issue of trust
 Shift from platforms to decisions
 Shift towards science of autonomy
 Emphasis on calibrated trust
 Awareness of testing and V&V
© 2015 The MITRE Corporation. All rights reserved.
2010
2010 SecDef
RDE Focus
2011 DoD
Integrated Roadmap
2013: DoD
Autonomous Research Initiative
2012
DSB Role Autonomy
| 11 |
Evolution of thinking
 Autonomy becomes S&T priority
 First systematic focus on autonomy
 Introduction of MUM-T
 Introduction of issue of trust
 Shift from platforms to decisions
 Shift towards science of autonomy
 Emphasis on calibrated trust
 Awareness of testing and V&V
2010
2010 SecDef
RDE Focus
2011 DoD
Integrated Roadmap
2013: DoD
Autonomous Research Initiative
2012
DSB Role Autonomy
 Emphasis and constraints on
autonomous weaponization
2013: SecDef
Autonomous Weaponization
© 2015 The MITRE Corporation. All rights reserved.
| 12 |
Evolution of thinking
 Autonomy becomes S&T priority
 First systematic focus on autonomy
 Introduction of MUM-T
 Introduction of issue of trust
 Shift from platforms to decisions
 Shift towards science of autonomy
 Emphasis on calibrated trust
 Awareness of testing and V&V
2010
2010 SecDef
RDE Focus
2011 DoD
Integrated Roadmap
2013: DoD
Autonomous Research Initiative
2012
DSB Role Autonomy
 Emphasis and constraints on
autonomous weaponization
 Focus on autonomy as decision
 Establishment of COI
2013: SecDef
Autonomous Weaponization
© 2015 The MITRE Corporation. All rights reserved.
2014: ASDRE
Reliance 21
| 13 |
Evolution of thinking
 Autonomy becomes S&T priority
 First systematic focus on autonomy
 Introduction of MUM-T
 Introduction of issue of trust
 Shift from platforms to decisions
 Shift towards science of autonomy
 Emphasis on calibrated trust
 Awareness of testing and V&V
2010
2010 SecDef
RDE Focus
2011 DoD
Integrated Roadmap
2013: DoD
Autonomous Research Initiative
2012
DSB Role Autonomy
 Emphasis and constraints on
autonomous weaponization
 Focus on autonomy as decision
 Establishment of COI
 Emphasis on HRI, Teams
 Emphasis on T&E/V&V
 Technology Offset
© 2015 The MITRE Corporation. All rights reserved.
2013: SecDef
Autonomous Weaponization
2014: ASDRE
Reliance 21
2015:
Autonomy Focus
| 14 |
Tenets of Autonomy Research
 Focus on uniqueness
– Where autonomy is unique and how this uniqueness impacts the tester
– Why current approaches insufficient, what new approaches are needed
 Recognize perspectives
– Understand how each looks at autonomy different, what drives them
– Develop an appropriate language to communicate across perspectives
– Pivot from development of autonomy to test of autonomy
 Drive with models
– Provide relationships between perspectives
– Assure completeness / gap analysis (don’t leave holes)
– Mechanism for decomposition (towards a plan)
 Make it actionable
– Use insights to drive investment, new technologies, methodologies
– Build in capability to measure progress
© 2015 The MITRE Corporation. All rights reserved.
| 15 |
Autonomy Definitions
 Many definitions, many opinions, many arguments
 All good but somehow all equally unsatisfying
© 2015 The MITRE Corporation. All rights reserved.
| 16 |
Autonomy Definitions
 Many definitions, many opinions, many arguments
 All good but somehow all equally unsatisfying
 Webster Dictionary (2015)
– “the state of existing or acting separately from others”
 AFRL Autonomy Science and Technology Strategy (2014)
– “systems which have a set of intelligence-based capabilities that allow it to respond to
situations that were not pre-programmed or anticipated in the design. Autonomous
systems have a degree of self-government and self-directed behavior”
 Towards Safety Assurance of Trusted Autonomy (2012)
– “the ability to reason and make decisions to reach given goals based on a systems
current knowledge and its perception of the variable environment in which it evolves”
 DoD Unmanned Systems Integrated Roadmap (2011)
– “autonomous systems develop, for themselves, the laws and
strategies by which they choose their behavior.”
 Army GO Workshop (2009)
– “a system which, by its appearance or movements,
conveys a sense that it has intent or agency of its own ”
 NIST Autonomy Levels for Unmanned Systems Framework (2004)
– “the condition or quality of being self-governing”
© 2015 The MITRE Corporation. All rights reserved.
| 17 |
Autonomy Definitions
 Many definitions, many opinions, many arguments
 All good but somehow all equally unsatisfying
 Webster Dictionary (2015)
– “the state of existing or acting separately from others”
 AFRL Autonomy Science and Technology Strategy (2014)
– “systems which have a set of intelligence-based capabilities that allow it to respond to
situations that were not pre-programmed or anticipated in the design. Autonomous
systems have a degree of self-government and self-directed behavior”
 Towards Safety Assurance of Trusted Autonomy (2012)
– “the ability to reason and make decisions to reach given goals based on a systems
current knowledge and its perception of the variable environment in which it evolves”
 DoD Unmanned Systems Integrated Roadmap (2011)
– “autonomous systems develop, for themselves, the laws and
strategies by which they choose their behavior.”
 Army GO Workshop (2009)
– “a system which, by its appearance or movements,
conveys a sense that it has intent or agency of its own ”
 NIST Autonomy Levels for Unmanned Systems Framework (2004)
– “the condition or quality of being self-governing”
© 2015 The MITRE Corporation. All rights reserved.
| 18 |
Autonomy Definitions
 Many definitions, many opinions, many arguments
 All good but somehow all equally unsatisfying
 Webster Dictionary (2015)
– “the state of existing or acting separately from others”
 AFRL Autonomy Science and Technology Strategy (2014)
– “systems which have a set of intelligence-based capabilities that allow it to respond to
situations that were not pre-programmed or anticipated in the design. Autonomous
systems have a degree of self-government and self-directed behavior”
 Towards Safety Assurance of Trusted Autonomy (2012)
– “the ability to reason and make decisions to reach given goals based on a systems
current knowledge and its perception of the variable environment in which it evolves”
 DoD Unmanned Systems Integrated Roadmap (2011)
– “autonomous systems develop, for themselves, the laws and
strategies by which they choose their behavior.”
 Army GO Workshop (2009)
– “a system which, by its appearance or movements,
conveys a sense that it has intent or agency of its own ”
 NIST Autonomy Levels for Unmanned Systems Framework (2004)
– “the condition or quality of being self-governing”
“I know it when I see it”
Justice Potter Stewart 1964
© 2015 The MITRE Corporation. All rights reserved.
| 19 |
Tenets of Autonomy Research
 Focus on uniqueness
– Where autonomy is unique and how this uniqueness impacts the tester
– Why current approaches insufficient, what new approaches are needed
 Recognize perspectives
– Understand how each looks at autonomy different, what drives them
– Develop an appropriate language to communicate across perspectives
– Pivot from development of autonomy to test of autonomy
 Drive with models
– Provide relationships between perspectives
– Assure completeness / gap analysis (don’t leave holes)
– Mechanism for decomposition (towards a plan)
 Make it actionable
– Use insights to drive investment, new technologies, methodologies
– Build in capability to measure progress
© 2015 The MITRE Corporation. All rights reserved.
| 20 |
Autonomy Levels
 Parallel approach is to divide
autonomy into levels
 Shift from singularity to
NASA (Function Specific
Level of Autonomy)
spectrum to spaces
Bloom Levels of
Autonomy
DoD 4 levels of
Autonomy (USIR)
SAE Levels of Driving
Autonomy
Army (Think, Look, Move, Talk, Work)
© 2015 The MITRE Corporation. All rights reserved.
Flavors of Autonomy
NIST Autonomy Space
Nice for categorization but
usefulness still in question
| 21 |
Tenets of Autonomy Research
 Focus on uniqueness
– Where autonomy is unique and how this uniqueness impacts the tester
– Why current approaches insufficient, what new approaches are needed
 Recognize perspectives
– Understand how each looks at autonomy different, what drives them
– Develop an appropriate language to communicate across perspectives
– Pivot from development of autonomy to test of autonomy
 Drive with models
– Provide relationships between perspectives
– Assure completeness / gap analysis (don’t leave holes)
– Mechanism for decomposition (towards a plan)
 Make it actionable
– Use insights to drive investment, new technologies, methodologies
– Build in capability to measure progress
© 2015 The MITRE Corporation. All rights reserved.
| 22 |
Autonomy Perspectives
 Autonomy assessment
– Listen to what agencies are saying
– Look at what they think autonomy will
enable
– No single group captures autonomy
– Collectively they build a full picture
– 30+ DoD, Service roadmaps, strategic
plans and investment strategies
referencing autonomy
Autonomy Survey Paper
© 2015 The MITRE Corporation. All rights reserved.
| 23 |
High Level
 Review of 100+ documents
– DoD, Agencies Federal and Civil,
International and Academic
– 30+ critical documents identified




















2015 OTI Technical Assessment Autonomy
2015 ATEVV Investment Strategy
2015 Unmanned Autonomous Systems Test Strategy
2015 Naval S&T Strategy
2014 AFOSR Technical Strategic Plan
2013 DoD Unmanned Systems Integrated Roadmap
2012 DSB The Role of Autonomy in DoD Systems
2012 DoD Autonomy in Weapon Systems Directive
2012 DoD Autonomy Research Pilot Initiative
2011 RSJPO Unmanned Ground Systems Roadmap
2011 DoD Unmanned Systems Integrated Roadmap
2011 DoD Science and Technology Priorities for Fiscal Years
2013-17 Planning
2010 NASA, Robotics, Tele-Robotics and Autonomous
Systems Roadmap
2010 Army Unmanned Air Systems Roadmap
2010 Air Force Technology Horizons
2009 DARPA Strategic Plan
2009 ARCIC Robotics Strategic White Paper
2009 Air Force Unmanned Aircraft Systems Flight Plan
2009 DoD Unmanned Systems Integrated Roadmap
2007 Navy Unmanned Surface Vehicle Master Plan
…
© 2015 The MITRE Corporation. All rights reserved.
| 24 |
Dual perspective
 Anticipation
– A closer look at what stakeholders are
anticipating from autonomy
– Look towards new capabilities enabled by
advances in technology and potentially new
applications and approaches
– Implicitly speaks to new challenges as well
Anticipation
 Trepidation
– The opposite perspective capturing the concerns
with adoption of unstructured autonomy
– Tends to be on concerns of consequences,
vulnerabilities and liabilities
Both sides essential in true understanding of autonomy
© 2015 The MITRE Corporation. All rights reserved.
Trepidation
| 25 |
Anticipations
 Reduction of Manpower - realizing the promise of unmanned systems to reduce
manpower and cost
– Mitigation of unmanned – reduce manpower, cost, logistics of existing platforms
– Reduction of operators – further reduction of manning and specially qualified operators to
control more than one platform or asset
– Information filtering – reduction of sheer data volume collected by unmanned systems.
Systems that make decisions on what not to show
© 2015 The MITRE Corporation. All rights reserved.
| 26 |
Anticipations
 Reduction of Manpower - realizing the promise of unmanned systems to reduce
manpower and cost
– Mitigation of unmanned – reduce manpower, cost, logistics of existing platforms
– Reduction of operators – further reduction of manning and specially qualified operators to
control more than one platform or asset
– Information filtering – reduction of sheer data volume collected by unmanned systems.
Systems that make decisions on what not to show
 Tactical Advantage - added advantages on the battlefield
–
–
–
–
Faster reaction time – local decisions faster than human cycle
Deeper penetration – operation in inaccessible or denied environments
Extended operation – can operate longer than human cycles
Agility and adaptation – ability to adjust to changing environment, changing mission
goals, ability to use in secondary missions
© 2015 The MITRE Corporation. All rights reserved.
| 27 |
Anticipations
 Reduction of Manpower - realizing the promise of unmanned systems to reduce
manpower and cost
– Mitigation of unmanned – reduce manpower, cost, logistics of existing platforms
– Reduction of operators – further reduction of manning and specially qualified operators to
control more than one platform or asset
– Information filtering – reduction of sheer data volume collected by unmanned systems.
Systems that make decisions on what not to show
 Tactical Advantage - added advantages on the battlefield
–
–
–
–
Faster reaction time – local decisions faster than human cycle
Deeper penetration – operation in inaccessible or denied environments
Extended operation – can operate longer than human cycles
Agility and adaptation – ability to adjust to changing environment, changing mission
goals, ability to use in secondary missions
 Trusted Companion - System capable of providing real-time, tactical and proximate
support to warfighters
– Faithful servant – utilization of competent mules, closer proximity to humans, operations
not in contact with adversary
– Loyal wingman – high tempo coordination and interaction, operations in contact with
adversary
© 2015 The MITRE Corporation. All rights reserved.
| 28 |
Trepidations
 Unmanaged Complexity - Complexity exceeds ability to design, monitor, and test
– Coupling of sub-systems – inability to understand interactions between systems,
complexity of test, difficulty in prediction
– Unstructured adaptation – implications of systems that can learn new rules
– Brittleness – superior behavior in ideal conditions, potentially catastrophic failure (or
behavior outside nominal
© 2015 The MITRE Corporation. All rights reserved.
| 29 |
Trepidations
 Unmanaged Complexity - Complexity exceeds ability to design, monitor, and test
– Coupling of sub-systems – inability to understand interactions between systems,
complexity of test, difficulty in prediction
– Unstructured adaptation – implications of systems that can learn new rules
– Brittleness – superior behavior in ideal conditions, potentially catastrophic failure (or
behavior outside nominal
 Unintended Consequences – difficulty in understanding true risks to mission or force
structure.
– Ambiguity of command – Complications on command and command structure
– Safety and liability – political and legal fallout from accident or damage
– Ethics of weaponization – implications on rules of engagement, long term complications
(consider current issue with drones)
– Complacency – difficulty in keeping human awareness at correct level of attention
© 2015 The MITRE Corporation. All rights reserved.
| 30 |
Trepidations
 Unmanaged Complexity - Complexity exceeds ability to design, monitor, and test
– Coupling of sub-systems – inability to understand interactions between systems,
complexity of test, difficulty in prediction
– Unstructured adaptation – implications of systems that can learn new rules
– Brittleness – superior behavior in ideal conditions, potentially catastrophic failure (or
behavior outside nominal
 Unintended Consequences – difficulty in understanding true risks to mission or force
structure.
– Ambiguity of command – Complications on command and command structure
– Safety and liability – political and legal fallout from accident or damage
– Ethics of weaponization – implications on rules of engagement, long term complications
(consider current issue with drones)
– Complacency – difficulty in keeping human awareness at correct level of attention
 Smart Adversary – impact from contact with an interactive and aware opponent
– Corruption – impacts of compromised systems including ability to detect
– Loss of control – adversary gaining control of asset or information
– The aware adversary – how engagement change when adversary aware they are
interacting with artificial system
– Asymmetry of development – concern of advancements of technology from an adversary
without the political, legal or self-imposed constraints we may be under
© 2015 The MITRE Corporation. All rights reserved.
| 31 |
Tenets of Autonomy Research
 Focus on uniqueness
– Where autonomy is unique and how this uniqueness impacts the tester
– Why current approaches insufficient, what new approaches are needed
 Recognize perspectives
– Understand how each looks at autonomy different, what drives them
– Develop an appropriate language to communicate across perspectives
– Pivot from development of autonomy to test of autonomy
 Drive with models
– Provide relationships between perspectives
– Assure completeness / gap analysis (don’t leave holes)
– Mechanism for decomposition (towards a plan)
 Make it actionable
– Use insights to drive investment, new technologies, methodologies
– Build in capability to measure progress
© 2015 The MITRE Corporation. All rights reserved.
| 32 |
DSB Recommendations
 “The DoD should abandon the debate over
definitions [and] of levels of autonomy”
– Focus on how autonomy impacts things not
what it means
© 2015 The MITRE Corporation. All rights reserved.
| 33 |
DSB Recommendations
 “The DoD should abandon the debate over
definitions [and] of levels of autonomy”
– Focus on how autonomy impacts things not
what it means
 “The Military Services should structure
autonomous systems acquisition programs to
separate the autonomy software from the
vehicle platform.”
– Autonomy is about decisions not platforms
© 2015 The MITRE Corporation. All rights reserved.
| 34 |
DSB Recommendations
 “The DoD should abandon the debate over
definitions [and] of levels of autonomy”
– Focus on how autonomy impacts things not
what it means
 “The Military Services should structure
autonomous systems acquisition programs to
separate the autonomy software from the
vehicle platform.”
– Autonomy is about decisions not platforms
 “AT&L should create developmental and
operational test and evaluation techniques
that focus on the unique challenges of
autonomy”
– Focus on unique properties of autonomy
– Focus on testing and role in adoption
© 2015 The MITRE Corporation. All rights reserved.
| 35 |
Autonomy in Motion
Acceptance Gap
© 2015 The MITRE Corporation. All rights reserved.
| 36 |
Autonomy Spectrum
Autonomy Spectrum
scripted
semantic
responsive
 Autonomy not sole purview of unmanned
© 2015 The MITRE Corporation. All rights reserved.
adaptive
self aware
| 37 |
Autonomy at Rest
Autonomy at Rest
“degree of
autonomy”
scripted
self aware
Siri
IBM Watson
High Frequency Trading
 Autonomy at rest
–
–
–
–
–
Cognitive assistants (siri)
Expert systems (high frequency trading)
Expert systems (watson)
C4I (next gen air traffic control)
C4I (Distributed battle management, Rspace …)
© 2015 The MITRE Corporation. All rights reserved.
Distributed Battle Mgt
Next Gen ATC
| 38 |
Autonomy Acceptance
Autonomy at Rest
scripted
self aware
Acceptance
Threshold
IBM Watson
 For any system, have some notional concept of acceptance
– Based on nature of autonomous system and environment it is in
© 2015 The MITRE Corporation. All rights reserved.
| 39 |
Autonomy Acceptance
Autonomy at Rest
Constrain
application
scripted
self aware
Increase oversight,
introspection
IBM Watson
 For any system, have some notional concept of acceptance
– Based on nature of autonomous system and environment it is in
 Adjustments
– Can constrain that autonomy
– Can mitigate with greater oversight
– Can develop new concepts for introspection and acceptance
© 2015 The MITRE Corporation. All rights reserved.
| 40 |
Autonomy Acceptance
Autonomy at Rest
scripted
self aware
Acceptance
Threshold
IBM Watson
 Result is a system that can be released in the wild and vetted through
employment
 Evolutionary approach to acceptance
– Adjust autonomy to match acceptance criterion
– Develop confidence over time
– Understand nature of application of autonomy
© 2015 The MITRE Corporation. All rights reserved.
| 41 |
Acceptance/Trust Gap
Autonomy in Motion
scripted
self aware
Squad Support
Robot Convoy
Driverless Taxi
Cargo Lift
Long duration ISR
Civilian Cargo
Fire fighting
ASW warfare
Wingman
 Unmanned autonomous systems (UAS) present a unique challenge to the
autonomy world
 Autonomy may require a minimum threshold just to operate
© 2015 The MITRE Corporation. All rights reserved.
| 42 |
Acceptance/Trust Gap
Autonomy in Motion
scripted
self aware
Squad Support
Robot Convoy
Driverless Taxi
Cargo Lift
Long duration ISR
Civilian Cargo
Fire fighting
ASW warfare
Wingman
 Some applications may be mitigated with oversight, co-human control
 Concept breaks when you remove the man (cant have a ½ autonomous system)
© 2015 The MITRE Corporation. All rights reserved.
| 43 |
Acceptance/Trust Gap
Autonomy in Motion
scripted
self aware
Acceptance
Threshold
Cargo Lift
 Consider Autonomous air lift (like AACUS)
–
–
–
–
–
–
Autonomous takeoff and landing
Landing zone assessment and selection
Obstacle avoidance and negotiations (powerlines, trees)
Payload negotiation and management
Environmental adjustment (weather, wind, updraft)
Adversarial interaction
© 2015 The MITRE Corporation. All rights reserved.
| 44 |
Acceptance/Trust Gap
Autonomy in Motion
scripted
self aware
Acceptance
Threshold
Cargo Lift
 Can restrict some of the operations but ultimately gap still remains
© 2015 The MITRE Corporation. All rights reserved.
| 45 |
Acceptance/Trust Gap
Autonomy in Motion
GAP
scripted
self aware
Acceptance
Threshold
Cargo Lift
 Result is an acceptance/trust gap that restricts release and operations
 Restricted release retards co-evolution
 Not just about making a more competent system, need to overcome trust gap
© 2015 The MITRE Corporation. All rights reserved.
| 46 |
Acceptance/Trust Gap
ATEVV Investment Strategy
Automated requirement (codable language)
Traceable evidence (pedigree)
Progressive sequential M&S, T&E (prediction)
Real time monitoring (watchdog oversight)
Just-in-time prediction (behaviors, actions)
Reusable assurance (actuaries)
ATEVV
Cargo Lift
Unmanned Autonomous Systems Test
Targeted testing (salience)
Smart test range (adaptive to system)
HMI efficacy (distributed SA)
Decision transparency (rules and impact)
UAST
 Two main groups focusing on test technologies
– ATEVV – under DoD ASDR&E
– UAST – under DoD TRMC
© 2015 The MITRE Corporation. All rights reserved.
| 47 |
Tenets of Autonomy Research
 Focus on uniqueness
– Where autonomy is unique and how this uniqueness impacts the tester
– Why current approaches insufficient, what new approaches are needed
 Recognize perspectives
– Understand how each looks at autonomy different, what drives them
– Develop an appropriate language to communicate across perspectives
– Pivot from development of autonomy to test of autonomy
 Drive with models
– Provide relationships between perspectives
– Assure completeness / gap analysis (don’t leave holes)
– Mechanism for decomposition (towards a plan)
 Make it actionable
– Use insights to drive investment, new technologies, methodologies
– Build in capability to measure progress
© 2015 The MITRE Corporation. All rights reserved.
Autonomy: A decade of understanding
Discussion
© 2015 The MITRE Corporation. All rights reserved.
| 48 |