Ethics Aspects Of Embedded And Cyber-Physical
Systems
Abhilash Thekkilakattil1, Gordana Dodig-Crnkovic1,2
1Mälardalen
University, Sweden
2Chalmers Technical University, Sweden
2University of Gothenburg, Sweden
Motivation
Increasing use of “intelligence” in cyber-physical systems
Navlab-1 from CMU’s
Navlab project,1980s
Mercedes-Benz Vario-based
VITA Autonomous Prototype
Eureka PROMETHEUS, 1980s
Robotic Volkswagen Passat
at Stanford University, 2009
Google’s latest self-driving
car prototype, 2015
Ethical aspects form important consideration: from design and
development to deployment and use of cyber physical systems
Traditionally: who is to be
blamed for accidents?
Now: should software also be held
responsible for accidents?
Addressing Ethical Challenges
Identify the moral problem
Examine alternative actions at hand
Choose a course of action
Record feedback
Attributing Responsibility to Software?
Typical arguments against holding software responsible:
– “Humans build the systems, and hence they are ultimately responsible
since they can control how the system works”
– “Humans decide under what settings the systems will operate”
– “Robots for which no human actor can be held responsible are poorly
designed sociotechnical systems.”
Most of the above arguments are true to some extent!
…however, not incorporating the possibility of software responsibility
in the discourse leaves a policy vacuum especially with the
increasing deployment of Artificial Intelligence (AI) in such systems
Policy Vacuum
Is mainly due to:
• Technology outpacing the legal infrastructure
– Legal support for autonomous driving came into effect only after its development
– The “bad sides” of many technologies like social networking and smartphones is
only becoming evident with use over a long time
• Easiness of creating “intelligent” technology
– Technology can be created relatively easily, e.g., Deep Blue computer that
defeated humans in chess
– Very likely that similar advances will be made in the safety-critical domain, e.g., a
robonurse
Dangerous situations can arise future “intelligent” systems are allowed to act
outside an ethical framework
Does not mean that human stakeholders are no longer responsible!
Demarcation Problem
To attribute responsibility to software there is a need for:
• Tracing anomalous behaviors into the responsible stakeholder
– Demarcate design errors from “unethical software behavior”
• Identify decisions taken by the software-agents
– Judgment by the software vs. design decisions
Highlights the need for a classification framework to enable traceability of
anomalous behaviors
Contributions
1. Framework for classifying different systems w.r.t automation and
autonomy
– Accommodating the notion of “software responsibility”
– Retaining the notion of “human responsibility”
– Demarcating situations where software can be held responsible
1. Recommendation for ethics as an extra-functional property
– Ethics is also an important property just like, e.g., reliability and timeliness, in the
context of “truly intelligent” systems
Terminology
• Stakeholder: agent who may interact with the cyber-physical system
at various stages of its design, development, deployment and
operation
• Developer: individual or organization associated with the
development of a cyber-physical system
• User: any stakeholder (person or software) that uses or operates a
cyber-physical system
• Software-agent: any software associated with the development
and use of a cyber-physical system, capable of making decisions
that affect the system itself and its behavior
Classification Framework
• Automatic systems
– Systems that typically replace hardware components
– No autonomy: simply imitate the behavior of the replaced component
– Example: drive-by-wire system
• Semi-automatic systems
– Automatic systems that are controlled and coordinated by humans
– Example: modern car with drive-by-wire and brake-by-wire
Human-in-loop
• Semi-autonomous systems
– Autonomously operates once assigned a task
– Human-in-loop assigns tasks that are autonomously carried out
– Example: modern drones operate autonomously once given a task
– Operates “completely” autonomously
– No human-in-loop to assign tasks or control behavior
– Example: Future robots!
cyber-physical systems
• Autonomous systems
No human-inloop
Assigning Responsibility
• Automatic system failures
– Automatic systems: No autonomy, simply imitate the behavior of the replaced
component
– No intelligence in the system: all decisions are take according to a predictable
algorithms e.g., steer-by-wire system
– Responsibility of failure is attributable to the manufacturers
• Semi-automatic system failures
– Semi-automatic systems: automatic systems that are controlled and coordinated
by humans e.g., an airplane
– Multiple stakeholders who are responsible:
• Manufacturer: responsible for correct functioning of the automatic systems
• User: responsible for operating the system according to the specifications
Assigning Responsibility
• Semi-autonomous system failures
– Semi-autonomous systems: human-in-loop assigns tasks that are autonomously
carried out
– There are 3 stakeholders:
• User: responsible for how the system is used
• Manufacturer: responsible for automatic subsystems
• Software-agent: responsible for operational decisions
• Autonomous system failures
– Autonomous Systems: operates “completely” autonomously without any humanin-loop to assign tasks or control behavior
– There are 2 stakeholders:
• Manufacturer: responsible for automatic subsystems
• Software-agent: responsible for operational decisions
Challenges
• Ethical considerations were accommodated by “human agents” like
engineers and users in traditional systems
• Need to include the possibility of an “ethics aware software-agent” in
future AI based systems
• This brings in some of challenges, e.g.,:
1. Communication challenge: communication between different stakeholders,
especially between “software-agents” and “human-agents”
1. Legal support for “software responsibility”: legislation to accommodate software
responsibility
Ethics as an Extra-functional Property
Ethics must be a part of the extra-functional property in future AI based
systems
– Software must be programmed to understand and apply ethics when AI
based technologies are deployed
– The ability of software to apply ethics is as important as other extrafunctional properties like reliability and security
– Allows for the seamless integration of ethics aspects into the software
engineering life cycle
Conclusions
 Reluctance to accommodate the notion of “software responsibility”
can leave a policy vacuum leading to dangerous situations
 Presented a framework to accommodate the notion of “software
responsibility”
– Enables the demarcation between “human responsibility” and “software
responsibility”
 We highlight the need to consider ethics aspects as an extrafunctional property just like e.g., reliability
Thank you !
Questions ?