Steven Arndt, Ph.D., P.E.
Chairman
Maryland State Board for Professional Engineering
September 16, 2015
Presentation Overview
 State Board for Professional Engineering
 Introduction to “catastrophic” software failures or
do software developer really need to be licensed
 Some catastrophic software failures
 How can licensing software engineers help?
 The development of the Software PE exam
 Update
 Questions
2
State Board for Professional
Engineering
 Qualifies and licenses persons seeking licensure to
practice as a professional engineer
 The Board regulates the practice of engineering
under the provisions of Business Occupations and
Professions Article, Annotated Code of Maryland,
Title 14 and the Board investigates complaints
against licensees, as well as complaints related to
unlicensed practice.
 The Board may issue a reprimand, suspend or
revoke a license and /or impose a fine
3
Which systems affect the health, safety
and welfare of the public?
• Typical domains
– medicine, transportation, infrastructure, commerce,
finance
• Typical applications
– implantable medical devices, automobiles, elevators,
power systems, financial and health record management
systems
• Others
– entertainment – e.g. amusement park ride
– consumer goods – e.g. microwave oven
– … etc.
4
Scenario
 Consider a closed loop insulin pump / artificial
pancreas
 reads blood glucose
 doses insulin automatically
 A software failure causes overdoses
 7 patients die before product is recalled
 Who is responsible?
 What happens next?
5
Real world
 Between 2005 and 2009 the FDA received ~ 56,000 reports of infusion
pumps failures
 numerous cases of injury and death
 87 infusion pumps recalled
 14 recalled due to “Class I” safety concerns
 devices could cause serious adverse health consequences or death.
 Software defects lead to
 over-and-under infusion
 alarms on pumps to fail in emergencies or activate in absence of a problem
 FDA offers manufacturers pre-market review of software code used in
infusion pumps
6
Famous Software Catastrophes
 Therac 25
 (1985-1987) a radiation therapy device malfunctions and delivers lethal
radiation doses to 6 patients
 Explosion of Ariane 5 rocket (1996)
 Loss of more that $350 million
 Caused by a common cause software exception (arithmetic overflow)
 Patriot Missile Battery Intercept Failure
 Death of 28 soldiers and injury of 98
 Caused by “software aging”
 Many so-called software failures are often found to be
 Human error
 Mechanical failure
 Environmentally caused
 Some combination
7
Another Disaster
 November 2000 -- National Cancer Institute, Panama City
 A therapy planning program developed by a US firm miscalculates
radiation therapy dosages
 Dosage errors caused by doctors misusing program
 doctors try to “trick” software into making calculations it was not intended to do
 software miscalculates and doubles dosages
 failure to manually verify treatment plan
 At least eight patients die
 20 receive overdoses likely to cause significant health problems
 Three radiological technicians are indicted for murder – two found
guilty
 What was the culpability of the software/software engineers?
8
Another Scenario
 110 million Target customers have had their personal
data stolen in a data breach
 Stolen information included
 credit and debit card data
 customer names and
 PIN (personal identification data) numbers
 One in three Americans affected
 Is this catastrophic?
 Who is to blame?
9
Critical IT Infrastructures
 telecommunication infrastructure
 water supply
 electrical power system
 oil and gas
 road transportation
 railway transportation
 air transportation
 banking and financial services
 public safety services
 healthcare system
 administration and public services
10
Have there really been many
failures?
 Just one major organization list thousands of software
failures 1985-present
 voluntarily reported failures
 whistleblowers
 examining corporate documents
 newspaper stories
 lawsuits
 privately settled cases
 government announcements
 Many are of critical infrastructures
 Some include injury and loss of life
11
Liability Perspective
 Most computer-related transactions are considered a sale
of "goods"
 Are covered by Uniform Commercial Code
 warranties, terms of the contract, preempt negligence claims
 Software agreements include exculpatory language
 integration clauses
 warranty disclaimers
 limits on remedies to the repair or replacement of defects
 Shifts the risk of software failure from the seller to the user
 How do we protect the public from harm from software
failures?
12
Is Software Engineering Really
Engineering ?
 Engineering is the creative application of scientific
principles to design or develop “something”
 Software has standards for design, construction, test,
maintenance, etc.
 Software engineering includes metrics and models for
measuring failure rate, failure consequence, etc.
 Software analyses methodologies help identify problems
before they appear in the real environment (should be used
for designing, testing and coding)
13
Why is Software Engineering so
Important Today?
 Engineered systems are becoming software intensive: flight




systems, financial systems, defense systems, energy related
systems, etc.
Software intensive systems are often safety critical: air
traffic control systems or medical systems, etc.
Cost of developing software is increasing
The share of software in systems is important, its cost
becomes a major contributor to the cost of the project.
The cost of software is essentially associated with corrective
actions. It varies between 40 and 70%
14
Why licensure?
• States require licensure of certain engineers to ensure
that any practitioner is at least minimally competent
• Intent is to protect the public from injurious
consequences of incompetent “engineers”
• Licensure is required if the engineer is involved in
building a system
– whose failure could cause significant harm
– is offering his services directly to the public
– and not through a corporation, or government entity
15
How does licensing software
engineers help?
 Raising the level of professionalism
 Creating a system of accountability
 Creating trust and confidence in the public
 Qualifying expert witnesses
 But …. only a small percentage of software
professionals will need to be licensed
 my thoughts on this…
16
Licensure requirements
 Bachelor’s degree in software engineering from an ABET




accredited program (21 of them in US)
Fundamentals of Engineering (FE) exam
Applicable, supervised work experience (typically, at least
four years)
Principles and Practices of Software Engineering (P&P)
exam
Evidence of good moral character
Continuing education to retain licensure
17
History of Software PE Exam
 In Texas, the licensing board for professional engineers
began offering licensure to software engineers in 1999
 There was no standardized exam covering software engineering in place.
 In 2006, the board changed its rules to require all applicants for licensure to
pass a PE exam
 This change effectively cut off the path to licensure for software engineers
in the state
 Texas board believes that software engineering is a critical
component of many engineering projects and that it’s
important to recognize the impact that software
engineering has on the public’s health, safety, and welfare
18
Examination Development
Process Overview
Need for
Examination
Identified
Task
Analysis
Performed
(PAKS)
Examination
Specification
Developed
Item
Writing &
Review
Examination
Assembly &
Review
Examination
Scoring
Standard
Setting
Study
Equating of
Examination
Examination
Administration
After Cut Score Established
20
Examination Development
Process Overview
Need for
Examination
Identified
21
Need for Examination or
Module Identified
 Request by no fewer than 10 Member Boards
 Proof of need
 Estimate of usage
 Impact on health, safety, and welfare
 Is not adequately tested by an existing exam
 Must be at least one EAC/ABET program
 Initially in partnership with a technical society
22
History of Software PE Exam
 In August 2009 the NCEES approved the development of a
new PE exam for software engineers
 10 NCEES member boards presented letters of support
 IEEE-USA, IEEE Computer Society, NSPE and the Texas
Board sponsored the development of the exam
 Sufficient ABET programs were identified
 The first exam was given in April of 2013
 Professional actives and knowledge study (PAKS)
 Body of knowledge development / Examination
Specification
 Exam questions
22
Examination Development
Process Overview
Need for
Examination
Identified
Task
Analysis
Performed
(PAKS)
24
Professional Activities and
Knowledge Study (PAKS)
 Procedure
 Committee structure
 Survey structure


A list of tasks, knowledge, and skills that committee feels may
be important to the safe practice of their profession at the
time of licensure
Survey generated based on the task statements and important
knowledge and skills from above and will be rated by survey
respondents
25
Examination Development
Process Overview
Need for
Examination
Identified
Task
Analysis
Performed
(PAKS)
Examination
Specification
Developed
25
Specification Development
 Specification Development Committee
 Based on analysis report
 Identification of knowledge areas
 Passing list (2.5 and above)
 Borderline list (2.4 to 2.5), may be included if strong rationale
 Failing list (less than 2.4)
 Appropriately grouped into subcategories (evaluated
for potential breadth-and-depth exam)
 This process establishes the “defensible link”
 Must be approved by the EPE Committee
26
Examination Specification
 Based on Body of Knowledge
 80 multiple-choice question
 Differentiates Software Engineers from other
specialties
 Currently 26 different exams including




Control Systems
Electrical and Computer: Computer Engineering
Electrical and Computer: Electrical and Electronics
Electrical and Computer: Power
27
Examination Specification
 Electrical and Computer: Computer Engineering
 Computer Systems (40)%


Numeric and Nonnumeric Formats
Computer Architecture
 Hardware (25%)




Digital Devices
Digital Electronics
Digital Circuits
Hardwar Description Languages
 Software (25%)
 System Software
 Development/Applications
 Software Maintenance
 Networks (15%)


Computer Networks
Physical Layers Implementation
28
Examination Specification
 Electrical and Computer: Computer Engineering
 Software

System Software






Development/Applications






Operating systems
Real-time operating systems
3Computer security
Device drivers
Interrupts
Software design and documentation methods
Quality assurance
Fundamental constructs
Programming language characteristics
Development tools
Software Maintenance



Configuration management
Software update
Change control
29
Examination Specification
 Software Engineering
 Requirements (17%)
 Design (14%)
 Construction (11%)
 Testing (13%)
 Maintenance (7%)
 Configuration Management (8%)
 Engineering Processes (7%)
 Quality Assurance (8%)
 Safety, Security and Privacy (15%)
30
Examination Specification
 Software Engineering
 Requirements






Software requirements fundamentals (e.g., concept of operations,
types of requirements, product and process requirements,
functional and nonfunctional requirements, quantifiable
requirements, system requirements, software requirements,
derived requirements, constraints, service level)
Requirements elicitation
Requirements specification
Requirements analysis
Requirements verification and validation
Requirements management
31
Examination Specification
 Software Engineering
 Engineering Processes





Process definition (e.g., software life cycle models [Agile,
Spiral, Waterfall, etc.], software life cycle processes, process
tailoring, process assessment, process improvement)
Software engineering standards (e.g., process, coding,
development, testing, reliability, architecture)
Estimation (e.g., software size and complexity, effort)
Measurement (e.g., metrics definition, collection, and
analysis, goal question metric)
Risk management (e.g., addressing uncertainty, opportunities,
decisions under risk, decisions under uncertainty)
32
Examination Specification
 Software Engineering
 Requirements (17%)
 Design (14%)
 Construction (11%)
 Testing (13%)
 Maintenance (7%)
 Configuration Management (8%)
 Engineering Processes (7%)
 Quality Assurance (8%)
 Safety, Security and Privacy (15%)
33
Examination Development
Process Overview
Need for
Examination
Identified
Task
Analysis
Performed
(PAKS)
Examination
Specification
Developed
Item
Writing &
Review
34
Item Writing and Review
 On-going process
 Sources of item writers
 Training
 Item submittal form
 Proper documentation
 Subject matter expert (SME) review
 Appropriateness of content
 Time to solve
 Solution and key
 Rationale for distracters
35
Examination Development
Process Overview
Need for
Examination
Identified
Task
Analysis
Performed
(PAKS)
Examination
Specification
Developed
Item
Writing &
Review
Examination
Assembly &
Review
36
Examination Assembly & Review
 Item bank
 In accordance with the specification
 SME and committee review
 Content overlap
 Adherence to specification
 Consistency and sensitivity
 Timed pre-test
37
Examination Development
Process Overview
Need for
Examination
Identified
Task
Analysis
Performed
(PAKS)
Examination
Specification
Developed
Examination
Assembly &
Review
Examination
Scoring
Standard
Setting
Study
Item
Writing &
Review
38
Standard Setting Study
 The passing score is determined in one of
two ways.
 For the first administration of a new exam or
specification change, a cut score panel
recommends the score. This becomes the
“anchor” exam.
 For subsequent administrations, a statistical
procedure known as “equating” is used to set
the score relative to the anchor exam.
39
Examination Development
Process Overview
Need for
Examination
Identified
Task
Analysis
Performed
(PAKS)
Examination
Specification
Developed
Item
Writing &
Review
Examination
Assembly &
Review
Examination
Scoring
Standard
Setting
Study
Equating of
Examination
Examination
Administration
After Cut Score Established
40
Current Status
 Licensure of software engineers now available in 43
states and jurisdictions
 Software P&P exam offered last 3 years (2013, 2014,
2015)
 Limited number of takers so far (12,14,16, respectively)
 Pass Rates are reasonable (63% for first time takers in
2015)
 Main issues are:
 Exam preparation (size of exam bank, etc.)
 Understanding when SW license is needed vs other
specialties
 Spreading the word
41