UNIVERSITEIT UTRECHT – KEYLANE|QUINITY
MSc Project Thesis
Requirements Engineering Optimization using
the Method Association Approach
wcrooijmans
22-7-2015
Abstract: The b2b product software company Keylane|Quinity is experiencing increasing
complexity in managing their Requirements Engineering (RE) process, in part due to a
growing product and an expected growth in the client base. The main question for the
company is how to balance their elicitation efforts in new implementation projects, so that
they are both effective (all critical functional requirements are found) and efficient (not all
known requirements are gathered again). In this project the Method Association Approach
(MAA), part of the Method Engineering domain, has been applied to assemble a new RE
method based on existing methods First we identified both the business needs for RE and
attributes of good RE from scientific literature. These so called ‘features’ were then associated
to the method fragments of candidate methods. The resulting patchwork method was
evaluated in lunch sessions with multiple experts, and subsequently improved. The result was
a theoretically complete. In addition to the standard MAA, additional efforts were made to
integrate the method into the company and its way of working. This resulted in a method that
is both theoretically complete and practically usable.
FOREWORD
For the opportunity to write this thesis and performe the studies within, I would first and
foremost like to thank Keylane|Quinity. The time at their offices has taught me lessons that I will
carry with me for the rest of my career. The list of names of all the people that helped me with
something, at some point, would make this foreword a 10-pager. Thank you all for the time and
input!
Thank you to Dennis Bron, external coach and long time Quinity consultant, for all the tips, tricks
and weekly sparring sessions. This thesis would not have been half of what it is now without
your support.
Thank you to Frans van Dorp, a sales manager willing to show that old dogs most definitely do
learn new tricks, for all the help and willingness to look into the opportunity of language
analysis.
Thank you to Karin Baars and Prashant Tapkhirwala for so graciously welcoming me to
Keylane|Leanapps, and the insights you provided into your work.
Thank you to my internal coaches, Prof. Sjaak Brinkkemper and Dr. Fabiano Dalpiaz, for your
expert guidance and sage wisdoms.
COMMON ABBREVIATIONS
ME – Method Engineering
ABME – Assembly Based Method Engineering
MAA – Method Association Approach
RE – Requirements Engineering
MDRE – Market Driven Requirements Engineering
GEAR – Good Enough Architectural Requirements
SQUARE – Security Quality Requirements Engineering
RESCUE – Requirements Engineering with Scenarios for a User centered Environment
KJ – Kawakita Jiro (for the KJ technique and SEI’s KJ+ method)
ACRE – Acquiring Requirements framework
MRETS – Method for Requirements Engineering Technique Selection
TT framework – Tsumaki and Tamai’s elicitation technique selection framework
QIS – Quinity Insurance Solution
RFI – Request For Information
RFP – Request For Proposal
PoC – Proof of Concept
NLP – Natural Language Processing
TABLE OF CONTENTS
Foreword ...............................................................................................................................................................................3
Common abbreviations ...................................................................................................................................................3
1 - Introduction ...................................................................................................................................................................1
1.1 Problem definition ...............................................................................................................................................1
1.2 On efficiency, effectiveness, and gaps............................................................................................................2
1.3 Objective ....................................................................................................................................................................3
1.3.1 Scope...................................................................................................................................................................4
1.4 Outline ........................................................................................................................................................................4
2 - Related literature.........................................................................................................................................................5
2.1 Requirements engineering ................................................................................................................................5
2.2 Method Engineering. ............................................................................................................................................6
3 – Research Questions & Approach ....................................................................................................................... 10
3.1 Research questions ............................................................................................................................................ 10
3.2 Data gathering methods ................................................................................................................................... 10
Phase 1 – sub question 1. ................................................................................................................................... 11
Phase 2 – sub question 2 .................................................................................................................................... 11
Phase 3 – sub question 3. ................................................................................................................................... 11
Phase 1 – Current situation......................................................................................................................................... 12
4.1 Meta modeling the current method ............................................................................................................ 12
4.1.1 Sales trajectory ............................................................................................................................................ 13
4.1.2 Project setup................................................................................................................................................. 14
4.1.3 Requirements Engineering..................................................................................................................... 14
4.2 Business needs ..................................................................................................................................................... 16
Phase 2 – Method Association Approach .............................................................................................................. 18
5.1 Project situations ................................................................................................................................................ 18
5.1.1 Front-Heavy. ................................................................................................................................................. 18
5.1.2 Amendment project................................................................................................................................... 19
5.2 Feature groups ..................................................................................................................................................... 19
5.2.1 Feature group sources.............................................................................................................................. 19
5.2.2 Feature group relations. .......................................................................................................................... 20
5.2.3 The feature groups..................................................................................................................................... 21
5.2.4 Feature group evaluation ........................................................................................................................ 23
5.3 Candidate method selection ........................................................................................................................... 26
5.3.1 Candidate method modeling.................................................................................................................. 26
5.3.2 Candidate methods overview................................................................................................................ 27
6 Method base .................................................................................................................................................................. 28
6.1 Method – Elicitation technique selection .................................................................................................. 28
6.2 Method - Joint Application Development.................................................................................................. 31
6.3 Method – RESCUE ............................................................................................................................................... 33
6.4 Method - Good Enough Architectural Requirements ........................................................................... 34
6.5 Method - Tropos .................................................................................................................................................. 37
6.6 Method - Security Quality Requirements Engineering ...................................................................... 38
6.7 Method - KJ+.......................................................................................................................................................... 39
6.8 Method – Keylane|LeanApps method ........................................................................................................ 42
7 – Association and method construction............................................................................................................. 44
7.1 Association ............................................................................................................................................................ 44
7.2 Preliminary method assembly ...................................................................................................................... 45
Using the association results, a preliminary method was assembled. See Figure 22 for a
summarized version. A full view of the preliminary method is included in ...................................... 45
Phase 3 – Validation ....................................................................................................................................................... 48
8.1 First review results ............................................................................................................................................ 48
8.2 Post review method changes ......................................................................................................................... 50
8.2.1 Agree on terminology ............................................................................................................................... 50
8.2.2 Identify stakeholders ................................................................................................................................ 50
8.2.3 Technique selection................................................................................................................................... 51
8.2.4 SMART goals ................................................................................................................................................. 51
8.2.5 Passive voice................................................................................................................................................. 52
8.3 Introduce more creative and interactive elicitation ............................................................................ 53
8.4 Impact of the evaluated method ................................................................................................................... 54
9 – Second Review round............................................................................................................................................. 57
9.1 Second review results ....................................................................................................................................... 57
9.2 Post-review method changes ......................................................................................................................... 57
9.3 Final method ........................................................................................................................................................ 60
10 - Other RE efforts ...................................................................................................................................................... 61
10.1 Requirements elicitation technique selection experiment. ............................................................ 61
10.1.1 Three very different selection approaches ................................................................................... 61
10.1.2 Lists upon lists of factors ...................................................................................................................... 62
10.1.3 Abandon ship! ........................................................................................................................................... 62
10.2 Crowd sourcing requirements engineering in a product wiki ...................................................... 63
10.2.1 Wiki’s and crowd sourcing .................................................................................................................. 63
10.2.2 Building a Wiki......................................................................................................................................... 64
10.2.3 Discussions ................................................................................................................................................. 65
10.2.4 Converting documents .......................................................................................................................... 66
10.3 Using Natural Language Processing for requirement and stakeholder identification ....... 66
11 – Discussion & Conclusion .................................................................................................................................... 68
11.1 Results .................................................................................................................................................................. 68
11.2 Conclusion ........................................................................................................................................................... 70
11.3 Validity.................................................................................................................................................................. 71
11.3.1 Credibility of the approach .................................................................................................................. 71
11.3.2 Confirmability of the results ............................................................................................................... 71
11.3.3 Dependability of the created method .............................................................................................. 71
11.3.4 Transferability of the results. ............................................................................................................. 71
11.4 Post-project review ......................................................................................................................................... 72
11.4.1 Working on a fuzzy problem ............................................................................................................... 72
11.4.2 Working with the Method Association Approach ...................................................................... 72
11.5 Relevance............................................................................................................................................................. 75
11.5.1 Scientific Relevance ............................................................................................................................... 75
11.5.2 Business Relevance. ................................................................................................................................ 76
References .......................................................................................................................................................................... 77
Appendix................................................................................................................................................................................1
Appendix 1 – Gap analysis meta model ...............................................................................................................1
Appendix 2 – Current situation process/deliverable tables .......................................................................2
Appendix 3 – feature groups ....................................................................................................................................7
Appendix 4 – Eliminated methods ...................................................................................................................... 13
Appendix 5 – Association table ............................................................................................................................ 16
Appendix 6 – preliminary method ...................................................................................................................... 19
Appendix 7 – post - First evaluation .................................................................................................................. 21
Appendix 8 – Final method .................................................................................................................................... 24
appendix 9 – Final method concepts and deliverables............................................................................... 27
appendix 10 – Wiki Screenshots .......................................................................................................................... 35
Addendum 1 – Natural language processing of requirements in client documentation ......................1
1 - Introduction ...................................................................................................................................................................2
2 - Domain .............................................................................................................................................................................2
3 - Method .............................................................................................................................................................................3
Keyword finding ............................................................................................................................................................3
RFP analysis using search words............................................................................................................................3
4 - Gap analysis test...........................................................................................................................................................4
5 - RFP analysis ...................................................................................................................................................................5
Discussion .............................................................................................................................................................................6
Conclusion .............................................................................................................................................................................6
References .............................................................................................................................................................................7
LIST OF FIGURES
Figure 1 - Requirement and gap sources .................................................................................................................2
Figure 2 - The Software Development Life Cycle ..................................................................................................5
Figure 3- The original situational method engineering approach .................................................................7
Figure 4 - Method Association Approach for Web Engineering Methods by Luinenburg et al. ........9
Figure 5 - Generalized MAA ...........................................................................................................................................9
Figure 6 - QIS implementation method.................................................................................................................. 12
Figure 7 - Keylane|Quinity Sales Trajectory ........................................................................................................ 13
Figure 8 – Keylane|Quinity setup phase ................................................................................................................ 14
Figure 9 - Keylane|Quinity design phase (start of the increments) ........................................................... 15
Figure 10 - Visual representation of feature group relationships .............................................................. 21
Figure 12 - Feature classification decision tree.................................................................................................. 23
Figure 12 - Elicitation technique selection decision support ....................................................................... 31
Figure 14 - Example JAD room layout prescription.......................................................................................... 31
Figure 15 - JAD meta model ........................................................................................................................................ 32
Figure 16 - RQ management phase of the RESCUE meta model .................................................................. 33
Figure 16 - GEAR meta-model ................................................................................................................................... 36
Figure 17 - Tropos meta-model ................................................................................................................................ 38
Figure 18 - Example of a SQUARE terminology survey question ................................................................ 39
Figure 19 - KJ+ meta-model ........................................................................................................................................ 41
Figure 21 - Keylane|LeanApps meta method ...................................................................................................... 42
Figure 21 - Association table ...................................................................................................................................... 45
Figure 22 - summarized preliminary method. .................................................................................................... 47
Figure 23 –Old Poster visualizing method changes .......................................................................................... 55
Figure 24 – Poster visualizing only the new method ....................................................................................... 56
Figure 26 – Deciding on notations in the review session ............................................................................... 59
Figure 26 - QIS wiki concept image ......................................................................................................................... 64
Figure 27 - A wiki talk page with LiquidThreads............................................................................................... 65
Figure 29 - NLP experiment envisioned outcome ............................................................................................. 67
Figure 29 - Summarized final method .................................................................................................................... 68
Figure 30 - MAA as used in the thesis ..................................................................................................................... 75
LIST OF TABLES
Table 1 - Keylane|Quinity business needs ............................................................................................................ 17
Table 2 - Feature group evaluation results .......................................................................................................... 24
Table 3 - Candidate methods overview ................................................................................................................. 27
Table 4 - Final technique selection framework techniques .......................................................................... 29
Table 5 - Technique selection framework changes........................................................................................... 30
Table 6 - Example RACI matrix.................................................................................................................................. 51
Table 7 - evaluation session results overview .................................................................................................... 60
Table 8 - Wiki platforms comparison ..................................................................................................................... 64
Table 9 - Thesis deliverables ...................................................................................................................................... 70
1 - INTRODUCTION
A good introduction of Keylane|Quinity and their product is given by its own website:
“The modernization of the application landscape of a non-life insurer is complex and risky.
Keylane|Quinity offers with the Keylane|Quinity Insurance Solution the best standard
solution for non-life insurers, but also ensures that it is implemented quickly and
successfully.
Underpinning it all: The Keylane|Quinity Insurance Solution, a modern, user-friendly system
that supports all primary insurance processes from quote request to policy and from first
claim notice to payment. When doing so we make optimum use of standard functionality,
which in turn ensures low operating costs.”
-
Quoted from www.KeylaneQuinity.com
In this chapter the exact nature of Keylane|Quinity’s problem is discussed, followed by a short
introduction of the main objective and challenges of this project. The chapter concludes with a
short outline of the other contents.
1.1 PROBLEM DEFINITION
After several years of producing custom software solutions for varying clients, Keylane|Quinity
moved towards the production of a single software product for the insurance market; the
Keylane|Quinity Insurance Solution (QIS).
The move to a standard software product was accompanied by many structural changes to both
the company and its processes. What worked for custom projects does not always translate well
to the delivery of a standardized software product.
When a software product needs to be implemented, many of a new client’s requirements are
likely already present within- and fulfilled by- that product thanks to past projects. In addition, a
procuct is targeted towards a specific market. This means that additional requirements can be
derived from that market as a whole, including standards, best practices, rules and regulations
that are common in that market. The different nature of Requirements Engineering for software
products rather than bespoke development has been a subject of debate in the academic world
as well [1].
During implementation projects for new clients, Keylane|Quinity didn’t always take the time to
explicitly gather requirements. Consultants often gather what they can from client initiated
sessions. This may work fairly well when dominant clients demand four weeks of demos and use
cases, but with less dominant clients there was a distinct lack in knowledge before design begins.
This put the burden of research on the designers themselves, leaving them with the task of
finding out what the client actually needs before being able to design a solution. In the future
situation this would preferably no longer be the case. With a dedicated requirements
engineering process the client’s needs can be identified consistently, regardless of whether
clients are dominant or passive.
1
The leading question during the requirements engineering process should be: ‘Does QIS fit the
client’s need?’. In the projects where an explicit form of requirements gathering was applied, it
became increasingly difficult to find a balance between efficiency and effectiveness.
By efficiency we mean that all (or at least most) of the major gaps are identified. Gaps are
functional needs of the client that QIS does not cover yet. Gaps constitute the most work for
designers and developers in any implementation project. If gaps are left unsolved, the client
may not be able to perform some of their core business processes, which severely impacts the
quality of the final QIS implementation. If gaps are missed and found later in projects, they can
result in large unforeseen costs for both the client and Keylane|Quinity.
By effectiveness we mean that we do not spend too much time gathering known requirements.
As mentioned before about software products, the product already covers many potential
requirements from varying sources. If the client mentions a need that QIS already covers, it’s not
necessary to capture, document and process it as rigorously as you would with a gap.
Besides balancing efficiency and effectiveness, requirements engineering is also becoming more
challenging because of the continuing growth of the QIS product and a recent initiative towards
internationalization. This is expected to yield many new clients and implementation projects. To
prevent Keylane|Quinity’s experts to be spread too thin, the company is willing to look into tools
or web-based ways of obtaining requirements with fairly little expert intervention.
In summary:
1. With a formal requirements engineering method, Keylane|Quinity is hoping to improve
consistency of requirements work across varying implementation projects.
2. This method should assist them in their balancing act between efficiency and
effectiveness.
3. This method could provide modern means of client interaction that reduce the burden
on their experts.
1.2 ON EFFICIENCY, EFFECTIVENESS, AND GAPS
A new client may have
10.000 requirements.
Only a fraction of these,
let’s say 2000, will be
discussed on the client’s
own initiative during
early information
sessions and in sales
documents. These 2000
are easy to assess and
filter thanks to
interaction with the
client, but what about
the other 8000?1
Figure 1 - Requirement and gap sources
1
Numbers are purely hypothetical for illustration purposes
2
Covering all 8000 of these unmentioned requirements is going to make for a very inefficient
elicitation trajectory, particularly when keeping in mind that QIS is already going to fulfill a large
share of these requirements. Other unmentioned requirements are the absolute basics of the
insurance domain (e.g. I want to register a policy) or completely irrelevant for the QIS product
(e.g. Can’t you store a database of all Dutch car-auction data in your system?).
The holy grail of unmentioned requirements are the unmentioned gaps. Knowing the major gaps
early enables Keylane|Quinity to plan their design and development better, and makes it
possible to deliver more accurate cost estimates to the client. Missing these gaps until much later
phases can have devastating results for projects. In fact, the costs of researching missed
requirements after the actual RE process can exceed the usual project costs anywhere between
10 to 200 times [2], as estimated by Barry Boehm.
From past experience Keylane|Quinity has learned that missed gaps are often related to the
following types of knowledge:



Tacit knowledge. (e.g. I forgot to mention our automated acceptance system. I’m so used
to it always running in the background that I don’t even notice it anymore).
False assumptions (e.g. Wait so your system doesn’t generate those reports
automatically? I assumed it would!).
Document knowledge that nobody made their own (e.g. Nobody knows why that system
works that way. The guy who made those decisions doesn’t work here anymore, but he
might have documented them before he left).
The problem of uncovering requirements that hide behind a client’s tacit knowledge is certainly
not new. In their work on the future of requirements engineering, Nuseibeh and Easterbrook
proposed that requirements engineers learn from the field of cognitive psychology, because:
“For example, problem domain experts often have large amounts of tacit knowledge that is not
amenable to introspection; hence their answers to questions posed by requirements analysts may
not match their behaviour” [3]. In later works Ratchev et al recognized the same problem: “The
conversion of this knowledge into explicit information depends greatly on the use of tacit
knowledge, i.e. personal knowledge that is implicit and cannot easily be communicated. The main
disadvantage of this form of knowledge is that it evades critical discussion” [4].
These missed gaps can be uncovered in various sources, such as documentation of the client’s
current system, documentation on the client’s processes, or a set of business rules. Others may
be available from stakeholders that weren’t involved in early contact, or the stakeholders that
were involved have knowledge that they did not deem important enough to share at the time.
This unshared knowledge can be elicited in interviews, workshops, questionnaires or demo
sessions performed on the initiative of Keylane|Quinity.
1.3 OBJECTIVE
The goal of this project is to provide an improved Requirements Engineering method that better
suits the current situation at Keylane|Quinity. The project will begin with an analysis of the
current Requirements Engineering practices. A new method will be established using the
Method Association Approach (MAA) as outlined by Luinenburg et al. [5]. The method will be
iteratively improved using expert evaluations. The new method will be considered successful
when a consensus with Keylane|Quinity’s experts is reached.
3
In addition to this approach, the external coach has requested some creative effort on my part.
Any additions made outside of the MAA on my own initiative are covered in chapter 10.
Initially we discussed the possibility of an evaluation by case study. There were no potential
cases available until the end of this project however, so we still retain the opportunity to write a
case study in a separate paper.
The resulting method, though intended for Keylane|Quinity, should be generalizable to similar
companies that operate in a similar context. This content and the opportunities for
generalization are further discussed in section 11.5.2 – Business relevance.
1.3.1 Scope. The scope of this project was limited to RE in the sales- and project setup phases of
implementation projects at Keylane|Quinity. Though design, development and testing phases are
briefly touched upon in this thesis, they are not part of the project scope.
For requirements specifically, we will limit our scope to only functional requirements. As
mentioned in the problem statement, these are the most valuable requirements both for Keylane
and it’s clients. Though quality requirements and constraints are briefly touched upon in this
thesis, they are also not part of the project scope.
1.4 OUTLINE
Chapter 2 will discuss the related literature from the fields of requirements engineering and
methods engineering. This is followed by chapter 3, where the research questions and methods
are discussed.
To streamline internal reporting we subdivided the work in this thesis into three distinct phases.
Chapter 4 marks the beginning of Phase 1; the analysis of the current situation. In this chapter
the current situation is discussed in detail. As part of the current situation, business needs are
also discussed in this chapter.
Chapter 5 marks the beginning of Phase 2; performing the method association approach. The
contents of this approach are covered in depth in chapter 2. It discusses the project situations,
feature groups and the selection of candidate methods. This is followed by chapter 6, the method
base, where all selected methods are presented. Phase 2 concludes with chapter 7, which
discusses the association of features and methods , and the assembly of a preliminary method.
Chapter 8 marks the beginning of phase 3; evaluation. In this phase the preliminary method was
evaluated and contextualized. Both lunch meetings and their results are discussed in chapters 8
and 9 respectively.
Chapter 10 discusses some additions made to the method on our own initiative outside of the
method association approach. Finally, chapter 11 contains a conclusion, discussion, validity and
the business and scientific relevance of this thesis.
4
2 - RELATED LITERATURE
In this chapter the scientific domain of this project will be elaborated upon. Chapter 2.1 briefly
discusses the topic of requirements engineering. For large and highly detailed works on the
subject please refer to some of the books referenced in this thesis ([6,7,8,9]). Chapter 2.2 will
discuss the field of Method Engineering (ME), which will be used to form a solution. For this
section we used a snowballing approach [10] to trace the lineage of MAA back to it’s origins.
2.1 REQUIREMENTS ENGINEERING
To understand how Keylane|Quinity understands requirements, we begin with the definition
used by Keylane|Quinity in its internal documentation (translated from the book ‘Success met
Requirements!’ [8]):
“Requirements involve the demands and desires of the customer: What should be the abilities of the
system be? A requirement describes what must happen, not how it must happen. That way,
requirements bring the scope of the design and realization trajectory into view.”
The book provides many examples of functional and non-functional requirements, but provides
no definition of either or the differences between them. A popular definition of functional and
non functional requirements is given by Wiegers and Beatty in their book on requirements[6]:
“A functional requirement is a description of a behaviour that a system will exhibit under
specific conditions.
A non-functional requirement is a description of a property or characteristic that a system
must exhibit or a constraint that it must respect.”
What is the engineering of these requirements then? Requirements Egineering (RE) is “the
discovering, documenting and managing of requirements”[6]. The Standish group, a well known
research advisory organization with a focus on software project management, has claimed for
years that having a clear statement of requirements is the number 3 contributing factor to IT
project success [11]. Hypothetically then, when RE is done right, project success is within your
grasp.
RE
Evaluation
Implementation
Design
Development
Figure 2 - The Software Development Life Cycle
But what is it’s place in the development of software? The Software/Systems Development Life
Cycle [12] (SDLC), as seen in figure 2, shows a common structure for an iterative software
5
development process. Though the names of process steps in different SDLC representations may
vary from author to author, RE always has a prominent role in this process[3][13]. The
requirements identified in the RE process feed directly into the design process, where solutions
are designed to fulfill them. After completing a cycle through development, implementation, and
evaluation, the evaluation results feed directly back into the RE process in the form of
shortcomings, oversights, bugs, and more.
Within RE itself some typical high level process steps are [12][13]:






Eliciting requirements.
Analysing requirements.
Specifying requirements.
Prioritizing requirements.
verifying requirements after development.
Requirements management to control the above.
Other sources may include steps like validation, impact analysis, negotiation, cost/value
analyses, quality assurance, change management and much more.
Developing product software has it’s influences on requirements engineering. Rather than
catering to all the requirements of individual customers in each project, a software product will
likely already cover some requirements that new clients may have. In addition, requirements
can also be identified from the broader market, including trends, standards, best practices and
legislation within that market. Hence, requirements engineering for software products becomes
more market-driven rather than client-driven. This is known as Market Driven Requirements
Engineering (MDRE) [1].
MDRE processes, though similar to the general RE process, can vary in several ways. The focus of
elicitation, for instance, shifts from single customers to entire markets. As determined by Regnell
et al. in a 1998 case study on MDRE: “[for software products] the RE process should be able to
invent requirements based on foreseen end-user needs and select a set of requirements resulting in
a software product which can compete on the market”[14]. Another example of a variation in
MDRE is the increased depencence on prioritization and analysis. With so many potential
requirements flooding in, some way of filtering is needed. In the words of Carlshamre: “In
market-driven software development, release planning is one of the most critical tasks. Selecting a
subset of requirements for realization in a certain release is as complex as it is important for the
success of a software product” [15].
2.2 METHOD ENGINEERING.
The Method Association Approach (MAA), which will be used in this project to create an RE
method, is part of the broader Method Engineering domain.
Methodology [sic] Engineering is a term originally coined by Bergstra in 1985 [16]. Method
Engineering was later defined as “the engineering discipline to design, construct, and adapt
methods, techniques and tools for the development of information systems” by Brinkkemper in
1996 [17]. In this work Brinkkemper already noted that not every project is the same, so good
methods should be built to work in varying project situations; what he calls ‘situational method
engineering’. This notion was supported in the same year by Rolland and Prakash [18]. The
6
original configuration process for situational methods proposed by Brinkkemper can be seen in
figure 3.
Figure 3- The original situational method engineering approach
Situational Method Engineering (SME) would later become a field of study in its own right.
Across years of research the field of SME spawned several classes of more specific approaches,
for example Method Configuration and Assembly Based Method Engineering.
Assembly Based Method Engineering (ABME) was originally defined by Ralyté et al in 2003 [19].
It was proposed as an ‘on-the-fly’ variant of the process seen in figure 2. Ralyté et al. interpreted
the ‘project environment’ and ‘characterisation of project’ steps as a form of requirements
engineering. Hence ABME was classified as a requirements driven approach. ABME consisted of
a three step process: specify method requirements, select method chunks that fit the
requirements, and assemble method chunks into a method.
ABME was put to practice in a case study by van de Weerd et al. three years later [20]. Van de
Weerd criticized the original ABME for lacking an explicit process step for the actual gathering of
candidate methods, and proposed new ABME process steps:
1.
2.
3.
4.
Analyze implementation situations and identify needs.
Select candidate methods that meet one or more aspects of the identified needs.
Analyze candidate methods and store relevant method fragments in a method base.
Select useful method fragments and assemble them in a new method by using route map
configuration to obtain situational methods.”
7
In the same work van de Weerd et al. also proposed a notation for meta-modeling selected
methods [21]. The Process-Data Diagram, or Process-Deliverable Diagram (PDD) as it would
later be named, consists of two linked models; one for processes and one for data
artifacts(deliverables) that are created in the process.
Deliverables have the same modeling structure (standard, open, closed). Relationships between
deliverables are either associations, aggregations (is a part of-) or generalizations (is a generic
form of-). Generalizations can be annotated to show whether the specific variants of a generic
deliverable overlap, or whether they are mutually exclusive.
The Method Association Approach (MAA) was originally proposed by Luinenburg et al. [5] as an
adaptation of van de Weerd’s ABME approach. MAA added the identifying of feature groups and
the association of these groups with candidate method fragments. Hence the ‘Association’ in the
name. The original MAA was created specifically for a project seeking to create a web design
method. An overview of the original MAA has been provided in Figure 5. Later in this project, a
generalized MAA was first published by Deneckère et al. in a collaboration between the
University of Sorbonne and University of Utrecht [22].
The MAA essentially added two new steps to the ABME approach defined by van de Weerd et al.:
1. identify project situations (renaming of “Analyze implementation situations and identify
needs”) (ABME step 1)
2. Identify feature groups
3. Select candidate methods (based on ABME step 2)
4. Model candidate methods (based on ABME step 3)
5. Associate feature groupings with candidate methods (based on ABME step 4)
6. Assemble situational web design method. (based on ABME step 4)
7. Validate situational web design method.
As far as we were able to find out, there are only two case studies available of a pure MMA
application, namely by Luinenburg et al. and Deneckère et al. themselves [5,22]. One other case
exists of ITAXEM, a method wich is reportedly derived from the MAA by Hunink et al. [23]. There
is obviously a clear lack in case studies for the Method Association Approach.
8
Figure 4 - Method Association Approach for Web Engineering Methods by Luinenburg et al.
Figure 5 - Generalized MAA
9
3 – RESEARCH QUESTIONS & APPROACH
In this chapter we explain the research approach used in this thesis. Section 3.1 introduces the
research questions, followed by the methods used in Section 3.2. A short planning is included in
section 3.3.
3.1 RESEARCH QUESTIONS
The main question leading this project is:
MQ – How can Keylane|Quinity improve its requirements engineering method in terms of efficiency
and effectiveness?
The desired end product of this project is a method, tailored for Keylane|Quinity, which is
adaptable to their varying project situations. We will use the method association approach as
outlined by Luinenburg, Jansen, Souer, Van De Weerd & Brinkkemper (2008) [5]. The MAA was
already introduced in Chapter 2.
The sub questions (SQ) covering this topic are the following:
SQ 1 – How can the current processes and future needs for early requirements engineering
of a product software company be identified?
To answer this question two activities (A) will be performed.
A1. Model the current process
A2. Evaluate the current process
SQ 2 – How can the steps of the Method Association approach be applied to inform the
essential processes and deliverables of a method for the elicitation, capture, and analysis of
early requirements?
To answer this question, the first six steps in the MAA (figure 6) will be performed. Each can be
considered its own sub activity.
A3. Identify project situations.
A4. Identify feature groups.
A5. Select candidate methods.
A6. Model candidate methods.
A7. Associate feature groupings with candidate methods.
A8. Assemble a situational implementation method.
The result should be a theoretically sound method, but still requires a view from practice.
SQ 3 – How can the resulting method be implemented and evaluated?
This question will be answered through group/expert reviews. In the first review we will
broadly validate the contents. In a second review we will present a hypothetical case to experts.
A9. Validate situational implementation method.
A10. Evaluate a hypothetical case.
3.2 DATA GATHERING METHODS
10
As a preparation for the project a model of Keylane|Quinity’s current method was created based
on Keylane|Quinity’s own documentation of their method. Using the PDD notation, each
documented activity or set of activities was modeled as a process fragment with its matching
deliverables. The original documents are only available for internal use and will not be made
public.
Phase 1 – sub question 1. Interviews held on location were semi-structured. Interviewees were
asked to describe their role and give an example of their work in a previous case of their
choosing. The interviews then proceeded with a walkthrough of the latest model version.
Interviewees were instructed to interject at any point with feedback, criticism, suggestions,
confusion and other remarks. Deviations from the model were then discussed in further detail to
identify their nature and relevance for a new model. (activity A1)
Interviews concluded with an opportunity for the interviewee to share their opinion on the
current situation. This included a discussion of any existing features of the current method they
would like to keep or additional features they would like the method to have. (activity A2)
Three additional short interviews were held with a manager, a project leader and a principal
consultant to validate the business needs for a new method. (activity A2)
Phase 2 – sub question 2. In addition to those already gathered from interviews, feature were
identified by studying literature on requirements engineering. For this stage of the project books
and maturity models were used (activity A4).
The selection of candidate methods was also performed by reviewing literature. In addition to
these methods from literature, an interview was held with two Keylane|LeanApps
representatives to elicit their method. The sister company’s method was pointed out by the
external coach as a specific point of interest. Once again the semi-structured interviewing
method was applied. (Activity A5).
After a large initial collection of candidates was gathered, the candidate methods were briefly
reviewed with the external coach. For each of the candidate methods that came out of this preselection, a PDD meta model was created (Activity A6). The candidate methods and their method
fragments were then associated with the feature groups. Whenever a method fragment satisfied
one or more features in a group, a note was made (Activity A7). After association of all methods
to all feature groups a preliminary draft method was constructed containing those method
fragments that satisfied the most features (activity A8).
Phase 3 – sub question 3. When the first requirements engineering method was established,
evaluation began (Activity A9). The first evaluation step was a large one-hour lunch meeting
with many of the phase 1 interviewees present. This meeting resulted in a lot of feedback,
making a need for contextualization apparent. The method fragments taken as-is from their
parent methods were not sufficiently applicable for Keylane|Quinity yet. To truly integrate this
method in the company, much more effort would be needed to create standards, guidelines and
other supporting materials.
11
PHASE 1 – CURRENT SITUATION
In this chapter the Keylane|Quinity RE method will be discussed. Section 4.1 begins with a
description of the meta-modeling process used to attain a complete view of the current method.
Section 4.2 describes the current business needs for method change (and preferences for
method chunks to keep), as elicited from experts in interviews.
4.1 META MODELING THE CURRENT METHOD
Elicitation of the current requirements practices began with an analysis of the company’s
documentation of the full implementation method. Figure 6 gives a high level overview.
Figure 6 - QIS implementation method
As outlined in the section 1.2.1 the scope of this project is limited to requirements engineering
leading up to the workshops. In figure 4 this includes the setup phase, requirements phase and
to a lesser extent the architecture phase (marked in a red box).
It quickly became clear from interviews that there were some discrepancies between the
documented method and reality. In some sections, the documentation was simply too generic to
accurately reflect reality. In other cases practices reported in documentation were used
inconsistently, rarely or in a completely unstructured fashion in real projects. The basis laid by
documentation was gradually expanded upon with experts in interviews, until eventually all
agreed that their piece of the puzzle was accurately represented.
The method as currently understood has been modeled in the Process Deliverable Diagram
meta-modeling notation, as mentioned in section 2.1.2. The concept and deliverable thesis can
be found in Appendix 2. The complete result is discussed in the following three sections.
12
4.1.1 Sales trajectory. The sales trajectory, though technically leading up to a project, already
provides several opportunities for early-requirements to be gathered. It is important to know
that all sub activities in this trajectory are optional and depend on the customer’s wishes. An
overview of the sales trajectory is given in figure 8.
Figure 7 - Keylane|Quinity Sales Trajectory
The sales trajectory usually kicks off with a Request for Information (RFI) and a Request for
Proposal (RFP). The request for information is a document sent out by prospective clients as a
broad net to assess software suppliers. The RFI typically contains a brief list of questions with
regard to general software capabilities, architecture, costs, contracts and the ability to interface
with legacy systems. Based on the RFI responses of software suppliers the prospective client will
decide whom to put on their short-list. The RFP will typically follow the RFI up with a long list
specific questions digging deeper into the product capabilities. Keylane|Quinity’s sales
department keeps track of these answers in an overview spreadsheet. Once the RFP has been
answered to the satisfaction of the prospective client, suppliers may find themselves on the
short-short list or with a definitive client.
RFI and RFP questions answered with a ‘no’ or similar may be indicative of a requirement,
because there is a customer wish not being covered by the product. The practice of deriving
preliminary requirements from RFI’s and RFPs is mentioned in the standard method
documentation. In practice however, interviewed experts agree that this is rarely or never done.
The prospective client may choose to book any number of additional information sessions with
potential suppliers. Contents of these sessions may include presentations, demos or even
13
business cases being executed in a test environment. During these sessions notes are taken by
Keylane experts, which may be a source for preliminary requirements.
After this initial phase has been passed successfully and the client has decided to become a
Keylane customer, they may want to set up a contract. In previous projects clients either sign a
preliminary contract for the setup phase (4.1.2), a full contract for an entire project, or no
contract at all (in which case an informal go-ahead is given).
In preliminary contracts, demo sessions and Proof of Concept (POC) sessions are both optional.
In a demo a generic walkthrough of QIS and its functionalities is shown. A proof of concept is a
specific variant that shows QIS performing a task in detail. The third contract component, a
global requirements estimate, is an optional short requirements phase. This thesis intends to
make that a permanent addition to all projects.
4.1.2 Project setup. In the setup phase (figure 6) more details of the project are set in stone.
The implementation method is explained to the customer, the project increments are defined
(see 4.1.3), and a planning of the project is created. The related deliverables will be made part of
the final contract. The project increments and planning also include the final cost estimation, so
explicit approval by the customer is needed at this point in order to continue.
Figure 8 – Keylane|Quinity setup phase
4.1.3 Requirements Engineering. When setup is complete the incremental phase can begin.
Increments consist of design, development, testing, and finally a go-live of the increment’s
results. Though not by-the-books agile, increments allow Keylane|Quinity to achieve some
flexibility in their projects without the shock and awe effect of a waterfall implementation. An
increment usually consists of one or more components of QIS, like policy administration, relation
administration or premium calculation. Other examples of possible increments are:
 Implement subsets of insurance products in each increment.
 Implement the front office first, then the back office.
 Implement based on channels (e.g. intranet first, then extranet, then internet).
 Implement one business line per increment.
All known requirements that are relevant for the current increment are collected and planned
for discussion in workshop style design sessions. First and foremost, the stakeholders that will
be participating in these workshops are trained. A preliminary training session prepares them
14
for the overall process, and additional trainings are given throughout to ensure all participants
remain up to speed.
Figure 9 - Keylane|Quinity design phase (start of the increments)
Known requirements are decomposed into their smallest parts, detail requirements, that are
used to inform design decisions. All detail requirements resulting from workshops are
documented, sent to the customer, and refined if necessary. The refined detail requirements are
processed in designs and captured in their own documents.
The next process step, the gap analysis, has been deemed out of scope for this project, but has
been modeled and elaborated nonetheless. For a PDD of the gap analysis see appendix 1. Simple
requirements that only need a configuration or process change are solved. The exact solutions to
actual gaps are discussed in more detail by functional and technical experts. Their designed
solutions are compiled into Request for Change (RFC) documents. After testing and a go-live
(WAY out of scope) the next increment begins.
15
4.2 BUSINESS NEEDS
Major business needs were derived from the method review section of interviews. The external
Coach (Principal consultant), a senior project manager, and the CEO of Keylane|Quinity reviewed
the content and priorities of these needs. After reviews some minor adjustments and additions
were made. The end result is captured in the table below.
Desire
Improving overall
effectiveness and
efficiency
Not missing gaps
(effective)
Not having to
literally gather all
requirements
(efficient)
Providing structured
increments
Make early
requirements
gathering more
formal
Make the method
independent of
individuals
Make the notation of
early requirements
more formal
Increase
interactivity in the
requirements
engineering
practices
Description
The overall goal for this project should be the improvement
of RE quality so that we don’t miss gaps (effectiveness), and
good use of time and resources by not exhaustively
gathering all requirements (efficiency)
Preferably there would be no more missed gaps. This
includes requirements uttered by the customer that were
not documented accordingly for subsequent stages of a
project. As indicated by the handbook ‘succes met de
requirements!’, costs of errors in the requirements solved
in a later phase exceed the norm anywhere between 10 to
200 times.
A mature software product satisfies much of the client’s
possible wants and needs already. Quinity does not want to
spend too much time on gathering and documenting every
requirement, when only a fraction of these will address a
novel need or gap.
QIS experts participating in the increments unanimously
prefer a structured approach. This structure should be
provided by project leaders by turning known
requirements into lists of ‘action points’ that guide
workshop discussion.
It has been indicated in interviews that the quality of early
requirements gathering varies from project to project.
Early requirements should be gathered formally and
consistently to support the desire for structured workshops
In the current situation Keylane|Quinity depends on a small
group of skilled individuals to do requirements research,
more so than they would like. In the future the company
wants to have a method that is more independent of it’s
practitioners. This can be achieved by providing good
method documentation and materials that teach or replace
skills.
In the current situation the way requirements and gaps are
written varies wildly from project to project. Which style is
used and which attributes are captured depends on who
leads the project. This obstructs traceability. In the future
situation there should be one standard base notation.
The current requirements engineering practices are mostly
of a ‘push’ or ‘pull’ nature. Push practices involve the
pushing of information in the form of presentations, demos
and/or trainings, where client experts have little room to
interact. Pull practices involve the client pulling in
information in the form of an RFI, RFP and/or Proof of
Concept case session, where Quinity experts have little
room to deviate from the determined path. A more
Priority
top
high
high
high
medium
medium
medium
medium
16
Consistently take
note of requirements
in sales sessions
interactive approach could potentially assist the need for
completeness, and could also save time if done correctly.
There are many possibilities for information sessions in the
sales phase of projects, including demo’s, PoC’s, use cases
and other presentations. Though technically the company
is still dealing with prospects in this phase, there is a lot of
information here that should not be lost once a contract is
signed.
medium
Table 1 - Keylane|Quinity business needs
17
PHASE 2 – METHOD ASSOCIATION APPROACH
In phase 2 we will discuss the results of the main method construction steps of MAA.
Section 5.1 reports on the identification of project situations. Project situations were identified
from various interviews held on location with Keylane|Quinity experts. Interviews were semi
structured, as reported in section 3.2 Data gathering methods.
Section 5.2 reports on the construction and evaluation of feature groups. Features were
gathered both from the interviews in section 5.1, and scientific literature.
Section 5.3 reports on the selection and preliminary evaluation of candidate methods. Candidate
methods were gathered from literature using a snowballing approach and searches performed
in the Google scholar search engine. The methods were then subjected to a preliminary
evaluation and modeled using a slightly simplified PDD notation.
In chapter 6 we present the method base and elaborate on the contents of candidate methods.
Chapter 7 begins in section 7.1 with a report on the association of candidate method fragments
with feature groups. Finally this chapter, and the report on the MAA construction steps,
concludes in 7.2 with the construction of a preliminary method based on association results.
5.1 PROJECT SITUATIONS
Project situations are classes of the same type of project that have distinctive features. Of course
each individual project is unique, but on a high level there may be broader characteristics at
play. Project situations were identified as part of the semi-structured interviews held with
Keylane|Quinity experts.
Most agreed that the biggest differences are caused by dominant or submissive clients. A
dominant client will push and pull for large amounts of information early in the sales trajectory,
which is why we term this situation front-heavy. A submissive client on the other hand only
wants to know the basics and trusts Keylane|Quinity to take care of the rest, which we refer to as
front-light.
The third project situation occurs when existing customers want to change or extend their
existing QIS implementation, an amendment project. In these projects the major source of
requirements becomes an existing implementation of Keylane|Quinity’s own product.
5.1.1 Front-Heavy. Front heavy projects are recognizable for their strong client initiative,
particularly during the sales trajectory. RFI and RFP documents are large, meetings are
numerous, and several Proof of Concept and/or demo sessions are booked.
Some noteworthy Front-Heavy projects are (anonymised for privacy purposes):


Company X: Demanded a large number of information sessions on six major categories,
spread out over a number of weeks. Each session included several of its own PoC
sessions. The global RQ analysis was not only performed by Quinity but also by the client
company itself.
Company Y. Sent out abnormally large RFI and RFP documents. In addition the RFP was
accompanied by an 80+ page requirements document provided by the client itself.
18
Opportunities for early-phase requirements gathering are abound in the RFI and RFP,
information sessions and PoC sessions. As a result, when the increments begin there is a laundry
list of known requirements to build the increment planning on. The Company X case even went
as far as making the formulation of complete RFC’s (change requests for generic solutions to the
standard QIS product) possible before the increments had even begun. This has obvious benefits
for overall project efficiency, as long as the gathering of requirements happens consistently.
Making the most out of the opportunities offered by a front-heavy project requires attention to
the early-phase requirements gathering and documenting processes
These projects are in clear contrast to the front-light project for Company Z, that deemed the
sales trajectory complete after two days of information sessions. Keylane|Quinity was so
deprived of knowledge at the time that an agreement was made with Company Z to do a oneweek requirements analysis project on location.
5.1.2 Amendment project. A finished implementation rarely equals a finished project.
Additional requirements popping up once the end-users get to work with a product are standard
fare for most software product companies. In some cases, particularly soon after new
implementations, it can be more efficient to handle a batch of new requirements as a separate
project. In addition, the modular design of QIS makes it relatively easy for existing customers to
have extra modules implemented. An extra project to process more requirements, or to
implement additional modules, is known as a ‘wijzigingsverzoek’ (amendment-request).
In both cases these projects are different from front-heavy and front-light projects. The client is
already working with an implemented version of QIS, which forms the primary source of new
requirements. In addition amendment projects differ in the following ways:





Stakeholders are familiar with Keylane experts.
Stakeholders are already familiar with parts of QIS.
Stakeholders only need module-specific training, or possibly none at all.
Stakeholders are familiar with Keylane|Quinity’s way of working.
Stakeholders are familiar with the implementation method.
5.2 FEATURE GROUPS
In this chapter the feature groups and their individual features are explained. A feature is an
attribute that a method should have, a capability that a method should provide, or an artifact a
method should deliver.
5.2.1 Feature group sources
Keylane experts. Some features have been identified in the interviews with Keylane staff. These
features include current capabilities and artifacts at both Keylane|Quinity and
Keylane|LeanApps, as well as desires for the method expressed by Keylane|Quinity staff and
management. In the feature groups excel table features that are fully present in the current
method are marked with an ‘X’. Features that are partially present or too inconsistent to
consider fully present are marked with a ‘/’ symbol.
Success met de Requirements! Success met de Requirements! (SmdR!) is the book used by
Keylane|Quinity as a source for their current requirements engineering practices [8]. SmdR!
provides a broad overview of the most common phases known in the RE domain, and discusses
19
common practices in each phase. The book has not been followed to a tee by Keylane|Quinity, so
some of its recommended practices are not (yet) implemented.
Engineering and managing software requirements. This 487 page book on RE is divided into
three parts [7]. The first is a study of the state-of-the-art in RE, divided into typical RE process
phases. The second is a study of up-and-coming practices. The third concludes with five case
studies of RE in practice. The book was compiled and edited by A. Aurum and C. Wohlin, and
chapters were contributed by various experts in the field.
SWEBOK Version 3. The third edition of the Guide to the Software Engineering Body of
Knowledge (SWEBOK) is an attempt by the IEEE computer society to standardize knowledge on
software engineering [9].15 knowledge areas in software engineering are covered and made
open to comment from both academic and industry experts. The RE knowledge area was
particularly interesting for this project.
Maturity Models. (Capability) maturity models do not just catalogue practices and features, but
add some nuance in the form of maturity levels. For example: noting down requirements in
natural language and in heavily formalized language have the same goal (getting requirements
on paper), but the latter is considered more mature than the former. The four maturity models
used for identifying features are the Capability Maturity Model Integration for Development
(CMMI-DEV) [24] by Carnegie Mellon University, the Requirements Capability Maturity Model (RCMM) by Beecham et al. [25], the Requirements Capability Maturity Model integration (R-CMMi)
by Solemon et al. [26], and the Unified Requirements Engineering Process Maturity (Uni-REPM)
model by Svahnberg et al. [27].
5.2.2 Feature group relations. In the domain of requirements engineering practices 13 feature
groups were identified. There are three types of features collected:



Keylane|Quinity’s desires for a method as outlined in chapter 4.2. These desires play a
crucial role in the project and meeting it’s goals.
Generalized practices. These practices were taken from literature and generalized when
necessary. For example: ‘discuss requirement in workshop’ is a generalization that
makes no assumptions with regard to workshop styles and structures.
Specific practices. Some specific practices have been included because they were deemed
particularly interesting for Keylane|Quinity. For example: ‘Discuss results in a JAD
workshop’ is a specific practice in which we left the prescribed structure.
The last two types of features were verified with the external coach for their applicability for
Quinity in general, the applicability to the expert’s function , and the expert’s interest in the
feature.
It is important to note that for the field of RE, feature groups are not independent. In the work of
Luinenburg et al. [5] it was possible, for example, to make a clear separation between authoring
features and personalization features of web content management systems. In this project there
is no such separation, as many RE practices are related in several ways. Requirements Elicitation
for instance, always feeds into Analysis, an analysis of Requirements interdependencies adds to
the more general Analysis, and Validation scrutinizes the practices performed in Specification.
Requirements management and Process Quality Assurance apply to all other feature groups,
20
including each other, and traceability follows the central flow of requirements from elicitation to
verification. For example: the feature ‘discuss requirements in a work shop can occur in
 Elicitation – the workshops are used to identify new requirements
 Analysis – the workshops are used to further elaborate on known requirements
 Specification – the workshops are used to drill down to the final form of requirements
We propose to make a distinction between these two different classes of methods in MAA
projects2:
 Product related MAA. When engineering a product related method, the final purpose of
that method is to create a product. In the case of Luinenburg et al. the MAA was used to
create a method specifically for the development of a web content management system.
 Process related MAA. When engineering a process related method, the final purpose of
that method is to define a process. In this project the MAA is used to create a
requirements engineering method first and foremost.
Figure 10 - Visual representation of feature group relationships
5.2.3 The feature groups
In this chapter we briefly introduce the feature groups, their names, and some examples of
features within them.
Requirements Elicitation. Elicitation is the process of learning, uncovering, extracting,
surfacing or discovering of needs from customers [28], but also documents (referred to as
extraction) [29] and other available sources. Common features include practices like workshops
and attributes like copious stakeholder interaction.
Requirements Analysis. Analysis is the process of investigating the problem domain and
requirements in order to develop a better understanding of stakeholders’ actual goals, needs and
expectations [7]. Typical features include practices like filtering early requirements and
The same conclusion was reached by Deneckère et al in later versions of their paper. The first they call a
‘product perspective’ and focuses on the deliverables of candidate methods. The second they call a
‘process perspective’ and focuses on the process fragments of candidate methods.
2
21
elaborating them for future use. Some sources recognize the Organizing of requirements as an
entirely separate set of activities [30], but we consider it part of analysis for brevity.
Requirements Interdependence. Interdependencies between requirements can take several
forms. Carlsharmre et al. recognize the following: AND (a group of requirements needs to be
satisfied entirely), REQUIRES (one is needed to have the other), TEMPORAL (one is needed
before the other), CVALUE (one disrupts the value of another), ICOST (one increases the cost of
another, or the entire project) and OR (at least one of a group needs to be satisfied)[31].
Common features are practices like mapping dependencies and creating an interdependence
map. Understanding interdependence can be considered part of the Analysis feature group, but
we consider it a separate set of activities because of Keylane|Quinity’s interest in this group.
Impact Analysis. When a standard product is being sold, novel requirements can lead to
solutions with a certain impact on that product. Adding a feature will likely have an impact on
other features [32]. Impacts can range from small configuration changes to a cascading effect
[33] that causes massive changes throughout the entire product. Typical features include doing
an analysis, and attributes like enabling insights into those cascading effects. Like
interdependence, impact analyses can be considered a part of the broader Analysis feature
group. We consider it a separate set of activities because of Keylane|Quinity’s interest.
Requirements Negotiation. Not all requirements represent the needs of all stakeholders.
When conflicting interests arise, it is up to the requirements engineer to help negotiate between
parties [34]. Typical negotiation features include practices like the search for mutually beneficial
solutions, and attributes like enabling concession making. Requirements negotiation can be
considered a separate feature group but may also occur during the elicitation or analysis phase.
Requirements Specification. Specification refers to the formal documentation of requirements
[35]. Of course requirements are also documented during earlier stages in the form of notes, but
in specification their notation becomes definitive. Features include practices like the use of
formalized languages, and artifacts like requirements specification documents.
Requirements Validation. Requirements validation is a class of practices for testing the quality
of specifications for the clarity, completeness, and corectness of their contents [36]. Typical
features include practices like proof reading of the documents and attributes like document
consistency.
Requirements Prioritization. When deciding which requirements to implement first,
prioritization is key [37]. Many practices and techniques exist within this domain with varying
degrees of formalization. Some famous example practices are MoSCoW [38], cost-value analysis
[39] and the 100 dollar test [40].
Release Planning. Release planning is a group of practices specific to product software. After
prioritization requirements are planned for development in a release of the standard product
[41]. Whether requirements become part of the next release or a later release depends on
factors that were used for prioritization, such as the value of a solution to clients and the
expected development costs. Aside from the actual planning of requirements into releases,
typical activities include evaluating release plans and aligning them with the product roadmap.
22
Requirements Verification. To conclude the software development process, a company might
want to go back to the original requirements specifications to see if they were actually fulfilled
[42]. This also often referred to as acceptance testing [43]. Typical practices include product
reviews by stakeholders and attributes include ensuring requirements fulfillment.
Requirements Traceability. Traceability is a set of activities that assist the tracking of
requirements during all major steps of their lifecycle, both within projects and between [44].
This includes tracking their elicitation, analysis, specification, prioritization and verification.
Typically traceability also covers the documentation of relationships between requirements
specifications and code changes, but that lies outside the scope of this project. Typical practices
include requirements version tracking and coupling requirements to other data (people,
meetings, decisions, deliverables, success factors, system tests etc.)
Requirements Management. Requirements management is a set of overarching activities that
has influence in all steps of a requirements lifecycle [7]. Typical practices include keeping
requirements traceable for future projects and planning all the other activities of a requirements
trajectory.
Process Quality assurance. Unlike validation, which is exclusively concerned with the quality
of specifications, Quality Assurance is concerned with the quality of both processes and
deliverables in all stages of RE and development as a whole [45]. So as not to overlap with
features under validation, the decision was made to limit the quality assurance feature group to
processes alone. Hence the name Process Quality Assurance. Typical practices include ensuring
compliance to protocols and establishing error handling processes.
5.2.4 Feature group evaluation
Feature groups and their individual features were evaluated with the help of the external coach;
a senior consultant at Keylane|Quinity. In the first three evaluation rounds features were
classified according to the following decision tree:
Fulfilled
Features
Feature
Partially
fulfilled
Should be
Not fulfilled
Might be
Won’t be
Figure 11 - Feature classification decision tree
23
Features deemed ‘fulfilled’ are already sufficiently satisfied by the current method. Features that
were considered partially fulfilled needed improvement. Features that were not fulfilled were
divided into an additional three categories. ‘should be’s were deemed necessary additions for
the future situation. ‘Might be’s garnered some interest, but more information was needed to
decide their fate. ‘Won’t be’s were deemed too advanced, unnecessary, or out of scope for this
project.
In the final evaluation round all features deemed ‘might be’s were discussed in more detail, and
a final decision was made on which features to keep. Their grouping was also refined in the final
evaluation session from the general RE process steps in the previous chapter to more specific
sub-groups (e.g. ‘Requirement status tracking’ or ‘involve stakeholders in analysis’ .
Evaluation round 1
Discussed 91
Added 1
Changed 14
Removed 11
Fulfilled
Partially fulfilled
Should be
Might be
Won’t be
31
32
3
7
6
Total features 312
before
Total features after 301
Evaluation round 3
Discussed 45
Added 0
Changed 6
Removed 18
Fulfilled
Partially fulfilled
Should be
Might be
Won’t be
15
4
2
5
0
Evaluation round 2
Discussed 26
Added 0
Changed 4
Removed 2
Fulfilled
Partially fulfilled
Should be
Might be
Won’t be
10
3
1
8
2
Total features 301
before
Total features after 299
Final evaluation
Groups merged 2
Major groups 7
Sub-groups defined 34
Features before 150
Merged/removed 37
Total features final 114
Total features 299
before
Total features after 1503
Table 2 - Feature group evaluation results
Due to their being out of scope, the feature groups Release Planning, Traceability, Verification and
Process Quality Assurance were excluded in their entirety.
3
24
Interdependence and Impact Analysis were initially recognized as separate feature groups from
Analysis due to company interest, but after evaluation so few features remained that they were
merged back into Analysis. For the full final collection of features, please refer to appendix 3.
25
5.3 CANDIDATE METHOD SELECTION
The initial strategy for method selection was to immediately continue investigation of
candidates that fulfilled the above criteria based on a personal qualitative assessment. After
feedback from the external coach, the decision was made to first create a pre-selection of
methods and elicitation techniques if they:





Provide value for a product software supplier.
Are suitable for b2b interactions.
Are suitable for a mature, complex product.
Are compatible with a focus on keeping the product standard.
Have sufficient documentation available to get a detailed understanding of fragments.
The full list consisted of 25 elicitation techniques, 4 workshopping methods and 15
requirements engineering methods. Another 15 methods were investigated but rejected for not
adhering to the selection criteria closely enough.
The list was then evaluated with the external coach. For the methods we covered each fragment
individually. The external coach used a more subjective selection approach, based on personal
experience, but some distinct criteria were used:



Compatibility with incremental approach. Keylane|Quinity is not willing to radically
change their incremental development approach, so method fragments that rely too
strongly on very agile or very ‘waterfall’ project structures were almost always rejected.
Compatibility with natural language notation. Keylane|Quinity strongly prefers
specification of requirements to be in natural language, because this enables easy
communication with stakeholders. Method fragments proposing the use of model
notations or formal grammars had to provide significant value in order to be accepted.
Performs under time constraints. Requirements work would be performed on the
client’s dime, so it has to be quick and to the point. Any method fragment with a slow
start or duration of a full week or more was rejected.
With sister company Keylane|LeanApps’ method included, the final candidate method base
contained 8 methods.
5.3.1 Candidate method modeling. At the beginning of this chapter we mentioned that an
adaptation of the PDD notation would be used. This decision was made in collaboration with
both coaches when the modeling process proved to be very time consuming. The following
changes were made:


Only relevant fragments are modeled. Modeling entire methods when we know that
only a subset of their fragments are interesting to us is not efficient. Some additional
fragments were modeled if they provide valuable context to relevant method fragments.
No modularity included on the deliverable side. In the standard PDD notation all
relations between deliverables include modularity (see 4.1 Meta modeling the current
method for an example). The modularity of a relationship is often very obvious to
experienced readers and does not contribute any new information. In addition, the
scattering of small script numbers across large models was deemed to have a negative
impact on the overall readability of those models.
26
5.3.2 Candidate methods overview. In this chapter we provide a brief overview of selected
methods in the table below.
Method name
Technique selection (modified version
based on the work of Carizzo et al.)
Year
2014
Main Source
Carrizo, D., Dieste, O., & Juristo, N. (2014).
Systematizing requirements elicitation technique
selection. Information and Software Technology,
56(6), 644-669.
Joint Application Development (JAD)
1995
Wood, J., & Silver, D. (1995). Joint application
development. John Wiley & Sons, Inc..
Requirements Engineering with
SCenarios for a User-centered
Environment (RESCUE)
2005
Jones, S., & Maiden, N. A. (2005). RESCUE: An
integrated method for specifying requirements for
complex socio-technical systems. Requirements
Engineering for Socio-Technical Systems, 245-265.
Good Enough Architectural
Requirements (GEAR)
2005
Schwanke, R. W. (2005, November). GEAR: A
Good Enough Architectural Requirements Process.
In WICSA (pp. 57-66).
Tropos
2004
Bresciani, P., Perini, A., Giorgini, P., Giunchiglia, F.,
& Mylopoulos, J. (2004). Tropos: An agent-oriented
software development methodology. Autonomous
Agents and Multi-Agent Systems, 8(3), 203-236.
Security QUAlity Requirements
Engineering (SQUARE)
2005
Mead, N. R., & Stehney, T. (2005). Security quality
requirements engineering (SQUARE) methodology
(Vol. 30, No. 4, pp. 1-7). ACM.
KJ+(extension of Jiro Kawakita’s
affinitization technique)
2014
Mead, Nancy; Konrad, Michael; & Stoddard, Robert.
“Eliciting Unstated Requirements.” Tutorial
presented at the 22nd IEEE International
Requirements Engineering Conference. Karlskrona,
Sweden, Aug. 26, 2014.
Keylane|LeanApps
2015
Tapkhirwala, P. – Director of Engineering. Personal
interview (2015, January 13).
Baars, K. – Product Manager. Personal interview
(2015, March 4).
Table 3 - Candidate methods overview
For an overview of eliminated methods please refer to Appendix 4.
27
6 METHOD BASE
In this chapter we describe the candidate methods included in the method base. For each we
briefly discuss the rationale, relation to business needs, history, contents, points of interest and,
if available, a case study. All PDDs mentioned are included in the attached folder.
6.1 METHOD – ELICITATION TECHNIQUE SELECTION.
Several methods for the selection of elicitation techniques exist. Initially we analyzed the
ACquiring REquirements (ACRE) framework which was selected for its reported adaptability to
project situations[46]. The ACRE framework was envisioned by Maiden and Rugg in 1996 to fill
a perceived gap in knowledge on the selection of elicitation techniques in software development
projects. Twelve technique are rated against six project attributes in a series of reference tables.
Selection is achieved by looking up the project attributes for your current project and looking up
the highest scoring techniques in those tables. Some project attributes didn’t make much sense
however. One example attribute asks whether you’re looking for implicit, semi-tacit or tacit
knowledge, which is often impossible to know up front. ACRE was eventually abandoned.
After snowballing by looking up references to ACRE another candidate method was identified.
The Methodology for Requirements Engineering Technique Selection (MRETS) was developed
by Jiang, Eberlein, Far and Mousavi [47]. MRETS contains a total of 46 techniques for the entire
requirements engineering process. Each of these techniques is given 36 attributes, which are
matched to 21 project characteristics. In addition the method proposes the use of clustering
algorithms to identify related techniques, and the use of an ‘objective function’ to make a
complex mathematical cost/value analysis. Though probably very complete, MRETS was also
deemed far too complicated to be practically useful.
In addition to MRETS, a third technique selection approach was identified in the snowballing.
Tsumaki and Tamai developed the ‘Framework for Matching Requirements Engineering
Techniques to Project Characteristics and Situation Changes’ with no catchy acronym provided,
so for the sake of brevity we’ll call it the TT framework. A total of six elicitation techniques are
classified on a two dimensional matrix:
 Static/Dynamic (systematic techniques for unchanging information vs. loose techniques
for dynamic information)
 Open/closed (the domain is stable and already quite well known vs. the domain is
unstable and unknown)
Six binary questions determine the current project’s position in the matrix allowing you to select
the closest technique. The paper does not actually report how methods are classified in the
matrix, particularly how methods can be placed in the grey areas between dimensions (e.g. 80%
dynamic, 40% open). Similarly it does not explain how binary questions can result in a position
in grey areas between the dimensions’ extremes. Though theoretically interesting, the TT
framework is also not practically useful
An attempt was made to consolidate these three methods into one single super method, but the
three proved too different to find any harmony between them. For more on our failed attempt
refer to chapter 10.1.
28
It wasn’t until much later that the fourth elicitation technique selection approach was identified.
An unnamed framework by Carrizo, Dieste and Juristo [48] was based on a review of other
technique selection methods (both for requirements elicitation and knowledge acquisition), and
sought to address some of the concerns in the other three approaches. The framework most
closely resembles ACRE by comparing fifteen techniques to sixteen project characteristics in a
reference table. This framework also recognizes that a project situation can be changed, albeit
sometimes at a significant cost for the eliciting company (e.g. when there is no experienced
requirements engineer available, an inexperienced engineer can be trained). It is by far the most
practically usable method available, despite some validity concerns with regard to the
information in the reference tables. Some additions and changes were made to it’s technique
base to better suit Keylane|Quinity’s needs. The full list of included techniques and their
descriptions is included in table 4. Additions and removals and other considered techniques
(and their motivations) are included in table 5.
Technique
name
Unstructured
interviews
Structured
interviews
Timed
elicitation
Laddering
Brainstorming
JAD
KJ technique
Observation
Protocol
analysis
Storytelling
Writing use
cases/
scenarios
Apprenticing
Questionnaires
Suggestion box
Description
Interviews with individual- or small groups of- stakeholders, without a pre-determined
protocol. Foster an open, conversation style response.
Interviews with a pre-determined set of speaking points and questions. Foster a Q&A style
response.
Interviews where the interviewee is asked to name a certain amount of concepts within a
set amount of time (e.g. your five main goals in three minutes, three critical requirements in
three minutes).
Ask a user pre-determined questions to elicit general goals, tasks or needs. Drill down on
answers (using why, what, when and how questions) to establish a hierarchy of knowledge
A session with a medium- to large group of stakeholders with broad exploration as a purpose.
Participants are usually presented with only one open question that needs answering.
A session with a medium- to large group of stakeholders, with structured discussion as a
purpose. Various sources provide guidelines with regard to room layout, use of materials,
and roles of participants.
A session with a medium group of stakeholders, with concept filtering and sorting as the
shared main purpose. Stakeholders create sticky notes based on one main question (e.g.
‘What are your crucial tasks?’). Stakeholders are then asked to collaboratively filter,
analyse, and group the full collection of sticky notes.
The Requirements Engineer observes an end user performing a pre-determined task or case.
The Requirements Engineer observed an end user performing a pre-determined task or
case. The end user speaks aloud about their actions and decisions.
An end user gives a natural language description of their role in a task or case. Less time
consuming than protocol analysis and does not need access to existing systems. Results
may be incomplete because there is no opportunity to witness tacit knowledge in action.
A scenario is defined as ‘a specific sequence of interactions with a system’. A use case is
defined as ‘a collection of scenarios occurring between a system and external actors,
characterized by the goal that the primary actor has toward the systems declared
responsibilities, showing how that goal might be delivered or might fail’ (A. Cockburn
’95). Both can be analyzed for requirements.
The requirements engineer analyzes training materials and/or participates in training sessions
that are intended for client employees.
A list of questions is sent out to any number of stakeholders. Questions may target
anything from a broad domain to a specific functionality.
A repository where end users and stakeholders may share their complaints, concerns, or
desires for the future system.
Table 4 - Final technique selection framework techniques
Technique name
Timed elicitation
Action
Added
Motivation
Though likely to miss many requirements, its a short and easy technique that
can be a valuable addition when time constraints are high.
29
KJ technique
Added
Storytelling
Added
Apprenticing
Added
Repertory grids
Removed
Nominal group
technique
Removed
Delphi technique
Removed
Focus groups
Removed
Prototyping
Removed
Ethnography
Rejected
Card sorting
Rejected
Analytical
Hierarchy Process
(AHP)
Rejected
Problem framing
Rejected
Brainstorming
Rejected
Abuse-/Misuse
cases
Rejected
Future workshop
Rejected
The KJ technique is a structured form of workshopping that hands over a lot
of work to the stakeholders and does not necessarily require an experienced
facilitator. This made the technique very appealing to Keylane|Quinity.
Observation, protocol analysis and apprenticing can be very time
consuming, and require access to current systems. Storytelling might not be
as effective nor reliable, but works better under time constraints.
An intimate knowledge of client processes can’t always be learned from
interviews and workshops. Though potentially time consuming, apprenticing
was deemed interesting enough for inclusion.
Repertory grids are used to rate concepts against attributes. It’s popular for
COTS selection, but this technique has no uses in implementation projects.
The goal of NGT is to find solutions for problems collaboratively with
stakeholders. Keylane|Quinity wants to avoid solution-based thinking in
these early phases and focus purely on requirements.
Similar to NGT, the goal of the Delphi technique is to find solutions. This is
achieved by exposing stakeholders to solutions created by other
stakeholders, hoping that this exposure will cause them to adapt and reach a
consensus. This technique was removed for the same reason as NGT.
Reportedly a workshop for reviewing products and services, but there is not
enough context provided in written materials to distinguish it from JAD or
KJ style workshops. Focus groups appear to be more of a concept than a
technique.
Keylane|Quinity’s complex software is not conductive to (rapid)
prototyping. This is a concern that may be addressed in the future, but
prototyping now is simply not viable.
Ethnography entails shadowing a client’s end-user for a longer period of
time. This technique was deemed to time consuming to justify it’s value. A
structured variant like protocol analysis or observation where a stakeholder
is given a specific case is much more effective.
A stakeholder sorts cards with concepts on them into either self defined
groups (open card-sort) or predefined groups (closed card-sort). Though
interested in the playful concept, the external coach saw no situation in
which this technique would add value for Keylane|Quinity, as it is mostly
intended for product selection. Not development.
Using interviews, (weighted) criteria are derived from goals. Potential
solutions are then rated against these criteria. Since Keylane|Quinity
maintains a standard product, they decide on solutions, making AHP
obsolete.
Using interviews, the perceptions and underlying reasons for the perceptions
of problems by different stakeholders are elicited. Seemed promising but
very little material exists to actually explain the elicitation of perceptions
and their underlying reasons.
Put stakeholders in a room and ask them to generate ideas, with very little
intervention. Rejected for it’s ‘loose canon’ type approach. When offering a
complex software product in a well known domain, exploratory techniques
like brainstorming are less efficient than it’s more structured alternatives.
A variant of use case specification where a workshop is held with the sole
purpose of thinking up ways to break or break into a system. Considered too
specific to the field of cyber security to be of use in implementation projects.
A workshop in which stakeholders are allowed to fantasize about their ideal
future situation. Depends to heavily on stakeholders’ ability to articulate
their desired future situation. In Keylane|Quinity’s experience their product
already contains advanced features that stakeholders wouldn’t even think of.
Table 5 - Technique selection framework changes
Using the adapted reference tables, a simple decision support system was constructed in
Microsoft Excel. An example situation with techniques and scores is included below in Figure 12.
This system was later dismissed in the first evaluation round. It was perceived as being too
complicated still, and would take away too much trust from the expert judgement of experienced
30
Keylane employees. If interested, the original Excel workbook is included in the attached folder.
The alternative course of action is further described in Phase 3.
Figure 12 - Elicitation technique selection decision support
6.2 METHOD - JOINT APPLICATION DEVELOPMENT
Joint Application Development (JAD) was selected for its resemblance to the global RQ analysis
which Quinity implemented successfully in a previous project (see Phase 1 – Current situation)
making it potentially easy to introduce with little resistance. JAD’s greatest reported benefit is
the facilitation of thorough and precise requirement specifications [49]. JAD could help
Keylane|Quinity achieve the following business needs:




Improved effectiveness.
Improved efficiency
Providing structured workshops
Increasing interactivity
JAD, originally named Joint Application Design,
was envisioned in the ‘70s by IBM systems
engineer Arnie Lind as a means of speeding up
the research phase of IBM’s software design
projects. One of the leading practice guides for
JAD was later published by Jane Wood and
Denise Silver in 1995 [50]. Today there are
many articles, books and whitepapers on JAD
available.
Figure 13 - Example JAD room layout prescription
Meta modeling perhaps doesn’t do JAD as much justice as it deserves. The defining feature of JAD
is not any of the processes or the deliverables, but the environment it seeks to create during the
sessions. JAD is intended to be as interactive as possible and attempts to do so through room
layout and resources [51,52,53]. Participants sit along U shaped tables [52,53]. This is intended
to give every participant a clear view of his colleagues and keep the attention on the speaker at
the front of the room. Other authors have done away entirely with rooms and instead opt for
virtual meetings through groupware [54] but classic JAD in a room still appears to be the most
popular.
31
Possible Speakers include the facilitator, project leaders and end user representatives sitting in a
circle around him/her [52]. When necessary the position can also be taken by an executive,
consultants, domain expert or end user [51,55]Scribes/documenters only take notes and have a
permanent position near the front of the room [51]. At the back of the room there should be
room for observers, and a table of refreshments
[52,55]. Most sources recommend making
plentiful space available for flipcharts,
whiteboards, blackboards, beamers, and
overhead projectors[52,53,55]. In addition
some recommend bringing in plentiful notepads
and post-its and one even stresses, perhaps
telling of the time, the importance of bringing a
Polaroid camera [55]. Some also recommend
the permanent presence of a desktop/beamer
combination with modeling software,
requirements engineering tools and other CASE
tools running and ready at all times [53,54,56].
The second defining feature of JAD is its clear
role structure:






Facilitator. An experienced and socially
apt expert that will lead the sessions.
Facilitators should be objective,
unbiased and good at conflict
resolution.
Project leaders/Lead developers/Lead
designers. An expert that leads a team
and is knowledgeable about (part of)
the product. He/she is capable of
judging requirements and estimating
their design and/or development costs.
Executive sponsor. A client executive
who champions the product within the
client company. Should have the clout to
free up the calendars of other client
participants. Is not required to be
present at every session but should
occasionally be present to show
support.
End users. Those that are directly
Figure 14 - JAD meta model
affected by the implementation.
Experienced end users in leadership roles are preferred.
Domain experts. Highly specialized individuals, both from the client and supplier
company, that can be brought in on an invitation basis to discuss their field.
Scribe/Documenter. Exclusively responsible for taking notes and making models during
the sessions. There so others don’t get distracted by having to take notes.
32

Observer. People who sit in to listen, but are not supposed to contribute, since too many
voices may cause confusion.
JAD leaves some points open for interpretation. An example mentioned earlier in this section is
for instance the duration of sessions.
 The traditional JAD session duration is considered to be several days of full-time lock up
[52,53], though in practice this often proves to be unfeasible
 JAD does not propose a structure for session contents. What to discuss and how openly
to discuss is for the practitioners to decide.
 Templates or structures for deliverables are not included. How to document results is
left to the practitioners.
In 1998 Davidson [57] performed a series of 20 case studies on JAD sessions by three different
companies. Only 40% of projects were performed using the traditional full-time lock up during
several days. 45% of projects actually used half-day sessions, and 15% limited sessions to ~2-3
hours. In 60% of projects sessions were spread out across a period of several weeks rather than
on consecutive days. Davidson reports that there is a possibility that the case companies might
not be getting the most benefit out of JAD for exactly those reasons, but his findings still show
reported improvements: “The findings of this exploratory study suggest that the three research
sites realized modest improvements in systems development processes and outcomes by using the
JAD method. Informants reported improvements in requirements definition, IS efficiency, mutual
learning between IS and users, and so on in 10-30% of the projects examined.”
6.3 METHOD – RESCUE.
Requirements
Engineering with
SCenarios for a Usercentered Environment
(RESCUE) was
developed by the
Centre for HumanComputer Interaction
(HCI) design at the City
University of London
[58]. The method was
developed for the
European organization
for the safety of air
navigation
(EUROCONTROL). The
method was initially
selected for its reported
focus on domain
analysis. After the
initial analysis it was
Figure 15 - RQ management phase of the RESCUE meta model
33
revealed that this part of the method consisted exclusively of I* modeling activities, and didn’t
prescribe an actual analysis as promised. RESCUE does however come with a very well
documented requirements management phase, and though this is not explicitly part of the
business needs in chapter 4.2, it may still have some valuable contributions to the current
practice. Requirements management in RESCUE consists of the following:









Specify requirements in the Volere shell (by Volere, part of the Atlantic Systems Guild).
Determine the requirement type (from a typology).
Relate requirements to use cases.
Writing fit criteria using SMART goals.
Decompose requirements into granular parts.
Chart interdependencies between requirements.
Manage the status of requirements.
Track changes in requirements.
Checking the quality of requirements.
All of RESCUE’s middle section is unfortunately unusable for Keylane|Quinity. The use case
modeling and scenario walkthoughs phases are not relevant, since Quinity will not be adopting
use case based RE. In additon, the impact analysis phase of RESCUE is simply too rigorous. The
creative design workshops phase was also excluded from the method since creative workshops
are better covered by ACRE and KJ+.
A series of case studies of RESCUE during its implementation were published in later years, both
with Neil Maiden as first author [59,60]. It appears no final report after the implementation of
RESCUE was ever written.
6.4 METHOD - GOOD ENOUGH ARCHITECTURAL REQUIREMENTS.
Good Enough Architectural Requirements (GEAR) was selected for its focus on architecture.
Keylane|Quinity also uses the architecture of their software as a reference, making GEAR very
compatible. GEAR could help Keylane|Quinity achieve the following business needs:





Improved effectiveness
Improved efficiency
Provide structured workshops
Make early phase requirements gathering more formal
Optimize session abstraction
GEAR was originally developed at Siemens Corporate Research, inc. by Robert Schwanke [61].
GEAR is an incremental method and aims to bring together concepts from model-driven RE,
Quality Attribute Scenarios and Global Analysis. The ‘good enough’ in the name comes from the
statement of the author that the increments should simply end when experts believe that “the
information is good enough to make the necessary decisions”. This is similar to thee often used
80/20 rule in agile development [62]. Globally GEAR contains the following increments:


Stakeholder analysis increment. Create a use case context model, identify stakeholders
and enough use cases to differentiate between them.
Quality Attribute increment. Engineers estimate likely quality attribute scenarios (QAS),
stakeholders are gathered in a workshop to review estimated QAS, the project plan and
34




the current system architecture. Stakeholders’ most important concerns are captured as
new QAS.
Global analysis increment. Engineers identify scenarios and capture them as UML
behavior models. A global analysis team analyses scenarios and QAS to identify
influencing factors between the two. Also catalog factors, issues and strategies for
communicating them to stakeholders.
Architecture principles increment. Architecture experts are brought in to relate required
functionalities to the system architecture and create an architecture principles document
to explain how they will be mapped to the existing system architecture. Can be reviewed
with stakeholders in an ATAM session.
Functional componentization increment. A decomposition of the software architecture is
used to specify the modules and components functionalities will become a part of, or
which new ones will be created.
Interfaces increment. A coarse grained specification is made of external and internal
interfaces of modules including all functionalities accessible through those interfaces.
Done when all use cases can be mapped to the architecture.
The first two phases of GEAR are considered irrelevant. The current practices of identifying
stakeholders suffice, and quality attribute scenarios are explicitly focussed on non-functional
requirements, which is out of scope for this project. The global analysis increment remains
useful if quality attribute scenarios are replaced with functional requirements. A particular area
of interest is the architecture principles increment, where functionalities are mapped to a
system architecture. This could allow Keylane|Quinity to easily visualize which requirements are
in and out of the scope of QIS by relating them to QIS’ architecture models.
The componentization increment was included for the sake of a complete view of GEAR, though
it is deemed unlikely that a single project’s requirements will ever result in a new module for
QIS. The mapping of functionalities to existing modules would simply create a more detailed
mapping than the previous increment.
35
Figure 16 - GEAR meta-model
The interfaces increment is, to a certain extent, already present in the current process. External
and internal module interfaces are always part of the goal architecture sketch that is created as
part of the workshops.
36
There are no case studies of GEAR available, though one would assume it was implemented at
Siemens Corporate research.
6.5 METHOD - TROPOS
Tropos was included for its introduction of a system as an actor in a domain. Similar to the
architectural approach of GEAR, this allows Keylane|Quinity to easily visualize which
requirements are in and out of scope for their QIS product in implementation projects. Tropos
can help Keylane|Quinity achieve the following business needs:





Improved effectiveness
Improved efficiency
Provide structured workshops
Make early phase requirements gathering more formal
Optimize session abstraction
Derived from the Greek homonym meaning ‘way of doing things’, Tropos is a broad software
engineering method resulting from the similarly named Tropos Project [63]. The project is
backed by six universities and has over twenty academics actively participating. Tropos depends
heavily on the i* modeling method, which was also seen earlier in RESCUE. To overcome some of
i*’s weaknesses a formalization (Formal Tropos notation) and accompanying validation tool
were made. For brevity only the requirements phase of Tropos will be described here:





Interview stakeholders (to elicit roles, goals, tasks and dependencies on others. Also
elicit sub goals, soft goals, methods used and resources used).
Create I* actor diagram(I* notation, formalized as formal tropos (FT) notation)
Create I* goal model (i* actor models merged including dependencies between actors.
Dependencies are shared goals, tasks and resources)
Introduce the system actor (model the system as an actor in the dependency model with
all its goals, tasks and resources)
Rework existing goals (change other actor goals as a response to the system)
After these steps Tropos delves further into architectural design, detailed design and eventually
implementation, but those steps lie outside of the scope of this project.
One of the major weaknesses of I* remains unaddressed: reliance on expert input and judgment.
What level of abstraction to model on, how to check for completeness, when to stop gathering
soft goals, and how many sub-goals to identify are all left to expert judgement. The FT notation
makes some headway in overcoming these, but the resulting models still appear highly
subjective. Aside from that, Tropos is particularly good for charting interdependencies between
actions, goals and actors.
Tropos has been a part of various case studies, either in parts or as a whole. Some of the most
prominent examples are: [64,65,66].
37
Figure 17 - Tropos meta-model
6.6 METHOD - SECURITY QUALITY REQUIREMENTS ENGINEERING
Security QUAlity Requirements Engineering or SQUARE was selected for the creation of a
common terminology with the client and developing supporting artifacts for the remainder of
the project. Other method steps are deemed out of scope. SQUARE can help Keylane|Quinity
achieve the following business needs:


Make early-phase requirements engineering more formal
Increase interactivity in the requirements engineering practices
38
SQUARE was developed by the Software Engineering Institute (SEI) in 2005 [67]. It was
intended to be a method for the elicitation, analysis and prioritization of security related
requirements in IT systems. Not to be confused with the 2011 ISO standard for Software Quality
Requirements and Evaluation (SQuaRE). Though generally too focused on security related
requirements to be of value for Keylane|Quinity, the first and third method fragments were
particularly well received in the first review.
The first method fragment, ‘agree on definitions’, proposes a simple yet highly interactive way to
set up a common terminology with the client. A survey style form is sent to the client containing
common ambiguous terms. For each term a checklist of definitions is provided containing the
supplier’s own definition and one to three generally accepted definitions. The last option is
always an open field where the client can enter their own definition if it differs. An example of a
survey question is included below. The completed survey is documented in a report or glossary,
which can then be shared between the two companies for the remainder of the project.
Figure 18 - Example of a SQUARE terminology survey question
The third method fragment, ‘develop supporting artifacts’, proposes a process for the creation
and sharing of documents, templates, reports, and other artifacts that will be needed in the
project. This is something that most software development companies do already, but it is rarely
formally recognized as a distinct part of their processes. The method fragment consists of a
planning activity by the supplier, followed by the creation and sharing of documents from both
supplier and client. The fragment ends with a verification activity to guarantee that all necessary
artifacts are completed and documents are received.
The method was validated in two case studies. First by a group of graduate students [68] under
the lead of the orginal author, Nancy Mead. Later it was applied by Carnegie Mellon students and
another SEI research group to the development of an IT asset management system for an
undisclosed branch of the United States Department of Defence [69].
6.7 METHOD - KJ+
KJ+ was included for its highly interactive workshop session design, and the way it relegates
work to stakeholders. KJ+ can help Keylane|Quinity achieve the following business needs:


Increase interactivity in the requirements engineering practices
Formalize the filtering of requirements.
39
Inspired by the KJ technique, KJ+ is another method developed by the SEI [70]. The original KJ
technique is a structured workshop design by Jiro Kawakita [71], which hands over most of the
elicitation and filtering efforts to stakeholders themselves. Though different interpretations
exist, KJ usually sits stakeholders down in a single room where they are asked to answer a focus
question on sticky notes. The notes are then grouped, filtered and sorted by the stakeholders
themselves in a series of exercises, where the facilitator merely takes a guiding role. KJ+ adds a
preliminary interview, focus question design, the identification of unstated needs and a step for
KANO analysis.
The KJ+ method does not interact well with other elicitation methods, as its demand for a series
of preliminary interviews would make it rather time consuming. The double grouping exercise
in the workshop design was also not a part of the original KJ technique, and may be too
complicated for inexperienced stakeholders. As such, the decision was made to include the
original KJ technique under the technique selection method, and to only include some minor
method fragments from KJ+.
In the preliminary interview the Kano analysis step was particularly well received. It is based on
the Kano model by Noriaki Kano [72], which proposes that there are three types of features any
product can have:



Must-be. A must-be feature is commonly expected by stakeholders. If it is not included in
your product it will cause great dissatisfaction. If it is included it will simply be taken for
granted and will be perceived indifferently by stakeholders.
Satisfier. A satisfier is a normal stakeholder need. If it is not implemented it will cause
dissatisfaction, but it will elicit a satisfied response if it is implemented.
Delighter. A delighter is an innovative feature that the stakeholders usually weren’t even
expecting to find in the product. If implemented it will lead to great satisfaction. If not
implemented stakeholders won’t even notice that its missing.
Kano analysis is often used as a supplement or alternative to a traditional cost/benefit analysis
and can serve as a basis for filtering or prioritization.
Though reported to be used internally at the SEI, there are no case studies available of the KJ+
method in use.
40
Figure 19 - KJ+ meta-model
41
6.8 METHOD – KEYLANE|LEANAPPS METHOD
The Keylane|LeanApps method was
included as a special interest for
Keylane|Quinity,
particularly
as
an
opportunity to gain a deeper insight in the
processes of their (at the time) new sister
company.
Keylane|LeanApps has a far more iterative
approach to software design, loosely based
on agile development practices and
principles. The company uses insurance
products as a baseline to measure the fit of
their software to client companies, whereas
Keylane|Quinity takes an architecture
focused approach. This distinction makes
sense, since Keylane|LeanApps operates
predominately in the life insurance and
pension sectors. Both life insurance and
pensions are subjected to far more
legislation than the damage insurance
sector in the Netherlands, which makes for
far more homogenous insurance products in
the market. This makes finding the few
differences that distinguish a new clients
products from the competition an effective
approach.
In
addition,
Keylane|LeanApps
predominately keeps itself occupied with
policies and relations. Keylane|Quinity pays
fare more attention to other insurance
operations such as workflow management,
premium calculation, request handling, and
more. A product-centred approach makes
sense for the needs of Keylane|LeanApps,
but since QIS also seeks to support many
other client processes this approach
wouldn’t fit Keylane|Quinity in it’s current
form.
There are still some lessons to be learned
from Keylane|Quinity’s sister company. The
clear separation between requirements and
solutions is also made, but not as explicitly
as it is made by Keylane|LeanApps. The first
Figure 20 - Keylane|LeanApps meta method
42
three phases also boast their own formal review cycles, both with the client and internal experts.
A fit test between Keylane|LeanApps’ standard insurance product definitions and the client’s
product definitions forms the primary source of requirements. The secondary source is an
elaboration session held with stakeholders to further understand the client’s products and
processes. These two sources of requirements are then processed in a fit test document and a very
basic prototype displaying a simple potential solutions for gaps. Stakeholders are asked to review
the fit test and attend a demo session, in which they determine whether their requirements have
been correctly understood. The results of the fit test- and prototype-reviews are compiled into a
fit test review document that forms the third source of requirements. If these reviews indicate a
lot of missing information, extra elaboration sessions are possible. As mentioned before, QIS is not
designed to be conductive for rapid prototyping. The other approach, the fit test, may be of use
however.
Once the client has fully approved of the fit test, Requirement Designs (RDs) are made. For each
major client requirement an RD is made, which decomposes the requirement into granular parts
(similar to the goal of Keylane|Quinity’s design workshops). RDs must be absolutely solution free
and should only state needs or goals. Product specialists then contribute an impact assessment
and complexity estimate to this specification. Highly or extremely complex requirements are
reviewed by a product manager who determines if more analysis and decomposition work can be
done. After product manager approval RDs are reviewed by the client.
After specification a Solution Design (SD) is made. An SD is content-wise equivalent to
Keylane|Quinity’s RFC. The SD outlines the specific solution that will be built to fulfill the
requirement. Once again there are two review opportunities: once by an internal review board
(which also determines the final impact) and once by the client. Fully approved solution designs
will be planned in the client’s own release, but will also be generalized by a functional expert team
for release in the standard LeanApps product.
43
7 – ASSOCIATION AND METHOD CONSTRUCTION
7.1 ASSOCIATION
In this step of the method association approach, the feature groups collected earlier were
compared to the method fragments of candidate methods. If the concept of a method fragment is
still unclear, a method fragment is any activity paired with it’s deliverable. In ALL PDDs
presented in chapter 6, a method fragment is a white rounded rectangle on the left combined
with it’s connected white rectangle on the right.
A two dimensional table was made to compare each of the 33 feature groups and 15 business
needs to a total of 54 method fragments (Figure 21). The feature groups were previously refined
in elaborate review sessions with the external coach, but methods had only been briefly
explained. This led to the realization that some method fragments would satisfy no features
whatsoever. As association progressed a total of 16 method fragments were excluded, resulting
in a total of 38 method fragments remaining. In Figure 21 the exclusions have been marked with
red names. Crosses in the figure represent a fulfilment of one or more features in a group by a
method fragment. This however, does not necessarily mean that the entire feature group is
satisfied by the entire method fragment. Red marked corners in the figure are Excell comments
in which I further elaborate on the relation between method fragments and features if there is
anything worth mentioning. For a larger version of the association table, please refer to
appendix 5 for the association table file included with this thesis.
Including the business needs identified in chapter 4.2 as features also proved to not be a very
wise decision. Some of these needs were not satisfied by method fragments because they were
too specific to Keylane|Quinity (e.g. consistently gather requirements from demo sessions,
actively involve FO and TO in RFI/P answering). Others were too abstract to be covered by
methods (optimize session abstraction, push for client objectivity when expressing
requirements).
Another concern presented itself in the two main business needs: effectiveness and efficiency.
This problem was predicted by an internal coach: how do we measure effectiveness and
efficiency? The assumption I made was that any method that allows for targeted elicitation
counts as effective. For example: the KJ technique has a group of stakeholders share their needs
and goals, but then puts them to work by having them filter those needs and goals until only the
crucial ones remain. This is opposed to an untargeted elicitation technique like brainstorming
session, where the usual goal is to just gather as much as possible of anything.
In the case of efficiency, I assumed that any method fragment that does not attempt to seek out
every single requirement could be considered effective. For instance: the ‘quality attribute
increment’ in GEAR attempts to identify an exhaustive list of non-functional requirements. This
would not at all be considered an efficient use of time for Keylane|Quinity. A good example of
efficient elicitation would be the use of technique selection frameworks, where a project
situation is always analyzed in order to select the optimal technique. As for the results of these
assumptions, please refer to Phase 3 – Validation.
44
Figure 21 - Association table
7.2 PRELIMINARY METHOD ASSEMBLY
Using the association results, a preliminary method was assembled. See Figure 22 for a
summarized version. A full view of the preliminary method is included in appendix 6. The
preliminary method is constructed from all candidate method fragments that satisfy feature
groups and adhere to one or more of the following criteria:
1. During association this fragment appeared to be a particularly good fit for this feature.
2. In the early review of candidate techniques and methods this fragment elicited a very
positive response.
3. Out of all the fragments that fulfill this feature, this fragment fulfills more other features.
These criteria helped prevent over-selection, but to not narrow our views too much at this stage
we also kept track of viable alternative method fragments. Looking at the feature group ‘quality
checking’ under the requirements validation group, there are 5 different method fragments that
satisfy it. It makes no sense to include all five, so we opted to only include the quality check from
RESCUE. Quality checking in the RESCUE method is part of the broader ‘Requirements
management’ method fragment, which also supports an additional 10 feature groups. As an
alternative the ‘requirements inspection’ fragment of SQUARE has been marked, as it was well
received in the early method and technique review.
The preliminary method contains the following method fragments:

The original top half of the sales trajectory (Keylane|Quinity). See Phase 1 for a
description.
45









Agree on definitions (SQUARE). In this method fragment a terminology survey is sent
to the client. The terminology includes common terms in the industry, Keylane’s
definition, a handful of alternative definitions, and room for the client’s definition. The
purpose of this method fragment is to establish a common terminology between supplier
and client.
Define Project (JAD/Tropos). As is common to many software development methods, it
all starts with the definition of the project. In this method fragment the client, the project
purpose, project objectives and critical stakeholders are captured.
Develop supporting artifacts (SQUARE). In this method fragment Keylane|Quinity
prepares all the documents (e.g. surveys, forms, reports, templates) that will be made
during the project. This is also when Keylane|Quinity and the supplier share relevant
documents that may be of use to eachother.
Gather known requirements (own contribution). Keylane|Quinity gathers
requirements from their own Sales department in an interview, and gathers up
everything learned from early documents and meetings. There was no apt method
fragment in any of the candidate methods, so I added this on my own initiative.
A mid section based on the project situations:
o Front Heavy (retains the bottom half of the sales trajectory). The only thing
Keylane|Quinity does here is taking notes. Dominant clients should be allowed to
fulfill their information needs however they see fit. If Keylane|Quinity still lacks
knowledge afterwards, they may refer to Front Light.
o Front Light (technique selection). When the client does not dominate the early
stages of a project, Keylane|Quinity can refer to a technique selection tool to plan
sessions and other events on their own initiative.
o Amendment project (technique selection/GEAR). In amendment projects the
client already has an implementation of QIS. Technique selection is applied like
in a front light project, potentially with the addition of GEAR’s function-toarchitecture mapping.
Workshop (JAD). This is a concluding workshop to review that all requirements are
understood. Similar to Keylane|LeanApps’ fit test reviews. We opted to use the workshop
section of JAD because it is well known and has plenty of guidance available.
Documenting requirements (various). It was at this point still completely uncertain
which attributes Keylane would and wouldn’t want to document. We included many
notations in this fragment, including activities like filtering, removing duplicates and
doing impact assessments for the sake of completeness.
Original Project setup & Increment phases (Keylane|Quinity). See phase 1.
Requirements Management (RESCUE). Earlier we established that requirements
management activities were lacking. We decided to include the very elaborate and well
documented management phase from RESCUE in it’s entirety to offer lots of choices.
The future of the Kano Analysis (KJ+) method fragment was still uncertain at this point. It was
perceived as an interesting addition during the technique and method review, but it could
potentially be far too time consuming.
The assembly of the preliminary method marked the end of phase 2, and provided the basis for
phase 3: evaluation.
46
Figure 22 - summarized preliminary method.
47
PHASE 3 – VALIDATION
In the third and final phase of the project the preliminary method resulting from phase 2 was to
be reviewed and perfected. The preliminary method was intentionally overly elaborate to give
the reviewers plentiful alternative options to consider. We intended not to make any major
fragment eliminations without consulting reviewers first.
This chapter continues with the explanation and results of the first review session, followed by
it’s resulting changes to the method. The second evaluation and it’s results are discussed in
chapter 9.
8.1 FIRST REVIEW RESULTS
To start off the validation of the preliminary method, a lunch session was held on location at the
Keylane|Quinity office. The purpose of this session was to have an open discussion about the
concept, goals and results of method fragments. The specifics of their contents was left for the
second review.
The first session was held on Tuesday the 14th of april. 10 participants attended including three
designers, three consultants, a team leader, the sales manager, Keylane|LeanApps’ product
manager and the external coach. What follows is a list of method fragments that were discussed
and the results of the discussion.
Agree on definitions –Though positive about the concept, the group agreed that establishing a
new list of terms for every project does not make sense. Instead the terminology should be one
standard set with occasional additions of new terms. Those responsible for adding terms should
take care not to turn it into a full thesaurus. The terminology should be limited to names of
entities that are known to have conflicting definitions. In addition, rather than including several
third party sample definitions, the document should only have two columns: The
Keylane|Quinity definition and room for client comments. Two method fragments were changed.
Define project – The identification of stakeholders in this early stage caused some discussion.
The main reasons for and against were:



-
It’s always good to have a list of contacts available when extra information is needed on a
requirement.
Can be used as a defense when unidentified stakeholders feel passed over later in the
project. A list of stakeholders established with the client can serve as evidence that
unidentified stakeholders were not pointed out by their own company.
Particularly practical if all decision makers are identified in the list. Those stakeholders
that have the mandate to act and decide on behalf of the client company should be in
there, so Keylane|Quinity doesn’t forget to contact important individuals.
New stakeholders will always be identified later in the project. This means a stakeholder
list would quickly become obsolete, or that we should expend time updating the list.
The decision was made to limit a stakeholder list to only those that are critical to the project:
stakeholders with the authority to make decisions, and stakeholders that must be consulted on
certain subjects. One method fragment was changed as a result. One of the session participants
remarked that this classification would be similar to the concept of an ARCI matrix. This subject
will be investigated in a later stage.
48
Develop supporting artifacts – The elaborate development of supporting artifacts (see
SQUARE in chapter 6) was considered too elaborate and was consequently removed. This
resulted in the removal of five method fragments.
Gather known requirements – As soon as the subject of interviewing Keylane|Quinity’s sales
department was introduced, the sales manager took the opportunity to provide an elaborate
description of the knowledge his department acquires. It was remarked that the handover of
knowledge between designers and developers, for example, is performed far better than the
handover between sales and consultants. Identifying the exact potential of the sales department
followed in a later stage, but for now one method fragment was considered changed.
Front Light and amendment projects – The fair remark was made that any form of elicitation
technique selection would have to compete with the few techniques that engineers are
comfortable with. This remark was quickly countered by a senior consultant with the fact that
relying on familiar techniques alone increases the risk of missing requirements. The group
agreed that a lack of knowledge or experience with alternative techniques can always be
mediated with training, so it was worth exploring technique selection.
The group was not particularly impressed with the decision support system built in Excel. One
participant aksed: “we are not going to fill out a form like that every time we want information
do we?”. Another remarked: “a textual description of techniques would fare better. If you explain
the technique and when to use it, we’d feel more comfortable with a decision than if it was made
with a calculator”. The group agreed that such a textual description would have to include for
each technique:





A clear guideline on how to do it.
A description of the context in which it works well.
A description of the circumstances in which it would not work.
The people and materials needed to do it.
For each technique, a list of complimentary techniques that combine well with it.
One method fragment was changed as a result.
Processing – At this point the participants started to get distracted by the earlier discussions,
but some points got through:
1) The removal of duplicates was considered unnecessary since no one has ever seen
duplicate gaps in a project. For non-gaps removing duplicates would be too inefficient.
2) Making sure that gaps are written using SMART criteria had unanimous support.
3) The creation of a requirements document had support but several participants remarked
that it would require a very clear document creation guideline.
4) Impact analyses are performed by the QIS group for generalized solutions. Specific
solutions are only made when they are absolutely critical to the client, regardless of
impact. Impact analyses are thus outside of the scope of implementation projects.
The subjects impact analysis, reviewing the document with the client were not discussed. Two
method fragments were removed and two were changed.
Requirements management – The use of the Volere Shell was removed since it does not make
sense to use an external standard document that is not tailored for Keylane|Quinity. Relating
49
requirements to use cases was also removed, since use cases are rarely written. Writing fit
criteria was considered part of the ‘refine notation’ and ‘check if requirements are smart’
method fragments in the processing stage. These three method fragments were removed.
The remaining five method fragments (decompose requirements, chart requirement
interdependencies, manage requirement status, track requirement changes, check requirement
quality) were considered due for later implementation. The current way of working in office
documents is not very conductive to these method fragments. The planned adoption of
Atlassian’s Jira and Confluence tools however, would make these tasks far easier. Five method
fragments were considered changed.
8.2 POST REVIEW METHOD CHANGES
After concluding the first review, we started writing the method as part of Keylane Quinity’s
Standard Development Environment (SDE). The SDE is an intranet folder containing documents
and guidelines for all of the company’s processes. Explaining the method as a part of the SDE
would make it easier to communicate it to Keylane experts.
In addition the following changes were made as a result of the review:
8.2.1 Agree on terminology
Rather than the original idea to include several common definitions from external sources (e.g.
dictionaries) in the terminology survey, the advice from the lunch session was followed. The
terminology survey will now take the shape of a simple three column table. The first column
contains a single term, the second the Keylane|Quinity definition, and the third has room for
client comments.
Making an actual terminology survey was considered outside of the scope of this project,
8.2.2 Identify stakeholders
In the project definition method fragment the step ‘identify stakeholders’ was included from the
Tropos project method. After the lunch session the method fragment has been renamed as
‘identify critical stakeholders’. Including an exhaustive list of stakeholders this early in projects
results in a list that requires a lot of updates and maintenance. In accordance with the meeting
proceedings the list will now only include decision makers and those who MUST stay informed.
To enable the selection of critical stakeholders, the suggestion will be made to use an ARCI
matrix [73,74]. The ARCI is one of many variations of responsibility assignment matrices,
specifically designed for classifying decision makers and influencers. The four letters stand for:




Approves. Those authorized to approve decisions
Recommends. Those responsible for recommending decisions
Consulted. Those whose opinions must be sought when making decisions.
Informed. Those who must be informed after decisions.
An example ARCI is included below:
50
Policy Requests
Acceptance
Relation
management
Product
manager
R, I
I
A, R
Project
Manager
A, R
A, R
C, I
Financial
consultant
I
C, I
Technology
team
I
I
C, I
Table 6 - Example RACI matrix
8.2.3 Technique selection
In accordance with the proceedings of the lunch meeting technique selection using a decision
support sheet in excel was deemed to complicated. Instead, this method fragment was replaced
with a simple textual description of elicitation techniques. For the sake of findability the
techniques have been classified in the following categories:




Interview techniques. Where stakeholders are interviewed in a one-on-one or one-onfew meeting.
Workshop techniques. Where stakeholders are interviewed in medium to large group
sessions.
Observation techniques. Where the requirements engineer observes and analyses
current operations at the client company.
Questionnaire techniques. Where stakeholders and end users are asked a broad range of
questions in paper or digital forms.
A document was created listing the techniques for internal use. Each of the techniques is given a
description, their ideal context for use, the obstacles that will impede their use, the required
people and materials, and its combination with other techniques.
These last four factors are not binding, meaning that your current situation does not have to
adhere to those criteria perfectly in order for a technique to be useful. Which techniques and
when to use them will largely depend on your expert judgment and experience. It is recommend
strongly that requirements engineers at least consider techniques that they are unfamiliar with.
Limiting their use to a select few favorites might increase the risk of missing requirements.
Finally, one last technique was added as a result of the lunch meeting: Demo observation. In a
demo observation session several stakeholders are educated in the basic functionalities of QIS
and are led through a workshop using a QIS demo environment. This direct involvement results
in highly detailed requirements, but also gives the participating stakeholders a good frame of
reference for future sessions.
8.2.4 SMART goals
In accordance with the lunch meeting proceedings the decision was made to give SMART goals a
more prominent place in the internal process guideline. SMART was originally envisioned by
Geoge T. Doran as a simple acronym of criteria for well written goals[75].
Though there are some variations in what the five letters actually stand for, it was decided in
collaboration with the coach to use the following:

Specific. It is clear which capability the customer really needs, and to what extent QIS is
going to be responsible for delivering that capability.
51




Measurable. The success or failure of the requirement’s solution can be measured.
Writing a measurable requirement can be done by formulating it as an in-or-out (the
system either does it or does not do it), or by including fit criteria.
Achievable. The requirement does not demand something impossible or unfeasible. For
instance: “the QIS system must detect errors in policy request data”. At first glance this
seems achievable, but how realistic is it really to expect not a single error to ever sneak
through into policy data? It is impossible to prevent every misspelling of names, every
wrong number in bank account data, and every other thinkable error that may occur.
Relevant. The requirement represents a need that matters and adds value to QIS when
fulfilled. The requirement must also make sense as a part of QIS. A popular example of a
goal that is not relevant is “the sales manager wants to make 50 peanut butter and jelly
sandwiches”. This may certainly be a personal goal for the sales manager, but this is not
something that QIS should support.
Time-bound. The requirement can be given a deadline. This is particularly important for
high-value gaps. Without a clear deadline, it may be delayed in favor of simple day–today problems and not receive the attention it deserves. Presenting a clear deadline to the
client is also a sign of commitment.
Rather than using complicated QA or inspection methods for checking the quality of
requirements, SMART is a lightweight and efficient alternative. Drawing more attention to it in
the SDE document is expected to result in better written requirements.
8.2.5 Passive voice
Though occasionally remarked upon before as a problem, the passive voice (lijdende vorm in
Dutch grammar) in requirements descriptions turns out to cause serious disruptions for
Keylane|Quinity. As a response this problem will also receive more attention in the SDE
document.
A sentence written in the passive voice places the subject in the position of the object. This
means the subject is undergoing an action, rather than performing it. Consider for instance:
(1) The thief was caught.
Here the subject (thief) is undergoing the action (catching). The typical role of a subject,
performing an action, is not fulfilled in this sentence, making it impossible to know who or what
is performing the action. Perhaps the thief was caught by a police officer, or an attentive citizen,
or a good security system. Now consider this example of a requirement:
(2) It must be possible to register a client
In this sentence the subject (client) is undergoing the action (registration). It is not clear which
role the QIS system should have in this requirement. Does QIS automate registration? Does an
end user register the client with partial support from QIS? Is the client registered in an external
system which communicates with QIS? Requirements should be written in the active voice
instead. For example:
(3) Qis must assist an acceptance-specialist with the registration of a client by providing
an entry form.
52
Gaps written in the passive voice will always cause confusion later in the project. Because the
system role is so unclear, designers are forced to retrace earlier steps and find out what a
stakeholder originally intended.
A short description of the passive and active voice have been added to the SDE document.
For a report of the changes made to the method please refer to table 7. For a full view of the
method after the first evaluation, please refer to appendix 6.
8.3 INTRODUCE MORE CREATIVE AND INTERACTIVE ELICITATION
After the first lunch meeting the external coach indicated that there was still a desire for more
creative solutions on my part, and more interactivity with stakeholders. Particularly any means
to get stakeholders to do their own elicitation work would be greatly appreciated.
In an earlier implementation by Keylane|Quinity, some success was already had with a closed
survey. The survey covered a broad range of subjects touching on all aspects of insurance
business, including some specific peculiarities (e.g. an unknown product mentioned on the
client’s website, references to unknown third party systems in the RFP). Replying would have
required broad knowledge of the client company’s operations so this survey was directed at
client management exclusively.
The closed survey used in that project has been distilled into a list of generic speaking points:
subjects that should be discussed in every requirements phase. This list was later expanded
using a broad introduction to QIS, some functional descriptions of QIS, and my participation in a
course on Relation Administration. A potential way of approaching completeness is to simply
make sure that all major process areas have been discussed. Checking off everything on the list is
no guarantee of completeness, but at least it’s another piece of evidence that completeness is
approached. For the full list please refer to the speaking points checklist.docx document in the
attachments.
The closed survey is another in a long line of techniques that target the management layer of
client companies. This led us to discuss the possibility of perhaps targeting end users with a
similar technique. End users were until now an untapped resource for Keylane|Quinity. In
scientific literature there are some advocates of involving end users (e.g. [76,77]). Others go as
far as making end users an integral part of design and development (e.g. participatory design
[78], end-user development [79]).To engage end users in the development process we made the
decision to introduce an open survey. The open survey would have to feature simple, open
questions and would have to be short in order to keep the target audience engaged. The survey
questions will be peer reviewed in the second review, so see chapter 9 for more.
The second tactic for engaging end users discussed was a Wiki. An open web platform similar to
Wikipedia, that hosts both text and media in encyclopedia style pages. Wikis natively come with
discussion functionalities, which would make it ideal for sharing Keylane|Quinity’s large
collection of documents in an interactive way. Other perceived benefits are:

Make documents more accessible. Nobody likes reading 120 page pdf files with
functional design decisions on all kinds of levels of expertise. In a Wiki, these documents
could be split up into more sensible pieces.
53



Improve navigation. Keylane|Quinity’s intranet contains well over 200 folders, sorted in
a way that doesn’t always make sense. A Wiki can use hyperlinks, even down to the
smallest subsections, to make navigating trees of folders obsolete.
Embed multimedia. For most of the basic functionalities of QIS there are training
materials and tutorials available. On a Wiki these could be embedded into pages to put
them within easy reach of readers.
Structure the body of knowledge. Most new clients won’t know where to start reading
once they’re given a stack of documents. A wiki can provide separate pages that provide
guidance, catered to specific groups of users based on their function, specialization and
degree of experience.
The trial wiki that was built as a part of this project is described in more detail in chapter 10.2
During this stage many other smaller revisions were also made to the PDD of the method. These
revisions are summarized in table 7. The revised PDD is included in Appendix 7.
8.4 IMPACT OF THE EVALUATED METHOD
During the evaluation of the preliminary method several participants remarked that it was not
clear to them when and where in the process my suggested method fragments would be taking
place. In fact, some remarked that they weren’t even familiar with the early phases of the current
method to begin with.
The poster on the next page in figure 23 was created to visualize both the point of this project
and a broad timeline of projects in general. Activities on the right side of the timeline were
added or replaced old activities from the left side. The entire requirements phase (voortraject) is
a completely new addition.
The large funnel on the left of the timeline visualizes the amounts of requirements coming in
during different stages of projects. As the sales trajectory with its RFI and RFP progresses, more
and more requirements come in. When Keylane|Quinity is finally selected as the software
supplier, the requirements phase begins. All previously known requirements are processed,
more requirements are gathered and a final report is made. The purpose of this phase is to
identify 90% of the major gaps, because more requirements may always pop up later.
The poster was reviewed with a handful of experts before its final unveiling during the second
lunchmeeting. As a result of these reviews we also created a second version (see figure 24), in
English, showing the entire process as a whole. Some minor adjustments were made to the
naming of phases and activities, but content wise not much was changed.
You may notice a little gray block all the way at the bottom of the project setup phase named
‘Finalize architecture’. This was an addition by the external coach, who envisions a new project
phase where the target architecture of implementation projects will be formally created. This
phase is outside of the scope of this project, and will not be created in this project.
54
Figure 23 –Old Poster visualizing method changes
55
Figure 24 – Poster visualizing only the new method
56
9 – SECOND REVIEW ROUND
The second review session for the method was held on the 9th of july, approximately two months
after the first. During this period we continued work on creative elicitation methods (speaking
points, open survey, wiki) and adapted the method as a response to the feedback in the first
session. In addition we began to seriously provide context to the method in the form of
checklists, templates guidelines and much, much more. This was all reviewed in the second
review, including the last open question: how are we going to write down requirements and
gaps.
In section 9.1 we describe some of the smaller results of the review, followed by method changes
in section 9.2. In section 9.3 we present the final method.
9.1 SECOND REVIEW RESULTS
During the second review the following method fragments were approved:
Interview sales. After having done some additional interviews with sales, we realized that sales
keeps track of every RFP’s answers very neatly in an answer overview. The value of this
overview was realized by all, though some remarked that the answers in there are too specific to
the customer. It might be a good idea to keep a separate RFP answer overview for internal use.
Terminology survey (formerly ‘agree on definitions’). The original terminology survey
contained, for each term, a Keylane|Quinity definition, two industry definitions and room for a
client definition. As a result of the first review this was changed to, for each term, one column
with the Keylane|Quinity definition and one column with room for client comments. This version
was much better received.
Some also mentioned that it is likely we will already pick up on some terminology differences in
the sales trajectory, which means that some comment columns can be pre-filled for the
customer. This was added to the internal guidelines document.
Keylane Wiki. The Keylane wiki prototype was very well received, particularly because it is a
fully functional genuine wiki that could, in theory, be put online as-is. The potential value was
also recognized, though many agreed that much work would still have to be done to create a
controlled environment for stakeholders.
9.2 POST-REVIEW METHOD CHANGES
The following method fragments were still deemed insufficient and changed.
Project definition – stakeholders list. The stakeholders list was originally described as ‘a list
of stakeholders’ and nothing more. In the first review the feedback was that a list like that can
quickly become unmanageably large. As a result we decided to only use stakeholders with a role
in the ARCI matrix. Approves, Recommends, Consulted and Informed.
We expected the Recommends role, those who recommend solutions, to be rejected (since
Keylane|Quinity already avoids stakeholders getting involved in making design decisions),
which it was. The informed role, those who should be informed of decisions, was also rejected,
since the prevalent opinion was that the list would still be too big.
In the final version we will only register the As and Cs:
57
Approves – Those who are responsible for reaching internal agreement about decisions.
Consulted – Those who must be consulted in the decision making process.
1 method fragment was considered changed.
End user survey. The end user survey was strongly rejected by many participants. Apparently
engaging end users is really uncommon early in projects for one or more of the following
reasons:
 The client company does not want it’s low level employees to be aware of a new software
acquisition until the exact specifications are known and they can start up a proper
change management trajectory internally.
 The client company wants to maintain control over the outgoing requests to the software
supplier. Involving large groups of low level employees risks a loss of that control.
 The client company’s politics call for a strict separation of responsibilities. Participating
in the acquisition of software is simply not part of end-user responsibilities.
 The client company does not want to risk disappointment among it’s low level
employees when their requests aren’t fulfilled. Since Keylane|Quinity sometimes has to
say no to feature requests to protect the integrity of the standard product, this is a likely
scenario.
As a response the closed survey was used to replace the open survey. Closed surveys are
targeted towards client management and should not run into those problems.
Since open surveys were replaced with closed surveys in the PDD, 1 method fragment is
considered changed.
Handling additional requirements in increments. I initially suggested to only write down
new gaps that arise in the increments. The group made the valid argument that requirements
that lead to product configurations should also be captured at all times. In fact, all requirements
should, lest your colleagues in future increments have to operate with a knowledgedisadvantage.
We collectively decided on the following approach to deal with new requirements and gaps
arising in the increments:



Requirements and gaps that apply to later increments should be captured so the
participants in those increments do not miss out on valuable information.
Requirements and gaps that apply to previous increments were missed, so they should
be documented in order to have them included in later increments.
Gaps that apply to the current increment should still be documented, so any resulting
changes to QIS can be traced back later. Requirements that can be easily solved in this
increment can be captured optionally.
In the resulting PDD one method fragment was changed and one was removed. The
requirements management method fragment was further cleaned up with the removal of one
method fragment.
Requirements notation. The last big open question was notation: what are we going to write
down? In past projects that had requirements documentation, the attributes differed wildly from
58
project to project. In this review
session we interactively decided upon
a single basic notation standard for all
future projects.
My initial assumption was that Gaps
needed an entirely different and much
larger amount of attributes than
requirements. I was proven wrong
when the group agreed that the
following would have to be registered Figure 25 – Deciding on notations in the review session
for both:






Name
Domain (from a standard list)
Subject (optional, own choice)
Author
Source (optionally with a role for human sources)
Description containing:
o One line statement of need
o Client rationale
o Our approach
o Related process (optional to add context)
o Related functionality if known (optional)
The attributes exclusive to gaps are:



ID number
Priority based on:
o Value to the client
o Expected impact
o Complexity
o Risk
o And including a one-sentence justification.
More optional elements In the description:
o Workarounds if available
o Scope if necessary
o System requirements if easily derived
We decided to structure requirements into three distinct parts:



A title consisting of an ID number (gaps only), name, domain, subject (optional), author
and source.
A subtitle containing a MoSCoW priority and a short justification (gaps only)
A description containing the client need, client rationale, our approach, and room for
contextual attributes using bullets.
An example gap and requirement would end up looking as follows:
59
Implement standard customer views – Reporting (Customer view) – Wesley Crooijmans,
in 12-7 workshop notes
Should have. This functionality can be configured out of the box and contributes to the quality of
client X’s communication processes.
 The client needs QIS to generate one single overview of each customer on demand,
across both life and damage insurance policies.
 Client X wants this to assist communications, by having all of a client’s policies readily
available.
 Client X has agreed to use the standard customer view functionality in QIS.
QIS-010 Advertiser as an intermediary – Intermediary administration (Intermediary
types) – Wesley Crooymans, from Bernard Brouwer (lead product-advisor Health)
Must have. This functionality is absolutely necessary to continue working with a vital group of
partners to the company.
 The client needs QIS to provide all intermediary administration capabilities for third
party advertisers that do not have the authority to submit requests. These advertisers
only act as Client X’s promoters and refer new end customers to the company.
 Client X wants them to be a part of the intermediary administration system so that they
can continue to keep track of- and pay commissions for assisting sales.
 The standard relation administration capabilities can be re-used if these advertisers are
added as a new intermediary type. These include:
o Adding a relation.
o Editing a relation.
o Coupling a relation to policies.
Four method fragments were considered changed.
9.3 FINAL METHOD
After processing of the second review results the method was finally completed. A report of all
changes made can be seen below in table 7.
When reworking the PDD with the latest additions, we realized that a major error was made in
the first review results PDD (appendix 7). The Front-Heavy method fragment (now renamed
Participate in sessions) takes place in the sales phase of projects, not the requirements phase.
This mistake led to the change of 4 method fragments and the removal of 1 method fragment.
In the final count of 46 method fragments we do not count the out-of-scope end of the
increments (design, develop, test, go live, and maintenance).
Total before
Added
Changed
Removed
Review one
62
0
12
10
Revision one
53
3
7
7
Review two
49
0
11
3
Total after
53
49
46
Table 7 - evaluation session results overview
The final method PDD has been included in Appendix 8. The concept and deliverable tables have
been included in Appendix 9.
60
10 - OTHER RE EFFORTS
Keeping in mind the external coach’s request for introducing some creativity (see 1.2 –
objectives), we experimented with automated elicitation technique selection and crowd
sourcing requirements in a wiki. The internal coach also recommended investigating the options
of using Natural Language Processing (NLP) for the identification of requirements, which is
included in this chapter’s 3rd section.
10.1 REQUIREMENTS ELICITATION TECHNIQUE SELECTION EXPERIMENT.
Recall chapter 6.1 method - Elicitation technique selection in the candidate methods chapter.
This chapter describes our initial findings of ACRE [46], MRETS [47] and the TT [80] framework,
and the reasons why these three were eventually discarded in favor of a customized version of
Carrizo et al.’s method [48]. We realized early on that none of the first three were very
practically applicable or reliable. Our initial course was to try and synthesize our own technique
selection method out of the three.
10.1.1 Three very different selection approaches
The first sign that a synthesis was likely to fail was the fact that selection followed a very
different process for all three approaches.
Selection in ACRE is achieved by simply filling out a set of six project factors and looking up the
resulting technique scores in reference tables. Scores are either not applicable, not very
applicable, applicable or very applicable, expressed as symbols. Where these scores actually
come from is somewhat questionable, as their sources nor creation are ever explained.
Validation of the scores was reportedly achieved by sending a survey to an undisclosed number
of experts. Nothing else is ever said on the subject.
The biggest complaint we had with ACRE was the absurdity of some of the project factors.
“knowledge type” for example asks the user whether they are looking for explicit, semi-tacit, or
tacit knowledge. Something that is impossible to know up front.
Where ACRE seems relatively simple, MRETS takes things to the other extreme. MRETS provides
a standard technique base containing 46 techniques (17 for elicitation). Techniques are given an
ad-hoc subset of 31 available attributes (7 for elicitation) measured on 6-point ordinal scales.
Selection criteria are 21 project factors that describe varying project and domain characteristics,
measured on 5-point ordinal scales (one exception is 4-point categorical).
For each configuration of the 21 project factors recommendation rules are proposed.
Recommendation rules are basically rules that prescribe 3 to 5 techniques for each specific
combination of project factors. In the paper the authors provide some of these rules, and
promise to complete the set in their future work. With 21 project factors, this means that there
should be a total of (5^20)*4 = 12.800.000 recommendation rules. We wish the authors good
luck.
A preliminary selection is achieved by picking the techniques that match your project’s
recommendation rule. This selection is then refined using a fuzzy clustering algorithm, a
cost/value calculation (objective function) and expert opinion. The first two are insanely
complicated, and an obvious ploy to feign objectivity by introducing complex logic and
61
mathematics into the paper. The end result should be 3 to 5 recommended elicitation
techniques.
The method was reportedly validated in three case studies. One of the three is elaborately
described in the paper, but it conveniently skips over the recommendation rule problem. The
other two case studies were never published.
Finally the TT framework classifies techniques on a two dimensional matrix. The x axis runs
from a static to a dynamic focus (very systematic to very loose and creative) and the y axis runs
from closed to open objects (the object of study is ‘stable and well known’ to ‘unstable and
unknown’). Six binary project factors determine your position on the same matrix, enabling you
to pick the closest techniques for the best results. How to get grey area results (e.g. 60%
systematic) from binary questions is not made clear. God only knows how techniques ended up
in the grey area either, since their position is never explained nor validated. No case studies
were ever published.
10.1.2 Lists upon lists of factors
There is only one similarity between the three methods, and that is that all three use project
factors as a crucial variable for selection. As mentioned above, both ACRE and the TT framework
use six, but MRETS uses 21. The second sign of our failing synthesis was how different those
factors really are. See the table below for a full listing of project factors:
Criteria
ACRE
MRETS
TT framework
Purpose of requirements (bespoke,
cots selection, contract procurement)
Project size, Project Category (communication,
embedded,
semi-detached,
organized),
Product Type, Team Size, Project Complexity,
Time Constraints, Organization and Customer
Relationship, Acquaintance with the Domain,
Product Quality Criteria, Cost Constraints,
Degree of Safety Criticality, Knowledge of RE
of the Teams, Degree of Knowledge of
Requirements,
Requirements
Volatility,
Availability of a Skilled Facilitator, Stakeholder
Heterogeneity, Degree of Innovation of the
Project, Customer Availability, Degree of the
Importance of Reusability, Degree of the
Importance of Eliciting implicit Knowledge,
Degree of Outsourcing
Application domain (stable,
unstable)
Knowledge type (behavior, process,
data)
Knowledge representation (new
system, non-tacit, semi-tacit, tacit)
Observable phenomenon (yes/no)
Constraints (meeting needed, time to
prepare, time for session, time to
obtain, # engineers, # stakeholders,
stakeholder friendliness, no technical
overhead)
Technique
interdependencies
(reported but not completed)
Requirements
engineers
type (logical, imaginative)
Information
resources
(abundant, scarce)
Desired user involvement
(little, much)
Requirements
quality
(completeness, behavioral
consistency,
logical
consistency, modifiability)
Non-functional
requirements (yes/no)
There is really no consistent theme in ACRE’s factors. They range from project purpose to
knowledge, and from project constraints to technique interdependencies. MRETS starts of
simple enough with some statistics, but then delves into the unmeasurable (e.g. Degree of
knowledge of requirements, Degree of the importance of eliciting implicit knowledge). The TT
framework then seems to stay focussed on simple attributes, but goes on to break it’s binary
pattern by asking for your desired requirements quality (why can’t I have all four?!). Not even
the scoring systems are compatible (ACRE: 4 point categorical, MRETS: 5 point ordinal, TT:
binary categorical).
10.1.3 Abandon ship!
62
After several days of headache inducing attempts to find a common thread; some way of tying
these three together into a logical whole, I eventually had to give up. MRETS was far too
complicated for practical use, with its millions of possible recommendation rules and completely
unnecessary mathematical clustering. The TT framework results would be impossible to justify
because of its weird black-and-white binary questions that would somehow lead to grey area
techniques. ACRE seemed so promising but it’s weak validation and odd choice of project factors
made it unusable. There was no way to make anything useful out of these three.
Consider then my relief at finding the paper of Carrizo et al.. Thanks to its similarities to ACRE it
has the same validity concern: How do you validate all those techniques scores for all of those
project factors? Luckily the authors are open and honest about their approach and admit to this
weaknesses in their paper, rather than citing an unavailable survey sent out to an unknown
number of anonymous experts.
10.2 CROWD SOURCING REQUIREMENTS ENGINEERING IN A PRODUCT WIKI.
10.2.1 Wiki’s and crowd sourcing
Crowd Centric Requirements Engineering (CCRE) is a term first coined by Snijders et al. in 2014
[81]. CCRE refers to the use of crowd sourcing [82] and gamification [83] principles throughout
RE processes. In the accompanying paper Snijders et al. propose a method for gathering
requirements from crowds of end-users, rather than a small group of high-ranking
representatives. The gaming aspects are added to reward users with digital points and badges.
Requirements elicitation is also covered within this paper.
This was not the first attempt at melding crowd sourcing and requirements elicitation. Earlier
attempts were made by Lim, Quercia and Finkelstein with their method StakeSource [84]. It was
initially intended for stakeholder identification, using principles from social networking theory.
One year later Lim, Damian and Finkelstein began development on the web-based tool
StakeSource 2.0 which incorporates the identification and prioritization of requirements [85]. A
demo was made available at http://www0.cs.ucl.ac.uk/research/StakeSource,but it appears that
the tool was never released.
Both of the above captured the interests of experts in this project, but in both cases the adoption
of a new tool was perceived as too great a barrier to entry. Still, the idea of a semi-self regulated
crowd doing the requirements elicitation autonomously struck gold. Crowd sourcing was proven
to have significant cost reducing effects on various types of projects in the past [86,87,88].
In search of a costless yet effective means of crowd sourcing elicitation the use of a Wiki had the
highest potential. A Wiki is essentially a collection of web pages created collaboratively by
groups of people [89]. The use of web technology enables the use of a broader range of contents
than traditional documents can, such as video materials, presentation slides and other media.
Wikis have been used for crowd sourcing purposes successfully in the past [90,91]. The use of
wikis in RE is not a novel concept either. Decker et al. used a wiki for the communication to- and
negotiation of- requirements to stakeholders [92]. Silveira et al. present a wiki-based
requirements documentation approach [93]. Fereira and da Silva propose a wiki for internal
requirements engineering as a replacement to expensive tools (e.g. RequisitePRO, CaliberRM)
[94]. There appear to be no wikis used specifically for the participatory elicitation of
requirements with customers, so we decided to make a prototype.
63
Building a wiki anno 2015 is not particularly taxing. There are plentiful free and open source
wiki frameworks, packages and builders available. Particularly popular variants are listed
below:
Engine
MediaWiki
Code
PHP
Database
MySQL, PostgreSQL,
SQLite
Foswiki
Perl
Text files
Pmwiki
PHP
Text files, MySQL,
SQLite
Examples
Wikipedia, wikiquote,
wikibooks,
Uncylopedia
Asap Library, W3C
brazil, Blackfin
consulting
Canadian air force
intranet, Netherlands
patent office intranet,
wikihistory
Table 8 - Wiki platforms comparison
10.2.2 Building a Wiki
The idea to build a proof of concept for Keylane|Quinity started as just that: an idea. The image
below shows a picture made in MS paint for an earlier presentation at the company. The initial
response was mostly positive.
Figure 26 - QIS wiki concept image
It wasn’t until much later, in the evaluation stage, that the time became available to actually get
to work on a genuine proof of concept. The decision was made to use MediaWiki, the same
engine that powers the famous Wikipedia, for it’s wide availability of documentation and plugins (not to mention its flexible open source licensing).
64
Setting up seemed difficult at first, particularly after looking at MediaWiki’s required software.
No less than three servers would be needed: MySQL for the databases, PHP for the code and
Apache for hosting the whole thing online. Luckily the popularity of Mediawiki means that
others solved this problem before. We opted to use the very user friendly Wamp Server stack,
which contains all the necessary materials: http://www.wampserver.com/en/
With Wamp and MediaWiki installed, you’re ready to go. That’s it. The Wiki is, albeit being a bit
empty, ready for use. MediaWiki’s markup language may take some time to get used to, but due
to its similarities to html even inexperienced web developers can get the basic jist of it pretty
quickly.
10.2.3 Discussions
To facilitate working with clients and discussing subjects within the wiki itself, the out of the box
functionality wouldn’t suffice. The default approach to discussion in MediaWiki is to make a
secondary page attached to every single page, a so called talk page. Discussion pages are
completely empty and do not provide any functionality to help discussions along. Users would
have to build them like they would any other page. This is no problem for those well-versed in
the MediaWiki markup, but it provides a significant barrier to entry for client users.
As a solution we included the LiquidThreads plugin:
http://www.mediawiki.org/wiki/Extension:LiquidThreads. LiquidThreads converts the
discussion pages of a Wiki into forums with full threading, comment and reply functionality. We
advise Keylane to look into the use of Flow in the future, which is the threaded discussion plugin
currently under construction for use in WikiPedia.
Figure 27 - A wiki talk page with LiquidThreads
65
Forum style discussion should lower the barrier to entry significantly, especially for
inexperienced users who won’t be learning Wiki markup just to make a simple comment.
In addition the external coach would have liked to see support for in-line commenting, but there
are currently no plugins that support similar functionality. MediaWiki’s static database structure
would suffer an enormous size increase if comment sections were added to each and every
single line of text.
10.2.4 Converting documents
The biggest perceived challenge with Wikis was the required effort of converting
Keylane|Quinity’s document collection into wiki pages. Several attempts have been made in the
past by MediaWiki users to enable smooth conversions from Word to Wiki markup, but none
appear to have succeeded. All MediaWiki plugins with this purpose are either unmaintained for
years, or perpetually in an unstable beta phase. Microsoft once made a disconcerted attempt at
building an official MS Word plugin, but it was so riddled with bugs and conflicts that it was
eventually abandoned.
Luckily, there is one somewhat reliable workaround available. Word documents can be saved
with the OpenDocument (.odt) format, which makes them compatible with the open source
Libre Office toolset. Libre Office’s document editor can in turn be used to export documents with
the MediaWiki markup in a .txt file.
Some minor issues still exist within these files like extra line breaks appearing and headings not
always converting well. Compared to the myriad of problems in other available solutions these
are easy to fix, and we don’t expect Keylane to start migrating its entire collection of functional
designs any time soon though. The wiki in its current form would be a good place to house
introductions, documentation of basic functionality , and some training materials.
For a bigger collection of screenshots please refer to appendix 10.
10.3 USING NATURAL LANGUAGE PROCESSING FOR REQUIREMENT AND STAKEHOLDER
IDENTIFICATION
Natural Language Processing of requirements was suggested as a potentially valuable way of
identifying requirements by the internal coach. His reasoning was as follows:
A software supplier generates a lot of documentation about its software product in the form of
functional descriptions, design documents, manuals and training materials. This documentation
is full of keywords related to the software’s functions and capabilities. Similarly the software
supplier’s clients have websites, process documents and sales communications that are rife with
keywords related to their business processes. Natural language processing could potentially
show the critical differences: keywords that are common in the client’s files, but rarely or never
appear in the software documentation. These missing keywords may be indicative of
requirements and gaps. We envisioned the results as pictured in figure 27 below.
66
Supplier
documentation
Gaps
Client
documentation
The, it, on, off, next,
to, up, down, left,
right,Supplier
over, under,
policy,
insurance,
Keywords
soft, board, legal,
ware
The, it, on, off, next,
to, up, down, left,
right, over, under,
Difference
policy,
The,insurance,
it, on, off, next,
soft, to,
board,
legal, left,
up, down,
ware right, over, under,
policy,
Overlap
insurance,
soft, board, legal,
ware
The, it, on, off, next,
to, up, down, left,
right, over, under,
Client Keywords
policy, insurance,
soft, board, legal,
ware
NLP
Figure 28 - NLP experiment envisioned outcome
Using the open source natural language processor AntConc we set to work on a large set of
Keylane|Quinity’s functional descriptions, and an opposing set of client process descriptions. As
a benchmark of success, we attempted to replicate a completed gap analysis report. The results
were somewhat disappointing.
Out of 66 general keywords that would likely lead to the known gaps, only 21 were present in
the client’s keywords to begin with. Out of 37 specific keywords that would definitely lead to the
known gaps, only 10 were found. Not only that, but these keywords appeared so low down in the
rankings that they would likely be overlooked anyway (ranging from 747th to 2628th). They were
not populating the top 100 or even the top 50 as we had hoped.
This experiment did however lead us to try two more ideas, which were met with slightly better
results:


RFP analysis. Using pre-selected searchwords and KeyWords In Context (KWIC) views
we attempted to extract requirements from RFP documents. Particularly the
searchwords ‘must’, ‘shall’ and ‘could’ performed well, resulting in over 50%
requirements per hit in one RFP.
Stakeholder identification. Using KWIC sorting on the words before the same
searchwords, we were able to identify 19 stakeholder groups from a 100+ page
document in under 10 minutes.
The full report on the NLP experiments is included in Addendum 1.
67
11 – DISCUSSION & CONCLUSION
In this chapter we briefly discuss the results in 11.1, and the answers to our original research
questions in 11.2. We continue with the validity and lessons learned in 11.3 and 11.4
respectively. Finally the business and academic relevance are discussed in 11.5.
11.1 RESULTS
Figure 29 - Summarized final method
68
The method still begins with the original sales trajectory. RFI’s and RFP’s are answered, and if
desired other information sessions are given to t he client. These three activities all form sources
of requirements.
Before we get to the actual search for requirements some setup activities are performed. The
agree on terminology step contains a terminology survey, in which standard terms are shared
with the client and feedback is received. This is followed by a short ~1 page project definition, in
which the project’s purpose, objectives, milestones and critical stakeholders are captured. The
wiki and closed survey can be presented to the client at this point. Since both have the client
gather their own requirements independently, we consider both to be part of the setup. Then, if
applicable, there is room for a sales interview and documents & systems analyses. The
requirements phase setup ends with the collecting of all known requirements from:
 The RFI and RFP
 From sales information sessions
 From sales employees
 From the terminology survey
 From the wiki and closed survey
 From client documents and systems.
In the actual requirements phase we begin elicitation with a quick check of the speaking points
checklist. For subjects that have not been discussed we select techniques that will be used to
elicit requirements about that subject. Derived requirements that have not yet been verified
with the client are also part of these sessions. Throughout this phase we document all
requirements in a central requirements dossier. Gaps are also captured in a separate overview
table for easy access. When all planned elicitation techniques have been performed, the
document gets one last check for structure, completeness and quality using a checklist. We then
share the dossier with the client and plan a final workshop to discuss the last remaining open
questions, undiscussed speaking points and unverified requirements.
On the one hand project managers set up the rest of the implementation project in the project
setup phase, followed by design, development, testing and go-lives in increments. On the other
hand we continue to manage the status and contents of requirements. In addition we process
new requirements arising in the increments and document them in the dossier and table.
After the contextualization step, an extra step that was added to the MAA, the following were
created:
Deliverable
Description
Requirements SDE
A process document in the style of the company’s style.
Method walkthrough
An elaborate walkthrough of an example project using the
method.
Dossier template
A template for the requirements dossier.
Gap table template
A template for the gap overview table
Speaking points checklist
A list of high level subjects that should be discussed with clients
Dossier checklist
A checklist for checking quality, consistency and completeness of
the dossier.
Standard terminology
A draft for the standard terminology list, with room for
conversion into a survey.
Elicitation technique list
A list of available elicitation techniques, including descriptions,
context, obstacles, resources and complementary techniques.
69
Table 9 - Thesis deliverables
11.2 CONCLUSION
In the method section of this paper we presented the following research questions:
MQ – How can Keylane|Quinity improve its requirements engineering method in terms of efficiency
and effectiveness?
SQ 1 – How can the current processes and future needs for early requirements engineering
of a product software company be identified?
SQ 2 – How can the steps of the Method Association approach be applied to inform the
essential processes and deliverables of a method for the elicitation, capture, and analysis of
early requirements?
SQ 3 – How can the resulting method be implemented and evaluated?
To answer the first sub-question we modeled the requirements engineering methods as they
were documented by the company, and then validated it in interviews with individual users of
the method. This turned out to be a very effective approach, since a lot of variations and
inconsistencies were revealed. The company was aware of the existence of some variations, but
was never exposed to such concrete evidence. Based on these findings needs for a new method
were established, with the help of a senior manager.
For the second question we applied the MAA with some minor variations. We tackle these
variations in depth in section 11.4. The end result was a method that, according to review
participants, does embody the essential processes and deliverables of a requirements
engineering method.
The third and last question was answered with two group reviews as evaluation and the
addition of the contextualization step to the MAA. This topic is also discussed in depth in section
11.4. We hoped to have a project available to use as a case study for the resulting method, but
sadly nothing became available in time. During the last weeks a candidate project was identified,
for which we retain the opportunity to perform a case in a separate paper.
It was the ‘efficiency and effectiveness’ part of the main question that proved hardest to answer.
We assumed that efficiency could be achieved by finding techniques that delegate more work to
the clients (e.g. wiki, survey, KJ technique). For effectiveness we proposed that the 90% biggest
gaps should be found in the requirements phase, which we hope to achieve with several means
(an SDE document containing tips for reaching completeness, a speaking points checklist, a
document checklist). We do recognize that without without a case study it is not possible to
measure an actual increase in ‘efficiency and effectiveness’, but participants in the reviews
appeared very positive about the outcome regardless.
70
11.3 VALIDITY
The validity of this thesis will be discussed on four points: Credibility of the approach,
confirmability of the results, dependability of the created method and transferability of the
results to other cases[95].
11.3.1 Credibility of the approach
The credibility of the approach used depends for the most part on the credibility of the Method
Association Approach, which was used in this research. The MAA was originally published in the
workshop proceedings of the 4th model driven web-engineering conference. Since then it has
received attention from prominent names in IS research, particularly from the Utrecht
University and the Pantheon-Sorbonne University (Paris 1). Another recent publication was
made by the latter in the proceedings of the 9th conference on Research Challenges in
Information Science, published by the IEEE.
During this thesis we took some liberties with the original MAA on three occasions. We added an
early review of the feature groups, a review of candidate methods, and added the
contextualization step (see section 11.4 for more). We strongly believe that these steps added
value to both the approach and the project as a whole.
11.3.2 Confirmability of the results
Though somewhat methodical, the MAA does not provide a lot of guidance for the identification
of feature groups and candidate methods. We must therefore conclude that, if repeated by
another, the resulting method would likely turn out different. Particularly because the field of
requirements engineering is so broad and rich with potential features and methods, it is entirely
believable that another researcher would find other features and methods to include.
However, confirmability depends more on the qualities of the method and it’s ability to fulfill the
internship company’s needs. By carefully capturing and updating these business needs
throughout the thesis we have created a method that is perceivd by the company as very
satisfying.
11.3.3 Dependability of the created method
With the MAA being a situational, the method should, in theory at least, be fully adaptable and
usable regardless of project situations. The two major project situations, dominant versus
passive clients, is one of the main goals we explicitly kept in mind during the creation of the
method. After all the tailoring done as contextualization this method should be dependable.
To fully validate the dependability of the created method a case study would have been perfect.
Though we discussed the possibility of a case with the external coach early on, no suitable
implementation trajectory presented itself during the thesis until the very last weeks. We still
retain the opportunity to complete a case study as a separate paper after finishing this thesis.
11.3.4 Transferability of the results.
Though tailored for Keylane|Quinity, some of the results should definitely be transferable to
other software product producing companies. We further discuss the possible applications in
section 11.5.2 – business relevance.
71
11.4 POST-PROJECT REVIEW
After 5 years of books and classes, a project of this size and scale was a whole new world to me.
Not surprisingly, I learned some valuable lessons in the process.
11.4.1 Working on a fuzzy problem
During the first few weeks of the project it already became clear that this was not going to be a
typical class assignment. There was no single correct answer to this problem. No perfect method
that would guarantee a 10/10 outcome. In fact, on several occasions during interviews and
interactions with Keylane employees, it became clear that even they were not fully aware of
what they needed all the time.
I quickly realized that my preconception of having to follow a set path had to go out the window.
This project was going to take a lot of creativity. Sometimes this meant coming up with ideas,
trying them out, and going back to the drawing board when they were inevitably rejected. This
wasn’t always such a bad thing, since the end results always ended up better.
11.4.2 Working with the Method Association Approach
The Method Association Approach, or MAA for short, was a creation of another MBI student back
in 2008. For as far as I could find, this approach had only been used twice before. Once by the
author himself and once by another fellow MBI student, who actually applied it to Quinity’s data
warehousing method just one year prior. In addition, Deneckère and Hug from the Sorbonne
University were collaborating with Onderstal and Brinkkemper to generalize the MAA, and I was
allowed an early look into their paper. The final version was published in May of this year [22].
With these three sources I thought I had all I needed to apply the MAA at Keylane|Quinity. For
the most part, this remained true, except for some minor tweaks and adjustments along the way.
(1) Evaluate your features early.
The original paper by Luinenburg et al kept it simple: gather features from literature. Their
features were never reviewed. This did not appear to be the case for Deneckère et al., but only a
single sentence on the subject is written: “We categorized these features in 18 groups and after
informal interviews with two experts we combined several feature groups, resulting in 10
feature groups.”
I started out the same way both papers did, by digging around in literature for feature groups.
Particularly maturity models proved to be goldmines for this type of information. Their
relevance for Keylane|Quinity however, was questionable at best. Sure, a maturity model’s
feature of writing all requirements as formalized models counts, but is it desirable for
Keylane|Quinity? No. Early on the company put it’s foot down and demanded natural language
requirements. Anything else would simply be too big a change. To keep a ‘formalized model
notation’ feature, use it in association, and have it influence the final method would simply make
no sense. Our review of features and the resulting eliminations are reported on in section 5.2.4.
(2) There are two types of MAAs, with subtle differences in features.
During the collection of feature groups I noticed it was significantly harder for me to keep a clear
distinction between feature groups, without features overlapping or being dependent on one
anhother. This seemed to not at all be the case for Luinenburg et al.. Over time a key difference
between my goal and theirs became clear. They created a method that would be used to create
72
product: namely a piece of software. I was creating a method that would be used to create a
process: namely a requirements engineering process. At the time I named these two situations
Product related MAA and process related MAA.
In a later version of the Deneckère et al. paper the same distinction popped up. During their
creation of an implementation method they noticed that Luinenburg et al. focussed on WHAT
artifacts are produced, whereas they focussed on HOW artifacts are produced. They name these
two situations the product perspective and the process perspective.
(3) Candidate method selection does not exist in a vacuum.
“Based on a literature study, four mature web modelling methods and a [sic] propriety web
design method were selected” report Luinenburg et al.. Deneckère et al. follow suit in their case
study by introducing method selection with “as this step is independent from the others”. Both
sources are in agreement that selecting methods is an independent activity.
This was not at all the case for me. I found that method selection was dependent on my
understanding of the company, and my previous work identifying project situations and
features. Perhaps this is more due to the high availability and variety of RE methods. Our initial
unrestricted search for any methodical way of working with requirements rapidly resulted in a
list of 30 method. Luinenburg et al. and Deneckere et al. probably had a lot less to work with. It
makes sense then for them to perceive method selection as a separate activity, whereas I had to
be critical and cut some candidates early.
(4) Don’t do so much modelling!
Before method selection I had already modelled the current situation in an elaborate PDD. After
method selection I fired up Microsoft Visio again for the fourth MAA step: ‘model candidate
methods’.
Creating a full PDD for some of these methods proved to be a very time consuming effort. Getting
all the concepts, deliverables, conditional activities, flows, relations and modularities right takes
time, time that the external coach deemed too valuable for the creation of 8 models. He urged me
to keep the models simple. As a result I eliminated modularity, modelled irrelevant processes as
closed, and made no process or deliverable tables. In the end I finished the candidate method
modelling MAA step right when planned.
(5) Association/assembly as a creative process
The fifth and sixth steps of the MAA, associating and building a preliminary method, proved to
be the hardest to explain properly. Both Luinenburg et al. and Deneckère et al. take wildly
different approaches.
Luinenburg et al. suggest two creative approaches. In the first, start from a feature group and see
which method fragments satisfy it. Which one of the candidates to select is not explained, but
Luinenburg et al. suggest that this approach can then be used to compare the satisfying method
fragments, which one would assume leads to a decision. The other approach starts from a
method fragment and sees how many feature groups it satisfies. The method fragments that
satisfy the most feature groups are deemed more valuable and are selected for assembly.
Deneckère et al. make a, quite frankly, bizarre attempt at doing a formal analysis of feature
groups and method fragments before association using rules based on logic. Some examples are:
 If a feature name or group name is the same as a method fragment name, they match.
73

If a method fragment’s name, goal or action is included in the name of feature (groups)
they match.
Similar rules are based on synonyms occurring in features and fragment names, or words from
one appearing within proximity in the other name.
Deneckère et al. make the claim that this analysis can be used to prove that all feature groups
can be satisfied with the gathered method fragments. I wonder how this proof could be
considered valid since they came up with the names for feature groups themselves. Any name
overlaps, synonyms or word proximities depend on the names they have chosen. There’s no
conceivable way to prove that naming was done objectively. I did not include Deneckere et al.’s
approach as a result.
(6) Added the ‘contextualize method’ step.
The result of method assembly is a preliminary method that should, in theory, satisfy all the
company’s needs. Some steps may be too elaborate, complicated or irrelevant, so there is an
evaluation step included in the MAA. After evaluation the result is a method that should both
theoretically and practically satisfy the company’s needs.
The first thing I noticed was that after evaluation, some method fragments still lacked the
contents needed to be useful or accepted by the company. ‘Send out the user survey’ for
instance, is hardly a useful method fragment without an actual user survey to send. Other
method fragments lacked integration into the company, like ‘define project’. Sounds like fun, but
how is the actual definition going to be done? Which contents constitute a good project
definition that this company can benefit from?
I added and named a new step to the MAA: ‘contextualize method’ to address these issues. In this
step a method engineer formally takes the time to create documents and training materials to
make the method ready for use. This step is also used to create supporting artefacts required by
the method, like questionnaires, surveys, checklists and templates.
Deneckère et al. performed a similar activity in their case, but declared it a a part of the assembly
step. This does not seem like a very efficient choice, since many method fragments will be
changed or removed during evaluation. Your contextualization efforts may be lost if the
associated method fragments don’t pass evaluation.
(7) Estimating planning
Estimating an accurate planning is never easy, but for this long project it was much harder than
I’d ever experienced in the two-month timeframe of University courses.
Originally I planned for association and preliminary method assembly to last a month each, not
having had any experience with the MAA at all. This turned out to be a gross overestimation,
since the combined effort for these two steps would barely take half a month. On the 25th of
March I began sending out the invitations to an evaluation session. An activity that was
originally planned for the first of June.
This was no problem however, since contextualizing the method (i.e. processing evaluation
results, creating supporting documents, checklists, templates etc.) would take the better part of
2 months. An activity that was originally planned for only one month.
Remember the generalized MAA presented in figure 6? The actual application of the MAA in this
project would have looked more like this:
74
Figure 30 - MAA as used in the thesis
11.5 RELEVANCE
11.5.1 Scientific Relevance. The first and obvious contribution this research can make is a new
case study of the MAA in practice. This will be the first ever application of the MAA to
requirements engineering, providing an opportunity for an interaction between situational
method engineering and requirements engineering.
Besides the relevance of the case study itself, this thesis has some contributions to offer to the
MAA itself. Mainly the added feature evaluation, early method evaluation and addition of
method contextualization:
 A feature evaluation helped us tremendously, by creating a feature list that was a lot
more concise and a lot more relevant to the company. We would recommend using it
when applying the MAA to a broad and mature field like requirements engineering.
 The early method evaluation did not just compact the method base, but also gave the
recipients of the would-be method a more concrete example of ‘method fragments’ and
what these fragments contain.
 Contextualization was an absolute necessity to make the resulting method ready for use.
Some of the fragments definitely needed some tailoring in order to be accepted by the
client company, and any method fragments that require supporting artefacts are
inherently useless if nobody creates those artefacts. Maybe contextualization is more of
an activity for Process Engineering or Change Management than Method Engineering,
but in our case it led to the delivery of a far better method.
75
11.5.2 Business Relevance. The first major contribution to the business world is specific for
Keylane|Quinity. The final deliverable of the internship will be a tailored method for the
company, adjustable to the different project situations and with supporting artefacts ready to go.
To the market of product software businesses, this research provides insight in the creation of
requirements engineering methods. This research also provides a proof of concept of the
Method Association Approach, creating opportunities for new cases. In addition to showing a
way of implementing the MAA, this research provides detailed descriptions of feature group
gathering, method selection, association and assembly that can be used to support future MAA
projects.
Lastly, the resulting method itself may be of use to other product software businesses.
Specifically the following:



Small to medium sized. The company in this thesis employs approximately 150 people.
For a company this size large scale requirements engineering tools such as IBM’s
Rational DOORS or Borland’s Caliber may be too big of an expense. The techniques used
in this method are low cost, low impact alternatives to the adoption of large tools, and
are conductive to agile tools like Atlassian’s Confluence.
Smaller than it’s clients. The company in this thesis is usually a lot smaller, both in
terms of personnel and revenue, than it’s clients. This means that the company has to:
o Actively compete for the opportunity to become a supplier, rather than having
clients come to them. The method assumes that a supplier selection process will
be the norm, and provides the means to make the most from that process.
o Follow the lead of clients, because the balance of power is not in their favor. The
method assumes that some clients will be dominant and others will be passive
when it comes to directing the flow of information. This would not be the case as
often for large software suppliers, who have the reputation and clout to be the
dominant party in most implementations. One of the method’s major goals is
ensuring the same quality of requirements regardless of client disposition.
Complex domain. The company in this thesis supplies to the insurance market. A high
complexit y market with no shortage on third party standards, government regulations
and rapidly changing trends. The method facilitates working with clients in this market
through techniques like a standard terminology survey and a speaking points checklist.
o The standard terminology survey prevents miscommunication by presenting
clients with a standard list of the company’s definitions for market specific
entities and objects (e.g. coverage, intermediary, insured object). The list
contains room for comment from the client, and can help both parties detect
potential miscommunications early in an implementation project.
o The speaking points checklist is a large list of high level concepts that can be
affected by an insurer’s software. Speaking points are divided into typical
functional areas (for insurance examples are premium calculation, request
administration and policy administration).
It is worth re-iterating that this project was scoped only to functional requirements and project
phases leading up to the design, development, and testing of software. Those looking for a
method that also covers non-functional requirements and/or requirements engineering
throughout the entire SDLC will likely find the method lacking.
76
REFERENCES
1.
Regnell, B., Brinkkemper, S.: Market-driven Requirements Engineering for Software
Products. Engineering and Managing Software Requirements, 287-308 (2005)
2.
Boehm, B.: Verifying and Validating Software Requirements and Design Specifications.
IEEE software, 32-41 (1991)
3.
Nuseibeh, B., Easterbrook, S.: Requirements Engineering: a Roadmap. In : Proceedings of
the Conference on the Future of Software Engineering (ICSE 00), Limerick, Ireland, pp.3546 (2000)
4.
Ratchev, S., Urwin, E., Muller, D., Pawar, K., Moulek, I.: Knowledge based requirement
engineering for one-of-a-kind complex systems. Knowlege-Based systems 16(1), 1-5
(2003)
5.
Luinenburg, L., Jansen, S., van de Weerd, I., Brinkkemper, S.: Designing Web Content
Management Systems Using the Method Association Approach. In Koch, N., Houben, G.,
Vallecillo, A., eds. : Proceedings of the 4th International Workshop on Model-Driven Web
Engineering, Toulouse, France, pp.106-120 (2008)
6.
Wiegers, K., Beatty, J.: Software Requirements 3rd edn. Pearson Education (2013)
7.
Aurum, A., Wohlin, C.: Engineering and Managing Software Requirements. Springer Science
& Business Media, New York, NY (2005)
8.
Cannegieter, H., Arensen, M.: Success met de Requirements! 2nd edn. Academic service,
The Hague, Netherlands (2008)
9.
Bourque, P., Fairley, R.: Guide to the Software Engineering Body of Knowledge (SWEBOK)
3rd edn. IEEE Computer Society Press, Los Alamitos, CA (2006)
10. Larson, E., Satterthwaite, R.: Searching the literature: A professional imperative. American
Journal of Infection Control 17(6), 359-364 (1989)
11. Johnson, J.: The Chaos Report. The Standish Group, West Yarmouth, MA (2008)
12. Langer, A.: System Development Life Cycle. In : Analysis and Design of Information
systems 3rd edn. pp. 10-20. Springer, London (2008)
13. Nurmuliani, N., Zowghi, D., Fowell, S.: Analysis of Requirements Volatility During Software
Development Life Cycle. In : Proceedings of the 15th Australian Software Engineering
Conference, Melbourne, Australia, pp.28-37 (2004)
14. Regnell, B., Beremark, P., Eklundh, O.: A Market-driven Requirements Engineering Process:
Results From an Industrial Process Improvement Programme. In : Requirements
Engineering 2nd edn. 3. Springer-Verlag, Berlin, Germany (1998)
77
15. Carlshamre, P.: Release Planning in Market-Driven Software Product Development.
Provoking an Understanding. In : Requirements Engineering 3rd edn. 7., Berlin, Germany
(2002) 139-151
16. Bergstra, J., Jonkers, H., Obbink, J.: A Software Development Model for Method Engineering.
Elsevier Science Publishers, North Holland, Netherlands (1985)
17. Brinkkemper, S.: Method Engineering: Engineering of Information Systems Development
Methods., 275-280 (1996)
18. Rolland, C., Prakash, N.: A Proposal for Context-specific Method Engineering. In : Method
Engineering. Springer-Verlag, New York (1996) 191-208
19. Ralyté, J., Deneckère, R., Rolland, C.: Towards a Generic Model for Situational Method
Engineering. In : 15th International Conference on Advanced Information Systems
Engineering, Klagenfurt, Austria, pp.95-110 (2003)
20. Van de Weerd, I., Brinkkemper, S., Souer, J., Versendaal, J.: A Situational Implementation
Method for Web-based Content Management System-applications: Method Engineering
and Validation in Practice. Software Process: Improvement and Practice 11(5), 521-538
(2006)
21. Van de Weerd, I., Brinkkemper, S.: Meta-modeling for Situational Analysis and Design
Methods. In : Handbook of Research on Modern Systems Analysis and Design Technologies
and Applications 1st edn. 3. IGI Global, Hershey, PA (2008)
22. Deneckere, R., Hug, C., Onderstal, J., Brinkkemper, S.: Method Association Approach:
Situational construction and evaluation of an implementation method for software
products. In : RCIS, IEEE 9th International Conference on, Athens, Greece, pp.274-285
(2015)
23. Hunink, I., Van Erk, R., Jansen, S., Brinkkemper, S.: Industry Taxonomy Engineering: the
Case of the European Software Ecosystem. In : Proceedings of the Fourth European
Conference on Software Architecture: Companion Volume, pp.111-118 (2010)
24. CMMI Product Team: CMMI for Development (CMMI-DEV) v1.2. Carnegie Mellon
University/Software Engineering Institute, Pittsburgh, PA (2006)
25. Beecham, S., Hall, T., Rainer, A.: Defining a requirements process improvement model.
Software Quality Journal 13(3), 247-279 (2005)
26. Solemon, B., Shahibuddin, S., Ghani, A.: Re-defining the Requirements Engineering Process
Improvement Model. In : Asia Specific Software Engineering Conference, Penang, Malaysia,
pp.87-92 (2009)
27. Svahnberg, M., Gorschek, T., Nguyen, T., Nguyen, M.: Uni-REPM: Validated and Improved.
Requirements Engineering 18(1), 85-103 (2013)
78
28. Hickey, A., Davis, A.: A unified model of requirements elicitation. Journal of Management
Information Systems 20(4), 65-84 (2004)
29. Finkelstein, A., Potts, C.: Evaluation of existing requirements extraction Strategies. Report,
FOREST Project, London (1985)
30. Bekkers, W., van de Weerd, I., Spruit, M., Brinkkemper, S.: A framework for process
improvement in software product management. In : Systems, Software and Services
Process Improvement: 17th European Conference. Springer Berlin Heidelberg, Grenoble,
France (2010) 1-12
31. Carlshamre, P., Sandahl, K., Lindvall, M., Regnell, B.: An industrial survey of requirements
interdependencies in software product release planning. In : IEEE International
Symposium on Requirements Engineering, pp.84-91 (2001)
32. van Gurp, J., Bosch, J., Svahnberg, M.: On the notion of variability in software product lines.
In : Working IEEE/IFIP Conference on Software Architecture, Amsterdam, Netherlands,
pp.45-54 (2001)
33. Fayad, M.: Accomplishing Software Stability. Communications of the ACM 45(1), 111-115
(2002)
34. Boehm, B., Egyed, B.: Software requirements negotiation: some lessons learned. In : IEEE
International Conference on Software Engineering, Dunedin, New Zealand, pp.503-506
(1998)
35. Vessey, I., Conger, S.: Requirements specification: learning object, process, and data
methodologies. Communications of the ACM 37(5), 102-113 (1994)
36. Paetsch, F., Eberlein, A., Maurer, F.: Requirements engineering and agile software
development. In : Proceedings of the twelfth IEEE WET ICE conference, Linz, Austria,
pp.308-313 (2003)
37. Karlsson, J., Wohlin, C., Regnell, B.: An evaluation of methos for prioritizing software
requirements. Information and Software Technology 39(14), 939-947 (1998)
38. Clegg, D., Barker, R.: Case method fast-track: a RAD approach. Addison-Wesley Longman
Publishing Co., Inc. (1994)
39. Karlsson, J., Ryan, K.: A cost-value approach for prioritizing requirements. IEEE Software
14(5), 67-74 (1997)
40. Berander, P., Andrews, A.: Requirements prioritization. In : Engineering and managing
software requirements. IV. Springer Berlin, Heidelberg (2005)
41. Günther, R., Saliu, M.: The art and science of software release planning. IEEE Software
22(6), 47-53 (2005)
79
42. Fischer, K., Walker, G.: Improved software reliability through requirements verification.
IEEE transactions on Reliability 28(3), 233-240 (1979)
43. Marciniak, J., Shumskas, A.: Acceptance Testing. John Wiley & Sons, inc. (1994)
44. Gotel, O., Finkelstein, A.: An analysis of the requirements traceability problem. In :
Proceedings of the First international conference on Requirements Engineering., pp.94101 (1994)
45. Galin, D.: Software quality assurance: from theory to implementation. Pearson education
(2004)
46. Maiden, N., Rugg, G.: ACRE: selecting methods for requirements acquisition. Software
Engineering Journal 11(3), 183-192 (1996)
47. Jiang, L., Eberlein, A., Far, B., Mousavi, M.: A methodology for the Selection of Requirements
Engineering Techniques. Software & Systems Modeling 7(3), 303-328 (2008)
48. Carrizo, D., Dieste, O., Juristo, N.: Systematizing requirements elicitation technique
selection. Information and Software Technology 56(6), 644-669 (2014)
49. Andrews, D.: A crucial dimension for rapid applications development. Journal of Systems
Management 42(3), 23-27 (1991)
50. Wood, J., Silver, D.: Joint Application Development 2nd edn. John Wiley & Sons (1995)
51. Duggan, E., Thachenkary, C.: Higher Quality Requirements: Supporting Joint Application
Development with the Nominal Group Technique. Information Technology and
Management 4(4), 391-408 (2003)
52. Knowledge Structures, Inc.: JAD Guidelines. (Accessed February 5, 2015) Available at:
http://goo.gl/ey88gq
53. Carmel, E., Whitaker, R., George, J.: PD and Joint Application Design: A Transatlantic
Comparison. Communications of the ACM 36(6), 40-48 (1993)
54. Yatco, M.: Joint Application Design/Development. In: System Analysis, Management
Information Systems (MIS) Program. (Accessed February 6, 2015) Available at:
http://www.umsl.edu/~sauterv/analysis/JAD.html
55. Mcconnell, S.: Rapid Development: Taming Wild Software Schedules. Microsoft Press, 449463 (1996)
56. Carmel, E., George, J., Nunamaker, J.: Supporting Joint Application Development (JAD) and
Electronic Meeting systems: Moving the CASE Concept Into New Areas of Development. In
: Proceedings of the Twenty-Fifth Hawaii Internation Conference on System Sciences,
Hawaii, vol. 3, pp.331-342 (1992)
80
57. Davidson, E.: Joint Application Design (JAD) in practice. Journal of Systems and Software
45(3), 215-223 (1999)
58. Maiden, N., Jones, S.: An Integrated User-Centred Requirements Engineering Process,
version 4.1. Design document, Centre for HCI Design, City University, London (2004)
59. Maiden, N., Robertson, S.: Integrating creativity into requirements processes: Experiences
with an air traffic management system. In : Proceedings of the 13th IEEE international
conference on Requirements Engineering, pp.105-114 (2005)
60. Maiden, N., Jones, S., Manning, S., Greenwood, J., Renou, L.: Model-driven requirements
engineering: synchronising models in an air traffic management case study. Advanced
Information Systems Engineering, 368-383 (2004)
61. Schwanke, R.: GEAR: A good enough architectural requirements process. In : The fifth
Working IEEE/IFIP conference on software architecture, pp.57-66 (2005)
62. Koch, R.: The 80/20 principle: The secret to achieving more with less. Crown Publishing,
New York, NY (2011)
63. Bresciani, P., Perini, A., Giorgini, P., Giunchiglia, F., Mylopoulos, J.: Tropos: An agentoriented software development methodology. Autonomous Agents and Multi-Agent
Systems 8(3), 203-236 (2004)
64. Giorgini, P., Massaci, F., Mylopoulos, J.: Requirement Engineering meets Security: A Case
Study on Modelling Secure Electronic Transactions by VISA and Mastercard. In :
Conceptual Modeling-ER, pp.263-276 (2003)
65. Bryl, V., Dalpiaz, F., Ferrario, R., Mattioli, A.: Evaluating Procedural Alternatives. A Case
Study in E-Voting. Electronic Government, an International Journal 6(2), 213-231 (2009)
66. Morandini, M., Nguyen, D., Perini, A., Siena, A., Susi, A.: Tool-Supported Development with
Tropos: The Conference Management System Case Study. In : Agent-Oriented Software
Engineering 8th edn. Springer Berlin Heidelberg (2008)
67. Mead, N., Stehney, T.: Security Quality Requirements Engineering (SQUARE) methodology.
ACM SIGSOFT Software Engineering Notes 30(4), 1-7 (2005)
68. Mead, N., Hough, E.: Security requirements engineering for software systems: Case studies
in support of software engineering education. In : Proceedings of the 19th Conference on
Software Engineering Education and Training, pp.149-158 (2006)
69. Chen, P., Dean, M., Ojoko-Adams, D., Osman, H., Lopez, L.: Systems quality requirements
engineering (square) methodology: Case study on asset management system. Technical
report CMU/SEI-2004-SR-015, Carnegie Melon University/Software Engineering Institute,
Pittsubrgh, PA (2004)
81
70. Mead, N., Konrad, M., Stoddard, R.: Eliciting Unstated Requirements: Tutorial presentation.
In : Workshop at the 22nd IEEE International Requirements Engineering Conference,
Karlskrona, Sweden (2014)
71. Kawakita, J.: The KJ method - A scientific approach to problem solving. Technical report,
Kawakita Research Institute, Tokyo, Japan (1975)
72. Noriaki, K., Seraku, N., Takahashi, F., Tsuji, S.: Attractive quality and must-be quality.
Journal of the Japanese Society for Quality Control 14(2), 147-156 (1984)
73. Lloyd, V., Rudd, C.: ITIL V3 - service design (vol. 5). TSO (2007)
74. Adlassnig, K.: ITIL Assessment in a healthcare environment: the role of IT governance at
Hospital Sao Sebastiao. In : Medical Informatics in a United and Healthy Europe:
Proceedings of MIE 2009. IOS press (2009) 76-80
75. Doran, G.: There's a SMART way to write management's goals and objectives. Management
Review 70(11), 35-36 (1981)
76. Moore, J.: Communicating requirements using end-user GUI constructions with
argumentation. In : 18th IEEE International Conference on Automated Software
Engineering, pp.360-363 (2003)
77. Lohmann, S., Ziegler, J., Heim, P.: Involving end users in distributed requirements
engineering. In : 2nd Conference on Human-Centered Software Engineering, Pisa, Italy,
pp.221-228 (2008)
78. Schuler, D., Namioka, A.: Participatory design: Principles and practices. L. Erlbaum
Associates Inc., Hillsdale, NJ (1993)
79. Lieberman, H., Paternò, F., Wulf, V.: End-User development: An emerging paradigm 1st
edn. Springer Netherlands (2006)
80. Tsumaki, T., Tamai, T.: Framework for matching requirements elicitation techniques to
project characteristics. Software Process: Improvement and Practice 11(5), 505-519
(2006)
81. Snijders, R., Dalpiaz, F., Hosseini, M., Shahri, A., Ali, R.: Crowd-Centric Requirements
Engineering. In : Proceedings of the Second International Workshop on Crowdsourcing
and Gamification in the Cloud, London, UK (2014)
82. Brabham, D.: Crowdsourcing. MIT Press, Cambridge, MA (2013)
83. Zichermann, G., Cunningham, C.: Gamification by design: Implementing Game Mechanics in
Web and Mobile Apps. O'Reilly Media Inc., Sebastopol, CA (2011)
84. Lim, S., Quercia, D., Finkelstein, A.: StakeSource: Harnessing the Power of Crowdsourcing
and Social Networks in Stakeholder Analysis. In : Proceedings of the 32nd ACM/IEEE
82
International Conference on Software Engineering, Cape Town, South Africa, vol. 2,
pp.239-242 (2010)
85. Lim, S., Damian, D., Finkelstein, A.: StakeSource 2.0: Using Social Networks of Stakeholders
to Identify and Prioritise Requirements. In : Proceedings of the 33rd International
Conference on Software Engineering, Waikiki, Hawaii, pp.1022-1024 (2011)
86. Kazman, R., Chen, H.: The metropolis model a new logic for development of crowdsourced
systems. Communications of the ACM 52(7), 76-84 (2009)
87. Kleemann, F., Voß, G.: Un(der)paid Innovators: The Commercial Utiliza-tion of Consumer
Work through Crowdsourcing. Science, Technology & Innovation Studies 4(1), 5-26 (2008)
88. Negri, M., Mehdad, Y., Giampiccolo, D., Marchetti, A.: Divide and conquer: Crowdsourcing
the creation of cross-lingual textual entailment corpora. In : Proceedings of the Conference
on Empirical Methods in Natural Language Processing, Edinburgh, UK, pp.670-679 (2011)
89. Wagner, C.: Wiki: A technology for conversational knowledge management and group
collaboration. The Communications of the Association for Information Systems 13(1), 265289 (2004)
90. Fritz, S., McCallum, I., Schill, C., Perger, C., Grillmayer, R., Achard, F., Obersteiner, M.: GeoWiki.Org: The use of crowdsourcing to improve global land cover. Remote Sensing 1(3),
345-354 (2009)
91. Bradshaw, P.: Wiki Journalism: Are Wikis the New Blogs? In : Future of Newspapers
Conference2007, Cardiff, UK, pp.1-22 (2007)
92. Decker, B., Ras, E., Rech, J., Jaubert, P., Rieth, M.: Wiki-based Stakeholder Participation in
Requirments Engineering. IEEE Software 24(2), 28-35 (2007)
93. Silveira, C., Faria, J., Aguiar, A., Vidal, R.: Wiki based requirements documentation of
generic software products. In : Proceedings of the 10th Australian Workshop on
Requirements Engineering, Melbourne, Australia, pp.42-51 (2005)
94. Ferreira, D., da Silva, A.: Wiki supported collaborative requirements engineering. In :
Wikis4SE Workshop, Porto, Portugal (2008)
95. Lincoln, Y., Guba, E.: Naturalistic enquiry. Sage, Beverly Hills, CA (1985)
96. Pohl, K., Rupp, C.: Requirements Engineering Fundamentals 1st edn. Rocky Nook, Inc.
(2011)
97. Ryan, K.: The Role of Natural Language in Requirements Engineering. In : Proc. of the IEEE
Int. Symp. on RE, pp.240-242 (1993)
98. Natt och Dag, J., Gervasi, V., Brinkkemper, S., Regnell, B.: Speeding up Requirements
Management in a Product Software Company: Linking Customer Wishes to Product
83
Requirements through Linguistic Engineering. In : proc. of the 12th Int. RE conference,
pp.283-294 (2004)
99. Yadla, S., Hayes, J., Dekhtyar, A.: Tracing Requirements to Defect Reports: An Application of
Information Retrieval Techniques. Innovations in Systems and Software Engineering 1(2),
116-124 (2005)
100. Sampaio, A., Loughran, N., Rashid, A., Rayson, P.: Mining Aspects in Requirements. In :
Early Aspects workshop 2005, Aspect-Oriented Requirements Engineering and
Architecture Design, Chicago, IL, vol. 1 (2005)
101. Anthony, L.: AntConc (version 3.4.3) [Computer Software]. Waesda University, Tokyo,
Japan http://www.laurenceanthony.net.
102. Liu, F., Pennell, D., Liu, F., Liu, Y.: Unsupervised Approaches for Automatic Keyword
Extraction Using Meeting Transcripts. In : Proc. of human language tecnologies: 2009
annual conference of the NAACL, pp.620-628 (2009)
103. Paquot, M., Bestgen, Y.: Distinctive Words in Academic Writing: A Comparison of Three
Statistical Tests for Keyword Extraction. Language and Computers 68(1), 247-269 (2009)
104. Rayson, P., Garside, R.: Comparing Corpora Using Frequency Profiling. In : proc. of the
Workshop on Comparing Corpora, 38th Annual Conference of the ACL, Hong Kong, pp.1-6
(2000)
105. Rayson, P.: Log-likelihood Calculator., Lancaster, UK
http://ucrel.lancs.ac.uk/llwizard.html.
106. Luhn, H.: Key Word-in-context Index for Technical Literature (kwic index). American
Documentation 11(4), 228-295 (1960)
107. Brennan, K.: A Guide to the Business Analysis Body of Knowledge (Version 2).
International Institute of Business Analysis (2009)
84
APPENDIX
APPENDIX 1 – GAP ANALYSIS META MODEL
* for the full method, see the attached Quinity1.3.vsd
This is the meta model for the GAP analysis activity, which was not part of the current method
section since it is technically out of scope. Included here as extra context to the Keylane|Quinity
development method.
1
APPENDIX 2 – CURRENT SITUATION PROCESS/DELIVERABLE TABLES
Two tables containing descriptions of all activities and deliverables as seen in the PDD models of the current method. Structure and contents as
proposed by Van de Weerd and Brinkkemper (2008).
Activity
Sales trajectory
Sub-Activity
Organize information
sessions
Class
Optional
Answer RFI
Optional
Answer RFP
Optional
Setup preliminary contract
Optional
Show proof of concept
Optional
Perform global RQ analysis
Optional
Description
The potential customer plans a number of information sessions with the sales
department. During sessions information about QIS and it’s capabilities is
exchanged. Session results are documented in SESSION NOTES that may contain
PRELIMINARY REQUIREMENTS.
A Request for Information (RFI) is answered by sales, in cooperation with product
experts. The RFI contains broad questions about the product, and is directed at
multiple suppliers as a means of supplier selection by prospective clients.
PRELIMINARY REQUIREMENTS can be gleaned from RFI questions. According to
experts, this is rarely done in the current process.
A Request for Proposal (RFP) is answerd by sales, in cooperation with product
experts. An RFP is a far more specific follow up to the RFI and can easily reach up to
a hundred pages. PRELIMINARY REQUIREMENTS can be gleaned from RFP
questions. According to experts, this is rarely done in the current process.
When a customer chooses to buy from Quinity, a PRELIMINARY CONTRACT can be
created if they deem it necessary. Preliminary contracts cover all activities leading
up to the increments.
If desired the customer receives one or more Proof of Concept (POC) presentations.
In these sessions a QIS demo environment is used to show client cases being
performed. In some cases a prototype with new functionality can also be created
specifically for the POC. The contents, costs and time consumed by the POC are
agreed upon in the PRELIMINARY CONTRACT if there is one.
If desired the customer can request a global RQ analysis. During a relatively short
period of time (for example one week) workshops are held to set up a GLOBAL RQ
ESTIMATE. The estimate contains a discussion of known requirements, expected
future requirements and the estimated time and money needed to fill gaps. The
contents, costs and time consumed by the global RQ analysis are agreed upon in the
PRELIMINARY CONTRACT if there is one.
2
Project setup
Increments
Explain QIS Implementation
method
Regular
Define project increments
Regular
Create plan of approach
Regular
Agree on project approach
Regular
Train the project team
Regular
Hold workshop
Regular
Document results
Share results with customer
Regular
Regular
Gap analysis
Regular
Estimate knowledge for next
increment
Irregular
The QIS implementation METHOD DESCRIPTION is given to the client, at this time
usually only to client management and project leaders. The METHOD DESCRIPTION
is included in the FINAL CONTRACT.
PROJECT INCREMENTS are defined in collaboration with the client. The PROJECT
INCREMENTS are included in the FINAL CONTRACT.
A PROJECT PLANNING is made with the customer. The activities in increments are
planned, including a cost estimation for solving the requirements and gaps in those
increments. The PROJECT PLANNING is included in the FINAL CONTRACT.
The METHOD DESCRIPTION, PROJECT GOALS and PROJECT PLANNING are
officially presented as part of the FINAL CONTRACT. If the client signs off on this
contract, the project officially begins.
Trainings are given to key stakeholders to prepare them for t their role in the
project. During trainings stakeholders will work with QIS demo’s. Trainings can
yield TRAINING NOTES that may contain KNOWN REQUIREMENTS.
Workshops are held with key customer stakeholders on the current increment’s
topic or a subsection of that topic (e.g. policy administration, premium calculation
etc.). In workshops the following things can be discussed:
 An introduction to QIS and a broad description of functionalities
 An analysis of the client’s current system and processes.
 Standard functionality of QIS modules related to the topic.
 Configuration options of QIS modules related to the topic.
 Elaboration on KNOWN REQUIREMENTS
During the workshop sessions WORKSHOP NOTES are made.
The WORKSHOP NOTES are refined into DETAIL REQUIREMENTS.
The DETAIL REQUIREMENTS are shared with the customer for their CUSTOMER
REVIEW. If deemed insufficient REFINED DETAIL REQUIREMENTS are made, or
additional workshops are held. If the customer approves of the requirements, QIS
experts document them in a REQUIREMENTS DOCUMENT.
The requirements in the REQUIREMENTS DOCUMENT are solved as shown in
Appendix 1.
Any knowledge (including requirements) that is relevant for the next increment is
documented and handed over. This is reportedly not always done well, but outside
of the scope of this project.
3
Gap Analysis full
version
(Appendix 1)
Solve by configuration
Regular
Solve by process change
Regular
Determine gap
priority/impact
Regular
Solve by adapting existing
functionality
Regular
Solve by adding functionality Regular
to the core
Solve by making custom
solution
DELIVERABLE
SESSION NOTES
PRELIMINARY REQUIREMENTS
DEMO NOTES
GLOBAL RQ ESTIMATE
PRELIMINARY CONTRACT
METHOD DESCRIPTION
Regular
If a requirement can be solved by configuration of existing QIS functionalities, the
configuration is made and the requirement is considered a SOLVED REQUIREMENT.
If a requirement is already solved by QIS, but in a way that is not compatible with
client processes, Quinity experts will push for the customer to adapt their
processes. A process change would mean that no custom development is needed,
which in turn is better for the standard product and less costly for the client. If
successful the requirement is considered a SOLVED REQUIREMENT.
Requirements that cannot be solved through configuration or process change are
considered to be GAPS. GAPS are always functional requirements that are currently
not or insufficiently fulfilled by QIS.
The first priority for dealing with GAPS is to solve them by adapting existing QIS
functionalities. Extensions, workarounds and adaptations all fall in this category. If
this is possible, a REQUEST FOR CHANGE is created and sent to developers.
The second priority for dealing with GAPS is to solve them by adding new
functionality to the core (standard) QIS product. If this is possible, a REQUEST FOR
CHANGE is created and sent to developers.
The absolute last resort for dealing with GAPS is to design a custom solution for the
client only. Again the resulting design is captured in a REQUEST FOR CHANGE and
sent to developers.
Description
SESSION NOTES document the results of information sessions in the sales trajectory.
PRELIMINARY REQUIREMENTS are requirements that can be gleaned from SESSION NOTES, RFIs and RFPs,
captured in natural language [1, 2]. Note: If a requirement is expected to require some form of development
to solve, it is recognized as a gap. In Quinity’s own terms the requirement will be named a gap.
DEMO NOTES are notes taken during QIS demo sessions.
A GLOBAL RQ ESTIMATE is a list of preliminary- and expected requirements, including an estimate of their
time and resource costs [3].
A PRELIMINARY CONTRACT contains booked demo sessions, a proof of concept, and any other activities
leading up to the increments. Contracts are optional at this point and can be subjected to change later in the
project [4].
A METHOD DESCRIPTION contains a description of Keylane|Quinity’s approach to implementations [5].
4
PROJECT INCREMENTS
PROJECT PLANNING
FINAL CONTRACT
KNOWN REQUIREMENTS
WORKSHOP NOTES
DETAIL REQUIREMENTS
CUSTOMER REVIEW
REFINED DETAIL
REQUIREMENTS
DESIGN DOCUMENT
SOLVED REQUIREMENTS
GAPS
REQUEST FOR CHANGE
PROJECT INCREMENTS are the major iterations in which a QIS implementation will be carried out. Each
increment has a theme and covers several requirements, QIS components or functionalities.
A PROJECT PLANNING sets deadlines for the project and start/end dates for the increments. This includes
deadlines for individual activities and deliverables[6].
The FINAL CONTRACT is the final legal definition of the project, agreed upon- and signed by the customer
[4].
KNOWN REQUIREMENTS have been previously identified from the PRELIMINARY REQUIREMENTS,
TRAINING NOTES, and DEMO NOTES [2].
WORKSHOP NOTES are made on the fly during design workshops.
After workshops DETAIL REQUIREMENTS are gathered from the WORKSHOP NOTES [7]. DETAIL
REQUIREMENTS are highly specific statements of what QIS must do, which are used to inform design
decision. Most of these will be derived from KNOWN REQUIREMENTS, but some new requirements may still
pop up.
A CUSTOMER REVIEW is the result of sharing DETAIL REQUIREMENTS with the customer. A form of
stakeholder participation in RE [8].
After processing CUSTOMER REVIEWs the detail requirements are refined and perfected into REFINED
REQUIREMENTS [2].
REFINED REQUIREMENTS are gathered in DESIGN DOCUMENTS. These documents are usually large
spreadsheet in which each designer gets his own tab [9].
SOLVED REQUIREMENTS are requirements that were solved by simple configuration of QIS or adapting
customer processes to QIS. If you were to subtract all SOLVED REQUIREMENTS from all KNOWN
REQUIREMENTS, only the GAPS would be left.
GAPS are those REFINED REQUIREMENTS that cannot be solved without development efforts [10]. Note:
only functional requirements can be considered GAPS.
A REQUEST FOR CHANGE is a definition of what needs to be done to solve a GAP [11].
[1] Ambriola, V., & Gervasi, V. (1997, November). Processing natural language requirements. In Automated Software Engineering, 1997. Proceedings., 12th IEEE
International Conference (pp. 36-45). IEEE.
[2] Yu, E. S. (1997, January). Towards modelling and reasoning support for early-phase requirements engineering. In Requirements Engineering, 1997., Proceedings of the
Third IEEE International Symposium on (pp. 226-235). IEEE.
[3] Carlshamre, P. (2002). Release planning in market-driven software product development: Provoking an understanding. Requirements Engineering, 7(3), 139-151.
5
[4] Umble, E. J., Haft, R. R., & Umble, M. M. (2003). Enterprise resource planning: Implementation procedures and critical success factors. European journal of operational
research, 146(2), 241-257.
[5] Lim, C. S., & Mohamed, M. Z. (1999). Criteria of project success: an exploratory re-examination. International journal of project management, 17(4), 243-248.
[6] Nutt, P. C. (1983). Implementation Approaches for Project Planning. 1. Academy of Management Review, 8(4), 600-611.
[7] Hull, E., Jackson, K., & Dick, J. (2005). Requirements engineering (Vol. 3). London: Springer.
[8] Decker, B., Ras, E., Rech, J., Jaubert, P., & Rieth, M. (2007). Wiki-based stakeholder participation in requirements engineering. Software, IEEE, 24(2), 28-35.
[9] Heninger, K. L. (1980). Specifying software requirements for complex systems: New techniques and their application. Software Engineering, IEEE Transactions on, (1),
2-13.
[10] Pol, P., & Madhup, P. (2011). SAP. Methods of Fit Gap Analysis in SAP ERP Projects. Bangalore, India: Infosys technologies limited.
[11] Bennett, K. H., & Rajlich, V. T. (2000, May). Software maintenance and evolution: a roadmap. In Proceedings of the Conference on the Future of Software Engineering
(pp. 73-87). ACM.
6
APPENDIX 3 – FEATURE GROUPS
All feauters and their groups as created in the second step of the Method Association Approach. Yellow features are either not done (not marked in
the Quinity column) or up for improvement (marked with a ‘/’ in the Quinity column). The source columns are marked with an ‘X’if a feature is
mentioned in that source.
The major group (e.g. elicitation, analysis, specification) is marked in bold italics in the top left corner of tables.
Quinity
LeanApps
Success md
requirements
From an RFI
From an RFP
From existing customer documents
/
/
X
X
X
X
X
X
X
X
From the analysis of existing systems
From the analysis of existing processes
From client demos
From business rules
From apprenticing
From a requirements questionnaire
X
X
X
X
X
X
X
X
X
X
X
X
From early information sessions
From product demos/prototypes (POC)
From trainings
From workshops
From interviews
From brainstorming/think aloud sessions
From laddering (game-ish)
From JAD sessions
X
X
X
X
X
/
Elicitation
From documents
A&W
SWEBOK
V3
Maturity
models
X
X
X
X
From exist. Sit.
X
X
X
X
X
X
X
X
X
From interact.
Sess.
/
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
From expertise
7
From our own expectations
Involve stakeholders
Identify the right stakeholders to involve
Push client to express requirements appropriately
Push for customer objectivity in trainings/prep
Requirements for certain groups with group members
present
Involve own marketing/sales
Notation
Note requirements down in customer language or source
language
Note the RQ source and author for traceability in later
stages
Note down as user stories
X
X
/
X
/
/
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Quinity
Success md
LeanApps requirements A&W
Filter Using common sense
Filter using triage, in/out scope, heuristics
X
X
Remove inconsistencies/overlap/duplicates
Per-requirement checklist
/
/
Assign requirements to experts and track
Formally catagorize requirements
/
/
Give requirements attributes
Iteratively improve on requirements
Track requirement version
/
X
Analysis
Filtering
X
X
X
SWEBOK Maturity
v3
models
X
Clean up
X
X
X
Organize
X
X
X
X
X
X
X
X
X
Expand
X
X
X
X
X
X
X
8
Determine requirement Impact (see impact analysis)
Determine requirement complexity
X
X
Derive a solution architecture
Relate requirements to the solution architecture
Obtain common understanding with different roles
X
X
X
Formally track interdependencies
Categorize interdependencies
/
/
X
X
Use design documents to mark affected areas
Interview FO, TO and developers for impact analysis
Use a checklist for impact analysis
Track the cascading effect of change (impact)
Use code slices to visualize affected areas
X
X
X
X
X
X
X
X
X
Early piority assesment
/
X
X
X
Goal
X
X
X
X
Interdepend.
X
Impact analysis
X
prioritisation
X
Tooling
Tools for analysis
Quinity
A&W
SWEBOK
v3
Identify conflicts
Define negotiable problems
Elicit stakeholder goals
/
X
X
X
Dicsuss alternative solutions
Assist concessions/trade-offs
X
/
X
X
X
X
Negotiation
Ident. Conflicts
Solutions
9
post conflict handling
Analyse negotiation results
Quality assurance review of results
Make results traceable
/
X
X
X
X
Quinity
Success md Aurum & SWEBOK Maturity
LeanApps requirements Wohlin v3
models
With experts
After explicit permission from a PM
X
/
X
X
Follow a written policy for notations
Always formulate as individual requirements
Natural language
As use cases/scenarios
/
X
X
In a central excel document
In a standard repository
X
/
Rationale behind the RQ (rejection)
Source (person or document)
Include relation to (QIS) product
With a unique ID number
/
Specification
Inclusiveness
X
X
Notations
X
X
X
X
X
X
X
/
X
X
X
X
X
X
X
X
X
X
X
X
Storage
Attributes
X
X
X
X
X
X
Tooling
Tools for notation
Tools for storage/retrieval
X
X
X
10
Validation
Content checking
Quinity
Succes met
de
Aurum & SWEBOK Maturity
LeanApps requirements Wohlin v3
Models
Check individual RQ completeness
Check RQ representativeness of customer wish
Use content checklists
/
X
X
X
Check correctness of RQ description
Check adherence to internal features
Use quality checklists
/
X
Check overall RQ consistency
Check RQ relevance for other customers
Inspection
Remove duplicates
X
/
/
X
X
X
Share Requirements document with customer
Establish a process to reach agreement with customer
X
X
X
X
X
X
X
X
X
Quality checking
X
X
X
X
X
X
X
Collection checking
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Inclusiveness
Tooling
Tools for content checking
Tools for quality checking
Tools for duplicate removal
Management
Planning
X
X
Quinity
Plan RE practices in a project
Assign ownership of RE practices to people
X
X
X
Succes met de Aurum & SWEBOK Maturity
LeanApps Requirements! Wohlin v3
models
X
X
X
X
X
X
11
RQ status tracking
Track requirement status
Review RQ status with (project)management
X
X
X
Post-project analysis
Close RQs when they become obsolete
Document RQ experiences
X
X
X
X
Define a formal RE process
Apply change mangement practices to RQs
Establish a formal process for including stakeholders in the RE process
/
X
X
X
Process definition
X
X
X
X
X
X
Glossary
Create a Product-wide glossary of terms
X
Stakeholder training
Train consultants/project leaders to recognize RQ engineering needs
Train other roles how to recognize RQs
Train customers how to recognize RQs and state them
X
X
X
X
X
X
12
APPENDIX 4 – ELIMINATED METHODS
This table contains all candidate methods that were rejected by the author or rejected in evaluation with the external coach. The rationale for their
elimination is included in the last column.
Method
Rational Unified Process
(RUP)
Source
Various
Accelarated SAP (ASAP)
http://wiki.scn.sap.com/wiki/display/SM/ASAP+Methodology
Exact’s method
Visser, C. (2009). Customer involvement in Distributed
Requirements Engineering. Master of Science Thesis
Oracle Unified Method (OUM)
http://www.oracle.com/partners/en/products/applications/oracleunified-method/get-started/index.html
V model
Forsberg, K., Mooz, H., & Cotterman, H. (2005). Visualizing
project management: models and frameworks for mastering
complex systems. John Wiley & Sons.
Boehm, B. W. (1988). A spiral model of software development
and enhancement. Computer, 21(5), 61-72.
Spiral model of Software
Development
Test-driven development
Extreme Programming
Feature Driven Development
Requirements Engineering
Process Selection (REPS by
Hewlett Packard)
Issue-Based Information
Systems (IBIS)
Beck, K. (2003). Test-driven development: by example.
Addison-Wesley Professional.
Beck, K. (2000). Extreme programming explained: embrace
change. Addison-Wesley Professional.
Palmer, S. R., & Felsing, M. (2001). A practical guide to
feature-driven development. Pearson Education.
Padula, A. (2004, September). Requirements engineering
process selection at Hewlett-Packard. In Requirements
Engineering Conference, 2004. Proceedings. 12th IEEE
International (pp. 296-300). IEEE.
Kunz, W., & Rittel, H. W. (1970). Issues as elements of
information systems (Vol. 131). Berkeley, California: Institute of
Urban and Regional Development, University of California.
Elimination rationale
No academic or free sources with enough granularity to
model method fragments accurately. Proprietary RUP
variants are restricted under copyright or expensive.
No sufficiently detailed free resources available. The
proprietary method is very costly.
The original hope was that this thesis provided enough
insights into Exact’s methods to create usable method
fragments. This was not the case.
Though touted as an accessible method for various
companies, Gold partnership or higher is needed to get
access.
The V model turned out to be an abstraction of a
method, not an actual usable method. More suitable for
informing development as a whole than just RE.
Like the V model, more of an abstraction than an actual
method. Leans too heavily on prototyping, which
Keylane|Quinity’s software does not support.
Seemed interesting at first, but way too agile to fit with
Keylane|Quinity’s incremental approach.
Completely ignores elicitation4 and how to do it. Other
activities are too agile to match Keylane|Quinity’s
incremental approach.
No detailed descriptions available for the creation of
method fragments. Too agile to fit with Keylane|Quinity.
Free sources are not detailed enough to yield clear
method framgents. Other sources are proprietary and
restricted under copyright.
Too much focus on ‘issues’ in every step to be useful.
Implementing an incomplete product only to refine it
based on real world issues is not a viable approach for
Keylane|Quinity.
13
Method
Open Web Application
Security Process (OWASP)
Source
Sindre, G., & Opdahl, A. L. (2005). Eliciting security
requirements with misuse cases. Requirements engineering,
10(1), 34-44.
Software Cost Reduction
(SCR)
Heitmeyer, C. L. (2002). Software cost reduction. John Wiley &
Sons, Inc..
Controlled Requirements
Expression
Finkelstein, A. (1988). Re-use of formatted requirements
specifications. Software Engineering Journal, 3(5), 186-197.
Elimination rationale
The concept of formulating misuse cases seemed
interesting, but the method turned out to be too focused
on security. No other method fragments offered any
value for Keylane|Quinity.
Ignores elicitation completely4. It’s proposed notation
seemed interesting but was far too convoluted to be
usable.
Ignores elicitation4 completely. The proposed model
notation is not acceptable for Keylane|Quinity.
Easy Winwin
Gruenbacher, P. (2000). Collaborative requirements
negotiation with EasyWinWin. In Database and Expert Systems
Applications, 2000. Proceedings. 11th International Workshop
on (pp. 954-958). IEEE.
The author made interesting promises in the abstract,
but the method turned out to focus almost exclusively on
negotiation.No attention was paid to elicitation, analysis
or specification.
Usage Oriented
Requirements Engineering
(UORE)
Regnell, B., Kimbler, K., & Wesslén, A. (1995, March).
Improving the use case driven approach to requirements
engineering. In Requirements Engineering, 1995., Proceedings
of the Second IEEE International Symposium on (pp. 40-47).
IEEE.
Flynn, D. J., & Jazi, M. D. (1998). Constructing user
requirements: a social process for a social context. Information
Systems Journal, 8(1), 53-83.
Use case models originally seemed interesting, but
deemed to complicated for communication with the
client, and too time consuming to justify for internal use.
User Led Requirements
Construction (ULRC)
ALM Rangers Requirements
Management (by practitioners
of Microsoft’s Visual Studio
Application Lifecycle
Management tools).
SQUARE-lite
https://vsarquickguide.codeplex.com/
Gayash, A., Viswanathan, V., Padmanabhan, D., Mead, R.
(2008). SQUARE-Lite: Case Study on VADSoft Project.
Technical Report.
Sounded promising but is too centered around the idea
of having users create EFDs. The external coach deemd
the efforts of training users and drawing models (claimed
by the paper to take 60 hours per stakeholder) an
unacceptable use of time and resources.
The actual method described in the guide was deemed
to specific to Microsoft products to be of any use. The
additional materials such as checklists were used as
inspiration in the contextualization of the method.
Originally professed as a lighter, more flexible variant of
SQUARE. After some analysis it turned out to be no less
complex than the original, so the original was selected.
By ‘ignores elicitation’ we mean that the source simply dismisses the process of elicitation. Rather than providing techniques or support, elicitation is reduced to a
simple statement along the lines of ‘and now you gather requirements’.
4
14
Method
Cooperative Requirements
Engineering With Scenarios
(CREWS) - Ecritoire
Source
http://crinfo.univ-paris1.fr/CREWS/Paris_chunk_index.html
COTS Aware Requirements
Engineering (CARE)
Chung, L., & Cooper, K. (2002, July). A knowledge-based
COTS-aware requirements engineering approach. In
Proceedings of the 14th international conference on Software
engineering and knowledge engineering (pp. 175-182). ACM.
Procurement Oriented
Requirements Engineering
(PORE)
Ncube, C., & Maiden, N. A. (1999, May). PORE: Procurementoriented requirements engineering method for the componentbased systems engineering development paradigm. In
International Workshop on Component-Based Software
Engineering (pp. 130-140).
Checkland, P., & Poulter, J. (2010). Soft systems methodology.
In Systems approaches to managing change: A practical guide
(pp. 191-242). Springer London.
Soft Systems Methodology
Elimination rationale
The external coach:
1. Deemed the use of ‘goals’ too abstract
2. Deemed the linguistic templates too complex
3. Deemed scenario authoring too time consuming
In hindsight CREWS L’ecritoire wouldn’t have fit with
Keylane|Quinity anyway. They never used scenarios in
the first place, so it wouldn’t make sense to adopt a
highly advanced scenario driven method.
A method from the Software buyer’s perspective.
Deemed interesting to learn about the client perspective.
After further analysis turned out to be best suited for the
acquisition of insanely complex multi-product software
landscapes.
Another method from the software buyer’s perspective.
Significantly lighter than CARE, but doesn’t actually say
anything about elicitation4 . Other method fragments
were too specific to buyers to be of any use.
Initially selected for it’s promise to help solve highly
problematic situations by the creation of ‘world views’.
Described as ‘te zweverig’ by the external coach. Uses a
lot of modeling, which Keylane|Quinity was not willing to
adopt.
15
APPENDIX 5 – ASSOCIATION TABLE
This appendix contains the association table used in the association step of the Method Association Approach. Features mark the rows and method
fragments mark the columns. If a method fragment’s name is red, then it was considered out of scope or rejected.
In the second view of the association table all red method fragments were removed. The remaining fragments are collored white (was considered but
not part of any method), yellow (was part of the preliminary method, but removed during evaluation) and green (is still part of the final method)
16
17
18
APPENDIX 6 – PRELIMINARY METHOD
The images below show the method after association, before evaluations had been made. Red marks indicate method fragment sources and potential
alternative method fragments. For brevity the deliverable side of the diagram was not modeled.
19
20
APPENDIX 7 – POST - FIRST EVALUATION
This appendix containts the first revision of the method. The first revision is the result of the first evaluation session. For brevity the deliverable side
of the diagram was not modeled.
21
3
1
2
22
4
5
23
APPENDIX 8 – FINAL METHOD
This appendix contains the final delivered method, including the deliverable side of the diagram
notation.
24
25
26
APPENDIX 9 – FINAL METHOD CONCEPTS AND DELIVERABLES
Activity
Sub activity
Description
Sales trajectory Answer RFI
Preserved from the old method. The RFI is a broad-net question list sent out by prospects to
(original)
potential suppliers. Questions are generic and address topics like hosting, costs, architecture,
support contracts and more. The RFI and RFI ANSWER can yield requirements.
Answer RFP
Preserved from the old method. The RFP is a higly specific question list sent out by prospects
to potential suppliers on their short list. Questions address very specific needs and desires with
regard to software. The RFP and RFP ANSWER always yield requirements.
Participate in Sessions
Dominant prospective clients will attempt to sate a large desire for information. The sessions
they book include generic presentations, demo sessions, use cases, proof of concept sessions
and more. We recommend following the client’s lead, as this shows commitment and builds
trust. In these sessions:
 Remind Keylane|Quinity participants of requirements guidelines and where to find
them.
 Make SESSION NOTES, preferably in one shared file.
 Collect potential requirements from the SESSION NOTES.
Setup preliminary contract
Included for completeness. When prospects decide to become clients they may want a
preliminary contract outlining early project activities, including the requirements phase. This
is not always the case.
Agree on
Send list of standard terms
Keylane|Quinity will be keeping a list of STANDARD TERMS. This list can be easily converted
definitions
into a survey and sent out to the client.
(based on
Client comments
The client is asked to add their comments, varying definitions and other remarks in CLIENT
SQUARE)
COMMENTS.
Document finalized set of
A FINALIZED SET OF TERMS is added to a new REQUIREMENTS DOSSIER.
terms
Define project
Determine project purpose
A PROJECT PURPOSE is written. Can be supplemented by the sales team and sales phase
(Based on Jad)
documents.
Determine project scope
A PROJECT SCOPE is written. This gives an early outline of the QIS components that will and
will not be implemented.
Determine project objectives
The PROJECT OBJECTIVES are determined. For now general deadlines and delivery estimates
will suffice.
Identify critical stakeholders
A list of CRITICAL STAKEHOLDERS is made. Collaboration with a client project owner or
(Based on Tropos method)
project manager is strongly recommended.
27
Not defined
Provide wiki access/closed
survey
Gather known
requirements
Interview sales
Analyze current systems &
documents
Elicitation
(Based on
ACRE)
Check speaking points list
Select elicitation technique
Processing
Workshop
(Based on JAD)
Elicit requirements
Write RQs into the dossier and
table
Validate document structure
Validate notation (Based on
RESCUE/SQUARE/GEAR)
Validate content completeness
(Based on Wiegers & Beatty –
Software requirements 3rd ed.)
Share dossier with the client.
(Inspired by LeanApps)
Determine information needs
Discuss open questions
The wiki and closed survey are both passive means of interaction, where the client does most
of their own requirements gathering and reports back to Keylane|Quinity. Though
recommended at this point in projects, these two means can be used later if desired. Yields
WIKI INTERACTIONS and CLOSED SURVEY ANSWERS.
In addition to known requirements, it may be a good idea to book an interview with
Keylane|Quinity’s own sales department. Sales has a lot of SALES KNOWLEDGE built up about
the client, sometimes going back way before the client even began looking for a supplier.
As an extra means of gathering requirements without having to burden the client, current
systems and documents can be analyzed. Sometimes current systems will be too outdated to be
worth analyzing, and some clients will have such low documentation standards that their
document’s aren’t worth analyzing, so both are optional. A project manager will determine
their worth. Yields DOCUMENTS & SYSTEMS KNOWLEDGE.
A list of SPEAKING POINTS can be used to determine whether there are any MISSED POINTS
left to discuss. For projects with passive clients this is a given. For dominant clients there may
be no missed points at all.
A TECHNIQUE LIST is provided with popular requirements elicitation techniques. The SELEC
TED TECHNIQUES can be selected by checking MISSING POINTS against the optimal usage
context and obstacles for usage for the techniques on the TECHNIQUE LIST.
ELICITED REQUIREMENTS are obtained using the SELECTED TECHNIQUES.
Any newly elicited requirements, just like known requirements, are written into the
REQUIREMENTS DOSSIER. GAPS are also covered separately in the GAP TABLE.
Once complete the structure of the REQUIREMENTS DOSSIER can be validated using the
requirements phase checklist.
The notation of requirements in the REQUIREMENTS DOSSIER can be validated using the
requirements phase checklist.
The completeness of requirements in the REQUIREMENTS DOSSIER can be validated using the
requirements phase checklist.
The REQUIREMENTS DOSSIER is shared with the client for review. Any questions that the
client has will be covered in the workshop.
Right before the workshop INFORMATION NEEDS are determined by looking up any remaining
UNVERIFIED GAPS or MISSING POINTS from the SPEAKING POINTS LIST.
Any open questions that still exist for both Keylane|Quinity and the client are discussed.
28
Discuss unverified
requirements
Discuss missing requirements
Take notes
Document requirements and
gaps
Requirements
management
(Based on
RESCUE)
Manage gap status
Track requirement changes
Register new requirements
Prepare knowledge for next
increment
Project setup
(original)
Explain QIS implementation
method
Design increments
Plan activities
Increment
(original)
Post project
Agree on project approach.
Train session team
Organize design sessions
Document detail requirements
Share results with customer
Design, Develop, Test, Go live
Maintenance
Any unverified requirements in the GAP TABLE are verified now.
Either party gets one last opportunity to discuss potentially missing requirements.
As recommended in JAD, a scribe sits in and makes WORKSHOP NOTES about all discussed
topics.
Any new requirements and/or gaps resulting from the workshop are extracted from the
WORKSHOP NOTES and written into the REQUIREMENTS DOSSIER. Gaps are also captured in
the GAP TABLE.
In the GAP TABLE gaps have a status attribute. Project managers manage this attribute to keep
track of progress on each gap.
Requirement changes are also tracked. Every time a requirement changes significantly a new
version of the REQUIREMENTS DOSSIER is stored and the change documented. This is the
same method that Keylane|Quinity uses for all document version tracking.
New requirements can always arise during the INCREMENTS. Project managers have endresponsibility for keeping the REQUIREMENTS DOSSIER and GAP TABLE up to date with the
latest.
Out of scope. Once an increment is completed some time is taken to prepare the next
increment. Gaps for the next increment that weren’t discovered until an earlier increment are
given some extra attention.
Out of scope. The QIS implementation method is explained to the client’s executive and
managing personnel.
The INCREMENTS are defined definitively. In each increment some subsection of QIS is
changed and implemented in the client company.
Activities like meetings, demos and other sessions with designers are planned into the
increments.
Out of scope. Keylane|Quinity agree on the approach, possibly in another contract.
Out of scope. Any client participants in the design session that weren’t previously trained are
given that training now.
Out of scope. Design sessions are held with the client to elicit DETAIL REQUIREMENTS
DETAIL REQUIREMENTS are documented.
DETAIL REQUIREMENTS are shared with the client for verification.
Out of scope. Solutions are designed, developed, tested and implemented.
Out of scope. Maintenance is performed on the client’s implementation in the form of bugfixes,
support, and quarterly releases (if the client is willing to update at least).
29
Document lessons learned.
Deliverable
RFI ANSWER
RFP ANSWER
SESSION NOTES
PRELIMINARY
CONTRACT
STANDARD
TERMS
CLIENT
COMMENTS
FINAL SET OF
TERMS
PROJECT
PURPOSE
PROJECT SCOPE
The LESSONS LEARNED from this project are documented and used to improve future projects.
Description
The answer to a prospect’s Request For Information [1]. Contains short answers to simple questions (e.g. ‘what hosting options
are available?’ – ‘On client location, on our location, via a third party apache host or through a SAAS plan’).
The answer to a prospect’s Request For Proposal [1]. Elaborate answers to specific questions. The base answers are:
 Yes we do this
 No, but we have this on the roadmap for future releases.
 No, but we are willing to make a solution.
 No, we won’t do this.
Followed by an elaboration and/or justification of the answer.
Session notes are made during various sales phase information sessions [2]. These sessions are booked by dominant clients
looking to satisfy their information needs, and make an informed software supplier selection. The following are possible:
 Regular presentation
 Demo session (Keylane|Quinity is asked to present QIS however they like).
 Proof of Concept (PoC) (Keylane|Quinity is asked to achieve a certain goal. How to get there in QIS is up to them).
 Use cases (Keylane|Quinity is given a scripted task to perform in QIS. Parts of the script that aren’t possible are
considered gaps)
A PRELIMINARY CONTRACT typically contains any activities leading up to the actual increments, like a requirements phase,
and the compensation for those activities. The exact contents of PRELIMINARY CONTRACTs are out of scope.
The STANDARD TERMS are entities and objects related to the insurance domain, that are often defined differently by
practitioners [3]. STANDARD TERMS are defined in a document with room for CLIENT COMMENTS.
CLIENT COMMENTS can be just comments, own definitions or any other response to Keylane|Quinity’s STANDARD TERMS. In
some cases CLIENT COMMENTS can even yield new gaps.
The FINAL SET OF TERMS [3] includes Keylane|Quinity’s definitions and CLIENT COMMENTS. This document is kept to prevent
miscommunication later in the project due to different definitions.
The PROJECT PURPOSE [4] is a short description of the client’s purpose behind acquiring software. The PROJECT PURPOSE is
written on the strategic level.
The PROJECT SCOPE [4] consists of the major QIS components that will be implemented during the project. The major
components are:
 Product definition
 Request administration
 Policy administration
30
 Relation administration
 Intermediary administration
 Authorisation
 Damage administration
 Premium calculation
 Financial systems
 Distribution channels
 External interfaces
 Communication
 MyEnvironment (MijnOmgeving)
 Workflow support
 File management
 Reporting
 Re-insurance
PROJECT
The PROJECT OBJECTIVES [4] are major deadlines and delivery dates. At this point broad indications are fine (e.g. before the
OBJECTIVES
end of Q4 2015). The purpose is to give readers a sense of scale and time for the project.
CRITICAL
The CRITICAL STAKEHOLDERS [5, 6] are stored based on two roles in the ARCI matrix. Approves: those responsible for
STAKEHOLDERS obtaining internal approvement of decisions. Consulted: those who must be consulted in the decision making process. The
Recommends role and Informed Role were deemed irrelevant for Keylane|Quinity.
WIKI
WIKI [7] INTERACTIONS are interactions with stakeholders within the Keylane WIKI. The wiki is currently only a prototype
INTERACTION
running on the MediaWiki engine and is still subject to change.
CLOSED
The CLOSED SURVEY is a survey with closed questions directed at client management. The best CLOSED SURVEY questions are:
SURVEY
 Simple questions that can be answered without elaboration sessions.
 Questions that serve as preparation for an elaboration sessions.
Closed survey questions should not be used to get the client to describe complex concepts across several pages.
SALES
SALES KNOWLEDGE can be elicited from Keylane|Quinity’s own sales department. Sales builds up knowledge of the client’s
KNOWLEDGE
needs long before there even was a project. Their knowledge may provide some requirements [8].
DOCUMENT &
DOCUMENT & SYSTEMS KNOWLEDGE is obtained by analyzing client documents [9] and their existing systems [10]. This
SYSTEMS
knowledge can include current processes, business rules, current solutions to problems, software components, architecture and
KNOWLEDGE
much more.
KNOWN
All requirements available before elicitation begins are considered to be KNOWN REQUIREMENTS [11]. KNOWN
REQUIREMENTS REQUIREMENTS can be found in:
 RFI ANSWERS
31
 RFP ANSWERS
 Sales INFORMATION SESSION NOTES
 WIKI INTERACTIONS
 CLOSED SURVEY responses
 CLIENT DOCUMENTS AND SYSTEMS
SPEAKING
SPEAKING POINTS are high level discussion subjects for each of the QIS components (see PROJECT SCOPE for the list). An
POINTS
undiscussed speaking point is a likely indicator of missed requirements, since the client is apparently not aware of that
component and it’s capabilities.
MISSED POINTS MISSED POINTS are SPEAKING POINTS that haven’t been discussed with the client.
TECHNIQUES
The TECHNIQUES LIST is a list of 16 well known requirements elicitation techniques. For each technique the list explains how
LIST
to do it, when it works best, which obstacles impede it, which resources are needed to do it, and which techniques complement
it well. Techniques are divided into interview techniques (e.g. structured, unstructured, laddering), workshop techniques (e.g.
brainstorm, KJ technique, JAD) and observation techniques (e.g. storytelling, protocol analysis, apprenticing). Based on [12, 13].
SELECTED
SELECTED TECHNIQUES are techniques chosen to address MISSED POINTS. For example, if the way new relations are
TECHNIQUES
registered is not yet understood then an observation technique would be a good choice to see how it is done now.
ELICITED
ELICITED REQUIREMENTS are new requirements elicited using a SELECTED TECHNIQUE. ELICITED REQUIREMENTS are
REQUIREMENTS written into the REQUIREMENTS DOSSIER.
REQUIREMENTS Included as context. A REQUIREMENT is any customer need, wish or desire with regard to Keylane software.
GAPS
Included as context. A GAP is any customer need, wish or desire for functionality that is not or not sufficiently covered by Keylane
software in it’s current state. REQUIREMENTS include GAPS.
UNVERIFIED
An UNVERIFIED GAP is any gap derived from documents, systems, or own expertise, that has not yet been communicated to the
GAPS
client. These UNVERIFIED GAPS should always be discussed to verify whether they still reflect a current need, wish or desire
[14]. We keep track of the verified attribute in the GAP TABLE.
INFORMATION
INFORMATION NEEDS are gathered right before the kickoff of the final workshop to determine what information is still
NEEDS
missing. This is the last opportunity in the requirements phase to do so. Information needs include:
 Any remaining open questions
 Any remaining UNVERIFIED GAPS
 Any remaining MISSING POINTS from the SPEAKING POINTS LIST
WORKSHOP
WORKSHOP NOTES are made by a dedicated scribe for all points discussed in the workshop. A Scribe is appointed to this task to
NOTES
minimize distraction for the other workshop participants.
REQUIREMENTS The REQUIREMENTS DOSSIER is used as a single repository for all requirements that will be shared with the client [15]. The
DOSSIER
REQUIREMENTS DOSSIER contains:
 A project definition (purpose, scope, objectives, stakeholders)
 An architecture draft (not in scope for this project)
32
 Requirements
 Gaps
A standardized notation has been made for requirements and gaps in the REQUIREMENTS DOSSIER.
GAP TABLE
The GAP TABLE only contains gaps and acts as an overview. In addition to the standard notation used in the DOSSIER the GAP
TABLE is also used to keep track of status, the ‘verified’ attribute, and whether solutions will be generic or client specific.
INCREMENT
The INCREMENT PLANNING determines in which increments QIS will be implemented, and what each increment contains.
PLANNING
Typically increments consist of:
 One or more components of QIS per increment
 One or more client products and matching QIS components per increment
 A channel and all it’s needed QIS components per increment (e.g. first intranet, then extranet, then internet.
 A client business line per increment.
ACTIVITIES
ACTIVITIES are design sessions planned per increment, with the purpose of eliciting DETAIL REQUIREMENTS.
DETAIL
DETAIL REQUIREMENTS (similar to design requirements in [16]). are made by designers when they decompose GAPS into their
REQUIREMENTS smallest functional parts. An example of a gap and It’s detail requirements:
 The system must provide a user with login data
o The system must generate a username containing the first five letters of an employee’s first name and the first
five letters of their last name.
o The system must generate a random password containing at least 10 characters from the full Unicode set.
o The system must auto-send an e-mail containing the username and password to the employee’s company
address
LESSONS
Any good project ends with the documentation of LESSONS LEARNED. In this case LESSONS LEARNED can also be contributions
LEARNED.
to CLOSED SURVEYS, SPEAKING POINTS, the REQUIREMENTS DOSSIER and GAP TABLE templates, the TECHNIQUE LIST, the
STANDARD TERMINOLOGY and more.
[1] Humphrey, W. S., & Sweet, W. L. (1987). A Method for assessing the software engineering capability of contractors: preliminary version (No. CMU/SEI-87-TR-23).
Carnegie-mellon Software Engineering Institute (SEI), Pittsburgh PA.
[2] Johnson, W. L., Feather, M. S., & Harris, D. R. (1992). Representation and presentation of requirements knowledge. Software Engineering, IEEE Transactions on, 18(10),
853-869.
[3] Mead, N. R., & Stehney, T. (2005). Security quality requirements engineering (SQUARE) methodology. ACM SIGSOFT Software Engineering Notes, 30(4), 1-7.
[4] Duggan, E. W., & Thachenkary, C. S. (2003). Higher quality requirements: Supporting joint application development with the nominal group technique. Information
Technology and Management, 4(4), 391-408.
33
[5] Bresciani, P., Perini, A., Giorgini, P., Giunchiglia, F., & Mylopoulos, J. (2004). Tropos: An agent-oriented software development methodology. In: Autonomous Agents
and Multi-Agent Systems, 8(3), 203-236.
[6] Lapào,L.P., Rebuge, A., Silva, M.M., Gomes, R. (2009). ITIL Assessment in a healthcare environment: the role of IT governance at Hospital Sao Sebastiao. In Medical
Informatics in a United and Healthy Europe: Proceedings of MIE 2009, 150, 76-80.
[7] Bradshaw, P. (2007).: Wiki Journalism: Are Wikis the New Blogs? In: Proceedings of the Future of Newspapers Conference 2007, 1-22.
[8] Barney, S., Aurum, A., & Wohlin, C. (2008). A product management challenge: Creating software product value through requirements selection. Journal of Systems
Architecture, 54(6), 576-593.
[9] Wong, K. Y., Casey, R. G., & Wahl, F. M. (1982). Document analysis system. IBM journal of research and development, 26(6), 647-656.
[10] Van Lamsweerde, A. (2001). Goal-oriented requirements engineering: A guided tour. In Proceedings of the Fifth IEEE International Symposium on Requirements
Engineering, 249-262.
[11] Hickey, A. M., & Davis, A. M. (2003, January). Requirements elicitation and elicitation technique selection: model for two knowledge-intensive software development
processes. In Proceedings of the 36th International Conference on System Sciences, 10- 19.
[12] Maiden, N. A. M., & Rugg, G. (1996). ACRE: selecting methods for requirements acquisition. Software Engineering Journal, 11(3), 183-192.
[13] Carrizo, D., Dieste, O., & Juristo, N. (2014). Systematizing requirements elicitation technique selection. Information and Software Technology, 56(6), 644-669.
[14] Boehm, B. W. (1984). Verifying and validating software requirements and design specifications. IEEE software, 1(1), 75.
[15] Wiegers, K., & Beatty, J. (2013). Software requirements. Pearson Education.
[16] Ball, L. J., Onarheim, B., & Christensen, B. T. (2010). Design requirements, epistemic uncertainty and solution development strategies in software design. Design
Studies, 31(6), 567-589.
34
APPENDIX 10 – WIKI SCREENSHOTS
35
36
37
38
ADDENDUM 1 – NATURAL LANGUAGE
PROCESSING OF REQUIREMENTS IN CLIENT
DOCUMENTATION
A N EXPERIMENT REPORT
Abstract. A software supplier generates a lot of documentation about its software product in the
form of functional descriptions, design documents, manuals and training materials. This
documentation is rife with keywords related to the software’s functions and capabilities.
Similarly the software supplier’s clients have websites, process documents and sales
communications that are rife with keywords related to their business processes. Natural
language processing could potentially show the critical differences: keywords that are common
in the client’s documentation, that rarely or never occur in the supplier’s documentation. These
keywords may be indicative of requirements. In two additional experiments we use NLP for the
identification of stakeholders and requirements in a client’s sales documentation.
1
1 - INTRODUCTION
In this report we will discuss the results of using Natural Language Processing (NLP) techniques
to extract information from industry documents. This research was performed as part of a
master’s degree thesis project performed in collaboration with a Dutch software supplier. This
supplier sells a software package that offers broad support for insurance-related processes,
products and services to damage/property insurance providers in Europe.
The software supplier was particularly interested in ‘gaps’: the functionalities that new clients
need, but are not or not sufficiently present in their current software product. Gaps are the most
valuable type of requirements, since clients need these functionalities in order to operate their
business. As estimated by Boehm, missing gaps until late in a software project can result in 10 to
200 times increased costs [1]. This has spurred the software supplier’s interest in new and
innovative ways of identifying gaps. As a part of the thesis we suggested the use of NLP
techniques.
The software supplier generates a lot of documentation about their software. Think for instance
about functional descriptions, design documents, and training materials. New clients, similarly,
often have a stack of documents about their products, processes, and policies. If we were to
extract keywords from the supplier’s documents, and compare them to keywords in the client’s
documents, there is bound to be a lot of overlap among insurance specific terms (e.g. policy,
premium, etc). What we are interested in are the differences. Because gaps are customer needs
that are not yet present in the software, the expectation was that their keywords would occur
significantly less in the software’s documentation
Supplier
documentation
Gaps
Client
documentation
The, it, on, off,
next,
to, up,
Supplier
down,
left,
Keywords
right,
over,
under, policy,
The, it, on, off,
next, to, up,
Difference
down,
The, it,left,
on, off,
right,next, over,
to, up,
under,
policy, left,
down,
Overlap
right,
over,
under,
policy,
NLP
The, it, on, off,
next,Client
to, up,
down,
left,
Keywords
right,
over,
under, policy,
Figure 31 - Envisioned NLP results
In section 2 we will elaborate further on the domains of software requirements engineering and
natural language processing, followed by the methods used in section 3. In section 4 we report
on the results of the keyword finding. In section 5 we report on several techniques used for
information extraction and their viability. Finally, we conclude this paper with a discussion and
conclusion in sections 6 and 7 respectively.
2 - DOMAIN
Wiegers and Beatty define a software requirement as “a statement of a customer need or
objective, or of a condition or capability that a product must possess to satisfy such a need or
objective” [1]. Requirements are often distinguished as either being one of the following:
2


Functional requirement. A description of a function that the software must support or
perform independently.
Non-functional requirement. A description of a property or characteristic that the
software must exhibit, or a constraint that it must respect.
NLP is an innovative field that has drawn a lot of attention from various other domains over the
years; Requirements Engineering (RE) has been no different. As early as 1993 Kevin Ryan urged
for a realistic attitude towards using NLP in RE [2]. Since then several applications of NLP have
been devised by other RE researchers [3,4,5].
3 - METHOD
All experiments outlined in this report were performed using AntConc[6]. AntConc is a freeware
corpus analysis toolkit that supports common NLP techniques such as concordancing, searching
collocates, searching N-grams, and generating keyword lists. Out of the broader offering of NLP
tools AntConc is one of the few truly (license-) free tools that boasts such a large feature set.
KEYWORD FINDING
Imagine taking a corpus of twenty documents and searching for their most used words; a
frequency list. The likely result is a top 100 containing ‘the, and, a, to, of, in, is, it’ and so on. The
most simple way of handling this problem is to use stop word removal [7]. With stop word
removal a list of very common words is used to remove common words from the frequency list.
This results in a list of keywords that are relevant compared to the stop word list, but what
about comparing a corpus to another corpus?
In keyword finding an attribute called ‘keyness’ is calculated. If a word occurs significantly more
in the target corpus than in the reference corpus, it’s keyness attribute is higher as a result.
Paquot and Bestgen put this technique to use by comparing a large collection of academic papers
to a corpus of English fiction [8]. They successfully identified the keywords that set academic
writing apart from fiction. AntConc by default calculates keyness using the log-likelihood
statistic of words in their respective corpus [9]. Lancaster University’s centre for computer
corpus research provides an explanation of this process including a simple log-likelihood
calculator[10]. We used this technique on a corpus of client process descriptions with a corpus
of supplier software descriptions as a reference. This experiment would be considered a success
if we could easily identify the same gaps found in the supplier’s gap analysis using the list of
keywords.
RFP ANALYSIS USING SEARCH WORDS
In the second experiment we attempted to rapidly
extract requirements and stakeholders from a
Request for Proposal (RFP). An RFP is typically a
100+ page questionnaire sent out by software
buyers to learn more about the offerings of
potential suppliers. RFP documents are rife with
requirements and mentions of important people,
but reading RFPs in their entirety may be very
time consuming. Why not just ctrl+f your way
through an RFP then? Antconc provides KWIC,
KeyWords In Context [11], surrounding search
hits to provide context as the name implies. In
addition, with a single click a search hit can be
viewed in its original position in a document,
even when searching through a large corpus of
Figure 32 - search hits for 'must' with KWIC sorting
3
multiple documents. We used these features to rapidly extract both requirements and
stakeholders from a sample RFP document..
4 - GAP ANALYSIS TEST
In a previously completed gap analysis with one software buyer, the supplier identified nineteen
gaps. Two of these gaps would not have been identifiable by keywords, leaving a total of
seventeen gaps to work with. Words from the gaps’ original Dutch names were extracted. To
prove that it is possible to replicate the gap analysis using NLP, these words would need to have
very high keyness ranks compared to their original frequency ranks. Some extra keywords were
added such as the parts of compound words and common synonyms, assuming that this would
increase the success of finding keywords (marked with a superscript * in the table below).
A total of 31 functional descriptions of the supplier’s software were uploaded into AntConc and
saved as a reference corpus. A collection of 22 rigorous process descriptions of the client’s
processes were uploaded as the target corpus, and word keyness was calculated using the
functional descriptions as a reference corpus. A selection of the results is shown below in table 1.
Keywords
Gezondheidsverklaring
Digitale
Gezondheid*
Verklaring*
Premiebetaling*
Creditcard
Credit*
Card*
Sms
Betaalbevestiging
Betalingsbevestiging
Communicatie
Bericht*
Telefoon*
Telefonisch*
Bevestiging*
Uitbetaling
Uitbetalen
Stop-Loss
Begunstigde*
Keyness rank
872
747
Not found
2630
2208
Not found
1874
1844
Not found
Not found
Not found
Not found
1675
2821
1281
2770
Not found
1212
2628
862
Frequency rank
968
795
Change
+96
+48
1958
2637
- 672
+429
2230
2191
+356
+347
305
2750
1005
255
-1370
-71
-276
-2512
1586
1921
784
+374
-707
-78
Table 10 - Sample keywords and rank mutations
Out of the total collection of 66 keywords, only 21 were actually found in the client’s documents.
For six gaps we didn’t find any keywords at all. Comparing the results from the keyness tables to
simple frequency tables, 13 keywords saw their rankings increase. The remaining 8 dropped
down in the rankings. The average rank mutation using keyness rather than frequency was -116,
meaning that the average keyword dropped 116 rankings. This can probably be explained by the
addition of more generic words (i.e. the split compounds and synonyms), which are more likely
to occur in the suppliers documents and thus result in a lower keyness.
Only looking at the specific keywords (without a *), hit rates were even lower. 10 out of 37 were
found, but their average rank change was 135, meaning that highly specific keywords had a
keyness rank that was 135 places higher than their frequency rank on average. This does not
take away the fact that only 10 out of 37 specific keywords were actually found in the client’s
4
documents. On top of that the 10 keywords that were found appeared so far out of the top 200
that it would have still taken quite some time to identify them. We hoped that using keyness
would put some, if not all, keywords in the top 100.
Total
Found in client’s process
descriptions
Average rank mutation
All Keywords
66
21
Specific keywords
37
10
-116
135
Table 2 - Sample keywords and rank mutations
5 - RFP ANALYSIS
In this section we will explore the possibilities of using NLP and AntConc to identify
requirements and stakeholders in client documentation. The source document used is a 134
page RFP document sent out by a Belgian insurance provider. This document was sent out to
various software suppliers as a means to research the products of potential suppliers. The
document contains natural language descriptions of the project context, project scope, the
company’s working methods, business requirements, IT requirements, contractual
requirements, localized market requirements and a short demonstration of their current
software. The document also contains an extensive list of questions directed at the software
supplier.
The first approach used is a targeted search for modal verbs, a subset of auxiliary verbs that
indicate properties of verbs. Requirements are often phrased using modal verbs. For example:
‘The system must support x’ or ‘A user should do y’. We opted to search the four modal verbs
used in the popular requirements prioritization technique MoSCoW [12]: must, should, could
and would and their present forms shall, can, and will. An example AntConc output for the
search word must can be seen in figure 1. The results in table 3 were gathered by reading
reading and assessing each search hit.
Hits
Must
Shall/Should
Can/Could
Will/Would
55
81
137
161
Functional
requirements
21
8
19
7
Non functional
requirements
15
30
16
8
%
65.5%
59.3%
25.5%
9.3%
Table 3 - RFP search words
For statements containing must, shall and should, more than 50% was indicative of a client’s
requirement for the software. Any statement of functionality that the system should have is
counted as a functional requirement. . Non functional requirements were identified in
accordance with the definition by Wiegers and Beatty [1]. This includes requests for system
qualities (e.g. “the database must respond to queries within 100ms”), constraints (e.g. “these
figures should not be made visible to the customer”) and strategic concerns (eg. “the system
should be aligned with our agile way of working”)
For can and could there is a steep drop-off, caused by prevalent use of those words in the
company’s working methods description. For example: “an expert can enlist the help of the sales
manager”. In this instance can refers to an activity that stakeholders perform amongst
themselves without needing any intervention from software. The drop was even steeper for will
and would, which were used often in the project context, project scope, project team role
descriptions, and the question list.
5
For the identification of stakeholders the same seven target words were used. By sorting the
KWIC to the left of search hits a pleasant, easily readable overview is created. An example of this
output can be seen in figure 3.
The identified stakeholders are the following:



















The bidder (software supplier)
Client (a customer of the insurance provider)
Additional party (third party acting on behalf
of the client company’s customers)
Direct insurers (the client and its
competitors)
The parent company
The system
The target audience
The project team(s)
Stakeholders
The product manager/Product management
The sales manager
Portfolio management
Team leaders
Figure 33 - Stakeholder identification in KWIC
Experts
output
Agents
Administrative agents
Partner organizations/partners
Derogation authorities
Subcontractors
DISCUSSION
The keyword-finding experiment did not at all turn out as hoped. We were only able to achieve a
rank increase on highly specific keywords, but the increase was not statistically significant. On
top of that less than 30% of our keywords were actually found in client documents, and they
occured so far down the keyness rank list that replicating the original gap analysis would have
been impossible.
The second experiment fared much better. Extracting software requirements from an RFP
resulted in many good finds. Particularly the modal verbs must, shall, and should, which
produced more than 50% requirements per hit. In addition, by sorting context words to the left
of modal verbs we were able to identify 19 stakeholder groups within minutes.
CONCLUSION
In the first experiment we attempted to replicate a gap analysis performed by a Dutch software
supplier at a client company. Keywords were formulated to identify the gaps in client documents
and their frequency rankings were calculated. The ‘keyness’ attribute of keywords was
calculated by comparing their frequency in client process descriptions to their occurrence in
supplier software descriptions. The keyness rankings were then compared to the original
frequency rankings. Our original hypothesis was that keywords related to gaps would have a
significantly higher keyness rank. This in turn would result in a significant increase in their
keyness. Using a combination of specific and generic keywords actually resulted in an overall
drop in rankings. Using specific keywords alone resulted in an average rank increase of 135
6
positions, but with nearly 3000 possible positions this increase was nowhere near as significant
as expected.
REFERENCES
1. Boehm, B.: Verifying and Validating Software Requirements and Design Specifications. IEEE
software, 32-41 (1991)
2. Wiegers, K., Beatty, J.: Software Requirements 3rd edn. Pearson Education (2013)
3. Ryan, K.: The Role of Natural Language in Requirements Engineering. In : Proc. of the IEEE
Int. Symp. on RE, pp.240-242 (1993)
4. Natt och Dag, J., Gervasi, V., Brinkkemper, S., Regnell, B.: Speeding up Requirements
Management in a Product Software Company: Linking Customer Wishes to Product
Requirements through Linguistic Engineering. In : proc. of the 12th Int. RE conference,
pp.283-294 (2004)
5. Yadla, S., Hayes, J., Dekhtyar, A.: Tracing Requirements to Defect Reports: An Application of
Information Retrieval Techniques. Innovations in Systems and Software Engineering 1(2),
116-124 (2005)
6. Sampaio, A., Loughran, N., Rashid, A., Rayson, P.: Mining Aspects in Requirements. In : Early
Aspects workshop 2005, Aspect-Oriented Requirements Engineering and Architecture
Design, Chicago, IL, vol. 1 (2005)
7. Anthony, L.: AntConc (version 3.4.3) [Computer Software]. Waesda University, Tokyo, Japan
http://www.laurenceanthony.net.
8. Liu, F., Pennell, D., Liu, F., Liu, Y.: Unsupervised Approaches for Automatic Keyword
Extraction Using Meeting Transcripts. In : Proc. of human language tecnologies: 2009
annual conference of the NAACL, pp.620-628 (2009)
9. Paquot, M., Bestgen, Y.: Distinctive Words in Academic Writing: A Comparison of Three
Statistical Tests for Keyword Extraction. Language and Computers 68(1), 247-269 (2009)
10. Rayson, P., Garside, R.: Comparing Corpora Using Frequency Profiling. In : proc. of the
Workshop on Comparing Corpora, 38th Annual Conference of the ACL, Hong Kong, pp.1-6
(2000)
11. Rayson, P.: Log-likelihood Calculator., Lancaster, UK http://ucrel.lancs.ac.uk/llwizard.html.
12. Luhn, H.: Key Word-in-context Index for Technical Literature (kwic index). American
Documentation 11(4), 228-295 (1960)
13. Brennan, K.: A Guide to the Business Analysis Body of Knowledge (Version 2). International
Institute of Business Analysis (2009)
14. Pohl, K., Rupp, C.: Requirements Engineering Fundamentals 1st edn. Rocky Nook, Inc.
(2011)
7