Name Best Practice
IDnr
Date adjusted
Description of the content
Type of document
ASL Process
Size of the application
management team:
Small (number fte < 10),
Medium (number fte 10 – 20)
and/or Large (number fte> 20).
Remarks
ASL BiSL Foundation
PO-box 9769
3506 GT UTRECHT
Quality Assurance Procedure
125_BP_E
18122006
This document contains the procedure for Planning and performing Quality Assurance reviews
for projects and programs ofam and
handling QA violations
Procedure
Quality Management
Small
Medium
√ Large
T: +31 (0) 30 753 1424
F: +31 (0) 30 755 1502
Chambre of Commerce Utrecht and surroundings:
I: www.aslbislfoundation.org
E: [email protected]
08106817
Quality Assurance Procedure
procedure
Place
Date
Author
Status
Amsterdam
181206
ASL
Concept 0.1
Quality Assurance
Procedure
181206
2/15
Version Control
Version
0.1
Date
Author
Author
Description
Initial document
Distributionlist
Version
0.1
Date
To
Quality Assurance
Procedure
181206
3/15
Quality Assurance Procedure
Overview
This document contains the procedure for:
Planning and performing Quality Assurance reviews for projects and programs ofam.
Handling QA violations (a QA violation is the non-execution of a mandatory QA review).
1. Quality Assurance in Application Management
1.1 Purpose
Ensure and deliver advice on that what is being delivered is provided on time, within budget,
meets the documented requirements and with high customer satisfaction.
Provide the Project Manager and AMS management with:
Advice on the proposed solution, contract, or work effort with respect to its management,
quality and financial exposures (via Solution Design reviews and the QA1/2/3 reports)
A health assessment and related advice on the project as it is being delivered (via
Solution Delivery reviews and the QA4/5/6 reports)
Target audience:
Opportunity Business Manager (OBM), Opportunity Owner (OO), Proposal Teamleader (PpTL),
Project Team Leader (PjTL), Quality Assurer (QA), Senior Business Area Manager (Sr. BAM),
Project Manager (PM)
1.2 Scope
This procedure applies to:
All projects and all programs.
This procedure describes only one area of Software Quality Assurance activities, the
independent review of project management and the related deliverables or work products.
1.3 Special Conditions
This procedure applies only to QA reviews on projects and programs performed byam.
1.4 Entry Criteria
An opportunity has been received and discussions are starting which will affect customer
expectations towards the solution or offering.
1.5 Exit Criteria
The project ended either successfully (project completion criteria are met) or unsuccessfully
(proposed solution has been rejected by the customer).
The end of the agreed program period is reached (normally end of year).
2. Overview of Quality Assurance
2.1 Description
The activities related to QA reviews are divided in two parts:
Quality Assurance
Procedure
181206
4/15
Activities related to the planning and execution of QA reviews
For a description of the different QA reviews see chapters 2.3 and 5.
Activities related to the detection and reporting of QA violations.
For a definition of a QA violation see chapter 2.4
2.2 Activity and Logic Flow
QA reviews
OO
PpTL/
PjTL
4. Respond
to Review
Findings
1. Plan for
Review
QA
2. Prepare
for Review
3. Perform
QA
Review
QA violations
OO
2. Respond
to QA
violation
PpTL/
PjTL
QA
1. Detect
and
Report
QA
violation
3. Track
and close
QA
violation
2.3 Activity List for QA reviews
Quality Assurance
Procedure
181206
5/15
QA reviews are referred to by e.g. a company´s Worldwide Quality Assurance designations of
QA1 through QA6. Different types of reviews (technical, project management, business) are
conducted at various stages of a project, as described below.
During Solution Design:
QA1 - Solution Assurance Review
QA2 - Business Assurance Review
QA3 - Proposal Assurance Review
During Solution Delivery:
QA4 - Project Management Review (Initial)
QA5 - Project Management Review
QA6 - Deliverable Readiness Review
Quality Assurance
Procedure
181206
6/15
Role(who will do)
PpTL/PjTL
Process-Activity
Step 1 Plan for Review
When (at what frequency)
At start of Solution Design for the QA1, QA2 and QA3 review
During Solution Delivery for the QA4, QA5 and QA6 review
Evidence
Where (will it get reported)
PROJECT TEAM DATABASE
BMRS
What (format will be used)
How (will it be done)
PpTL/PjTL
Refer to appendix C for the frequency of QA reviews and to the
activity lists for the timing of the reviews.
Refer to 'AM SQA plan template for Quality Planning activities.
Contact the QA team to plan a specific QA review,
Update Business Management reporting with the planned QA review
dates
Step 2 Prepare for Review
When (at what frequency)
For a QA3 review: 5 days before the planned review.
For the other reviews: 2 days before the planned review.
How (will it be done)
Provides requested project materials to the Quality Assurer.
Ensures that corrective actions and deficiencies from previous
reviews that should have been completed or corrected have been
completed or corrected.
Prepares a short presentation on the project background and current
status, if necessary.
Works with the Quality Assurer to identify specific review participants
and roles they will play.
QA
Step 3 Perform QA Review
When (at what frequency)
At date planned (see step 1)
How (will it be done)
Reviews materials.
Holds in-person interviews and review meetings with the Project
Team as needed to resolve questions or issues. Leads interviews and
review meeting(s). Obtains additional materials from the PpTL/PjTL as
needed.
Creates the QA Review Report, the findings, recommended actions
with their completion dates and the lessons learned.
Completes the QA Risk Assessment
(Optional) Reviews preliminary version of the QA Review Report with
the PpTL/PjTL and team and finalizes the report.
Provides final version of the QA Review Report and Risk Assessment
to the OO and the PpTL/PjTL.
In case of troubled projects (health rating C or D) or high/exceptional
SQA plan
Where (will it get reported)
PROJECT TEAM DATABASE
What (format will be used)
For QA1-3:
Technical and Proposal
Risk Assessment
Project Management Plan
(including auxiliary plans)
(Peer) Review results.
For QA4-6:
Delivery Risk Assessment
Detailed PMP
Actuals
(Peer) Review results.
Where (will it get reported)
PROJECT TEAM DATABASE
(via PjTL)
QA Tracking database (for
projects between 1000 and
5000 hours and for programs)
QA Workbench GAMSD (for
projects >= 5000 hours)
What (format will be used)
QA Review Report containing:
Findings
Recommended actions
Completion dates for
actions
Lessons Learned
Rating (for delivery projects
> 1000 hours)
Quality Assurance
Procedure
181206
7/15
PpTL/PjTL and OO
risk projects, reporting will be extended to the Sr. BAM, and for large
projects also to the QA (refer. appendix B).
Step 4 Respond to Review Findings
When (at what frequency)
QA Risk Assessment.
Where (will it get reported)
At or before completion dates for recommended actions.
PROJECT TEAM DATABASE
BMRS
How (will it be done)
What (format will be used)
Reviews report from the Quality Assurer.
Performs the recommended actions before the requested completion
dates.
Informs QA of completion of actions.
Document non-concurrence with the recommendations and review
this with QA and management.
Documents the rating and actual review date in BMRS
Updated QA Review
Report
Other documents as
appropriate
2.4 Activity List for QA violations
A QA violation is the non-execution of a mandatory QA review.
For the mandatory QA reviews see appendix A.
Quality Assurance
Procedure
181206
8/15
Role(who will do)
QA
Process-Activity
Step 1 Detect and report QA violation
When (at what frequency)
Detect: during scheduling of QA reviews by QA team or during a QA
review.
Report: within 2 working days after detection of violation
How (will it be done)
PpTL/PjTL
Detect a violation
A portfolio of projects is maintained by QA. This portfolio is derived
from several sources (PpTL/PjTL, OO, Business Operations). If a
QA1-3 review is performed then the SQA plan is known. Based on
the project portfolio and/or the SQA plan a violation can be detected.
Report a violation
For projects > 1000 hours, the violation will be reported to the OO
with a cc to:
The PpTL/PjTL
The Teamleader of the Independent QA team foram
The Manager of QA Netherlands
The OBM
For projects < 1000 hours, the violation will be reported as an issue in
the QA3-6 report.
Step 2 Respond to QA violation
When (at what frequency)
Within five working days after receiving the violation report.
How (will it be done)
QA
Reviews report from the Quality Assurer and determines root cause of
violation.
Determines the possible corrective actions that will alleviate the root
cause determined for the violation.
Defines action plan.
Reviews action plan with Quality Assurer and management.
Step 3 Close QA violation
When (at what frequency)
At completion of action plan.
How (will it be done)
Track the action plan to completion.
Close the QA violation.
Evidence
Where (will it get reported)
PROJECT TEAM
DATABASE: Violation
Report (via PjTL)
QA team database:
Violation Status
Overview (projects >
1000 hours)
What (format will be used)
QA report issue (projects
<= 1000 hours)
Violation report (projects >
1000 hours)
Violation Status Overview
(projects > 1000 hours)
Where (will it get reported)
PROJECT TEAM DATABASE:
Action Plan
QA team database: Violation
Status Overview (projects >
1000 hours)
What (format will be used)
Action plan
Violation Status Overview
(projects > 1000 hours)
Where (will it get reported)
PROJECT TEAM DATABASE:
Action plan (via PjTL)
QA team database: Violation
Status Overview
(projects > 1000 hours)
What (format will be used)
Updated Action plan
Violation Status Overview
(projects > 1000 hours)
3. Measurements
Quality Assurance
Procedure
181206
9/15
Number of QA reviews held within a business unit.
Health rating of the projects within a business unit.
Risk rating of the projects within a business unit.
Number of QA violations within a business unit.
The basic data for these measurements is stored in the QA Tracking database, the QA Workbench
and the QA team database.
4. Verification
All review results and violations are reported in the monthly meeting between the teamleader of
Independent QA and the Sr. Business Area Manager
An overview of review results and violations is reported by the teamleader of Independent QA in
the monthly meeting of the Sr BAM with his OO's.
An overview of review results and violations is reported in the periodic meeting between the AM
Manager, the QA manager and the teamleader of Independent QA
An overview of review results and violations on projects > 5000 hours is reported monthly to the
European QA manager (excluded program reviews).
A more detailed description of the findings and action plan on these projects is reported in the
following cases:
The project has a health rating of C or D (troubled projects)
The project has a risk rating of 7 or higher (High or Exceptional)
5. Tailoring
No tailoring is defined for the Activity List for QA Violations.
The Activity List for QA Reviews is completed with the additional descriptions in Appendix A, B and
C (defining clip levels, related review authorization and review validation details, report distribution
and review frequencies). The following tailoring is possible on this set of rules:
For Program reviews, full tailoring is allowed.
Project reviews below the clip level have to be done by the project team, and can be conducted
using a QA Checklists for Small Projects and following a Peer Review procedure. In addition, the
same tailoring as with projects above the clip level is allowed.
Project reviews above the clip level are mandatory, but some tailoring is allowed. For QA1, QA2
and QA6, the review topics need to be assured by the PpTL/PjTL, but reporting may be
combined with another appropriate review (e.g. QA3 for QA1/QA2, or QA5 for QA6) or in
another format. For QA4 and QA5 or QA5 and QA6, it can be decided during QA3 whether
these reviews may be combined.
6. Quality Records
Quality Assurance
Procedure
181206
10/15
Quality Record
Quality Assurance Review Reports
Risk Assessments
SQA Plan
Other Quality Documents
Violation report
Violation status report
Where Stored
PROJECT TEAM DATABASE
QA Tracking db for projects between 1000 and 5000 hours and
programs
QA Workbench GAMSD for projects >= 5000 hours
PROJECT TEAM DATABASE
QA Tracking db for projects between 1000 and 5000 hours and
programs
QA Workbench GAMSD for projects >= 5000 hours
PROJECT TEAM DATABASE
PROJECT TEAM DATABASE
PROJECT TEAM DATABASE
Mailbox of taskid 'AM QA'
QA team database
Retention Period
One year after project
completion
One year after project
completion
One year after project
completion
One year after project
completion
One year after project
completion
One year after project
completion
7. References
7.1 Standards & Procedures
Worldwide Quality Assurance Policies and Procedures
Quality Management System
Peer Review procedure
7.2 Definitions / Acronyms
QA: Quality Assurance
BMRS: Business Management Reporting System (central database for storing business
management information used to get overview on the business and support decision
making)
Independent QA: Quality Assurance, independent from the Line of Business
Aligned QA: an independent QA'er aligned to a business unit in AM.
7.3 Other References
Capability Maturity Model for Software, Software Engineering Institute, Carnegie Mellon
University
8. Tools, templates and forms used in the Process
8.1 Forms and Templates
AM SQA plan template
AM QA Checklists for Small Projects
Solution Design, QA123 review report
Solution Delivery, QA456 review report
8.2 Tools
QA Workbench : on server XXXX (only for independent QA)
QA Tracking database: on server XXXX (only for independent QA)
AM QA mailbox: on server XXXX (only for independent QA)
Quality Assurance
Procedure
181206
11/15
QA team database: on server XXXX (only for independent QA)
Appendix A. Clip Levels and authorized reviewers
The Worldwide Quality Assurance (WWQA) process mandates quality assurance reviews, the
requirements for which are documented in the following table.
For projects below the clip level the validation activities as mentioned below in column 'Review
Validates' have to be performed by the PjTL/PpTL and have to be recorded in a Project Control
Book, but do not need to be reported formally.
For program reviews, no clip level is defined.
Quality Assurance
Procedure
181206
12/15
Authorized reviewers by Project Clip Level
Type of Review
Review Validates
Above 1000 hours
On and Below 1000
hours
QA1. Solution
Assurance Review
Proposed solution:
fulfills customer's requirements
is technically viable
complete and reasonable estimates and schedules
technical risks identified, assessed and controlled.
PM
Informal
(peer review)
QA2. Business
Assurance Review
Verification of:
solution satisfies both the companies´ and customer business
objectives
complete cost case, contingencies identified
complete and reasonable estimates and schedules
Cost, planning, and technical baselines previously reviewed have
been updated to reflect changes, and can now be regarded as final.
Documentation clearly defining the nature, scope, special
conditions, delivery schedule, responsibilities and dependencies of
all parties (including payment streams) is in place to establish clear
project baselines. Overall agreement that the offer can be
responsibly made and represents good business for the offering
organization reached in QA2 is verified, if applicable.
The project planning baselines, and all other project management
documentation sufficient to commence work on the project, are in
place and represent an adequate project management foundation.
Draft schedules for performance of quality assurance reviews have
been established.
Project performance is evaluated to ensure that activities are all on
track, key dates will be met, quality deliverables are being provided,
processes, standards and procedures are being adhered to,
financial and resource commitments are being met, risks are being
mitigated, and issues are carried forward for Service Line
Management focus as appropriate. QA5 reviews are repetitive
based on the review classification (health) assigned to the project.
Adequate and appropriate pre-delivery testing has been performed
by the delivery team and validated by the project SMEs. This review
is scheduled before major delivery milestones and
before overall project delivery and completion.
PM
Informal
(peer review)
Independent QA
(also review of
QA1/QA2)
Informal
(peer review)
QA3. Proposal
Assurance Review
QA4. Project
Management
(PMR) Review Initial
QA5. Project
Management
(PMR) Review
QA6. Deliverable
Readiness Review
Independent QA
Informal
(peer review)
Independent QA
Informal
(peer review)
Independent QA
Informal
(peer review)
Note: Scope change may result in a proposal or project previously below a particular clip level
passing into a new clip level. If this occurs the PpTL/PjTL informs the assigned Quality Assurer. The
review should be held in accordance with the new clip level.
Appendix B. Project Health Rating for QA4, QA5 and QA6
Quality Assurance
Procedure
181206
13/15
Health
Rating
A
Description
Project is under control. Minor problems may exist, but the PpTL/PjTL
has an effective plan in place to solve the problems. No major existing
or potential problems have been identified.
B
Project is currently under control. However, existing or potential
problems have been identified which will require positive management
attention in order to keep the project under control.
C
Significant problems exist. Probability exists for exceeding estimates
or budgets, customer dissatisfaction, and/or limited financial
exposure. Aggressive management action is required to bring the
project under control.
D
Major problems exist with definite, serious financial exposure and/or
customer dissatisfaction.
Report Distribution
PpTL/PjTL
OO
Other review
participants
PpTL/PjTL
OO
Other review
participants
PpTL/PjTL
OO
Manager of OO
Other review
participants
For projects of > 5000
hours:
EUROPEAN
GAMSD QA
Manager
QA Manager
PpTL/PjTL
OO
Manager of OO
Other review
participants
Next Review
No later than six (6)
months past the
current project review
date.
No later than five (5)
months past the
current project review
date.
No later than four (4)
months past the
current project review
date.
No later than three
(3) months past the
current project review
date.
For projects of > 5000
hours:
EUROPEAN
GAMSD QA
Manager
QA Manager
Note 1: QA1, QA2 and QA3 reviews do not report a Health Rating. The advice can be 'Proceed', 'Do
not Proceed', or 'Complete actions, then proceed'.
Note 2: C & D Projects are considered 'troubled'.
Note 3: Each QA review also produces a Risk Assessment. High Risk projects are subject to the
same management attention as troubled projects.
Appendix C. Frequency of reviews
QA1-4: Once during the lifecycle of a project
QA5: See Appendix B, column 'Next Review', but at least once during the lifecycle of a project
and no later then 30 days before end of project
QA6: Before sending a major deliverable to a customer and at the end of each project
Quality Assurance
Procedure
181206
14/15
The timing of each type of QA review is prescribed in the Activity Lists of AM Management System
End of Document
Quality Assurance
Procedure
181206
15/15