E UROPEAN M IDDLEWARE
I NITIATIVE
EMI 1 U P D AT E 9
CODING
AN D
TESTING METRICS REPORT
Date:
24/11/2011
Activity:
SA2.7 – Quality Control
Lead Partner:
CINECA
INFSO-RI-261611
2010 © Members of EMI collaboration
PUBLIC
1 / 11
EMI 1 UPDATE 9 – CODING AND TESTING METRICS REPORT
Date: 24/11/2011
Copyright notice:
Copyright (c) Members of the EMI Collaboration. 2010.
See http://www.eu-emi.eu/about/Partners/ for details on the copyright holders.
EMI (“European Middleware Initiative”) is a project partially funded by the European Commission. For more
information on the project, its partners and contributors please see http://www.eu-emi.eu.
This document is released under the Open Access license. You are permitted to copy and distribute verbatim
copies of this document containing this copyright notice, but modifying this document is not allowed. You are
permitted to copy this document in whole or in part into other documents if you attach the following reference to
the copied elements: "Copyright (C) 2010. Members of the EMI Collaboration. http://www.eu-emi.eu ".
The information contained in this document represents the views of EMI as of the date they are published. EMI
does not guarantee that any information contained herein is error-free, or up to date.
EMI MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, BY PUBLISHING THIS DOCUMENT.
INFSO-RI-261611
2010 © Members of EMI collaboration
PUBLIC
2 / 11
EMI 1 UPDATE 9 – CODING AND TESTING METRICS REPORT
Date: 24/11/2011
TABLE
OF
CONTENTS
1. SUMMARY ........................................................................................................................................................5
2. METRICS ...........................................................................................................................................................6
2.1. NUMBER OF TEST PLANS ...........................................................................................................................6
2.2. NUMBER OF TEST REPORTS .......................................................................................................................6
2.3. NUMBER OF MANDATORY TESTS ................................................................................................................7
2.4. NUMBER OF RFCS TRACKING A DEFECT WITH ASSOCIATED REGRESSION TEST ..........................................8
2.4.1 Product Name.................................................................................................................................9
2.5. NUMBER OF RFCS TRACKING A NEW FEATURE WITH ASSOCIATED FUNCTIONALITY TEST ..........................9
2.5.1 Product Name............................................................................................................................... 10
2.6. NUMBER OF DEVELOPMENT TASKS TRACKING A NEW FEATURE WITH ASSOCIATED FUNCTIONALITY TEST10
2.6.1 Product Name............................................................................................................................... 11
2.7. NUMBER OF PASSED CERTIFICATION CHECKS.......................................................................................... 11
2.8. NUMBER OF REDUCED LINES OF CODE (KPI KJRA1.3) ............................................................................... 12
2.8.1 Product Name............................................................................................................................... 13
INFSO-RI-261611
2010 © Members of EMI collaboration
PUBLIC
3 / 11
EMI 1 UPDATE 9 – CODING AND TESTING METRICS REPORT
Date: 24/11/2011
1. SUMMARY
Quality
characteristic
Metrics
Required Level
Number of Test Plans per EMI One per EMI software
product.
software product.
Nº per
software
product.
Number of Test Reports per One per released
software product.
released EMI software product.
Nº per released
product.
Number of mandatory tests per Ideally 100%, however
an improvement per
EMI software product.
% per released
product.
Number of RfCs tracking a defect Ideally 100%, although
an increased value per
with an associated regression test.
% per released
product.
Number of RfCs tracking a new Ideally 100%, although
feature
with
an
associated an increased value per
product per release is
functionality test.
% per released
product.
Number of development tasks Ideally 100%, although
tracking a new feature with an an increased value per
product per release is
associated functionality test.
% per released
product.
Number of passed certification 100% for those checks
that are considered
checks.
% per released
product.
KPI KJRA1.3 Number of Reduced > 33% (1/3) reduction
over the three-year
lines of code.
% per released
product.
product is also a good
indicator.
Testability
Assessment
Actual Result
product per release is
also a good indicator.
EMI
also a good indicator.
also a good indicator.
mandatory according to
the Production Release
Criteria.
Maintainability
compliance
INFSO-RI-261611
activity. The reduction
can be consequence of
removing components or
replacing them with
commercial or
community alternatives
[R4]
2010 © Members of EMI collaboration
PUBLIC
4 / 11
EMI 1 UPDATE 9 – CODING AND TESTING METRICS REPORT
Date: 24/11/2011
2. METRICS
2.1. NUMBER OF TEST PLANS
The table below shows for each of the released products in Release X (Update Y) whether a test plan
has been provided:
Product
Test Plan available?
L&B, v. 3.1.0
YES
StoRM v. 1.8.0
YES
Link to available test plans: https://twiki.cern.ch/twiki/bin/view/EMI/QCTestPlan
Link to the EMI 1 Update 9: https://twiki.cern.ch/twiki/bin/view/EMI/EMI1Update9
2.2. NUMBER OF TEST REPORTS
The table below shows for each of the released products in Release EMI 1 (Update 9) whether a test
report has been provided and whether it complies with the test report template defined in the EMI
Testing Policy:
Product
Test Report available?
Test Report complies with SA2?
Product 1
YES
YES
Product 2
YES
YES
Product 3
YES
YES
The following trend diagram shows the number of test reports, compliant and non compliant with the
EMI Testing Policy, per EMI Update:
The following graphic shows the overall percentage of test report compliance with the EMI Testing
Policy for all the products released in EMI Updates.
INFSO-RI-261611
2010 © Members of EMI collaboration
PUBLIC
5 / 11
EMI 1 UPDATE 9 – CODING AND TESTING METRICS REPORT
Date: 24/11/2011
2.3. NUMBER OF MANDATORY TESTS
The table below shows for each of the released products in Release X (Update Y) whether the set of
mandatory tests defined in the EMI Testing Policy have been executed or not for this update:
Product
L&B, v. 3.1.0
StoRM v. 1.8.0
INFSO-RI-261611
Mandatory Tests
Executed?
Unit Tests
NO
Clean Deployment Tests
YES
Upgrade Deployment Tests
?
Basic Functionality Tests
YES
Automated Regression Tests
NO
Unit Tests
NO
Clean Deployment Tests
YES
Upgrade Deployment Tests
?
Basic Functionality Tests
YES
Automated Regression Tests
YES
2010 © Members of EMI collaboration
PUBLIC
6 / 11
EMI 1 UPDATE 9 – CODING AND TESTING METRICS REPORT
Date: 24/11/2011
The following graphics show the overall percentage of executed, non executed and non applicable
mandatory tests for all the products released in EMI Updates.
2.4. NUMBER OF RFCS TRACKING A DEFECT WITH ASSOCIATED REGRESSION
TEST
The graphic below shows for each of the released products in Release EMI 1 (Update 9) whether the
RfCs tracking defects that have been included in this release have an associated regression test.
INFSO-RI-261611
2010 © Members of EMI collaboration
PUBLIC
7 / 11
EMI 1 UPDATE 9 – CODING AND TESTING METRICS REPORT
Date: 24/11/2011
2.4.1 Product Name
For each of the released products in Release X (Update Y), the following trend diagram should be
calculated
2.5. NUMBER OF RFCS TRACKING A NEW FEATURE WITH ASSOCIATED
FUNCTIONALITY TEST
The graphic below shows for each of the released products in Release EMI 1 (Update 9) whether the
RfCs tracking new features that have been included in this release have an associated functionality
test.
INFSO-RI-261611
2010 © Members of EMI collaboration
PUBLIC
8 / 11
EMI 1 UPDATE 9 – CODING AND TESTING METRICS REPORT
Date: 24/11/2011
2.5.1 Product Name
For each of the released products in Release X (Update Y), the following trend diagram should be
calculated
2.6. NUMBER OF DEVELOPMENT TASKS TRACKING A NEW FEATURE WITH
ASSOCIATED FUNCTIONALITY TEST
The graphic below shows for each of the released products in Release X (Update Y) whether the
development tasks tracking new features that have been included in this release have an associated
functionality test.
2.6.1 Product Name
For each of the released products in Release X (Update Y), the following trend diagram should be
calculated
INFSO-RI-261611
2010 © Members of EMI collaboration
PUBLIC
9 / 11
EMI 1 UPDATE 9 – CODING AND TESTING METRICS REPORT
Date: 24/11/2011
2.7. NUMBER OF PASSED CERTIFICATION CHECKS
The following graphic shows statistics on the passed certification checks for Update 9.
NOTE: it should be considered whether it is interesting to present these statistics for all the past
updates.
2.8. NUMBER OF REDUCED LINES OF CODE (KPI KJRA1.3)
The graphic below shows for each of the released products in Release X (Update Y) the number of
lines of code (kSLOC).
INFSO-RI-261611
2010 © Members of EMI collaboration
PUBLIC
10 / 11
EMI 1 UPDATE 9 – CODING AND TESTING METRICS REPORT
Date: 24/11/2011
2.8.1 Product Name
For each of the released products in Release X (Update Y), the following trend diagram should be
calculated
INFSO-RI-261611
2010 © Members of EMI collaboration
PUBLIC
11 / 11