Visual Studio Quality Tools Deployment Plan Template Develop Tools Deployment Planning Services Page 1 Visual Studio Quality Tools Deployment Plan Template, Version 1 Table of Contents 1 2 Getting the most from your Deployment Plan ............................................................ 5 1.1 Deepen your understanding of the Visual Studio Quality Tools ............................................ 5 1.2 Find additional resources........................................................................................................ 6 1.3 Give Microsoft feedback ......................................................................................................... 6 Executive Summary ................................................................................................... 7 2.1 Existing Best Practices ........................................................................................................... 7 2.2 Key Areas for Improvement .................................................................................................... 8 2.2.1 Current State – Urgent Issues ............................................................................................ 8 2.2.2 Current State - Additional Issues ........................................................................................ 9 2.3 Future State ............................................................................................................................ 9 3 Key Areas for Improvement ..................................................................................... 10 4 Roadmap ................................................................................................................. 11 5 Detailed Findings ..................................................................................................... 14 6 Conclusion ............................................................................................................... 21 Appendix ....................................................................................................................... 22 Page 2 Visual Studio Quality Tools Deployment Plan Template, Version 1 1 Getting the most from your Deployment Plan Guidance: This template includes guidance blocks and wording examples. Prior to handing over the document, remove the guidance blocks (like this one) and replace any highlighted sample text in <brackets> with your findings and recommendations. Our recommendations for optimizing the deployment of the Visual Studio Quality Tools in your environment are detailed within this document. Please take your time to review the findings and ask any follow-up questions necessary. Depending on the capabilities of your test team, you may select to keep the deployment in-house or contract with an outside consultant. In either case, this plan should be given to the party responsible for the work and used as an implementation guide. 1.1 Deepen your understanding of the Visual Studio Quality Tools As a benefit for Visual Studio Quality Tools Deployment Planning Service customers, Microsoft developed materials that will facilitate your transition. These items have been installed on-site and were reviewed with <Insert the name of the customer SME here>. The materials include: Guidance: Use the “Visual Studio Quality Solutions – Post Engagement Resources Guide” to locate the resources below. VS2010 ALM Demo VHD - This virtual machine (VM) includes Visual Studio 2010 Ultimate, Visual Studio Team Foundation Server 2010, and a sample application along with sample data which supports 9 hands-on-labs. This VM includes everything you need to learn and/or deliver demonstrations of many application lifecycle management (ALM) capabilities in Visual Studio 2010. VS2010 Training Kit – Easy-to-use training materials that will show you how to get you up and running once you deploy. Visual Studio ALM – A compilation of industry leading white papers and case studies that describe the Visual Studio Application Lifecycle Management solution. Visual Studio 2010 Quality Solutions – Hands On Labs (HOLs) o Visual Studio 2010 Quality Solutions – HOLs Presentation o How Do I: Plan Tests – Video o How Do I: Author And Run Tests – Video o How Do I: Track Quality – Video Visual Studio Test Professional 2010 Evaluators Guide Microsoft Visual Studio 2010 Quality Offering Datasheet Microsoft Visual Studio 2010 Test Professional Datasheet Page 5 Visual Studio Quality Tools Deployment Plan Template, Version 1 1.2 Find additional resources You can find additional Microsoft Visual Studio Quality Solutions resources in the following locations: Guidance: Use the “Visual Studio Quality Solutions – Post Engagement Resources Guide.docx” document to identify additional resources you want to share with the customer based on their needs. 1.3 MSDN Product Homes: o Visual Studio o ALM MSDN How-Do-I videos: o Visual Studio Test Tools MSDN Library: o ALM o Testing MSDN Learn: o ALM (includes Testing) Give Microsoft feedback This Planning Service has been provided as part of your Microsoft Software Assurance benefits. Please use the link below to tell Microsoft about your experience with the engagement and any improvements you would like to see made to it. The results of the survey will only be viewed by the Planning Services team at Microsoft. http://www.surveymonkey.com/s/dtdps_cs Page 6 Visual Studio Quality Tools Deployment Plan Template, Version 1 2 Executive Summary Guidance: The audience for this section will be interested in being able to read and digest this quickly. Try to make the text in this section concise. At the request of <Customer Name>, <Partner name> conducted a Developer Tools Visual Studio Quality Tools Deployment Planning Services assessment with the following objectives: Document existing ALM Quality topology Create a baseline measurement of the current test capability Identify existing best practices Uncover opportunities for improvement Identify the highest impact areas to the business Document ideal end-state for the Visual Studio Quality Tools deployment Generate and present a roadmap to implement the Visual Studio Quality Tools The Application Lifecycle Management (ALM) model was used as a framework to develop a vision and sustainable approach by which <Customer Name> can prioritize IT investments that fuel business growth. The engagement focused on understanding existing development processes and recommending process improvements. Technology and people/knowledge requirements were then identified to support the process. The following issues with the current development capability were articulated at the start of the assessment. Guidance: This list can come from the ALM Assessment Builder Tool (Presenting Problems) Quality issues No visibility into project status Projects delivered late The following business priorities were articulated at the start of the assessment. Guidance: This list can come from the ALM Assessment Builder Tool (Business Priorities) 2.1 Improve quality Improve customer satisfaction Existing Best Practices Our interviews surfaced the following Best Practices that are being used by teams at <Customer Name> today. These practices are: Page 7 Visual Studio Quality Tools Deployment Plan Template, Version 1 Guidance: The list below can be generated from the ALM Quality Assessment tool. Note: when using the tool you need to complete the “Best Capabilities” field. You will also need to clearly define which capability and practice area the best practice relates to, as the tool won’t do it for you. Business Analysis: The client uses Team Foundation Server 2010 to manage all project requirements. Requirements Management: By utilizing the capabilities of TFS, particularly the alerting mechanism, which effectively and efficiently keeps all team up to date. Code Reviews: Code reviewing is integrated into the automated check-in policy. No code can be checked in without a code review taking place. Build Management: The sophisticated approach to build management is well defined and documented. The build processes are fully automated and parameterized where required, to support multiple target environments. The automated build and deployment approach to production is replicated in the test environments. Test Team Organization: The use of virtualization for test environments. Test Planning: Test plans are well defined and linked to project plans. We recommend that these practices continue to be employed and are continuously evaluated to promote process optimization. 2.2 Key Areas for Improvement 2.2.1 Current State – Urgent Issues During our on-site interviews, we uncovered the following practices that should be considered critical and essential to improving the testing capability in relation to its impact on the business. These practices were: Guidance: If the ALM Assessment Builder Tool is used, the information below can be derived from the Impact Map, Impact Benefits and Impact Observations. Testing and Quality Assurance o Test Types/Functional Testing: The current approach to functional testing of focusing solely on requirements in isolation is leading to numerous defects being found later in the lifecycle and in production. Best practice would be to also identify scenarios that may not readily be identified at individual requirement level. Exploratory testing is a beneficial mechanism for unearthing hard to find and unusual defects. o Test Management/Requirements Linking: The linking of test cases to the requirements they are testing increases the visibility of test coverage and test progression. This should increase customer confidence. Page 8 Visual Studio Quality Tools Deployment Plan Template, Version 1 o o 2.2.2 Test Management/Test Case Prioritization: Test cases should be prioritized during the test analysis phase so that they accurately reflect the customer’s needs. Test Management/Defect Management: Using a single, integrated and efficient defect management system will reduce unnecessary levels of complexity and duplication. Current State - Additional Issues Guidance: If the ALM Assessment Builder Tool is used, the information below can be derived from the Impact Map, Impact Benefits and Impact Observations. During the course of the assessment, we were able to identify additional issues that we believe are having a material impact on Application Lifecycle Management within <Customer Name>. These include: 2.3 Testing and Quality Assurance o Test Planning/Stabilization Window: Stabilization windows should be zealously guarded. The overall quality of a product and the perception of the end users of that product can be seriously damaged by decreasing the window in-flight. Where stabilization windows are reduced the use of test case prioritization becomes essential. Requirements Management o Requirements Analysis/UAT: Ensuring that the UATs are made available early in the project, the test team can incorporate them into the testing lifecycle. This should ensure a smooth UAT phase. Development o Quality Metrics/Code Coverage: Defects found early in the lifecycle are cheaper to rectify than those found later. By integrating code coverage standards into the check-in policies the number of expensive late cycle defects is reduced. o Code Analysis/Performance Testing: By integrating performance testing into the iterative development lifecycle the end product becomes much more robust. Future State The eventual roll out of process and tools to address the opportunities mentioned in this assessment should follow a tight metrics driven approach to improvement. While we recognize that change takes time and reinforce that change requires commitment, it is important to create both a strategic and tactical plan for addressing these challenges, as well as answering the questions of where do we want to be in three, six and twelve months and how will we quantify our success. Page 9 Visual Studio Quality Tools Deployment Plan Template, Version 1 3 Key Areas for Improvement Our interviews revealed multiple areas for improvement. These were rated by impact to the business (High, Medium or Low) across the maturity levels. These are shown in the Impact Map. The x-axis defines the maturity level of the service area. The categories are: Basic - processes are implemented in an ad-hoc, undocumented and potentially inconsistent manner. Standard - a process has been defined and is generally followed. Tools are used in some cases to assist, but may not be integrated and used throughout the organization. Advanced - usage of tools to drive the process is in wide use and usage guidelines are documented and understood. Dynamic - the organization is bringing new and innovative methodologies to the practice area and may be setting industry standards. The y-axis defines the relative gain that would be obtained from improving the practice. Guidance: During the assessment the areas that need the most attention are cited in more detail in section Current State. These areas are represented in the upper left cell which denotes practices with a basic level of maturity and a high impact to be gained from improvement. The majority or practice areas fall into a basic level of maturity with a medium impact to be gained from improvement. Basic High MATURITY Standard Advanced Urgent Improve Test Management Test Types Improve Mediu m IMPACT Code Analysis Enhance Low Enhance Dynamic Maintain Enhance Maintain Maintain Maintain Quality Metrics Improve Requirements Analysis Test Planning Enhance Business Analysis Requirements Management Code Reviews Build Promotion Project Organization Project Tracking Test Team Organization Build Management Table 1: Current Impact and Maturity Rating Page 10 Visual Studio Quality Tools Deployment Plan Template, Version 1 4 Roadmap Based on our observations and discussions, we recommend that the following iterative roadmap be implemented in order to better align the testing capability with the business, and enable testing efforts to drive increased value to <Customer Name>. Please note that the areas for improvement mentioned in the prior section which are marked as urgent may not be addressed immediately. In some cases, the foundations for improving a particular service area will not be in place in the first or second iteration. Guidance: The table below is generated from the ALM Assessment Builder Tool. . Iteration 1 - Start Iterations Iteration 2 - Improve Iteration 3 - Refine Identify pilot project for implementation of Microsoft Test Manager Test Management Test Team Organization Improve test management Increase code functional quality Test Management Code Coverage Initiative Title Implement Microsoft Test Manager Increase breadth of functional testing Increase code non-functional quality Capabilities Test Management Test Team Organization Test Planning Train Test Team Test Types Code Analysis Test Types Test Management Test Team Organization Test Planning Test Types Requirements Analysis Initiative Title Capabilities Initiative Title Capabilities Integrate UAT earlier into the testing lifecycle Table 2: Initiatives Roadmap When embarking on an effort to optimize the development capability we recommend that strong leadership sponsorship is secured, that the overall goals are clearly communicated to all stakeholders, and clear metrics and milestones are established and agreed to. Our full report details all findings and recommendations. Page 11 Visual Studio Quality Tools Deployment Plan Template, Version 1 Initiative Details Guidance: The table below is generated from the ALM Assessment Builder Tool. Iteration 1: From: 01/06/2011 To: 30/06/2011 Identify pilot project for implementation of Microsoft Test Manager Initiative Goals: To identify a suitable project that will enable the implementation of Microsoft Test Manager Initiative Activities: Identify friendly project On-board project team with initiative Initiative Cross References: Impacted Capabilities: Test Management Test Team Organization Implement Microsoft Test Manager Initiative Goals: To have Microsoft Test Manager deployed to all relevant workstations and configured to the pilot project Initiative Activities: Deploy Visual Studio 2010 Ultimate on relevant machines Deploy Visual Studio 2010 Test Professional on relevant machines Initiative Cross References: Impacted Capabilities: Train Test Team Initiative Goals: Initiative Activities: Initiative Cross References: Impacted Capabilities: Test Management Test Team Organization Test Planning To ensure the test team is comfortable using Test Professional prior to starting the pilot project Identify and obtain suitable training in Test Professional Test Management Test Team Organization Test Planning Page 12 Visual Studio Quality Tools Deployment Plan Template, Version 1 Iteration 2: From: 02/07/2011 To: 30/07/2011 Improve test management Initiative Goals: To improve the test management by ensuring a closer bond between the test cases and requirements Initiative Activities: Link test cases to requirements Prioritize test cases based on customer need Integrate defect management Initiative Cross References: Impacted Capabilities: Test Management Increase breadth of functional testing Initiative Goals: To ensure greater coverage through enhanced functional testing. For example, adopting exploratory testing Initiative Activities: Use the capabilities of the test runner to allow exploratory testing Implement scenario based testing Initiative Cross References: Impacted Capabilities: Test Types Integrate User Acceptance Testing Initiative Goals: Integration of UAT earlier into the testing lifecycle Initiative Activities: Work with requirements and client to identify UATs Integrate UAT tests into test plan in Microsoft Test Manager Initiative Cross References: Impacted Capabilities: Test Types Requirements Analysis Iteration 3: From: 01/08/2011 To: 30/08/2011 Increase code functional quality Initiative Goals: To reduce the number of unit test level defects promoted into system test Initiative Activities: Define code coverage standards Implement code coverage standards into automated check-in policies Initiative Cross References: Impacted Capabilities: Code Coverage Increase code non-functional quality Initiative Goals: To ensure developers build code that meets the necessary performance requirements Initiative Activities: Use load testing capabilities of Visual Studio 2010 Ultimate Use code analysis tools with Visual Studio 2010 Ultimate Initiative Cross References: Impacted Capabilities: Code Analysis Test Types Page 13 Visual Studio Quality Tools Deployment Plan Template, Version 1 5 Detailed Findings The following table provides a snapshot of the development tools and platforms currently in use at <customer name>. Category Requirements Architecture Quality and Testing Software Configuration Management Databases Product Management Tools Tools TFS 2010 Rational Rose Rational Robot / Rational Manual Tester TFS 2010 Oracle / SQL Server 2008 TFS 2010 Detailed Findings Guidance: The table below is generated from the ALM Assessment Builder Tool. Requirements Management Summary : Business Analysis Maturity Observations: Business Analysis: The Client uses Team Foundation Server 2010 to manage all project requirements. Assessed Score: 3.50 Maturity Level Rating: Advanced Impact Level Rating: Low Impact Observations: Best practice is already being followed. Best Practices: Business Analysis: The Client uses Team Foundation Server 2010 to manage all project requirements. Impact Benefits: The customer is already using TFS to manage all project requirements. There would be little or no benefit to making changes to the current process. Requirements Analysis Page 14 Visual Studio Quality Tools Deployment Plan Template, Version 1 Maturity Observations: Formal User Acceptance Tests are used. However, it is often the case that the UATs are identified late in the development / test cycle, leaving little time for integration into the daily process. Assessed Score: 2.30 Maturity Level Rating: Standard Impact Level Rating: Medium Impact Observations: Overall the customer’s client is happy with the quality of the product deliveries. However, the customer’s client is frequently finding niggling issues when performing UAT. This causes embarrassment to the customer. Impact Benefits: By working with the client to ensure that the UATs are made available to early into the project lifecycle, the test and development teams can ensure that the key deliverable areas are identified. Requirements Management Maturity Observations: The customer demonstrates a high level of maturity around the Requirements Management practice area. Assessed Score: 3.50 Maturity Level Rating: Advanced Impact Level Rating: Low Impact Observations: Changes may have a detrimental impact. Best Practices: By utilizing the capabilities of TFS, particularly the alerting mechanism, which effectively and efficiently keeps all team up to date. Impact Benefits: No benefit in changing the current practice. Development Summary : Code Analysis Maturity Observations: Performance testing should be integrated into the development lifecycle, rather than left as a separate test phase at the end of the Page 15 Visual Studio Quality Tools Deployment Plan Template, Version 1 development lifecycle. The development team is conscious of the benefits of code coverage and unit testing. However, given the timescales for the iteration, this often falls by the wayside. There is no entry criterion regarding code coverage when passing into system test. Assessed Score: 1.00 Maturity Level Rating: Basic Impact Level Rating: Medium Impact Observations: The customer's client routinely raises performance issues after the product has been released. These issues are generally known by the customer due to the performance test phase. However, because that phase occurs so late there is never any time to make changes prior to release. There is a service pack approach to fixing such issues and the customer’s client is accepting of this approach. The lack of sufficient code coverage prior to promoting a build to the test environments has a detrimental impact on the test team’s capability of executing their systems test within the iteration. Impact Benefits: By integrating performance testing into the iterative development lifecycle, the end product becomes much more robust. This potentially negates the need to perform an expensive performance test phase just prior to release. By including code coverage into the automated check-in policy, the number of bugs raised by the test team should be reduced. This not only saves time in the long run, but should reduce cost. Code Reviews Maturity Observations: Development does good code reviews. Assessed Score: 3.20 Maturity Level Rating: Advanced Impact Level Rating: Low Best Practices: Code reviewing is integrated into the automated check-in policy. No code can be checked in without a code review taking place. Quality Metrics Page 16 Visual Studio Quality Tools Deployment Plan Template, Version 1 Maturity Observations: The check-in policies are well documented and quite thorough. One area that should be looked at for improvement is code coverage. The team does not use the available quality metrics to drive improvement. Assessed Score: 2.40 Maturity Level Rating: Standard Impact Level Rating: High Impact Observations: The customer sees the detrimental impact of poor code coverage within unit testing as a major blocker to the test team being able to perform all the system testing within the iterations. The customer currently operates a reactive approach to coding issues. They really wants to start taking a more proactive approach and using quality metrics to guide future decisions is a step in the right direction. Impact Benefits: Adding code coverage to the check-in policy would reduce the number of issues that come out of the test team. By utilizing the metrics around code quality the customer should be able to analyze the areas which have the most issues and address the process appropriately. Software Configuration Management Summary : Build Management Maturity Observations: The customer has a sophisticated automated build process in place. Assessed Score: 4.20 Maturity Level Rating: Dynamic Impact Level Rating: Low Best Practices: The sophisticated approach to build management is well defined and documented. The build processes are fully automated and parameterized where required, to support multiple target environments. The automated build and deployment approach to production is replicated in the test environments. Page 17 Visual Studio Quality Tools Deployment Plan Template, Version 1 Deployment & Operations Summary : Build Promotion Assessed Score: 3.40 Maturity Level Rating: Advanced Impact Level Rating: Low Project Management Summary : Project Organization Maturity Observations: Very mature Assessed Score: 3.30 Maturity Level Rating: Advanced Impact Level Rating: Low Project Tracking Maturity Observations: Pretty mature Assessed Score: 3.80 Maturity Level Rating: Advanced Impact Level Rating: Low Impact Observations: Little impact. Customer is comfortable with current approach. Impact Benefits: Testing & Quality Assurance Summary : Test Team Organization Maturity Observations: The test organization is independent and well-staffed. Although the test leads are involved in the early stages of the product, they do not Page 18 Visual Studio Quality Tools Deployment Plan Template, Version 1 have any input into the estimation. Assessed Score: 3.20 Maturity Level Rating: Advanced Impact Level Rating: Low Best Practices: The use of virtualization for test environments. Test Planning Maturity Observations: Consideration to the various functional and non-functional aspects of testing are considered and catered for within test plans. Each test phase has an associated test plan. Although the time allocated to the stabilization period at the start of projects is sufficient this is often reduced actuality if development slips. The release dates are not often moved, rather testing is squeezed. Assessed Score: 2.30 Maturity Level Rating: Standard Impact Level Rating: Medium Impact Observations: The perceived level of quality is a big pain issue for the customer. When stabilization windows are reduced the number of bugs found in production increases and undermines the confidence of the customer's client. Best Practices: Test plans are well defined and linked to project plans. Impact Benefits: Protection of the stabilization window would reduce the risk of releasing a product that has not undergone sufficient quality assurance. Test Management Maturity Observations: There are varying levels of maturity being displayed. Of particular significance are the lack of linkage between test cases and requirements, the lack of an integrated defect management tool and the lack of test case prioritization. Assessed Score: 0.4 Page 19 Visual Studio Quality Tools Deployment Plan Template, Version 1 Maturity Level Rating: Basic Impact Level Rating: High Impact Observations: Test case prioritization - This is essentially to ensure that the most important tests are executed first. This takes on additional importance when stabilization windows are reduced. Requirements linking - The customers client is very critical of the lack of visibility when it comes to which requirements are being tested. Defect Management - The time expended using tools that are integrated into the process is impacted both the testing and development areas. Impact Benefits: Test case prioritization - All test cases should be prioritized during the test analysis phase, so that they accurately reflect the customer's needs. Test-Requirement Linking - Linking test cases to requirements is an essential enabler of test/project tracking progress. Defect Management - Using an integrated defect management system will reduce unnecessary level of complexity and duplication Defect management - Utilizing the capabilities within Microsoft Test Manager and TFS would greatly reduce unnecessary duplication of effort and complexity Test Types Maturity Observations: Functional testing is performed at the most basic level. Maturity Level Rating: Basic Impact Level Rating: High Impact Observations: The client has big issues with the level of functional testing currently performed. Impact Benefits: Improved functional testing will improve the overall quality and increase the client's confidence with the customer. Page 20 Visual Studio Quality Tools Deployment Plan Template, Version 1 6 Conclusion We recommend that the implementation of the practices outlined in this document be validated during the initial deployment and as projects and teams are brought on board the system. Test teams constantly have to adapt and change their processes as the development environment changes around them, Microsoft Test Manager allows projects and test plans to be highly customized to accommodate these changes. To encompass all of the recommendations in this document, a schedule for all of the relevant tasks should be created. Complete implementation and customization should be done by <Customer Name> operations staff. Page 21 Visual Studio Quality Tools Deployment Plan Template, Version 1 Appendix As mentioned we highly encourage <Customer Name> to accelerate the implementation of Microsoft Team Foundation Server. The below chart defines some key functionalities as the map to core areas discussed in the assessment to the functionality and approach of Team Foundation Server Key Improvement Area Team Foundation Server Capability Description Application and Production Change Management Collaborative Development Team Foundation Server provides a central repository that teams of development team may share in an effort to share information and data to improve team collaboration and communication Database Change Management Team Foundation Server provides support to combine source code management with Database change management to ensure migration change management quality at every promotion state of the application. Most importantly, it ensures quality build management of source combined with data schemas at the production state. Version Control Repository Team Foundation Server provides a version control environment that ensures workspace isolation with enriched concurrent development. Release Management Team Foundation Server provides automated deployment services that ensure release quality with automated build verification at ALL promotion states. Build Management Team Foundation Server provides world-class automated build capabilities triggered by check-ins and scheduled events that ensures build quality at each level of promotion within your production environment. Change Management Deployment Environment Management Team Foundation Server Supports change management at all levels. It provides protection to build and release environments through code and build audits and isolation. It provides traceability from requirements to tasks to code to tests through the lifecycle giving efficient impact analysis. Database Deployment Team Foundation Server treats your database schemas as it would source code and gives your database all the same change management benefits as source code. Project Monitoring and Control Stakeholder Management Test Planning Test Management Team Foundation Server provides rich project management capabilities that give your stakeholders direct visibility into project progress, risks, and accomplishments. It also provides a common interface from which to plan, design, and execute your tests. Page 22 Visual Studio Quality Tools Deployment Plan Template, Version 1 Page 23 Visual Studio Quality Tools Deployment Plan Template, Version 1
© Copyright 2026 Paperzz