CSPF Scenarios
Running in the Pipeline and How to Read the Results
Michelle L. Crane and Frédéric Fusier
DevOps Transformation and Enablement
18 Feb 15
Contents
 CSPF Scenarios and UCD D1 Environment
 Alm.60.all Build
– Moving from Alm.60.all to individual scenario
– Scenario build information (tabs)
 Problems
– How to investigate
– How to handle
 Miscellaneous
– How to access servers
2
Scenarios and UCD D1 Env
CSPF Scenarios in the Pipeline
 Currently running scenarios
– CLM BVT (also launched by RelEng)
– CLM BVT NOPLE
– CLM Licensing
– JTS BVT
– QM BVT
– RM BVT
– SVT GOV
– SVT MTM
– SVT MTM NOPLE
 New
– DCC BVT
– SVT NONPLE RDNG
4
UCD D1 Environment
• Four machines
1. CLM all-in-one
2. LQE for reporting
3. DB2
4. IHS
1
4
3
2
https://w3-connections.ibm.com/wikis/home?lang=en#!/wiki/Rational%20SVT%20Automation/page/UCD%20%20CLM%20Departmental%20Linux%20DB2%20-%20Stream%20600
Scenario Build Activities
• These are the steps used to
Create D1
Env
Create
DataGen Env
Deploy to D1
Env
Deploy to
DataGen Env
build/run the scenarios in the
pipeline
• When deploying a manual UCD
D1 environment, it is a two step
process:
1. Create a new UCD environment
…wait…
2. Deploy D1 to that environment
• Same idea for pipeline,
…wait… between Create and
Deploy
Publish
Properties
• That wait is waiting for IWD to
provision the environments
• Timeouts
• 150 minutes for D1
• 60 minutes for DataGen
• Personal builds do not time out
6
Run Tests
ALM.60.all Build
ALM.60.all Build
• The main build watched is the
ALM.60.all build
• Current status of the build is noted in
the Tracking Build Item, e.g., Track
CLM 6.0 Sprint 9 (January 19 , 2015
- February 13, 2015) (343344)
• The ALM.60.all builds are usually
run twice a day, 1130 EST and 1930
EST.
• The CSPF scenarios are run as
children of this build
• The scenarios depend on SSE
(which depends on CLM),
CDSETUP and CSPF builds. If one
of these fails, the scenarios will not
run
8
Sample ALM.60.all build
• Let’s look at sample
ALM.60.all build ALM.60.all
ALM60-20150210-2056
• Look at the build-log and you
can see the status of all of the
child builds, including the
scenarios
• Easiest way to get more detail
is to use External Links tab
and follow to the Build
Tracking Item
9
Tracking Build
• These tracking items cover
all of the scenarios being
run, including the CSPF
scenarios
• Approved = Pass
Rejected = Fail
• See links to the specific
scenario child builds in the
comments
• Follow a link to see
information about a specific
child
10
Overview Tab
• Overview tab
1. Pass/fail overall
2. List of work items already
filed against all scenarios
3. Link to create a new
‘problem with build’ work
item against scenario
11
1
2
3
Activities Tab
• Activities Tab
1. Shows activities that make
up this build
2. And how long they took
1
2
12
Tests Tab
• Tests Tab
1. Shows exactly what step
and test have failed
2. Shows how long tests take
as well
3. Stack trace details; linked to
code in your workspace
1
2
3
13
Logs Tab
• Logs Tab
1. Log file for every activity,
plus master build log
1
14
External Links Tab
1
• External Links Tab
1. Back to build tracking item
2. RQM result, where more
details, including
screenshots, are found
3. UCD processes
4. UCD environments
2
3
4
15
Pass Along Properties Tab
• Pass Along Properties Tab
1. URL of deployed server
•
Actually, IHS server
2. URL of DataGen server
• DataGen =
rqmAdapterFQDN =
clmSeleniumAdapter
16
2
1
Problems with Scenarios
Tags
 Four tags
– Implies four types of problem
Issue
Tag
Test defect (test/framework has a
defect)
cdp_test_issue
Product defect
cdp_product_issue
Test adoption defect (test needs to
adopt product UI changes)
cdp_adopt_test
Infrastructure defect
cdp_infra_issue
18
Product Defect
2
1
1. Overview: see failure
2. Tests: see exact test
and stack trace
3. External Links: follow
link to RQM result
4. See screenshot
3
4
run.cspf.tests CPSF600-CLM_BVT_NO_PLE-20150211-0345
How to Handle Product Defects
 Find/file defect against the product
 Find/file ‘problem with build’ work item against build
– Link the two work items
– @ffusier, @craneml
– Tag with “cdp_product_issue”
– Tag with “clm_blocker”
 If it’s a quick fix for the product, okay to let scenario fail
 But if fix is slow, best to workaround the failure in the scenario, e.g., not run that
test or automate some manual workaround to get around the issue
– Slow > 1 week
20
Web UI Change
or Adoption
2
1
1. Overview: see failure
2. Tests: see exact test
and stack trace
3
3. External Links: follow
link to RQM Result
4. See screenshot
4
run.cspf.tests CPSF600-CLM_BVT-20150114-0227
How to Handle UI or Adoption Defects
 Find/file “problem with build” work item against build
– @ffusier, @craneml
– Tag with “cdp_adopt_test”
– Make sure link to RQM result is included (and/or attach screenshots and logs)
 Scenario owners: fix  Some fixes can be quick (e.g., name of GC product);
some are slow (e.g., introduction of GC in the first place). Do as much as you can
and call of @ffusier and @craneml if required
22
UCD
Infrastructure
Problem
2
1
1. Overview: see failure
2. Logs: see error in
d1Deploy.log
3. In log, see that ‘application
process failed’, i.e., the UCD
process failed to deploy D1
4. External Links: follow the link
to the UCD process
3
4
5. Drill down to find the step that
failed
5
6. Check the output log for clues
6
run.cspf.tests CPSF600-JTS_BVT-20150126-0545
How to Handle UCD Infrastructure Problems
 Two kinds of problems with UCD
1. Something wrong with application or script in UCD itself, e.g.,
 Change in product affecting how product is installed/configured in WAS
and UCD process has not caught up yet
2. Something wrong with something UCD depends on, e.g.,
 Site it down (broken or maintenance) e.g., jazz.net, java, other install sites
 IWD (separate topic)
 Solutions
1. Find/file “problem with build” work item against build
 @ffusier, @craneml
 Tag with “cdp_infra_issue”
 Main IWD application contact is Ye (Jennifer) Liu
2. Site down, maintenance window: nothing Jennifer can do to fix
 Wait and try again later
* Note: Michelle or Frederic are monitoring the daily CT pipeline for infrastructure
issues; contact them first before filing a new defect.
24
IWD
Infrastructure
Problem
1
2
(Special case)
1. Overview: see failure
2. Logs: see failures in create*Env
logs
3. In log, see “Timed out waiting”
3
error
1. D1: 9000 sec = 150 min
2. DataGen: 3600 sec = 60 min
4. [email protected] can log
into IWD gss5 to see what’s
wrong with the environment
* Timeouts only affect pipeline builds, not personal builds.
run.cspf.tests CPSF600-SVT_GOV-20150210-0528
How to Handle IWD Infrastructure Problem
 Not much to do about IWD issues except
– Comment in TBI about infrastructure issue
– Wait and try again
 Let @ffusier and @craneml know of issues
– FYI, we do:
 IWD defects
 Filed against ITE at https://csnext.ibm.com:8001/jazz/web/projects/ITE
 Current problem:
https://csnext.ibm.com:8001/jazz/web/projects/ITE#action=com.ibm.team
.workitem.viewWorkItem&id=46047 - GSS5 requires to be rebooted
every day to maintain the pipeline runs
* Note: Michelle or Frederic are monitoring the daily CT pipeline for infrastructure
issues; contact them first before filing a new defect.
26
Test or Framework Problem
 Similar to product defect or adoption issue – failure shows up as a test failure
 But not a product defect or change
– Instead, something’s wrong with the scenario or framework
– e.g., NPE or something unexpected
 Could be triggered by product change, but not the product’s fault
– Instead, scenario/framework didn’t handle every possibility
27
How to Handle Test or Framework Problem
 These are our most important problems – we cannot allow these!
 Find/file “problem with build” work item against build
– @ffusier, @craneml
– Tag with “cdp_test_issue”
– Make sure link to RQM result is included (and/or attach screenshots and logs)
 These problems must be fixed right away. Scenario owners should fix any
scenario problem, but call on @ffusier or @craneml if required. Feel free to fix
framework issue if you can find the problem.
 Should be fixed the same day.
28
Miscellaneous
Miscellaneous
 D1 and DataGen environments do not last very long.
– DataGen environments can be deleted almost as soon as they finish
– D1 environments for a successful run can be deleted quickly
– D1 environments for a failed run can last a day or so
 Much information is preserved in the build logs and the UCD application process
itself
 But if environment is needed (e.g., IWD information, or contents of DataGen
machine), need to work fast
30
Accessing Servers
 Possible to use VNC and ssh to connect to CLM, IHS and DataGen servers
– Only do this when you are sure the environments still exist!
 All properties show one URL for the deployed server: the IHS URL
 VNC Ports, Usernames and Passwords
31
Server
Protocol
VNC Port
Username
Password
CLM/IHS
vnc/ssh
1
clmadmin
ec11ipse
DataGen
vnc/ssh
3
root
aut0mat10n
Getting CLM URL from IHS
 Sometimes you need the CLM URL, e.g., to get logs
 Easiest way is to ssh into the IHS machine and read the URL from its
configuration
ssh -l clma fit-vmXX-YYY.rtp.raleigh.ibm.com
Answer yes to prompt and enter clmadmin’s password; can also use root to
log in
cd /opt/IBM/Plugins/config/webservermerge
grep rtp.raleigh plugin-cfg.xml
This searches the config for what other URLs are being used. As of 6.0, there
will be two pairs of URLs returned. One URL will be for the CLM server, the
other for the LQE machine
 Log into <url>:9443/jts/admin to double-check that you have the CLM URL
32
Orchestrator Properties File
& Build Definitions
 Build definition
33
 Properties file
Build definitions to test changes
 Test major or risky changes before requesting review
– Analyze changes impacts
34
Build definitions to test changes
– Impacts can be serious
35
References
References
 UCD D1 Environment Instructions: UCD - CLM Departmental Linux DB2 - Stream
600 - Rational SVT Automation Wiki
 Working With CSPF Page: https://jazz.net/wiki/bin/view/Main/CspfWorkingWith
 Working With the UCD Pipeline:
https://jazz.net/wiki/bin/view/Main/CspfWorkingwithUcd
37
Questions???