Useful Tools for Testing
Aled Smith
Useful Tools for Testing
•
This presentation will be looking at two useful tools for
performing tests:
1. Virtual Machine Creator
•
https://svn.cern.ch/reps/en-ice-svn/trunk/utilities/SIC/virtual-environment-setup/vm_creator/
2. Unit-Test-Harness
•
•
https://svn.cern.ch/reps/en-ice-svn/trunk/utilities/SIC/virtual-environment-setup/Unit_Test_Harness
A use case for each of these tools will also be
demonstrated.
VM Creator: Overview
What is it?
• The Virtual Machine Creator is a Python based script that fully automates the process of creating and
configuring OpenStack virtual machines.
How does it work?
• The VM creator uses three main components:
• Fabric - A Python based SSH wrapper.
• JenkinsAPI - A Python based API for Jenkins.
• Nova - A Python based API to OpenStack.
SSH
Jenkins
OpenStack
Fabric
JenkinsAPI
Nova
VM Creator
What can it do?
• Fabric provides the Virtual Machine Creator with a Python based SSH interface to a virtual machine so that
any SSH tasks like installing and configuring software can be executed from a Python script. It also provides
built-in functions to perform tasks like sudo commands and file transfers.
• JenkinsAPI allows the Virtual Machine Creator to interact with Jenkins and performs tasks like registering
new slave nodes and executing Jenkins jobs.
• Nova allows the Virtual Machine Creator to interact with OpenStack and perform tasks like creating, deleting
and modifying virtual machines.
VM Creator: Use Case
• This use case will create a CERN Scientific Linux 6 Jenkins slave virtual machine running with WinCC-OA
installed for the purposes of running test projects using Jenkins.
Generate new
key-pair
Create virtual
machine
Add ‘jenkinst’
service account
Expand main
partition
Install
X-Window
Install
WinCC-OA
Install
PVSSBootstrapper
Install
Subversion
Install
Java
Connect VM to
Jenkins
• This basic flow chart shows the main tasks that the
Virtual Machine Creator must complete for this
particular use case.
• However, the Virtual Machine Creator is not restricted
to any specific use case and the user can pick and
choose the functions they’d like to use and also add
their own.
VM Creator: Use Case
• This use case will create a CERN Scientific Linux 6 Jenkins slave virtual machine running with WinCC-OA
installed for the purposes of running test projects using Jenkins.
Generate new
key-pair
Create virtual
machine
Add ‘pvss’
local account
Expand main
partition
Install
X-Window
Install
WinCC-OA
Install
PVSSBootstrapper
Install
Subversion
Install
Samba
Install
Gmoon
• This basic flow chart shows the main tasks that the
Virtual Machine Creator must complete for this
particular use case.
• However, the Virtual Machine Creator is not restricted
to any specific use case and the user can pick and
choose the functions they’d like to use and also add
their own.
• For example, in this use case, a CERN Scientific
Linux 6 virtual machine is created with WinCC-OA
installed that is not a Jenkins slave, has a local
account called ‘pvss’ and has Samba and Gmoon
installed.
VM Creator: Use Case
• The WinCC-OA Jenkins slave creator use case described previously has been implemented and incorporated
into a Jenkins job.
Jenkins Job: https://icejenkins.cern.ch/job/CREATE_JENKINS_SLAVE/
Unit-Test-Harness: Overview
What is it?
• The Unit-Test-Harness is a tool that automates the process of unit testing framework components within a
WinCC-OA project.
What can it do?
• It can be invoked from command line using Python or from a GUI panel within the project.
• It can automatically create a new WinCC-OA project and install the required components
• It can automatically run the unit tests and publish the results in Junit XML format.
How does it work?
• The user creates a ctl script containing the unit tests.
• If executed from the GUI panel, the Unit-Test-Harness finds any unit tests within the project and presents
them to the user. The user can then select the tests that they want executing.
• If executed from the command line, the user needs to describe the components that need to be
installed/tested, their sources and the location of the unit test ctl script in an XML file.
• The Unit-Test-Harness can then parse the XML and either create a new test project or use an existing one.
• It then performs the desired units tests and publishes the results in Junit XML format.
• Finally, the Unit-Test-Harness can also delete the project and clean-up any downloaded files.
Unit-Test-Harness: Overview
CTRL
Python
Start via GUI panel.
Start via command line.
Find available unit tests
within project.
Use existing project?
User selects unit tests.
Yes
Create project, install
desired components
and start project.
Check project is
registered and start it.
Run the unit tests.
Start Unit-Test-Harness
within project.
Publish the results of
the unit tests.
Copy results from
project and stop project
Clean-up project?
No
End via GUI panel.
No
End via command line.
Yes
Delete project
Unit-Test-Harness: Use Case
• This use case is used to test whether an RDA3 client correctly receives different
formats of data while the RDA3 server continuously iterates the data it publishes.
• It checks that the client is receiving the data correctly by using a listener that
checks each iteration of data that is received and records any missed or incorrect
iterations.
• This use case requires some logic to be run outside of the Unit-Test-Harness
such as starting the RDA3 Server.
• The Unit-Test-Harness is therefore split into two parts:
1. ‘setup_project.py’ - which handles the creation of the projects and
installation of the components.
2. ‘run_tests.py’ - which executes the units tests and retrieves the results.
• This use case was also incorporated into a Jenkins job and is shown in the
following flow chart:
Unit-Test-Harness: Use Case
CTRL
Python
Start via GUI panel.
Start via command line.
Find available unit tests
within project.
Use existing project?
User selects unit tests.
Jenkins
setup_project.py
No
Start Jenkins job
Initialise listener values
Yes
Create project, install
desired components
and start project.
Check project is
registered and start it.
Start RDA3 client
Start RDA3 server
Run the unit tests.
Start Unit-Test-Harness
within project.
run_tests.py
Publish the results of
the unit tests.
Copy results from
project and stop project
End via GUI panel.
Clean-up project?
Start listener and leave
running for fixed time
Stop listener
Yes
Delete project
Publish test results
No
End via command line.
End of Jenkins job
Unit-Test-Harness: Use Case
•
•
•
In this screenshot, you can see the
XML code within the XML
configuration file that is passed to the
Unit-Test-Harness through the
command line.
The XML file describes the
components that need to be installed,
their sources and also defines the
location of the test suite file.
Specifying just the test suite tells the
Unit-Test-Harness to execute all of
the unit test cases but these can also
be specified individually using the
<testCase> tag.
Unit-Test-Harness: Use Case
•
•
•
•
In this screenshot, you can see a snapshot
of the code within the test suite file.
In this ctl script, the user defines the unit
test cases and their logic.
Each unit test should have an assert
routine.
The Unit-Test-Harness supports the
following assert routines:
•
•
•
•
•
•
assertError()
assertEmpty()
assertEqual()
assert()
assertTrue()
assertFalse()
Jenkins Job: https://icejenkins.cern.ch/job/RDA3_CLIENT_MONITOR/
Any questions?
Useful Links:
Virtual Machine Creator:
•
https://svn.cern.ch/reps/en-ice-svn/trunk/utilities/SIC/virtual-environment-setup/vm_creator/
Unit-Test-Harness:
•
https://svn.cern.ch/reps/en-ice-svn/trunk/utilities/SIC/virtual-environment-setup/Unit_Test_Harness
Fabric:
• http://docs.fabfile.org/en/1.10/
JenkinsAPI:
• http://jenkinsapi.readthedocs.org/en/latest/
Nova:
• http://docs.openstack.org/developer/python-novaclient/