Automation Test Suite
Neeraj Jain
Cavisson System Inc
Agenda






Features
Directory Structure
Commands
Reports
Demo
Q&A
Features




Run several tests (test set)
Tests to run may be selected by user
On Failure of a test

Test Set progress may continue

Abort the test set

Jump to certain test in test harness*

Skip certain tests*
For each test set run

Test summary results are saved

Detail logs for each test are saved
Features (Continued)


Reporting

Results summary as text, html, XL and pdf *

Results may be emailed to subscriber on completion *

Results may be emailed to subscriber on each failure *

Compare summary results against past test-set run(s)
Test Run Management

Show available test runs

Manage test run data
Directory Structure
Automation is installed in /home/netstorm/work directory which is called home directory.
BIN DIRECTORY

All programs are kept in bin directory under home directory of netstorm. All programs
under this directory are replaced when upgrade is done.
TEST CASE DIRECTORY

All test cases are kept in testcases directory under home directory of netstorm. There
is one directory per test case. This directory is NOT replaced when upgrade is done.

All test cases have four files which are in the test case directory:

testcase.conf: This file have two keywords





SCENARIO_NAME- To give scenario name. Scenario name should be without .conf extension
DESCRIPTION – To give description of test case.
pre_test_setup (Optional) - This script for making any pre test setup.
post_test_setup (Optional) - This script for making any post test setup.
check_status (Optional) - This script is for checking the status.
Directory Structure (Continued)
TEST SUITE DIRECTORY

All testsuites configuration files are kept in testsuites directory under home directory
of netstorm. Name of testsuite configuration file will be with conf extension. This
configuration file will have following keywords:



TEST_CASE_NAME: This keyword is for given the test case name, which we want to run. This keyword
also have second field Continue or Abort. This is for to make decision for continuing or aborting test
cases on any failure (By default it is Continue). This keyword can multiple times in configuration file.
EMAIL_ADDRESS: This keyword is for giving the email address.
REPORT_FORMAT: This keyword is for giving format of report. Presently we are supporting only text
format.
This directory is NOT replaced when upgrade is done.
LOGS DIRECTORY

All logs are kept in tsr directory under logs directory in home directory of netstorm. In
this directory there will be directory named on test suite run number. In this directory
there will be summary.report and progress.report file. tsr directory is NOT replaced
when upgrade is done.

How To Run Test Suite?





login as netstorm.
Go into work directory
Run the ts_run command as following
bin/ts_run –n testsuites/<test suite name>
For Example:



bin/ts_run –n testsuites/Sample_test_suite1
On running above command it will show the test suite run number (TSR).
After completion Test case there will be directory as TSR under tsr directory in logs.
Commands
ts_run

This command is used to run test suite. Command runs in following manner:

bin/ts_run –n testsuites/<test suite name>
ts_show_test_suite

This command is used to get the information about the test suite. Command runs in
following manner


bin/ts_show_test_suite –n <test suite name>
Sample output:
Test Suite Name: Sample_test_suite
Test Case Name
Description
-----------------------Sample_test_case1
Test Sample
Sample_test_case2
Test Sample2
Commands (Continued)
ts_show_logs

This command is used to get the logs of a specfic or all tsr. Command runs in
following manner


bin/ts_show_logs –n <tsr | ALL>
Sample output:
Test Suite Name: Sample_test_suite
Test Suite Run Number: 1086
Start Date/Time: 09/14/07 13:59:57
Test Case Name Test-Run# Start Date/Time End Date/Time Status
-------------- --------- --------------- ------------- -----Sample_test_case1 7867
09/14/07 13:59:57 09/14/07 14:00:19 Pass
Sample_test_case2 7868
09/14/07 14:00:19 09/14/07 14:00:41 Pass
Commands (Continued)
ts_show_test_cases

This command for showing information about the test cases which are in testcase
directory. Command runs in following manner


bin/ts_show_test_case
Sample output:
Test Case Name Pre Test Script Post Test Script Check Status Scenario
-------------- --------------- ---------------- ------------ -------Sample_test_case1 Yes
Yes
Yes
test_casesuits
Sample_test_case2 Yes
Yes
Yes
test_casesuits
Commands (Continued)
ts_compare_tsr

This command is for comparing results of two or more test suite runs. Command runs
in following manner


bin/ ts_compare_tsr <tsr1> <tsr2> …….
Sample output:
./ts_compare_tsr 2285 2286 2287
Test Suite Compare Report
Test Case Name
2285
tx_am_tiny_1req
Pass
2286
Pass
2287
Pass
Reports
There are two reports summary.report and progress.report in tsr log file.
Sample output of summary.report:
Test Suite Name: Sample_test_suite
Test Suite Run Number: 1086
Start Date/Time: 09/14/07 13:59:57
Test Case Name Test-Run# Start Date/Time End Date/Time Status
-------------- --------- --------------- ------------- -----Sample_test_case1 7867
09/14/07 13:59:57 09/14/07 14:00:19 Pass
Sample_test_case2 7868
09/14/07 14:00:19 09/14/07 14:00:41 Pass
Reports (Continued)
Sample output of progress.report:
Starting test suite Sample_test_suite, TSR Number = 1086
Starting execution of test case - Sample_test_case1
Starting pre_test_setup script
spawn su -c /home/netstorm/work/bin/nsu_assign_ip '192.168.12.0/24|192.168.12.5:7|-|-|eth0'^M
Password: ^M
IP assignment completed!^M
spawn su -c /home/netstorm/work/bin/nsu_assign_ip -s '192.168.12.0/24|192.168.12.15:1|-|-|eth0'^M
Password: ^M
IP assignment completed!^M
spawn echo Calling pre_dut_setup^M
Starting scenario
Checking status
TRIED = 1 SUCC = 1 FAIL = 0
Test 1 Passed...
Starting post_test_setup script
Demo
Live Demo
Q&A
????