Automated Testing
Nathan Weiss
April 23, 2007
Overview





History of Testing
Advantages to Automated Testing
Types of Automated Testing
Automated Testing Tools
Disadvantages to Automated Testing
History of Testing
History of Testing

Four main phases in testing
methodology:




Debugging (1950’s)
Exhaustive Testing/Prove it Works
(1960’s)
Prove it Does Not Work (1970’s)
Defect Prevention & Testing Process
(1980’s)
Debugging


No distinction between debugging and
testing.
Consisted of making code compile and
changing code so that it would not
crash.
Exhaustive Testing/
Prove it Works

Exhaustive testing in terms of code
paths and given input.



Found to be impossible
Write tests to prove the program
works.
Testing defined as “what is done to
demonstrate correctness of a
program.”
Prove it Does Not Work


Testing not for proving correctness, but
for finding defects.
Started testing to see if a program did
what it was not suppose to do, not just
test to prove that it does what it was
suppose to do.
Defect Prevention &
Testing Process



“Testing” traditionally referred to what
was done to the system after working
code was produced.
Testing moved towards an activity done
during a phase, not a phase by itself.
Studies show that 50% of bugs are
created in requirements and design
stages.
Advantages to Automated Testing





Test scenarios impossible/unfeasible
using manual testing.
Lowers testing time and costs in an
iterative environment.
Remove the human error factor.
Introduces stability in regression tests.
Increase in testers’ morale.
Types of Automated Testing

Split into two categories


Black box tests
White box tests


PC applications
Embedded Systems
Black Box Automated Testing




GUI testing
Always been prime candidates for
manual testing.
Need to automate has created many
commercial tools.
Two ways to create the tests:


Scripts
Recordings
Black Box Tests - Scripts



Less of a testing tool/environment and
more of a testing library.
Flexible.
Easy to run batch scripts to run
multiple tests.
Black Box Tests - Recordings




Full testing tool/environment.
Tester performs a test once while tools
is recording his actions.
Tester can add ending conditions to
test the success of a test.
Very useful for testing GUIs of stand
alone applications.
White Box Automated Testing



Module testing.
More APIs then environments.
Embedded systems introduce more
complexity to white box testing.
White Box Tests – PC Apps



Tests usually written in the same
language as the system being tested.
Tools usually come in the form of an
API containing asserts and other
testing utility methods.
Tests can be performed on any level of
abstraction required by the project.
White Box Testing –
Embedded Systems



Test code usually not stored/run on the
same hardware as the system code.
Do not have direct access to system
code or memory.
Requires an interface with an
embedded system debugger.
Automated Testing Tools




xUnit
SilkTest
Selenium
Custom
xUnit



Compilation of popular code driven
testing frameworks.
Frameworks based on the design by
Kent Beck.
Originally implemented for SmallTalk
as SUnit.
xUnit

Partial List of Frameworks





NUnit (.net)
JUnit (java)
CUnit ( C )
CPPUnit (C++)
SUnit (SmallTalk)
xUnit

xUnit test execution



Setup
Test body
Teardown
xUnit
SilkTest


Created by Borland.
Works with:





Web Browsers(IE and Firefox)
Java GUIs
.Net GUIs
Win32 GUIs
Price : 4-6k
Selenium



Testing tool for web applications.
Comes in API and tool form.
Supports:



Windows: IE, Firefox, Seamonkey, Opera
Mac OS X: Safari, Firefox, Camino,
Seamonkey
Linux: Firefox, Konqueror
Selenium


Can create tests using the Selenium
library in many languages (Java, .NET,
Perl, Python and Ruby)
Or you can create tests using their
Selenium IDE which is implemented as
a Firefox extension.

Once created in Firefox, the tests can be
run in any of the suported browsers.
Selenium
Custom



Write your own custom testing
environment or API.
Used when you have very specific
testing needs.
Often used for embedded systems.
Disadvantages to Automated Testing

Initial cost.

Who tests the testers?
References





Borland. (2007). Borland SilkTest: An Automated Regression
and Functional Software Testing Tool. Retrieved April 2, 2007,
from
http://www.borland.com/us/products/silk/silktest/index.html
[2] Dustin, E., Rashka, J., Paul, J. (1999). Automated Software
Testing: Introduction, Management, and Performance.
Addison-Wesley.
[3] Lewis, E., W., (2005). Software Testing and Continuous
Quality Improvement. CRC Press LLC.
[4] NUnit. (2006). Nunit. Retrieved April 2, 2007, from
http://www.nunit.org
[5] OpenQA. (2006). OpenQA: Selenium. Retrieved April 2,
2007, from http://www.openqa.org/selenium/index.html