Tool Look
qq
This article is provided courtesy of STQE,
the software testing and quality engineering magazine.
QUICK LOOK
■ Robust as a capture/replay tool
■ Idiosyncratic when used
as a scripting tool
E
A look at QARun, a GUI
test automation tool
by Noel Nyman
arly in 1998, while working in the Microsoft WindowsNT Group, I evaluated several
automation tools for the Applications Test
team. That team uses retail workstation applications such
as office suites, graphics programs, and utilities to test
the version of WindowsNT under development. I used the
tools to create tests that ran across a variety of hardware
and system configurations. I ignored features of the tools
that weren’t useful to the Applications Test team. My hope
is that you’ll find this short review valuable, even with
those limitations. The opinions expressed here are mine
alone. They are neither an endorsement nor a critique of
the products by Microsoft. New versions of the tools,
released after I evaluated the products, may have different
features than those I describe.*
less of its location. Many tools use “aliases” to name
objects. The alias names, assigned by the tool or the user,
may be independent of changes in the user interface
labels for the objects.
A standard Windows “combo box” control is an edit
box combined with an arrow button and a multi-line list
box. One item from the list box appears in the edit box.
When you click the arrow, the list box drops to show you
the other items available. Most automation tools can identify standard combo box controls during capture mode.
Corel Ventura™ (Corel, http://www.corel.com) uses a
custom “combo box” control to select text font color.
Product Summary
QARun and QADirector are part of QACenter
(Compuware, http://www.compuware.com). Together,
QARun and QADirector organize test plans, record
Graphical User Interface (GUI) automation scripts, and
provide an editor for modifying recorded scripts. QARun
has special commands to test applications created with
PowerBuilder (special commands for SAP, Oracle, UNIFACE, Visual Basic, and others were not part of my evaluation) and to communicate with external databases using
ODBC and SQL. All QARun script information, resources,
and logs are held in a project database. QACenter also
includes tracking, and load and stress testing products,
which were not part of my evaluation. I tested QARun version 4.4.
Capture/Playback
Most GUI automation tools have capture/playback—the
ability to record a tester’s mouse movements and keystrokes, and play them back later. In a mixed configuration environment the playback mechanism must adapt
well to changes in control size and location, screen resolution, and color depth to be useful. So, GUI automation
tools try to identify objects in the application instead of
just recording the locations of mouse clicks and drags. On
playback, user actions are directed at the object, regard* Version 4.5 of QARun was released in the spring of 1998.
Ventura’s “edit box” contains a colored bar instead of text,
and the drop down “list” box is a grid of colors and a command button that will show you even more colors when
it’s clicked. Many automation tools are confused by this
object. They may see the control, but can’t identify it
because there’s no text in the “edit box” part.
QARun sees the “Others…” text string and guesses that
the entire custom control is a PushButton. QARun uses the
text string to identify this control, but the label “Font color”
makes more sense to the tester. Dialogs in QARun allow the
tester to create the alias “Font color” for this control, and tell
QARun it’s something other than a PushButton. Those
changes are registered in the same QARun database that
holds everything else about the test project.
Other GUI automation tools are much less robust about
handling custom controls. This QARun feature could be a
deciding factor for some test teams choosing an automation
tool.
54
www.stqemagazine.com
Software Testing & Quality Engineering
Jan./Feb. 1999
This article is provided courtesy of STQE, the software testing and quality engineering magazine.
BitmapSelect
Another strong feature of QARun is its ability to find bitmap
images. The tester uses the QARun editor to select the area
on the screen and assigns an identifying name to it. QARun
translates clicks on this area into BitmapSelect commands.
On playback, QARun searches the entire screen for the
matching bitmap and performs the specified click action. I
tested BitmapSelect by defining several bitmaps on application toolbar buttons. BitmapSelect worked quite well over a
surprisingly large range of screen resolutions, from
1152x864 pixels down to 640x480. It did not survive color
depth changes, except at the original screen resolution.
Rolling Your Own
QARun scripts can be edited only with the QARun editor.
You can also export/import them as .txt files. That’s a hassle, but may be worth the trouble. The QARun editor has
no undelete feature! Without undelete, inadvertent mouse
movements or keyboard clicks may lose quite a bit of
work…irretrievably.
not be able to test to make sure the Close All menu item
was available before choosing it, for example.
The Check command compares the current state of
the specified window with the state the tester recorded
when the test was created.
Variables in QARun
QARun supports an odd conglomeration of public, private, and local variables. Variables don’t have to be
declared before use, so a typo may create a new variable
when you thought you were assigning a value to an old
one. Variables are not strongly typed. For example, you
can assign a string to a variable, then later assign an integer value to the same variable. If you do that by accident,
QARun won’t tell you.
There’s no typing of variables returned by functions.
As a result, function calls can’t be used as parameters
because QARun can’t determine at run time what type the
parameter will be. For example, this line in a script fails:
Print “Output from StringFunction is: ” + StringFunction
QARun Script Code
Here is a function I wrote to open a file in WinEdit
(Wilson WindowWare, http://www.windowware.com):
Even though StringFunction returns a string, we get this
error message:
Script2(3): A numeric/string expression was expected
Win32 API Calls From QARun
I rely on calling Win32 API functions from my automated
tests. Regardless of a tool’s sophistication, there are
always features you need that only the OS can provide.
QARun can call some Win32 API functions, but its abilities
are limited. QARun defines special variable types specifically for Win32 API prototyping, such as “int,” “uint,” and
“str.” Missing are variables like “dword,” hexadecimal conversions, and structures. Here’s the QARun code to
retrieve the NT build number using GetVersion().
I created the LogOut function to write strings to my
own log file. QARun automatically logs errors it finds to
the project database for later reporting and tracking.
Key to all QARun actions is Attaching to a window,
usually identified by its caption and the class QARun
assigns to it. QARun maintains an object database where
each window is named and its parameters are listed. The
tester can tell QARun to ignore specific parameters when
finding the window. I could not get wild cards to work for
captions, and that limited me when Attaching to error
dialogs that might refer to different file names at different
times. This reflects the record/playback orientation of the
tool, where the path and file names are “always” the same
and no wild cards are needed.
Variables can be used for Attach window names, as
shown in the last Attach command.
MenuSelect is the only command for working with
menus. There are no commands to test for existence of a
menu, menu count, enabled, grayed, or selected. I would
Declare DLLFunc “uint GetVersion() kernel32” GetVersion
Function Main
Print Int(GetVersion()/65536)
End Function
In QARun, GetVersion() returns “90505220” in decimal format (NT 4.0, build 1381). I’d like to use the Win32
HIWORD macro to extract the build number. HIWORD is
not available in QARun. Dividing by the max value of the
LOWORD and using Int() to strip the fractional remainder
sort of accomplishes the same thing. Extracting parts of
other Win32 API function returns is not so easy, and many
calls are impossible.
QADirector
QADirector is the mechanism intended to run QARun
scripts. The tester can run any combination of the scripts
or the entire project. If desired, several projects can be
created with different collections of scripts for sanity
tests, printing tests, etc. When scripts fail, the failure and
script name are reported to the tester. QARun must be
started to view the log to see the exact failure.
55
Jan./Feb. 1999
Software Testing & Quality Engineering
www.stqemagazine.com
This article is provided courtesy of STQE, the software testing and quality engineering magazine.
Final Analysis
QACenter provides an integrated solution that will help you
test GUI applications and track the bugs you find. As with
most tool suites, you get the best results if you use all the
features. If you don’t need some parts of QACenter, the
integration is less important to you, as it was for us. Then
the strengths and weaknesses of the individual tools, like
QARun, are more significant. STQE
Noel Nyman has worked in software product development and testing for over 20 years on an eclectic project mix including embedded controllers, shrinkwrap
applications, and operating systems. “He tests, therefore he is.”
STQE magazine is produced by STQE Publishing, a division of
Software Quality Engineering.
56
www.stqemagazine.com
Software Testing & Quality Engineering
Jan./Feb. 1999