2010 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery
Scenario-Based Approach
for Blackbox Load Testing of Online Game Servers
Chang-Sik Cho, Dong-Chun Lee, Kang-Min Sohn,
Chang-Joon Park
Ji-Hoon Kang
Dept. of Computer Science and Engineering
Chungnam National University
Deajon, Korea
[email protected]
Smart Game Team
ETRI
Deajon, Korea
{cscho, bluepine, sogarian, chjpark}@etri.re.kr
online games. To meet such requirements for game servers,
many game testers are involved in testing, and a large
number of game players actually participate in beta testing
before deploying a game. In order to reduce such testing
time, the use of testing automation is recommended [2-5].
Traditional server load testing technologies such as
LoadRunner [6] have been applied on online game testing,
but various game actions cannot be easily simulated by
simply and partly modifying the game packets of actual
players. Thus, The Sims Online (TSO) automation [3], [7]
and the Virtual Environment Network User Simulator
(VENUS) system [8], [9] used a virtual client engine with a
subset of actual game clients. Both the TSO automation and
the VENUS system have saved many person-hours, leading
to better game quality, and have been more efficient than
manual testing. However, they both have some weak points.
First, both use a subset of the main game client for testing
the client code. In other words, blackbox testing is not
supported. Though using an existing game client has
minimized the need to create new code, the test client code
should be rewritten for new game testing. Their approaches
can be applied to game developers only and are not
adequate for game publishers because the developed game
client code must be supplied and rewritten for game testing.
These days, testers in game publishing side, need to test the
games without the game source code.
Second, the TSO automation and the VENUS system do
not support scenario-based testing. Their testing has been
achieved using the unit of the simple command script.
Though both have provided more flexible testing actions
compared with traditional server load testing technology, it
is difficult for them to test interactive actions such as multiuser collaborative play.
Complex and various test scenarios such as party play
and waypoint movement still have not been supported.
However, game testers want to simulate complex scenarios
and mimic real testing environments instead of doing
simple load testing.
In this paper, we propose blackbox testing and scenario-
Abstract— Simply having a large beta test cannot
consistently provide stability and performance to game
servers, which are major issues in online game
development. Therefore, test automations have been used in
order to reduce the testing time of online games by
simulating highly repetitive tasks and emulating server
loads. However, in previous approaches, blackbox testing
and scenario-based testing are not supported because they
use prerecorded packets of real players as templates, or
reuse a subset of the main game client for the test client. In
this paper, we propose blackbox testing and scenario-based
testing of online games as well as simple load testing.
Instead of rewriting the virtual client dummy code, only the
game description language and virtual game map are redefined when a new game is to be tested. In addition, an
actual testing environment can be mimicked more closely,
because complex and various scenarios such as attack,
party play, and waypoint movement can be tested through
combining actions. We have applied our tools on several
online games to verify the effectiveness of our method.
Online Game Testing; Load Test; Massive Virtual Users;
Blackbox Testing; Scenario-based Testing; Game Description
Language; Virtual Game Map (keywords)
I.
INTRODUCTION
A massively multiplayer online game (MMOG) is a
multiplayer video game that can be played via a game
server over the Internet with players from around the world.
MMOGs can enable players to cooperate and compete with
each other on a large scale, and include a variety of
gameplay types, representing many video game genres[1].
MMOGs are deployed using a client-server system
architecture and can actually be run on multiple separate
servers. The servers must be able to handle and verify a
large number of connections. Thus, the stability and
performance of game servers have become major issues in
978-0-7695-4235-5/10 $26.00 © 2010 IEEE
DOI 10.1109/CyberC.2010.54
259
based testing of online games, as well as simple load testing.
The game description language and virtual game map are
used to achieve these goals. The massive virtual clients
automatically generate packet loads to test the stability of
game servers as in the existing approaches. However, by
describing game logics and test scenarios using the game
description language and virtual game map, the test client
code itself does not need to be rewritten for new game
testing. The virtual clients automatically generate packet
loads according to the defined game description language,
and complex scenarios such as attack, party play, and
waypoint movement can be tested by combining actions.
The rest of this paper is organized as follows. Section II
shows examples of game testing scenarios and actual game
packets to help explain the conceptual model of our method.
Section III explains the syntax of the game description
language and section IV shows the implementation results.
Section V shows the comparison with previous approaches.
Finally, section VI concludes our paper with a description
of future work.
II.
game. The TSO team used a subset of the main game client
to create a test client in which the GUIs are mimicked via a
script-driven control system. LoadRunner has been used to
control up to thousands of simulated users against a given
candidate server cluster. A bridge code has also been used
to hook LoadRunner into a TSO test client. The method of
TSO automation cannot be used on the publishing side
because testers cannot access the game source code. Also,
in the TSO automation, testing has been achieved by a
simple command script, not by test scenarios.
The VENUS system has been developed to provide a beta
test environment to game developers in order to test online
games efficiently. Using a virtual client engine in the
VENUS system, the user can easily simulate online games
for beta testing. The load-limit point and bottle-neck of a
game server can be verified by simulating a massive
number of virtual clients in an online game environment.
Further, the VENUS system is very helpful because it
stores an internal game data from the game server in a
database. The VENUS system has been applied to many
different online games to test its usefulness. However,
dummy game clients should be rewritten by using the
VENUS Software Development Kit (SDK) for new game
testing. Also, its testing is focused on simple load testing
instead of scenario-based testing.
RELATED WORK
Developing and deploying a permanent massively
multiplayer world is very difficult due to the distributed
nature and large scale nature of the game system. In general,
game developers use a set of beta tests to achieve stability
for the online game server. Selected beta testers play the
game and send feedback information to the game testers.
Then, the game developers analyze the error reports from
the feedback and debug the game applications. However,
this ordinary testing process consumes a lot of development
resources such as cost and time. In many cases, the error
reports from the beta testers provide only vague clues of a
particular error to the testers, which makes it quite difficult
for the developers to repeat the error environment under the
same conditions that the beta tester experienced [2].
Server testing technology using virtual clients has been
actively studied. Industry-standard load testing solutions
such as LoadRunner [6], QALoad [10], and e-Load [11]
can emulate hundreds or thousands of concurrent users by
reusing captured packets. These solutions enable measuring
end-to-end performance, diagnosing applications and
system bottlenecks, and tuning for better performance.
They help reduce the costs and time required to test and
deploy new applications and systems. However, their key
infrastructure components are Web servers, database
servers, and so on [12-14], so cannot be applied easily to
game servers. It is difficult to test the stability of an on-line
game server application using a conventional packet reuse
method because an on-line game server employs
complicated logic compared with a general server
application [15-16]. Online game server testing has
introduced new research issues in this area.
Game test automation has been used in The Sims Online
III.
ONLINE GAME TESTING SCENARIOS
Conceptually, a test scenario is a sequential list of actions
with corresponding positions. The actions are meaningful
user activities such as logins, movement, combat, and so on.
Fig. 1 shows an example of a test scenario. As shown in the
example, a test scenario is made up of a sequence list of a
position and action pair.
Scenario = ( List (position, action))
Figure 1. Example of a game testing scenario.
260
The test scenario of Fig.1 is described as follows.
((A,
Generate_Users)
(B,
Gather_Minerals) (D, Sell_Items))
Hunt_Mobs)
(C,
A, B, C and D are the position data and Generated_Users,
Hunt_Mobs, Gather_Minerals and Sell_Items are the names
of actions.
In our method, the position information is represented in
a virtual game map and the actions are represented in the
game description language. That is, all the game logics are
described by defining the game description language and
virtual game map, so the test client code does not need to
be rewritten when a new game is to be tested.
A. Virtual game map
Figure 2. Virtual game map.
A virtual game map is a collection of x, y, and z positions
and contains the texture information and building objects.
The process by which a virtual game map is generated is
as follows. At first, the virtual game map is blank. The
actual game players or virtual users randomly and
massively move around the game world. The game world is
gradually searched using the position information gathered
by game users’ movements. The packet analyzing tool
determines which areas the game users are and are not
allowed to visit. Finally, the tester can modify the map
textures and place building objects such as villages, stores,
and hunting grounds manually.
B. Example of game packets
To explain the game description language conceptually,
Fig. 3 shows an example of ELMA game protocols. ELMA
is a typical MMORPG in its alpha testing period, and is
used to test the feasibility of our system.
In ELMA, the packets always start with 0x21, and the
timestamp, packet length, protocol ID, and parameters
follow. Its endianness is little endian, and it has no
encryption rules. The protocol ID is in 3-byte hexadecimal
form, and the number of parameters is dependent on each
protocol ID.
As shown in Fig. 3, the IDs of the move protocol are
0x46, 0x6, and 0x1, and those of the attack protocol are
0x51, 0x7, and 0x2. The move protocol consists of four
parameters, and the attack protocol consists of five
parameters. The parameters of the attack protocol are
character ID, attack type, damages, target ID, and
animation type, and the data types are short, integer, integer,
byte and integer.
In a virtual game map, game objects such as virtual users,
actual game players, and NPC objects are displayed. The
testers can view the position and status of game objects by
selecting them in the virtual map. Also, the tester can
command the actions for a designated game object.
Waypoints can be pointed to by mouse clicks, and a group
of virtual users can move according to the particular
waypoint.
Figure 3. Example of ELMA game packets.
261
IV.
string is defined with the size.
GAME DESCRIPTION LANGUAGE
The game description language consists of actions, packet
protocols, and packet generation rules. The actions consist
of a series of packet protocols, and the packet protocols
define the individual packet messages transmitted between
the game server and game client. The packet generation
rules define the overall packet structure rules such as
encryption, endianess, timestamp, and so on. Fig. 4 shows
the syntax of the game description language, and Fig. 5
shows the relationship between the scenario and game
description language.
Examples of ELMA elements are as follows.
(Starter, byte, 1, (0x21))
(TimeStamp, Int, 1, (0xcf, 0xc, 0x5, 0x4)
(Move_protocol_ID, byte, 3, (0x46, 0x6, 0x1))
(Characte_ ID, Short, 1, (0xb2, 0x7, 0x2))
(Attack_Type, Int,1, (0x4, 0x0, 0xde, 0x3))
(Target_ID, byte, 7, (0xd1,0x48, 0xd0,0x3f,0x5d,0xf5
0x2a))
The protocol consists of a name and one or more
elements.
Game_Description_Language=
(Protocol,
Action,
Generation_Rules)
Protocol = (Element)
Action
=
(Time_Constraint,
Socket,
Protocol,
Parameter)
Generation_Rules = (Endianess_Rule, Encryption_Rule,
TimeStamp_Rule,
HeartBeat_Rule, ...)
Protocol= (name, list (element))
The ELMA attack protocol example shown in Fig. 3 is as
follows.
(ATTACK,
(Starter,
TimeStamp,
Length,
Attack_protocol_ID, Attack_Type, Damages, Target_ID,
Animation_ID))
Element = (Name, Type, Length, Byte_string)
Figure 4. The syntax of the game description language.
An action is a sequence of protocols. It consists of the
time constraint, socket, and protocol. Time constraints
describe the time relationship for two protocols. They
include send, receive, sendWait, and delay. There are many
kinds of servers such as a gateway server, login server,
world server, and so on, so the socket description should
also be defined with each protocol.
Figure 5.
Action = (Time_Constraint, Socket, Protocol, Parameter)
Time_Constraint = (Send, Receive, SendWait, Delay)
Send : Send protocol
Receive : Receive protocol
SendWait : Send protocol and Wait until
receiving server’s response Protocol
specified in parameter
Delay: Wait and delay the time intervals (ms)
specified in parameter
Socket: Network IP and port description
Protocol: Previously defined protocol name
The relationship between scenario and game description language.
An element is the basic component of the game
description language. It consists of name, type, size and
byte_string.
An example of ELMA login actions is as follows.
Element = (Name, Type, Size, Byte_string)
##(Send)(Socket1)(SendDefault_ID)##
##(SendWait)(Socket1)(Login _ID)(Login_ID_Responce)##
##(Delay)(Socket1)300##
##(SendWait)(Socket1)(Char_List)(Char_Lis_Responce)##
##(Receive)(Socket1)(Live_Msg_ID)##
##(SendWait)(Socket1)(Enter_World)( Enter_World _Responce)##
The byte_sting is for the reuse of actually captured
packets. Not all packet protocols need to be defined in the
game description language. In some cases, the captured
byte string can be reused as it is. In this case, the date type
is fixed with the byte, and the length of the captured byte
262
V.
IMPLEMENTATION RESULTS
virtual game map.
The process of defining the game description language is
as follows. First, the position and length of the packet
header and protocol ID is analyzed. Then, the packet
generation rules such as encryption and endianess are
analyzed. After the overall structure and the packet
generation rules are defined, individual protocols such as a
move, attack, and skill are analyzed. Finally, actions are
defined.
Fig. 7 shows screenshots of the packet analyzing tool. It
shows the process of defining the ATTACK protocol,
defining in particular the Attack_Type parameter. After the
user selects a packet range in the individual packet data
pane, the name, type, and size of the element are defined.
On the right-hand side, there is a protocol view pane listing
the already defined protocols.
We have implemented a VENUS II system, which is
named after the previous VENUS system, and it has been
applied on four online games to verify the effectiveness of
our method. The applied game genres [1] include MMOG,
multiplayer online game (MOG), and casual game. In
MOG and casual game, the testing is rather simple,
includes entering, leaving, and re-entering rooms
repetitively, and a virtual game map is not used. In this
section, the application to ELMA will be explained in detail.
A. Overall system architecture
The VENUS II system consists of a packet analyzing tool
and virtual user control tool. Fig. 6 illustrates the overall
architecture of the VENUS II system. The actual game
packets transmitted between a game client and game server
are captured and stored. Then, the stored packet lists are
analyzed, and the game description language and virtual
game map are generated. The Winpcap [17] library has
been used for filtering and capturing the game packets.
Game Client
Packet Analyzing Tool
Virtual
Game Map
Virtual Game
Map Analyzer
N
E
T
W
O
R
K
Packet
Capure
Game
Description
Language
Analyzer
Game
Description
Language
Virtual User Control Tool
User Interaction Manager
Virtual Game
Map Manager
Virtual User Manager
Virtual
User 1
……
Figure 7. Packet analyzing tool defining the game description language.
Virtual
User n
C. Virtual user control tool
Game Server
A virtual user is an activity entity of a single online game
user. To the main game server, these test clients look
identical to an actual connected game player. To generate a
heavy load and provide a set of stresses to the game servers,
a plurality of virtual users should be created and controlled.
Fig. 8 shows a scene of the virtual user control tool. In
the virtual user control tool, there are three window panes,
a user interaction pane, virtual game map pane, and virtual
user property pane.
In the virtual game map in the center, the virtual users
and mobs are displayed according to their position using
circles and triangles. The red bar below the shape indicates
the HP value of the game objects, and the color of the shape
indicates the states of the objects such as in combat,
walking, dead, and so on. When the tester selects a virtual
user in the virtual game map, the detailed properties of the
virtual user are shown in the property pane window.
The virtual user control tool generates an actual load
when the tester inputs the actions in the user interaction
Figure 6. The system architecture of VENUS II.
The game description language and virtual game map are
referred to in the virtual user control tool. The virtual user
control tool consists of a user interaction manager, virtual
game map manager, and virtual user manager. The virtual
game map manager shows the position and status of game
objects. The user interaction manager constructs test
scenarios by selecting the game objects in the virtual game
map and inputting the actions sequentially. The virtual user
manager generates actual packet loads according to the
defined game description language.
B.
Packet analyzing tool
The packet analyzing tool captures and analyzes packet
data, and generates the game description language and
263
position and hunting mobs, i.e., it is a waypoint scenario.
The activities of the virtual users can be viewed and
verified from the viewpoint of an actual player’s game
client.
pane. A large number of virtual users can be generated with
group and action commands being applied to the group of
virtual users at the same time. The group of virtual users
can select and attack a victim mob at one time, or each
virtual user can select and attack its own mob
independently.
The action is converted into a sequence of protocols
internally, and the generation rule is applied to each
protocol. Sending of the packet protocol can be delayed for
the designated time interval according to the time constraint
defined in the actions.
The login/logout, directional movement, and attack
actions are provided with button control, and test scenarios
such as party play and waypoint movement are supported.
In our experiment, 1,000 ELMA virtual users have been
generated and are controlled in a 2.8Ghz Intel Core2 Quad
CPU PC.
Figure 9. A screenshot of the actual ELMA game client.
VI.
COMPARISON OF ONLINE GAME TESTING
Table 1 shows a comparison of existing game testing
approaches and the proposed method. Our approach is
compared with existing ones in the viewpoint of the
blackbox testing and scenario-based testing.
The TSO automation and the VENUS system are based
on whitebox testing. Their approaches can be applied to
game developers only and are not adequate for game testers
because the developed game client code must be supplied
and rewritten for game testing.
Figure 8. Virtual user control tool
Fig. 9 shows a scene from an actual ELMA game client
when the virtual users are moving toward a designated
TABLE I.
Approaches
Categories
Blackbox
testing
Scenariobased
testing
COMPARISON WITH PREVIOUS APPROACHES
HP LoadRunner [6]
TSO Automation [7]
VENUS [8]
Our Paper
Reuse game client
code partialy
whitebox
Capture, analyze and
generate GDL
blackbox
Main idea
Reuse captured data
Backbox/Whitebox
Requiring
game client code
Easy application to
another game
blackbox
Reuse game client
code partialy
whitebox
No
Yes
Yes
No
Hard
Impossible
Impossible
Easy
Game map support
No
No
No
Yes
Unit of testing
Protocol
Action
Action
Scenario
Flexibility to various
test scenario
Dynamic game data
support
Hard
Easy
Easy
Easy
Parameterization and
correlation
ActionScript
(proprietary)
SDK
(code itself)
GDL
(action and protocol)
264
REFERENCES
HP LoadRunner supports blackbox testing by reusing
captured packet data. And it supports partially the dynamic
content with parameterization and correlation. The
dynamically created server responses and the correlated
data can be handled easily in the general server such as web
server, but it is hard for them to be applied to game testing
because an on-line game server employs complicated logic
compared with a general server application.
The TSO automation and the VENUS system reuse the
game client code itself, so various scenarios can be tested
easily, but the game map is not supported in scenario
testing and their testing unit is action.
Our approach overcomes the weakness of previous
approaches and supports blackbox testing and scenariobased testing. Game logics are easily expressed with game
description language and game map, and testing scenarios
are easy and intuitive.
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
VII. CONCLUSIONS
[10]
[11]
[12]
In this paper, along with simple load testing, we propose
blackbox testing and scenario-based testing of online games.
In previous test automation, the developers must provide
their client code to the game testers, because a test client
code is based on a dummy game client code without a GUI.
In our approach, the game logic is described using a game
description language and virtual game map, so the test
client code does not need to be rewritten when a new game
is to be tested. The virtual clients automatically generate a
packet load according to the game description language.
Moreover, complex scenarios such as attack, party play, and
waypoint movement can be tested by combining actions.
To verify the effectiveness of our method, we have
applied the VENUS II system on several online games.
Among them, application to the ELMA online game has
been explained in detail. The login/logout, directional
movement, and attack actions are defined using the game
description language in the packet analyzing tool, and the
virtual user control tool has generated and controlled 1,000
ELMA virtual users in a 2.8Ghz Intel Core2 Quad CPU PC.
The stability of the ELMA game severs has been verified
using our tools.
Future research is planned to improve our VENUS II
system, provide more intelligent test scenarios, and make it
a standard model for online game testing.
[13]
[14]
[15]
[16]
[17]
ACKNOWLEDGMENT
This work was supported by the IT R&D program of
MKE/IITA. [2009-S-041-01, Online Game Quality
Assurance Technology Development using Scenario
Control of Massive Virtual User]
265
Game
genres.
[Online].
Available
at
http://en.wikipedia.org/wiki/Game_genres.
C. P. Schultz, R. Bryant, and T. Langdell, Game Testing All In One,
Thomson Course Technology PTR, 2005.
L. Mellon, B. Dalton, G. Walton, and D. West, "Large-scale engineering
for online and offline games," in GDC 2007, Spring 2007.
Y. Deshpande, R. Jenkins, and S. Taylor, “Use of simulation to test
client-server models”, Proceedings of the 1996 Winter Simulation
Conference, pp. 1210-1217, November 1996.
D. Wangerin, C. DeCoro, L. Campos, H. Coyote and I. Scherson, “A
modular client-server discrete event simulator for networked
computers”, Proceedings of the 35th Annual Simulation Symposium, pp.
125-133, April 2002.
LoadRunner. [Online]. Available: http://www.hp.com
L. Mellon, “Automated testing of massively multi-player systems:
Lessons learned from The Sims Online,” GDC 2003, Spring 2003.
B. H. Lim, J. R. Kim, and K. H. Shim, “A load testing architecture for
network virtual environment,” Proceedings of 8th International
Conference on Advanced Communication Technology, pp. 848-852,
February 2006.
Y. W. Jung, B. H. Lim, K. H. Sim, H. J. Lee, I. Park, J. Chung, and J.
Lee, “VENUS: The online game simulator using massively virtual
clients,” in Proceedings of AsiaSim 2004, pp. 589-596, October 2004.
QALoad. [Online]. Available: http://www.compuware.com/
Empirix, OneSight. [Online]. Available at http://www.empirix.com/
S. Elbaum, S. Karre, and G. Rothermel, “Improving web application
testing with user session data,” Proceedings of 25th International
Conference on Software Engineering, 3-10, pp. 49-59, May 2003.
D. A. Menasce, “Load testing of web sites,” IEEE Internet Computing,
pp. 70-74, July-August 2002.
F. Ricca and P. Tonella, “Analysis and testing of web applications,”
Proceedings of the International Conference on Software Engineering,
pp. 25–34, May 2001.
W. Feng, F. Chang, W. Feng, and J. Walpole “Provisioning online
games: A traffic analysis of a busy Counter-Strike server,” Proceedings
of the second ACM SIGCOMM Workshop on Internet measurement
workshop, pp. 151-156, November 2002.
T. Fritsch, H. Ritter, and J. Schiller, “Network effects on games: The
effect of latency and network limitations on MMORPGs : A field study
of everquest2”, Proceedings of 4th ACM SIGCOMM Workshop on
Network and System Support for Games NetGames '05, ACM Press, pp.
1-7, October 2005.
WinPcap: The Windows Packet Capture Library. [Online]. Available at
http://www.winpcap.org/