CheesePi: Measuring Home Network
Performance Using Dedicated Hardware
Devices
Abstract
Internet users may not get the service quality promised by their providers, and also
may not know what service they can receive. When users experience poor Internet
connection performance, it is not easy to identify the source of the problem. We
develop CheesePi, a distributed measurement system that measures the Internet
connection experience of home users based on some network performance
attributes (e.g. latency, packet loss rate, and WiFi signal quality). The CheesePi runs
on a Raspberry Pi (a credit card sized computer) connected to the user’s home
network as a measurement agent. It is important to measure the network
performance from the user’s side since it is difficult to measure each individual’s link
from the operator (provider) side. Each measurement agent conducts measurement
periodically without disturbing the user’s Internet quality. Measurements are
conducted during big media events from SICS (Swedish Institute of Computer
Science) labs and student accommodations. The measurement results show
customers with an Ethernet connection experienced significantly better latency and
packet loss compared to WiFi users. In most of the measurements users at SICS lab
perceived better latency and packet loss compared to the users at the student
accommodation. We also quantify how customers experienced lower performance
when streaming from websites which do not use CDN technology compared to the
websites which do use CDN, particularly during big media events.
1 Introduction
This chapter gives an overview of the master’s thesis focus area to help readers
understand what the project is essentially about. Next the existing problems that
motivate this work will be discussed, followed by the project’s goal. Thereafter the
specific goals and contribution of this master thesis in the project is going to be
presented. At last the organization of the report will be explained.
1.1 Motivation
The Internet is one of the very essential needs in the modernised world. We use the
Internet for communication, sharing information, watching videos and TV, shopping,
and many other activities. According to the Internet World Statistics [22], there are
about 9.2M Internet users in Sweden, 604M in Europe, and 3.3B around the world.
1
Approximately 85% of Internet users watch online videos [1]. Users become
frustrated when videos pause while the system downloads additional data before
resuming video playback. According to a new study from the University of
Massachusetts Amherst and Akamai Technologies [23], many broadband Internet
users give up on an online video if it doesn’t display within two seconds (probably not
true in developing countries). Most Internet users, when they perceive a connection
problem, they have little idea about its root cause. The problem can be due to a
variety of reasons, sometimes it can be a local problem due to poor WiFi connection
in the user’s environment. It could be also due to the Internet Service Provider’s
(ISP) lack of bandwidth, or several other reasons. Measuring the Internet connection
is important for the home user to ascertain if they are receiving the promised service.
It is also important from the service providers and regulators point of view, to
characterize the whole network and assessment of potential degradation in the
services they provide. In other words it is monitoring the network and services they
provide. Researchers have been trying to measure end-to-end quality-of-service
(QoS) for many years, and we want to add our contribution for users’ better
experience.
1.2 Goals
The main goal of this project is to build a measurement platform (CheesePi) that help
home Internet users to characterize their connections and with collated data to help
ISPs and operators to fix network problems.
The main goals of this master thesis are:
1. Design the database of the different measurement tools in the MA (Raspberry
Pi) and in the central server.
2. Build test modules for ping, httping, and traceroute network measurement
tools.
3. Run several scheduled measurements periodically.
4. Perform specific measurements during big media events to popular streaming
websites and integrate the results into the database.
5. Conduct analysis on the collected results and draw conclusion on the
observations.
1.3 Organisation of the report
Chapter one introduces the motivation and objective of the project, and the
contribution of the master thesis. Chapter two introduces the area of the thesis, and
introduce the network tools used on this project. Chapter three discusses the related
projects. Chapter five is going to explain the methodology followed for building this
project and conducting the experiments. Chapter five presents the results of the
experiment and small discussion about the observations. Chapter six discusses the
observations and the followed approach during the master thesis. Chapter seven
2
draws conclusion about this master thesis. Chapter eight presents the possible
future works of this project. Chapter nine presents the lessons learned during this
master thesis. The last part of this report presents the references of articles and
technical reports used for this master thesis and an appendix.
3
2 Background
First in this chapter, the general purpose of the Internet and how it works will be explained
first. Then, different way of accessing the Internet are going to be introduced. After that,
different network performance measurement approaches followed by the tools used for
measuring network performance will be discussed. At the end, different content delivery
techniques used by different media streaming services will be introduced.
2.1 Internet Protocol (IP) Networks
The Internet is a network of many networks. The connected devices and networks
are differ in hardware, software and design, so they need a common protocol in
order to communicate with each other. The Transmission Control Protocol/Internet
Protocol (TCP/IP) protocol enables inter-connected networks to perform end-to-end
communication, so that any node on the Internet should have the ability to
communicate with any other node.
Figure 2.1: Internet building blocks [2]
The TCP/IP protocol suite is divided into four layers for division of labor and easiness
of implementation and is called a protocol stack [3]. The four layers are: Application,
Transport, Network, and Link layers. The application layer provides a service that
enables a user to communicate over the TCP/IP model, for example services such
as: HTTP (web), FTP (file transfer), and SMTP (mail). The transport layer provides
an end-to-end data transfer between applications or peers, and the most used
protocols are the Transport Control Protocol (TCP) and the User Datagram Protocol
(UDP). Data is multiplexed up the network stack at this level from information in the
4
port numbers. This is because network flows arriving at this machine, but are being
used by different applications. The Network layer transfers data from source to
destination by performing routing functions, and the IP is the most prominent
protocol at this layer. In order to demultiplex flows at the router level, the IP
addresses are used. And the Link layer is the interface to the network hardware, for
example, 802.11, ATM, and X.25. For example, a WiFi user connected to the home
router wirelessly wants to send data to a target server. The frame is send to the first
hope (WiFi router), then the router sends it to the next router, and it goes like that till
it reach the target server. The frame contains source and the next hop MAC (Media
Access Control) address (Link Layer), source and destination IP address (Network or
Internet Layer), the source and destination port (Transport Layer), and the data. The
destination port tells about the used application layer protocol, for example, if the
destination port is 80, it is an HTTP protocol for accessing the web. Unlike the IP
addresses, the source and destination MAC addresses changed every time during
the path till it reaches the destination host.
Figure 1.2: The TCP/IP Model
IP defines an addressing schema to uniquely identify a host and devices on a
network. When a packet is sent from the source to a targeted destination, it should
include the IP address of the sender and the receiver. The routers in the network
select where to forward the packet based on the destination IP address. The IP
version 4 (IPv4), a widely deployed protocol. IPv4 uses a 32 bit addressing schema
to represent a device in the network, which allows more than four billion (2^​32​)
addresses, whilst IP version 6 (IPv6), created to resolve shortage of addresses in
IPv4, uses a 128 bit addressing schema. IP does not provide delivery guarantees,
5
and hence is called a ‘best effort’ delivery, this means packets can be lost, corrupted,
and received out of order. It is up to the upper layers like Transmission Control
Protocol (TCP) to guarantee the (eventual) delivery.
2.2 Access technologies
Users link to the network via different technologies. There are several choices if a
user wants to access the Internet namely: WiFi, Ethernet (typically via a cable), and
Mobile Data (Cellular Network) are common available choices in 2015.
2.2.1 Ethernet
We are not going to discuss the details of the Ethernet protocol in the link layer of the
TCP/IP stack. Ethernet is used to connect computers in a home or company to a
modem (or router) for Internet access. It typically uses a cable to connect a computer
to a modem (router) and does not suffer from medium contention [5]. Some users
prefer Ethernet for speed, security or even (simpler) configuration reasons.
2.2.2 WiFi
The popularity of laptops and smartphones make WiFi a widely used access method
to the Internet. Many people at homes, offices, and public areas use WiFi technology
to access the Internet. In WiFi, there is no physical connection between the mobile
device and a fixed device that has permanent, often non-mobile and stationary
connection to the Internet. Most home users are familiar with a wireless router or an
access point. When a user connects to the Internet all the data transferred to the
access point is sent over unlicensed radio spectrum. A user should be within the
coverage range in order to have access. A periodic beacon sent by the router, needs
to be seen to join the wireless network, via a secure protocol exchange. For
example, a wireless router that runs the 802.11n standard, a user should be in the
range of 70 meters indoors or 250 meters outdoors [6].
2.2.3 Mobile/Cellular connection
In a cellular network the data from a mobile device is transmitted on licensed spectra
via a radio-to-air interface [7]. The signal is received at a fixed base station. Each
base station is responsible for a physical area, often called a cell. Frequency
allocation is an important issue, so not to interfere with adjacent cells and competing
providers in the same physical area. 3G/4G networks use CDMA division to achieve
separation (as well as frequency bands), however 5G technologies will use CDMA
and BDMA. The coverage area of the base station depends on different factors:
frequency, number of users, antenna, location (existence of mountains and
buildings), and other parameters, but they cover larger areas compared to access
points using WiFi technology.
2.3 Network performance measurements
It is not easy for customers to monitor the quality of the service they are provided.
Currently, from the customer perspective, the performance of the Internet typically
6
means measuring the speed, “how fast is my network?” or “how quickly can I
download the data?”. But does speed mean everything? The answer is simply no,
because speed is not the performance requirement of every online application. The
most popular performance measurement metrics for different applications are:
1. The upload and download speeds
2. The delay or latency between two end points
3. The variation of the delay between selected packets, which is called jitter
4. The ratio of discarded packets to the total number of packets sent (packet
loss ratio)
5. To what extent a sequence of packets is reordered within the network, or
even duplicated by the network (packet error ratio)
The purpose of network monitoring is to observe and quantify the network
measurement results based on the performance metrics. The two common network
monitoring approaches are based on passive and active measurements. The active
measurement approach injects test packets into the network by sending packets to
servers (testing network reachability and performance) and applications (network
reachability and server software, e.g. Apache). The passive approach does not inject
test traffic into the network and does not rely on test servers. Instead, passive
monitoring uses a device in promiscuous mode to sense existing data traffic. Both
approaches have advantages and disadvantages. In active measurements, the
traffic is artificial and creates extra load on the network, whilst passive monitoring
measures the real traffic originated from the user and does not add extra traffic to the
network. Since the passive approach may require sensing all packets on the
network, there can be privacy or security issues about how to access/protect the
data gathered [12]. The active monitoring provides control on generating the traffic
which is not possible in the passive approach, and it helps to have a full control on
what and when to conduct measurement.
2.4 Network performance measurement tools
This section introduces some of the tools and device used in a network performance
measurement.
2.4.1 Ping
Ping is used for measuring the network delay or the round trip time (RTT) taken for a
packet from a client to a chosen destination. It is supported by most operating
systems. Ping uses the ICMP protocol to send a so called ‘echo’ request and reply
packets. It puts a timestamp in the packets to measure the RTT, and a sequence
number to calculate the packet loss for the unreceived packets. The main problem of
ping is: it can be ignored by some servers due to the suspicious nature of malicious
attacks, albeit of a high frequency nature, significantly higher than the default one
second inter packet time.
7
2.4.2 Httping
Httping is used for measuring the latency of the network path and behaviour of a web
server. Since the httping requests a full HTTP response from the server, the returned
size of the data might be different for each Uniform Resource Locator (URL). Httping
works on most operating systems, plus there is an Android version. Httping can
include the DNS (Domain Name System) lookups delays as well.
2.4.3 Traceroute
Traceroute is a network diagnostic tool that tracks the route (path) of a packet from a
source to a destination and measures its transit delays across an IP network. It is
used to trace the path a packet follows in a network, and can be used to identify
connectivity problems. By default it sends a sequence of UDP packets addressed to
a destination host; ICMP Echo Request or TCP SYN packets can also be used [13].
Traceroute starts by sending a packet with a Time-To-Live (TTL), also known as hop
limit, at a value of one. The first router decrements the TTL value, and sends an
ICMP "time exceeded" reply to the source due to the TTL value in the packet being
zero. Then the source sends the next packet with a TTL value incremented by one.
The previous router forwards the packet (as the value is now valid) but the next
router drops the packet as the previous one decremented the counter to zero. In
other words, traceroute acts like an expanding ring search, increasing the scope of
the packets. This continues until the destination is reached and returns an ICMP
"Port Unreachable" reply or a max (30 or 64) is reached. This is because traceroute
packets should not circulate the Internet indefinitely. Three probes (by default) are
sent at each TTL setting, and the terminal (console) displays the address of the
gateway or the router in the path and the round trip time of each probe. It is
supported (included) by most of the modern operating systems. Some routers block
ICMP/UDP packets.
2.4.4 Measurement Device
There are a number of ways to host a measurement suite. Some of them will be
discussed in the state of the art section. We have chosen a standard hardware
device, with our dedicated software platform, CheesePi. The Raspberry Pi is a credit
card sized single board small computer developed in UK by the Raspberry Pi
foundation. The price of a Raspberry Pi is very affordable and about 5 million units
have been sold [11]. A Raspberry Pi2 models cost around $35. Due to its
robustness, low price, low power consumption, size, and other factors Raspberry Pis
are becoming popular and start to be used in many projects including network
measurement projects (NetBeez, ARK) see below.
2.4.5 Storage
One can store measurement data in a number of different ways. Common log or
CSV files are still the predominant format. However within network measurements
8
there has been a trend towards more structured data (M-lab for example). A
database is one way to store the data. The data is organised in a way it can easily
be accessed, managed and updated by other computer programs. Relational
databases, a table-based database represents entities as a schema and allows
users to relate data fields and tables with others stored in the database [15].
Recently, non-relational databases, document based and key-value pairs with wide
column fields are becoming a popular alternative technology for analytics of
large-scale data.
2.5 Content Delivery Approaches
When users visit a website to watch a movie or read a news, they don’t know how
the content is delivered to their device. In this section we are going to discuss
different content delivery approaches.
2.5.1 Single Server
The website has only one server, the request for every service from the connected
clients is processed by a single server.
2.5.2 Multiple Servers
One of the most commonly used applications of load balancing is to provide a single
Internet service from multiple ​servers​. For Internet services, the load balancer is
usually a software program that is listening on the ​port where external clients
connect to access services. The load balancer forwards requests to one of the
"backend" servers, which usually replies to the load balancer. This allows the load
balancer to reply to the client without the client ever knowing about the internal
separation of functions.
2.5.3 Content Delivery Network (CDN)
A Content Delivery Network or Content Distribution Network is a system that
replicates digital content over mirrored servers, that are located in suitable places in
the Internet [8]. The goal of the CDN is to deliver fast and reliable data by distributing
the content to caching servers located closer to the end consumers than the original
servers. Most of the popular web streaming like BBC, SVT, YouTube, and Netflix use
CDN technology to deliver their content. If a user requests an object or a file from a
website that uses CDN technology, instead of the hosting server the CDN takes care
of it. This is typically done through the DNS request returning an address to the CDN
server, despite the user typing in the commonly known address. In the UNIX system
the command “host” can reveal where a CDN is being used. The CDN will consider
the geo-location of the visitor and the object will be served from the closest caching
server. If the assigned cache server does not have the requested object, it asks
other cache servers, then the original (host) server [9]. Note, sometimes the CDN
might have to remove some content, if the total catalog size it holds is large and the
popularity of items is highly variable.
9
3. State of the art within network monitoring
This chapter introduces network monitoring projects and how they are related to
CheesePi. It also explains the network tools used during the project. We begin with
monitoring systems that use a hardware-based measurement agent. Then it explains
the software based network monitoring systems.
3.1 RIPE ATLAS
RIPE Atlas is a project measuring Internet quality around the world using
measurement probes. The goal is to measure Internet traffic while providing users
with information about their connection [24]. Probes perform measurements using
ping and traceroute with support from DNS and SSLcert. Users with active hardware
probes produce rendezvous points that can be used to set up user defined
measurements involving other probes. The source code and RIPE data is available
to the public. Around eight thousand probes have been deployed as of 2015. An
open source approach is common with CheesePi, but RIPE uses dedicated
hardware whilst CheesePI uses commodity hardware.
3.2 ARK
The Archipelago Measurement Infrastructure (ARK) uses 120 Raspberry Pis
deployed to perform network monitoring to quantify topology (connectivity) and
dynamics (congestion) using active measurements for both IPv4 & IPv6 [25]. As of
August 2015, the developers UCSD, have deployed 120 nodes. Quantifying the
congestion is done by a Time-Sequence Ping (TSP), a crafted sequence of pings
along a selected network path. The project conducts measurements on the global
ARK overlay network predominantly using traceroute. Variations in the diurnal
variation in delay have been quantified. ARK project is a pure networking research
project, but CheesePi is a measurement system targeting home Internet users. As
said both ARK and CheesePi use Raspberry Pi as the measurement agents.
3.3 SamKnows whitebox
It is a project that measures broadband performance for consumers, ISPs, and
government (regulators) around the world [26]. They use Whiteboxes (a modified
router) to measure the performance of users’ broadband. The hardware is connected
to the user’s home router and run tests based on the client’s interest without
disturbing the Internet experience. It also sends measurement data to a central
server for permanent storage. They also use a web-based measurement similar to
the hardware based measurements (using Whitebox), which run tests from the
user’s computer. The Whitebox is configured to run several performance
measurements. It measures download and upload speed (throughput) between the
10
Whitebox and closely located test node (server). It also measures VoIP (Voice over
IP) quality, UDP packet loss and latency under normal and loaded conditions, FTP
throughput, DNS resolution latency, peer to peer throughput, Internet connection
lost, web page loading latency, email relaying latency, bitrate reliability and startup
latency during video streaming, video quality of experience for popular media
streaming web servers (e.g. YouTube, Netflix, and BBC iPlayer), performance of
multicast IPTV over RTP.
They use a dedicated device for continuous measurement and a dashboard for
visualizing the results based on user’s interest, which makes it similar to CheesePi.
But customers have no freedom on controlling the Whitebox, and can not use the
hardware for other purposes. Whereas the Raspberry Pi which runs the CheesePi
platform can be used for many different purposes. They have one or more test nodes
(servers) in the popular cities around the world from network operators [27], and the
stored data in the central server is used by users, ISP, Internet regulators and
academia. Samknows has a good collaboration with different organizations, such as
Ofcom in the UK and EU research projects.
3.4 NetBeez
NetBeez is a distributed network monitoring system, which is mainly targeted at
companies with offices in different locations and a centralized IT department [28].
The NetBeez agent (Beez) is deployed behind a switch in every remote location as if
it were a normal user. The agents then simulate users in the organisation and
monitors the network, if a problem occurs it makes troubleshooting easier by using
the diagnostic information from the Raspberry Pi. The measurement agent, which is
a Raspberry Pi, is configured to run several tests. It runs ping to verify the
connectivity and round trip time between two agents (or other chosen host), the DNS
resolution latency, traceroute to check if there is a change in the network path and
observe the bottleneck hops, Iperf to test the stress of the bandwidth, and measure
the performance of an HTTP server by evaluating the HTTP test packets. All the
measurement tests are controlled by the NetBeez server, it also collects the test
results from the NetBeez agents. The users are able to control the functionalities
through a NetBeez web based Dashboard, and they can visualize the performance
on different aspects of the network (locations, targets, historical data).
Similar to CheesePi, NetBeez also uses Raspberry Pis as measurement agent.
However, CheesePi targets are home users and NetBeez customers are companies
who have offices in different locations. CheesePi is free measurement platform while
NetBeez offers subscription based services.
11
3.5 Software-based network monitoring systems
In this section monitoring systems, which do not use dedicated hardware for
conducting the measurements, or follow a software based approach are going to be
presented. Deploying software is somewhat easier than a hardware-based approach
and can achieve larger scale measurements and coverage. We explain those that
are either well known, or are similar in vain to CheesePi.
3.5.1 SpeedTest
Speedtest is a web service that provides free analysis of Internet access parameters
[29]. Probably from the user perspective it is the most well known one, globally. It
was founded by Ookla in 2006, The service measures the bandwidth and latency of
a visitor's Internet connection against closest host servers. As of October 2015
[30],They have 4085 distributed host servers around world. This makes it somewhat
similar to Bredbandskollen (a Swedish web-based speed test) and SamKnows. Each
test measures the data rate for the download direction, i.e. from the server to the
user computer, and the upload data rate, i.e. from the user's computer to the server.
The tests are performed within the user's web browser using the HTTP protocol. As
of 2015, over 7 billion speed tests have been completed. The site also offers detailed
statistics based on test results. This data has been used by numerous ISPs and
researchers in the analysis of Internet access around the world.
3.5.2 ICSI Netalyzr
Netalyzr is a network measurement and debugging service/tool that evaluates users’
Internet connectivity [31]. The tool aims to be both comprehensive in terms of the
properties measures, and easy to use as it is web-based for users with little technical
background. Netalyzr is a signed Java applet that communicates with a suite of
measurement-specific servers. Traffic between the two then probes for a diverse set
of network properties, including outbound port filtering, hidden in-network HTTP
caches, DNS manipulations, NAT behavior, path MTU issues, IPv6 support, and
access-modem buffer capacity. In addition to reporting results to the user, Netalyzr
also forms the foundation for an extensive measurement of edge-network properties.
Along with describing Netalyzer’s architecture and system implementation, they
presented a detailed study of 130,000 measurement sessions from 99,000 public IP
addresses that the service has recorded since being made publicly available in June
2009.
3.5.3 Fathom
Fathom has been used in home network environment [32] which has similar goals to
ours, but differs in the software/hardware and the external/embedded measurement
approaches. Fathom is a Firefox extension that implements the measurement
primitives of Netalyzer to measure network characteristics using JavaScript. It adds
12
3% to the times to load a web page and can timestamp network events to within one
millisecond.
3.5.4 PingER
The PingER (Ping End-to-end Reporting) project monitors the end-to-end
performance of Internet links worldwide [33]. Developed by the IEPM group at the
SLAC National Accelerator Lab in California, PingER has a decade old data
repository of network performance measurements from and to sites around the
world. Currently, there are over 60 active monitoring hosts in 23 countries monitoring
over 300 sites in over 165 countries that cover 99% of the world’s Internet connected
3.2 billion population [22].
13
4. Methodology: a monitoring approach
In this chapter we will discuss the research methodology used in the CheesePi
project. We applied a quantitative methodology [14] to conduct the research. We
used several networking tools to measure the quality of the Internet service and
design a database to store the results. Then we conducted analysis on the collected
results. This chapter will discuss how the measurement tools and the database have
been designed and the important techniques used for the sampling and scheduling
the performance measurements.
4.1 CheesePi Architecture
The MA runs on a Raspberry Pi connected to a home network. This measurement
agent run tests periodically without disturbing the perceived Internet experience of
the user. In practice this means using as little capacity as possible. A controller
distributes schedules on how tests should run on the MAs. There is a database for
storing the measurement result of the four tools namely; ping, httping, traceroute,
and VoIP in both the MA and the central server. A collector organizes the results
collected from the MAs and stores it in the central server database. A tool recorded
the WiFi contention is also part of the system. User can visualize the real-time and
history of the experienced Internet performance on a web-based dashboard, which is
hosted locally on the Raspberry Pi. The detail of each test modules will be explained
later.
14
Figure 4.1: CheesePi Architecture
4.2. ICMP-based measurements
We use the default installed ping on Raspbian, an optimized Debian based operating
system for Raspberry Pi. The controller in the central server schedules the MA to do
the delay measurements. The schedule consists of the number of measurements to
perform, the landmarks (the target destinations), and when to start the
measurements.
In CheesePi, the controller decides how and when the measurements should be
conducted. For this master’s thesis, we use a crontab, a wall-clock job scheduler in
Unix operating systems to schedule the measurements. For example, the cron job
schedule can be, every five minutes conduct ten ping measurements to BBC web
server with an interval of one second. “​*/5 * * * * /home/pingMeasurement.py”
The default ping packet size we used was 64 bytes, which is eight bytes of the ICMP
header and 56 bytes of (null) payload. Before we choose the ping packet size we
conducted an experiment to see how the packet size affects the network
performance? We sent 100 probes each to ​www.bbc.com with packet sizes of 64,
128, 256, and 512 bytes in parallel. The experiment was taken at SICS lab, which
the download and upload speed is around 100 Mbps (Megabits per second). The
average round trip times (RTTs) were 30.75, 30.79, 30.88, and 31.44 ms for the ping
measurements with 64, 128, 256, and 512 byte packet sizes. The results indicate
15
that the latency is almost similar for the different packet sizes and we decided to use
64 bytes. We conduct the experiment by considering users with limited data plans, or
some form of constrained access.
The cron job runs the Python module, that conducts delay and packet loss
measurements, according to a the schedule, for example every five minutes. The
Python module works as follows:
1. It runs a number of probes, for example, “ping -c 10 www.facebook.com”
2. It refines and organizes the ping results in a suitable format for store in the
database.
3. It stores the necessary measurement metric results and the MA information,
which helps to uniquely identify the MA in the central database, in a local
database. For example, average RTT, IP address of the MA, destination
domain name, MAC address of the MA, and other information.
4. It also stores the tool output in a file (optional)
According to the schedule provided by the controller, the uploader module sends
collected ping measurement results to the CheesePi central server. For example, if
we decide to send the data every 23:00 UTC (Coordinated Universal Time), at 23:00
the program checks if there are accumulated results since the last submission and it
transmits the data to the central server. The controller is responsible for the
distributing of the data submission time. Because if all MAs (we assume the number
of MAs is large) send their data at the same time, the central server can be loaded
and this can lead to improper data handling.
4.3 HTTP-based measurements
We installed httping on all Raspberry Pis for measuring the performance of a web
server. Similar to ping, the controller in the central server schedules when the HTTP
measurements should be conducted. For this master thesis, the cron job runs the
python module that performs httping measurements at the same time as the ping
measurements. By running both the httping and ping measurements parallely, we
can ascertain whether the network or the web server is the source of the low
performance during visiting to particular web server. Unlike ping, we do not choose
the packet size for the httping probes. Since the httping program returns the HTTP
header of the target web server, the packet size can be different for different
destination (landmarks).
The httping python module runs httping measurements in a very similar way to ping
measurements, which organizes the results in a suitable format for the database,
stores them in a local database and a file (optional). Then the module uploads the
results to the CheesePi central server in a similar manner as ping results uploaded.
16
4.4 Traceroute delay measurements
We use the default installed traceroute in Raspbian on all MAs. Unlike ping and
httping, traceroute sends three packets at a time and the packets may follow
different paths. Therefore the Python module, that does traceroute measurements,
has to take into account this potential hurdle. The traceroute module works as
follows:
1. It runs traceroute command e.g. “traceroute ​www.facebook.com​”.
2. It refines and organizes the traceroute results in a suitable format to be stored
in the database as in Figure 4.3.
3. It stores the traceroute results and other necessary information, which
uniquely identifies this measurement in the central database, in the local
database.
4. It also stores the results in a log file (optional).
Then, according to the schedule chosen by the controller, the upload module sends
the accumulated results to the CheesePi central server.
4.5 Database design
This section describes how the database was designed in the MA and the central
server. Choosing an appropriate database for CheesePi was evaluated and
relational (SQL, MySQL) and non-relational (Time series, Influx) databases were
evaluated during the development of this project. The different databases considered
in this project and the scientific approach followed for choosing them will be
discussed next. Since CheesePi is a system that helps users to characterize their
real-time network performance, how these databases in the MA were designed and
what are the advantages and disadvantages of each will be discussed next.
4.5.1 Measurement Agent design I
Relational databases are well suited to carrying out complex queries. And MySQL,
an open source relational database was chosen for CheesePi because of its open
source freedom, online support, scalability and flexibility, high performance, strong
security options for data protection, popularity, and management ease. Below are
some cases where a single query is used to retrieve data from multiple tables. These
are used during a visualization task and analysing the network performance.
17
Figure 4.2: Ping and httping database design
In the MA the ping and httping measurements are each represented by a single
table. The table designs of httping and the ping are the same since the metrics used
by both tools are similar. Within each tool there can be differences in the output,
however to keep the tables manageable we use similar metrics. Some of the
attributes like delay (RTT) and packet loss (%) are performance metrics. Other
attributes like timestamp and starting and ending time are used in management of
the data and measurements. The source and destination addresses plus the target
hostname identifies the two end points. And the Ethernet MAC address is used to
identify an MA in the central server.
18
Figure 4.3: Traceroute database design
As one can see in Figure 4.3 the traceroute measurement result is represented by
three tables due to its different behaviour from the previously discussed tools. The
tables: Traceroute_Seq, Traceroute, and Hop are related by an attribute ID. The Hop
table which stores the details of the results, the routers in the path and their delay,
should be related with the Traceroute table which stores the measurement
information about the time when the measurement took place, and the details about
source and destination nodes. So a unique ID (Ethernet MAC address of the MA +
auto-increment integer) is the primary key in the Traceroute table to uniquely identify
every traceroute measurement and a foreign key in the Hop table to relate the details
for each measurement. We make the ID a combination of the Ethernet MAC address
and auto-increment integer to uniquely identify the MA’s result in the central server.
Since it is impossible to auto-increment a text (varchar), before a new row is inserted
into the Traceroute table, a trigger auto-increment the integer ID in the
Traceroute_Seq table and concatenate it with the Ethernet MAC address of the MA.
For example, if the last inserted ID in the Traceroute_Seq and Traceroute was 9 and
b8:27:eb:16:7d:a6-9, in the next traceroute probe before the results inserted the ID in
the Traceroute_Seq table incremented by one and the ID in the Traceroute table
becomes b8:27:eb:16:7d:a6-10.
19
4.5.2 Measurement Agent design II
We don’t want to limit users to store the measurement results only on mysql. It is a
good approach to let users store their results in any database or other file formats.
CheesePi uses Grafana, an open source application for visualizing large-scale
metrics data, as a dashboard for displaying the measurement results. Currently,
Grafana supports a few time series databases (Graphite, InFluxDB, and Open
TSDB). Even though InFluxDB is a schemaless time series database, which is
incapable of retrieving results from multiple tables with a single query, there is no
need of complicated queries for displaying the results in the dashboard. So, the
CheesePi project decided to use InFluxDB as a default database for storing the
measurement results, and a user has a decent looking (and editable) dashboard.
The attributes (fields) in all the performance measurement tools are similar to the MA
design I (MySQL).
Figure 4.4: The CheesePi Dashboard
4.5.3 The central server design
Most of the measurement analytics are done in the central server, so it is important
to design the database carefully in order to create relations among tables so that
data can be retrieved with relatively simple efficient SQL queries. For example, a
query might want to retrieve the number of users in Stockholm who experience a
network delay (RTT) value greater than 250 ms during measurements to
www.bbc.com on 20th of May 2015 between 13:00-14:00 UTC. So we need a
number of tables which are related by key fields to get this information.
20
Figure 4.5: Central server database design
The tables are PI, User, Connection, Operation, Ping, Traceroute, Traceroute_Hop,
and Httping. The PI table stores the Pi ID and its Ethernet MAC address to identify
the Pi (MA) uniquely. The User table stores the Pi ID, email address of the user, and
alias (temporary name) to identify the user of each MA, as a user can have multiple
MAs. Another table is a Connection table, which stores the connection ID, Pi ID, and
MAC address. During measurements the MA can use different access technologies,
either WiFi or Ethernet, and the connection table identifies the connection type
during the measurements. We need to maintain which operation was run, which tool
was used during the measurement, the connection type, the start and end time and
the success/failure, and the Operation table is responsible to storie such information.
The ping, httping, and traceroute tables in the central server have similar design to
the tables in the MA’s, except the additional Pi ID and Op ID attributes.
We want CheesePi to be robust and scalable, the MA was built to store the
measurement data in many formats. As one can see in Figure 4.1, the MA can store
the measurement results in any format based on user’s choice, but in the central
server the collector will organize the data in a format optimised for complicated
queries.
4.6 Sampling of network packet​s
In passive monitoring real data is used to measure the performance of the network,
while active monitoring generates its own data for measurement. Ideally, the
sampled measurement probes, the packet size and the frequency of the
measurements, should represent the real user experience. But CheesePi should not
21
disturb the browsing experience of the user, so active sampling should be as
lightweight as possible, while still gathering useful data. As was mentioned in section
4.2 an experiment was conducted to choose the minimum packet size for network
performance measurement with a minimum extra load on the network. The result
shows that no significant difference in the delay and packet loss for the different
chosen packet sizes, which are the key metrics of CheesePi for network
performance measurement. So the default packet size for a network (ping)
measurement in CheesePi is 64 Bytes, but the user can change the default size.
In the HTTP measurement it is not possible to select the packet size. The only
alternatives are the HEAD and GET HTTP requests. If the httping requests an HTTP
GET, the complete page/file is returned. However, the main purpose of using httping
in this project is to measure the latency and packet loss during the connection, not
the throughput of the web server. So head is used as a default HTTP request option
since its packet size is less than the get one.
4.7 Measurement frequency scheduling
CheesePi runs the user's Internet performance measurement periodically. The
frequency of the measurements and the sampling duration is different for different
applications, so it is important to consider what can happen within the measurement
duration and in the interval between the two successive measurements. In
CheesePi's case, the system is trying to measure the general trend of network
performance at the Transport, Network and Link layers of the TCP/IP stack. If the
interval between two successive measurements is large, it is possible to miss
important performance information, and it is also possible to disturb the user’s quality
of experience if the interval is small.
We conducted an experiment in order to choose a measurement schedule that
balances the capturing of the connection behaviour and not disturbing the user's’
Internet connection. As one can see in Table 4.1 and Table 4.2 First we ran a
continuous one hour network measurement of 3600 probes, and the interval
between each probe was one second (1 X 3600). This is good to capture the
connection behaviour but such large probes can disturb the user’s connection
experience.
We decided to minimize the number of the number of probes, so the next step is
comparing the 60 probes of every minute (first minute, second minute, third minute
and so on) to sample the network behaviour by smaller probes. And as one can see
in Table 4.1, the results show the average network delay and packet loss of every
minute (60 probes) in a five minute measurement is almost similar to the average
five minute (300 probes), which is around 32 ms delay and 0% packet loss. Then we
decide to represent the five minute network performance by one minute
22
measurement. But if the measurement is to several different targets, still the number
of probes can influence the user experience. So, is it possible to sample the network
behaviour with less number of probes? We took the first ten probes of every minute
measurement and we compare them with the average five minute measurement,
and can be seen in Table 4.2. The average delay and packet loss of the ten probes
of every minute is almost the same (around 32 ms and 0%) to the average delay and
packet loss of the total five minute measurement (300 probes). At the end we
decided upon ten probes at an interval of one per second to sample the network
performance over a 5 minute interval.
The path of the network doesn’t change often and we decided to only run the path
(traceroute) measurement once every 15 minutes.
Average ping RTT (ms)
Ping packet loss (%)
1 hour (3600 probes)
32.42
0
first 5 minutes (300 probes)
32.37
0
first minute (60 probes)
32.38
0
second minute (60 probes)
32.36
0
third minute (60 probes)
32.34
0
fourth minute (60 probes)
32.39
0
fifth minute (60 probes)
32.36
0
Table 4.1: Network measurement from an MA in SICS lab to www.bbc.com in
different time intervals.
23
first 10 probes of every
minute
Average Ping RTT (ms)
Ping Packet loss (%)
first minute (10 probes)
32.32
0
second minute (10
probes)
32.30
0
third minute (10 probes)
32.33
0
fourth minute (10 probes)
32.48
0
fifth minute (10 probes)
32.32
0
Table 4.2: First few probes of the first five minutes network measurement from an
MA in SICS lab to www.bbc.com
4.8 Data analysis tools
In this master thesis we used different Python packages for plotting and analysis. For
example, we used numpy package for analysis the results, and matplotlib for plotting
the graphs. All the latency vs. time graphs in this report are smoothed by a window
size of 50.
4.9 Measurement agent's location
All the measurement for this master’s thesis were conducted in Stockholm. Most of
the experiments were done at the SICS lab and student apartments in Kista Galleria
and Lappis. The Internet provider in the student apartments and SICS are
Bredbandsbolaget and SUNET respectively.
Measurement Agent
Connection Type
Location
rpi1
Ethernet
SICS lab
rpi2
Ethernet
SICS lab
rpi3
WiFi
SICS lab
rpi4
WiFi
Student Accommodation (Kista)
rpi5
Ethernet
Student Accommodation (Lappis)
rpi6
Ethernet
Student Accommodation (Kista)
rpi7
Mobile Data (3G)
Home (Stockholm)
Table 4.3 Measurement agent's location and Internet connection type.
24
25
5. Network Performance Evaluation
This chapter will discuss the motivations behind the conducted measurements. The
measurement results gathered from several locations in Stockholm are also going to
be discussed. We are also going to look how the Internet performance of home
users behaves during big media events.
5.1 Measurement motivation
In order to determine the source of a user’s Internet performance degradation, the
quality measurement should consider different access technologies (cable, WiFi,
mobile data) allowing us to determine their differing performance characteristics. We
can then identify if the performance problems are due to the access technology. As
an example, we can measure the Internet performance of two neighbors who have
the same service provider with similar service level agreements, but one of them
uses a WiFi router to access the Internet and the other one is directly connected
through Ethernet. Suppose the network and HTTP measurements are taken from the
neighbours at the same time, if the delay and packet loss rate is high in the user who
uses WiFi technology, then the problem can be narrowed down to the WiFi network.
Further diagnosis such as channel interference, speed limit of the router, contention
and so on can be a separate measurement effort.
Another approach is to determine the Internet performance of users while targeting
different server destinations. In this case, different popular media streaming websites
(servers) are chosen as a destination landmarks for measurement from our MAs.
The target server can influence the user’s Internet performance due to several
reasons:
● The distance (network topological) between the MA and the target server
also. The more the MA is closer (less number of hops) to the target the better
the performance.
● The technology used by the server (Single server or CDN). If the server uses
CDN technology, its performance will probably be more stable. But a single
server’s performance will be highly influenced by a large number of visitors.
● The memory and processing capacity of the target server also has an
influence on the performance. The better processing capacity the server has
the better performance can be obtained.
It is also good to compare the measurements collected from different service
providers. For example, the measurement results of two users from the same
geographic location but with different service providers. We assume the target server
for measurements use a CDN technology, both the users are connected to the
Internet through Ethernet cable, and both of them are located in the same area. If
one of the users has higher delay and packet loss rate compared to the other one, it
26
is highly probable the service provider to be the cause. During this master thesis we
didn’t conduct experiment from home Internet users with different operators (service
providers), but the service providers of SICS and the student accommodation are
different operators.
Finally, we want to observe, how the Internet performance or popular streaming
servers behave during big media events: sports, music contests, etc. To see whether
Internet users experience more during big sport games than any other normal days
on video buffering (i.e. pauses). The streaming servers become very busy serving
the huge number of customers. The performance experienced by the Internet user
during such events depends on the network and how well the server handles the
requests and what technology it supports. Later on this chapter, we are going to
discuss on the measurement results obtained from some popular media events.
5.2 Major Media Events
We were lucky during this master’s thesis, quite a few big media events happened
such as: a boxing match between Mayweather and Pacquiao (‘Fight of the century’),
Eurovision 2015, the Champions League final between Barcelona and Juventus, and
an Open House event at SICS with more than 400 guests (results in appendix).
Apart from SICS Open House, all the big media event measurements were not
conducted with the standard CheesePi tools. As it was explained in section 4.7 we
designed CheesePi to sample the network performance by conducting small number
of measurements, but in the big media events the measurements were continuous
with high frequency (1200 probes per hour), and we were improving the way of
measurements based on the lesson learned from the previous media event. So we
decided to conduct separate measurements for the big media events and at the end
to import the results to CheesePi database (this might not be the ideal approach but I
didn’t want to miss the measurements). Some of the more interesting results of the
events will be presented in this section.
5.2.1 Boxing - Fight of the Century
On the 3rd of May 2015 a boxing match between Mayweather and Pacquiao
happened at Las Vegas. The match started at 04:00 UTC and it took 12 rounds,
which lasted around one hour. The media calls the fight “Fight of the Century”,
reports indicate that Mayweather vs. Pacquiao was the most-watched Pay Per View
(PPV) event of all time [16]. For those who live in USA it was costing $89.95 to order
"Mayweather vs. Pacquiao" and an additional $10 for the high-definition (HD)
experience [17]. In such matches major providers struggle to handle the load of last
minute demand, and this was a good opportunity for us to measure the network
behaviour.
After we heard about this event the next task was to choose which streaming servers
27
should be measured. We choose five landmarks among the listed streaming
websites by a promotional article in The Guardian, and the reasons for selecting the
landmarks can be explained as follows:
● www.hbo.com
● Home Box Office (HBO) was the organizer of the event.
● One of the top American premium cable and satellite television
network.
● www.sho.com
● Showtime (SHO) is one of the top American premium cable and
satellite television network.
● www.philstar.com
● Pacquiao (one of the boxers) is from Philippines, and Philstar is a
Philippine news and entertainment portal for the Filipino global
community
● my.sky.com
● UK's largest pay-TV broadcaster with 11 million customers as of 2015.
● www.viaplay.se
● This thesis is conducted in Stockholm, Sweden and Viaplay is a major
broadcaster in Sweden.
The measurement was conducted at Stockholm in a student apartment and SICS
laboratory. The measurement in most of the MAs started around 2015-05-02 23:10
UTC and ended around 2015-05-03 21:43 UTC.
The ping packet size was the default 64 bytes, but the size of the httping probes
were different for all the landmarks based on server replies for the HTTP head
request. During the measurement the httping packet size for each target was as
follows:
●
www.hbo.com (477 bytes)
● www.sho.com (297 bytes)
●
www.philstar.com (569 bytes)
● my.sky.com (165 bytes)
● www.viaplay.se​ (286 bytes)
During this measurement, we were not checking whether the servers are CDN
technology enabled or not. But after some event measurements we started to
consider CDN as one of the selecting criteria for the target. We were using CDN
Finder ​http://www.cdnplanet.com/tools/cdnfinder/ as a tool to check if the target uses
CDN or not (i.e. single server). According to CDN finder and MaxMind
https://www.maxmind.com/en/geoip-demo the location of the target servers and their
CDN supportability is listed as follows:
28
● www.hbo.com is located in Boardman, Oregon, United States, North America,
and no CDN technology was found behind the hostname.
● www.sho.com located in Amsterdam, North Holland, Netherlands, Europe and
Akamai, a leading CDN service provider, was running behind the hostname.
● www.philstar.com located in Tempe, Arizona, United States, North America
and no CDN technology was found behind the hostname.
● my.sky.com located in Amsterdam, North Holland, Netherlands, Europe and
Akamai is running behind the hostname.
● www.viaplay.se located in Dublin, Dublin City, Leinster, Ireland, Europe and
no CDN technology was found behind the hostname.
Among the five targets, the HBO and Viaplay servers were not responding to ping
requests. So for those two servers, we only conduct httping measurement. It would
have been desirable if we could have run ping tests specifically for HBO, since it was
the organizer of the match , probably has the most customers (at least in the US).
Figure 5.1: HTTP latency to different targets from SICS Ethernet during the boxing
match between Mayweather and Pacquiao.
Except for HBO, the httping latency from SICS Ethernet to all the targets did not
behave differently during the match (4-5 AM). The average httping RTT of a probe
from SICS Ethernet to HBO before 04:27 AM was around 461.6 ms, but after 04:27
till the end of the measurement there were only 21 probes with RTT of less than 500
ms and the average RTT was 580.9 ms, and this is clearly shown in Figure 5.1. This
latency increase is observed in all MAs. As we can see in table 5.1 the average
httping latency to HBO was 479, 582, 515 ms in SICS Ethernet, student apartment
WiFi, and student apartment Ethernet respectively. After the match began, in all MAs
29
the latency started to increase at the same time. After the match began, the average
httping latency increased by 21.6%, 22.5%, and 38.6% for SICS Ethernet, student
apartment WiFi, and student apartment Ethernet respectively and it remains almost
constant till the end of the measurement. This latency increase can be related to
different causes. It could be due to a path change or a congested router on the path
creating the delay. It can be also due to the delay created by load on the target
server. It was my mistake not conducting traceroute measurement during this event,
and this creates complexity in finding out the source of the cause.
RTT (ms) Before
the match (21:00
- 04:00)
RTT (ms) During
the match (04:00
-05:00)
RTT (ms) After
the match (05:00
-22:00)
Academic
environment (SICS,
Ethernet)
478 ± 86
582 ± 26
581 ± 65
Student apartment
(WiFi)
582 ± 364
713.37 ± 86
717 ± 276
Student apartment
(Ethernet)
515 ± 141
714 ± 22
714 ± 104
Table 5.1: Httping latency to HBO from different MAs during boxing match between
Mayweather and Pacquiao.
In order to observe if Boxing had an effect on the obtained performance, I did the
same measurement one week after the match , even though I repeat the same
mistake by not including traceroute measurements. But as it is shown in Table 5.2
and Figure 5.2, unlike the average latency to HBO during the match day, the
average latency to HBO one week later was almost similar in the time before, during,
and after the match, and the latency was also less variable. This tells us the source
of the latency increase that happened during the match day can be the packet
filtering measurement taken by HBO, in order not to disturb the quality experience of
the subscribers (It is a personal guess).
SICS Ethernet
Latency (ms)
Before the match
(21:00-04:00)
Latency (ms)
During the match
(04:00-05:00)
Latency (ms) After
the match (05:00
-22:00)
498 ± 68
499 ± 28
493 ± 47
Table 5.2: HTTP latency to HBO from SICS Ethernet one week later after the boxing
match between Mayweather and Pacquiao.
30
Figure 5.2: Httping latency variability to HBO from SICS Ethernet during and one
week later after the boxing match between Mayweather and Pacquiao.
The HTTP and network latency to my.sky.com and www.sho.com were much smaller
compared to the other targets. A traceroute that taken after this measurement
showed the SHO and Sky servers were located closer to the MAs, the number of
hops in order to reach the destination was less compared to the other targets.
The HTTP latency from SICS Ethernet to Philstar started behave differently around
14:00. The RTT value started to increase in the time between 14:38 and 18:00. Then
later it stopped increasing and relatively stayed stable in the time between 18:00 and
20:28. Finally, the RTT value started to decrease till it reaches the average value
before the change, and it happened between 20:28 and 21:28. Such behaviour is
observed in all MAs, and as it is indicated in Figure 5.3, a similar behaviour is also
reflected in the ping measurement. Since the behaviour is also reflected in the ping
result, the increased latency was likely due to the network. The Philippines time zone
is UTC+8, so the local time at which the latency started to increase was
approximately 22:00, which may be the time people usually start to watch TV. But
did this behaviour reflected one week after the match?
31
Figure 5.3: Network and HTTP latency to Philstar from SICS Ethernet.
As it is shown in Table 5.3, unlike the Philstar latency results during the match the
latency obtained one week later was relatively smaller and consistent, especially in
the time between. So the latency increased to Philstar during the match day could be
the congestion created on the links before the target server due to the replay of the
match or there was some other live TV program.
Httping
latency (ms)
(23:00-14:00)
Httping
latency (ms)
(14:00- 22:00)
Ping latency
(ms)
(23:00-14:00)
Ping latency
(ms) (14:0022:00)
During the
match
374 ± 87
484 ± 155
175 ± 8
227 ± 37
One week
later
352 ± 37
352 ± 48
170 ± 0.5
170 ± 0.7
Table 5.3: Latency to Philstar from SICS Ethernet during and one week later after
the boxing match between Mayweather and Pacquiao.
The measurement results for HBO in Table 5.1 showed that the latency before the
match from the Student apartment connected by a WiFi is higher and greater
variance than the latency from the MA in student apartment connected via a cable.
And it is similar for all the other targets. The laptops and smartphones of the two
students who live in the apartment were connected to the Internet through WiFi. The
contention and channel interference in the WiFi network likely created a higher
latency compared to the Ethernet connection.
32
HBO
SHO
Sky
Philstar
ViaPlay
SICS
Ethernet
544 ± 87
79 ± 130
24 ± 62
418 ± 128
113 ± 70
Student
appr.
Ethernet
649 ± 148
75 ± 105
19 ± 86
351 ± 227
105 ± 71
Student
appr. WiFi
668 ± 313
108 ± 233
48 ± 221
No data
collected
144 ± 232
Table 5.4: HTTP latency comparison among different MAs during boxing match
between Mayweather and Pacquiao.
In the Boxing event we observed, the CDN enabled streaming websites are rarely
affected by the match. While those which are not CDN enabled, showed different
characteristics during the match day. The MA from the student Ethernet experienced
better latency than the MA connected by WiFi. The network latency from the
wirelessly connected MA was 22.8% larger than the latency of the MA connected by
cable, and also less variable.
5.2.2 Eurovision 2015
The next big event we found was Eurovision song contest. Eurovision is the longest
running annual TV song competition held, primarily, among the member countries of
the ​European Broadcasting Union (EBU) since 1956. It is also one of the most
watched non-sporting events in the world. For example Eurovision 2015 was seen by
around 197 million viewers [18]. Vienna was the host of Eurovision 2015 and the
contest was held in 3 stages (semi-final one, semi-final two, and grand final) on 19,
21 and 23 of May 2015, respectively.
In this event, we measured from different places in order to get a variety of locations
and access technologies. One good aspect of this event was the three stages of the
contest, we got a chance to adapt and rethink the measurement approach for the
grand final contest. For example, the HTTP was crashing during the continuous high
frequency measurements in the first and second semi-final contests, and so were
were able to follow a better approach for the final contest.
5.2.2.1 Semi final one
The Eurovision 2015 semi-final one was on 19th of May 2015. It started at 19:00
UTC and ended at 21:00 UTC. In most of the MAs the measurements started and
ended at 17:00 and 23:59 UTC respectively. As it was mentioned before the
measurement was conducted from different places (team members apartment and
SICS lab) and the event was not during the working hours. So there was a bit
33
difference in the starting time of the measurement, but it does not affect the analysis
since all measurement nodes cover the whole events.
Choosing targets for the measurement is tricky because it was not easy to identify
which website users would use to watch the event. We tried to look for popular
streaming websites of many participant countries, and also asked friends to get
suggestions and check the popularity of the chosen targets. The semi-final one
measurement targets were the following:
● www.eurovision.tv - a European broadcasting union’s streaming website for
Eurovision. We choose this target due to its popularity and it was also an
organizer of the event.
● www.svtplay.se - SVT (Sveriges Television AB), a Swedish public TV
broadcaster. The measurement is conducted in Stockholm (Sweden), and
Eurovision is very popular in Sweden. So we decide to include a popular
streaming website which broadcast Eurovision in our targets.
● tvthek.orf.at - Österreichischer Rundfunk (ORF) is an Austrian national public
broadcaster. The reason we chose the target was: its dominance in Austrian
broadcasting media and Austria was also the host of the event.
In order to compare and analyse the results, it is good to know the location of the
target servers and the technology they support. The location of the target servers
and its CDN technology supportability was as follows:
● www.eurovision.tv - CloudFlare, one of the top CDN providers, distributes the
contents of the target. The target server during the measurement was located
in Stockholm, Sweden.
● www.svtplay.se - Akamai distributes the content of SVT and the target server
during the event was located in Stockholm, Sweden.
● tvthek.orf.at - According to CDN Finder and MaxMind, ORF does not use
CDN technology to distribute its content and it is located in Austria.
The ping and httping were running in parallel and the interval between each probe
was three seconds. The packet size of a ping probe for each target was 64 bytes,
and 479, 206, and 477 bytes for each httping probe of EurovisionTV, SVT, and ORF
respectively.
34
Figure 5.4: HTTP latency to different targets from an MA at SICS (Ethernet) during
Eurovision 2015 semi-final one.
From the previous boxing event, we observed that performance from the Ethernet
connection was better than the performance from the WiFi connection. So we
decided to first start looking at the results for each target from the MA at SICS lab
connected to the Internet via a cable. As it is shown in Figure 5.4 and Table 5.5 the
HTTP delay of SVT was consistently small between 6 to 8 ms with a small variance
but there were very few large ( > 205 ms ) delays during the contest in the time
between 20:00 and 21:00, and after 23:00. The latency of ORF and EurovisionTV
vary a lot especially ORF. From the SVT delay results we can understand that the
CDN server performance was good, there was no congestion in the path between
the MA and the target. Next we are going to look at EurovisionTV and ORF results.
EurovisionTV
SVT
ORF
Mean Medi
an
Stdv.
Mea
n
Medi
an
Stdv
.
Me
an
Medi
an
Stdv.
Before
Program
(17:00 - 19:00)
23.87 8.44
203
7.83
6.43
5.68
181
94
865
During
Program
23.75 8.31
241
17
8.09
205
147
94
591
35
(19:00 - 21:00)
After Program
(21:00 - 00:00)
8.35
7.97
1.7
11
7.84
93
370
93
1772
Whole
Measurement
(17:00 - 00:00)
17.21 8.17
169
12
7.67
125
251
94
1287
Table 5.5 Comparison of HTTP latency from SICS Ethernet to different targets
before, during, and after the program of Eurovision semi-final one.
Figure 5.5: HTTP latency from a MA at SICS (Ethernet) to EurovisionTV during
Eurovision 2015 semi-final one.
The HTTP delay of EurovisionTV from SICS Ethernet started to vary many times
from 17:20 till the end of the contest (21:00) as it is clearly shown in Figure 5.5 and
Table 5.5. This tells the event affected the HTTP delay to EurovisionTV to some
extent. But we can not be certain the increase in the RTT value was due to the delay
created by the server or traffic created on the network. So to get an idea about the
causes we should compare the network delay with the HTTP delay (network and
server’s computation latency).
36
Figure 5.6: Comparison of Network and HTTP delays from a MA at SICS (Ethernet)
to EurovisionTV during Eurovision 2015 semi-final one.
The network delay was 1.84 ms with a tiny variation of 0.53 ms, as it is clearly shown
in Figure 5.6 the delay was small and consistent. The larger network delays mostly
happened a few minutes after the contest started with peaks 44.44% larger than the
average network delay during the whole measurement. While the HTTP delay was
varying before and during the contest. As it is shown in Figure 5.6 and Table 5.5 the
deviation before and during the contest was around 10 times the average delay.
Many of the peaks were more than five times the average latency of the httping
measurement to the target. So the results indicate the large HTTP delays were due
to the delay created in the server. ​But what about the large HTTP delay variance to
ORF?
37
Figure 5.7: HTTP latency from a MA at SICS (Ethernet) to ORF during Eurovision
2015 semi-final one.
There are many servers which do not respond to a ping request, and ORF’s server is
one of these servers. It is difficult to draw a conclusion based only on HTTP
measurements. But since httping results are the performance of the whole TCP/IP
stack, it is possible to have a general understanding of the connection’s
performance. As one can see in Figure 5.7 the HTTP delay to ORF are peculiar. It is
common for the latency to increase and fluctuate a lot while watching a big media
event from a website which uses a single server to handle all the user requests. But
the average HTTP latency (147 ms) and its variance (591) during the contest was
relatively better compared to the latency and variance before (181 ms, 865 ms) and
after (370 ms, 1772 ms) the contest. When we looked the path based measurement
results to find the cause, we found that, the IP address of the target was keep
changing between 194.232.116.172-176. During this measurement we did not
include “-r” parameter in the httping command, which makes it only resolve the
hostname once at the start. This tells we were measuring five different ORF servers
and this might be the cause of the frequent and large latency variation. So ORF was
using several servers for load balancing to handle user request, but still it does not
save users from experiencing larger delays for some moments.
38
Figure 5.8: Comparison of HTTP latency to EurovisionTV from different MAs during
Eurovision 2015 semi-final one.
We compared the latency experienced by different MAs. The HTTP delay to SVT
and ORF experienced by the MAs in SICS lab and the student accommodations is
more or less similar except some higher delays for the wirelessly connected MAs.
But as one can see in Figure 5.8 the HTTP latency experienced by SICS lab and the
student accommodations is different. The middle subplot of Figure 5.8 shows, the
HTTP delay from both the students apartment have a similar pattern and an average
latency. This is due to both MAs served by the same operator, and after the second
hop the packets follow a similar path to reach the target. However, there is a big
latency difference between the results obtained from SICS lab and the student
apartment. Even though the traceroute shows both of them have almost similar
number of hops, SICS uses Nordnet network which is directly connected to the
designated edge CDN server (Stockholm), whereas due to the students
accommodation ISP routing policy, the packets from the student apartment were
leaving Sweden network to Norway and came back to Sweden network. So the main
reason for the large latency difference from the student apartment and SICS lab was
the delay created by the path followed to reach the target.
The next step was to look how the event affected the latency in the student
apartment and comparing the latency between the WiFi and cable connection. The
average HTTP delay to Eurovision from both the Ethernet and WiFi connected MAs
in the time from the beginning of the measurement till the end for the contest (17:00 21:00) was 311 ± 446 and 324 ± 396 ms respectively. This latency is much higher
and deviates more compare to the average HTTP delay experienced after the
contest (21:00 - 00:00) by both the Ethernet and WiFi connected MAs, which is 90 ±
39
89 and 109 ± 133 ms respectively. This behaviour is clearly shown in Figure 5.8.
Such pattern is also reflected in the network measurement, so this indicates there
was higher traffic in the network during and few hours before the contest. We will see
in the next contests if this latency variation was due to the Eurovision effect on the
Telenor network that goes to the target. From the presented results the Ethernet
connection in the student apartment was as bad as the WiFi in terms of delay and
deviation.
5.2.2.2 Eurovision 2015 semi-final two
The Eurovision 2015 semi-final two was held two days after the semi-final one on
21st of May 2015. It started at 19:00 UTC and ended at 21:00 UTC. In most of the
MAs, the measurements started and ended at 16:05 and 23:05 UTC respectively.
During this contest, we add one more MA which used 3G to connect to the Internet.
All the targets were the same targets as the semi-final one.
Figure 5.9: Comparison of HTTP latency to SVT among different MAs during
Eurovision semi-final two.
Before the contest, the average HTTP delay to SVT from the student
accommodation in Kista with an Ethernet connection was 16 ± 320 ms, the large
deviation is due to a single large delay that happened in 18:19 which was around 19
seconds. The average HTTP delay of the SICS Ethernet was 15.4 ± 171 ms which is
40
almost similar to the student accommodation. But in the student accommodation
during the contest at around 19:54 the HTTP delay increased by large amount and
continued like that till 20:38, the average delay during that time was 215 ± 115 ms.
As one can see in Figure 5.9, such behaviour was also observed in the WiFi
connection of the student apartment. Similar case but at different time also
happened another student accommodation in Lappis, which started at 19:12 and
ended at 19:36. smooth except for some moments during the contest in rpi4, rpi5,
and rpi6 (the MAs in student apartments). In all the MAs, the cause for the delay
increase was a continuous 6 packet loss, and after that the target IP address
changed and the latency increased by more than 15 times. The average HTTP delay
from the 3G connection was three times than the Ethernet connection in the student
apartment.
Figure 5.10: Comparison of HTTP latency to EurovisionTV among different MAs
during Eurovision semi-final two.
In the MAs at the student accommodation the HTTP delay to EurovisionTV started to
increase when the contest was about to begin and it started decreasing after the end
of the contest. And becomes relatively stable after 22:00, which is similar to
Eurovision semi-final one. During the HTTP delay measurement to EurovisionTV, the
MA with a 3G connection experienced a better average delay and deviation (85 ±
41
115 ms) compared to the MA at the student accommodation (148 ± 194 ms). This is
clearly shown in Figure 5.10. The latency increase pattern observed in both
Eurovision semi-final one and two in the measurements from the student
accommodation to EurovisionTV will be analysed later in the next sections.
5.2.2.3 Eurovision 2015 Final
The Eurovision 2015 grand final was held on 23rd of May 2015. The program started
at 19:00 UTC and ended at 23:00 UTC. The voting started at 21:14 and ended at
21:40 UTC. The result announcement started at 21:52 and ended at 22:49 UTC. In
most of the MAs, the measurements started and ended at 16:30 and 04:00
respectively.
During this contest we wanted to make some changes in the targets due to ORF
website was not streaming the grand final, and both EurovisionTV and SVT were
using CDN technology. Thus we decided to remove ORF and add more targets, for
example, SBS (Special Broadcasting Service), an Australian public broadcasting,
and the BBC’s iPlayer. We added SBS because it was the first time for Australia to
participate in Eurovision and SBS is one of the top broadcasting services in
Australia. We dedicated a MA at SICS with an Ethernet connection, to measure the
performance by probing to SBS, BBC iPlayer, and BBC UK. The other MAs were
probing to SVT, BBC UK and EurovisionTV.
Figure 5.11: Network and HTTP latency of different targets form SICS Ethernet
during Eurovision 2015 final.
42
As one can see in Figure 5.11, except the large HTTP delay (5042 ms) at 21:12, the
network and the HTTP delay to BBC iPlayer from the SICS Ethernet was consistent,
which is 9 ± 46 ms and 1.6 ± 0.3 ms average HTTP and network delays respectively.
It was also similar for the SVT measurement but when the contest started (21:00 21:10) five large delays with an RTT value more than 5 seconds are recorded. This
might be related to large number of users request, related with the beginning of the
voting. The HTTP delay to EuroVision (8 ± 21 ms) was deviating less after 21:00
compared to the measurement before 21:00 (16 ± 158 ms).
Figure 5.12: Network and HTTP latency of different targets form SICS Ethernet
during Eurovision 2015 final.
The SBS server was located in Australia and the network and HTTP delay was large
356 ± 3 ms and 713 ± 182 ms respectively. Out of 12600 httping probes to SBS 90
of them recorded with more than 2000 ms delay. No special pattern was observed
during the contest, except 3 probes with more than 5 second HTTP delay. But this
large delay can affect the quality of different online applications. In the
measurements to SBS we observed, the more the target is far from the client in the
Network topology the more the performance degraded. We also compared the
latency between BBC UK and the BBC iPlayer. The HTTP delay of BBC iPlayer (12
± 111 ms) was smaller compared to BBC UK (78 ± 65 ms). The small delay of BBC
43
iPlayer is due to it uses Amazon CloudFront for distributing its content. This is clearly
shown in Figure 5.12.
As one can see in Figures 5.13 and 5.14 the HTTP delay to BBC iPlayer and
EurovisionTV from the student apartment is higher than the HTTP delay from SICS
lab. As we previously discussed in Eurovision semi-final one, it is due to the routing
policy of the service providers. In the SICS network the DNS server returns the
Amazon CloudFront server nearest to the MA and the packets were reaching the
target after few hops. However the DNS server for the student accommodation was
returning different Amazon CloudFront server, which is located in Amsterdam
(according to the DB-IP Database) which can not be reached close to the MA. From
this delay comparison we can understand that, the routing policy of the service
provider or its agreement with CDN providers affects the user’s Internet
performance.
Figure 5.13: Comparison of HTTP delay to EurovisionTV from different MAs during
Eurovision 2015 final.
However the HTTP delay to SVT from the student apartment (16 ± 102 ms) is similar
to the HTTP delay from SICS lab (14 ± 171 ms). In this case, even though both the
DNS servers in the student accommodation and SICS network return different CDN
servers and did not share common path to reach the target, the probes from both
networks are able to reach the target quickly. This might be due to 108,507 Akamai
servers deployed worldwide in over 110 countries and within more than 1,400
44
networks around the world [35]. The distribution of the CDN servers responsible for
distributing the streaming websites content and the agreement between the ISP
(operator) and the CDN provider influence the Internet performance of a user.
Figure 5.14: Comparison of HTTP delay to BBC iPlayer from different MAs during
Eurovision 2015 final.
5.2.2.4 One Week After Eurovision 2015 Final
In order to understand the effects of the big media event on the performance of the
Internet quality, we repeated the experiment exactly one week after the Eurovision
2015 final to observe if some of the performance changes during the measurement
were due to the media event. The time the measurement conducted, the places, the
targets, and the frequency of the measurement was the same as the Eurovision
2015 final.
The red graphs in the top part of Figure 5.15 are the HTTP delay of different targets
from the SICS Ethernet connection during Eurovision 2015 final, while the bottom
measurement graphs in blue are the HTTP delay to the targets after one week.
During the Eurovision Final the average HTTP delay to SVT was (14 ± 171 ms), the
HTTP delay of 87.12% of the probes were less than 10 ms, and for the
measurement conducted one week after 87.85% of the probes recorded less than 10
ms HTTP delay. There were few peak points observed around the beginning of the
program (19:00) and voting time (21:15) which did not happen in the measurement
conducted after one week. So the graph tells us the large latency of the few probes
45
may indeed be due to the load on the web server or the link during large number of
users around the mentioned time. But the average latency in both measurement is
almost similar, which is 14 ± 171 ms and 11 ± 60 ms for the Eurovision Final and the
one week after measurements respectively .
Figure 5.15: Comparison of HTTP delay from SICS Ethernet to different targets
during Eurovision 2015 final and one week after the final.
The average HTTP delay to BBC iPlayer from SICS Ethernet during the Eurovision
Final was 10 ± 46 ms. However, when the voting was about to start there was a peak
with a latency around 250 ms and we initially thought it was due to an increase in
number of visitors at that time. In the measurement conducted one week later, until
18:07 the average HTTP delay was 11.9 ms but suddenly it increased to an average
of 220 ms and stayed for about 2 hours till 20:03. We try to analyse the reason
behind the sudden latency increase. First we checked if it was due to continuous
packet drop or a target IP address change, but there was no packet loss and the IP
was changing frequently before the latency started to increase. When we checked
the traceroute, the path was changed at the moment the latency started to increase.
During the smaller latency the httping packets were reaching the target after few
hops with a very small RTT. However during the higher latency period the number of
hops increased because it was following a different route. When the path to the
target returned to the previous path the latency decreased closer to its previous
average delay (26 ms). This artifact is clearly shown in the bottom right plot of Figure
5.15. The cause of the sudden latency increase may be due to target IP address
46
changing its load balancing. The new path for reaching the target requires more time
and affects the end to end delay.
The httping latency from SICS Ethernet to EurovisionTV during the Eurovision Final
has fewer peak points than the experiment conducted one week after the final. For
example, in the measurement during the Eurovision Final there were 29 HTTP
probes recorded with delay greater than 100 ms, whereas one week later there were
60 probes with HTTP delay of more than 100 ms. During the Eurovision Final
measurement we thought the high latencies were due to users visiting EurovisionTV
to check the programs and watch the contest at the time before the contest started
and as the program started. But the measurement that conducted one week after
Eurovision Final shows the high latency were probably not due to the event. The
high latency in the time between 16:30 and 22:00 can be due to network congestion
or the CDN server is relatively busy everyday during the given time interval. In order
to determine the cause of the behaviour we should look at the ping latency from
SICS to EurovisionTV and other MAs’ httping latency. This helps to check if the
behaviour (pattern) is due to the characteristics of the network, or the target server.
(a) From SICS Ethernet.
(b) From Student apartment WiFi.
Figure 5.16: Comparison of HTTP latency to EuroVisionTV from student apartment
WiFi and SICS Ethernet between the measurements during Eurovision Final and one
week after the final.
As one can see in Figure 5.16 (a), the network latency to Eurovision TV from SICS
Ethernet during Eurovision final and one week later showed low variation, as the
coefficient of variation (the standard deviation / mean) is less than 1, a thumb rule for
the variation in dimensionless values, given the measured values of 1.77 ± 0.4 ms
and 1.72 ± 0.28 ms respectively. So it is unlikely the network was the cause of the
47
previously observed large httping delays. But if we look at HTTP latency from
student apartment (WiFi) to EurovisionTV, in both measurements the latency is
higher in the time between 16:30 to 22:00. As it is clearly shown in Figure 5.17 this
behaviour is observed in all EuroVision measurements. So this higher latency in the
time between 16:30 and 22:00 could be due to the load on the CDN server everyday
during the specified time, or it may give a priority to other tasks instead of HTTP
head request packets.
Figure 5.17: HTTP latency from student apartment to EurovisionTV during different
measurements.
5.2.3 2015 UEFA Champions League Final
The UEFA Champions League, known simply as the Champions League, is an
annual continental club ​football competition organised by the ​Union of European
Football Associations (UEFA) and contested by ​top-division European clubs​. The
2015 UEFA Champions League final was the final match of the ​2014–15 UEFA
Champions League​, and it was played at the ​Olympiastadion in ​Berlin​, Germany, on
6 June 2015, between the Italian side ​Juventus and the Spanish side ​Barcelona​. A
massive global audience on TV and social media confirmed the UEFA Champions
League final's status as the world's most watched annual sporting event. We thought
this big media event would create a behavioural change in the Internet quality of
users, and we conducted measurements during the event to observe how it affect
the Internet performance of users.
48
It started at 18:45 UTC and ended at 20:45 UTC. And the trophy ceremony ended at
21:00 UTC. In most of the MAs the measurements started and ended at 16:00 and
04:00 (the next day) respectively. From the previous measurements we learned
targets which use a CDN technology are less affected by big media events. So
during this measurement we tried to focus more on popular free streaming targets
which do not use CDN technology. And we select four targets: one subscription
based websites, and three free streaming websites which do not use CDN
technology. Unfortunately we realized later two of the free streaming targets were
using CloudFlare to distribute their content. The chosen targets were the following:
● www.espnplayer​.com - ESPN (Entertainment and Sports Programming
Network) is a U.S.-based global ​cable and ​satellite ​television channel owned
by ​ESPN Inc​. We choose ESPN as our target because as of February 2015,
ESPN has approximately 94,396,000 subscribers, and it was one of the
Champions League final broadcasters. The crashing during most of the world
cup 2014 watched live streaming [21] was another reason for choosing ESPN.
It uses Akamai to distribute the content on large live events but during the
Champions League final the target server was not an Akamai server. The
server was located in USA.
● www.​cricfree.sx - CRiCFREE is a popular free sport streaming portal website.
According to [20] it has millions of visitors every month. We chose the target
to observe how free streaming websites handle user requests during big
media events. It uses CloudFlare to distribute its content and the closest
server was in UK.
● www.crichd.in - CricHD is also a poplular free sport streaming portal. After we
conducted the measurement, we realized it is another CRiCFREE with
different domain name with a small changes in the front page of the website.
● www.atdhe.ws​ - ATDHE is a another popular free sport streaming portal. It
doesn’t use a CDN technology to distribute the content and the target server
is located around Netherlands.
In some of the previous httping measurements we had faced difficulties with some
conclusions due to frequent IP address change of the target. During this
measurement we include the “resolve hostname once” parameter in the httping
command, so the measurement was only to one web server. This is helps the
measurement to be more stable when compared to a measurement to a hostname
with several IP addresses for load balancing.
Based on the knowledge of networking and communication systems, when large
number of users visit a specific server at the same time, the server becomes busy
handling all the requests and creates high traffic over the link to the target. So during
a big media events we expect there will be higher latency and packet loss during the
match. Now let’s look at the result obtained from different MAs during the match.
49
First we should see how the Champions League final affected the performance
measurement to ATDHE. Before we look on all MAs, let’s look the results collected
from SICS Ethernet. Figure 5.18 shows the latency of the HTTP and the network,
and its relations to the ping packet loss. The red line graph in the figure represents
the HTTP latency, the blue graph in the middle subplot represents the network
latency, and the green graph in the bottom subplot of the figure represents the
network packet loss rate in every five minute across the whole measurement. The
HTTP delay was 94 ± 19 ms. However when the match was about to start at 18:22,
the HTTP delay variance increased (104 ± 84 ms) and continued until 20:30. The
coefficient of variance for these two events are 0.2 and 0.8 respectively, which again
captures, somewhat simply, the change in the experienced network service. This
was just before and during the match, which is similar to our expectation. Then it
went back to its previous “state” (94 ± 36 ms), and there were few spikes between
21:30 and 22:30 as one can see in Figure 5.18.
Now, let’s look at the ping latency and packet loss to try and understand the cause of
the latency (change). A network latency of 44.7 ± 0.9 ms throughout the whole
measurement phase except for four probes with delays between 100 and 104 ms,
could be taken as “stable”. There were also few ping packet losses throughout the
whole measurement. From the results we understand the latency increase during the
match are due to the latency created by the server, because the behaviour observed
in HTTP delay measurement was not reflected in the network measurements. So this
can be due to large number of visitors to the ATDHE website during the Champions
League final, and this load on the single server created the HTTP delay increase.
This artifact is clearly shown in the top subplot of Figure 5.18.
50
Figure 5.18: Network, HTTP, and network packet loss relation of measurement from
SICS Ethernet to ATDHE during UEFA 2015 Champions League Final.
From the previous measurements we observed that some behavioural patterns
observed only on specific Networks (MAs). So let’s look if the HTTP delay to ATDHE
was larger during the match in all other MAs. As it is shown in the blue graph of the
top subplot in Figure 5.19, the SICS WiFi also experienced higher latencies (110 ±
142 ms) around the time the match took place from 18:35 - 20:45. Whereas, the
average HTTP delay before the match (16:00 - 18:35) was (96 ± 28 ms). The WiFi
and Ethernet connection of the student apartment also experienced higher httping
delay during the match but there were also latency spikes every few minutes. During
the measurement to ATDHE, the student apartment experienced lower HTTP delay
(59 ± 93 ms) compared to the delay from SICS lab (96 ± 48 ms). According to the
traceroute results, the student apartment packets were passing through fewer hops
to reach ATDHE server compared to the packets from SICS. But as it is shown in the
middle and bottom subplots of Figure 5.19 the HTTP delay from the student
apartment was deviating more both in the WiFi and the Ethernet environments,
especially during the match. For example, during time 18:35 - 21:00 the HTTP delay
was 79 ± 87 ms and 65 ± 118 ms for the WiFi and Ethernet respectively. We doubt
this latency variation was due to the network congestion on the path, and thus we
decided to look the latency of ping from the student apartment and SICS lab.
51
Figure 5.19: HTTP latency of different MAs to ATDHE during UEFA 2015 Champions
League Final.
The first subplot of Figure 5.20 shows the network latency from SICS WiFi and
Ethernet to ATDHE. The network latency of the Ethernet was very smooth (44 ± 0.9
ms), while the latency from the WiFi had few larger delays (47 ± 10 ms). This tells us
the latency on the network was very stable while there were some delay increase
during the match and due to the WiFi. The network results tell us, the larger HTTP
delay during the match was not due the network congestion, it is due to the load on
the single ATDHE server.
As it is shown in the middle and bottom subplots of Figure 5.20 the student
apartment experienced less ping latency than SICS but there were a lot of spikes
throughout the whole measurement, especially from the WiFi. These spikes from the
WiFi might be due to high wireless-medium contention. Two students were watching
the match and a movie using wirelessly connected laptops, and there were also two
other smartphones connected to the Internet through the WiFi. The contention
among wireless devices and the channel interference from neighbor routers, who
use the same channel as the WiFi router of the student apartment, could be the
cause of the spikes.
52
Figure 5.20: Network latency of different MAs to ATDHE during UEFA 2015
Champions League Final.
We have examined how the Champions League Final affected the measurement
performance to a popular server which does not use CDN technology to distribute its
content. What about the targets which use CDN technology to distribute their content
and ESPN (supposed to use a CDN)?
Figure 5.21: Httping latency of different MAs to CricHD during UEFA 2015
Champions League Final.
53
Even though CricHD uses CDN technology the average HTTP delay from every MA
was more than 200 ms. This is shown in all the subplots of Figure 5.21. The CDN
server, which was handling the HTTP request to CricHD, was located in UK and the
average network (ping) delay from all the MAs in Stockholm was around 40 ms. We
did not expect a big difference among the ping and the httping latencies since
CricHD uses CDN (CloudFlare), however, the average latency was 160 ms more.
These httping latencies were consistent and never went below 180 ms for any of the
MAs. It can’t be also due to the packet size (326 Bytes), because we tried the ping
measurement with different packet size up to 526 Bytes but the difference was not
significant. It seems there may have been some load balancing or reverse proxy
behind the CDN server that makes the HTTP delay this much larger.
Beside the consistent large HTTP latency from SICS to CricHD there were also
some spikes a few minutes before and during the match. While the latency from
student apartment to CricHD was higher (301 ± 461 ms) during the match (18:35 21:00) compared to the average HTTP delay (220 ± 363 ms). We try to figure out if
the latency increase is due to the network or the target server. As it is shown in the
middle and bottom blue subplots of Figure 5.22 the ping latency to CricHD from the
student apartment started to increase at around 17:00 and became higher (66 ± 13
ms) between 18:35 and 21:20 (during the match) compared to the average network
delay (42 ± 17 ms) and the delay after 22:55 (28 ± 1.5 ms). This tells the larger delay
observed in the HTTP performance measurements are influenced by the network
performance. Then we tried to find which network link is creating the latency by
looking the traceroute results. The router in Norway “tiscali-1.ti.telenor.net” was
creating these larger delays. It might be because the router was busy at that time or
there were some priority to other packets. When the latency to the
“tiscali-1.ti.telenor.net” started to decrease and later became stable, the ping latency
to CricHD also became stable. So it seems a specific router on the path can strongly
influence the end-to-end delay.
54
Figure 5.22: Network latency of different MAs to CricHD during UEFA 2015
Champions League Final.
Some unofficial video stream portals use different domain names with a small front
page, designed to keep streaming if the main domain of the website is seized by
intellectual property protection organizations. In our case, CRICFREE and CricHD
looked the same, and providing similar services with a slight difference on their front
page. We also trace the route to both targets, and they follow the same path till the
last hop to the target. CRICFREE also uses CloudFlare to distribute the content, and
the closest server to our MAs was located in the UK similar to CricHD. As it is shown
in Figure 5.23 the general behaviours observed during httping and ping
measurements to CricHD also happened while measuring to CRICFREE. However,
the average HTTP delay to CRICFREE was smaller compared to the CricHD HTTP
delay. For example the average HTTP delay to CRICFREE from student apartment
of Ethernet connection (147 ± 167 ms) was smaller than the average HTTP delay to
CricHD (220 ± 363 ms). In most of the MAs, the average HTTP delay to CRICFREE
was less than 150 ms. This indicates, the delay created in the CRICFREE server
was smaller compared to the delay created by CricHD server.
55
Figure 5.23: HTTP delay of different MAs to CRICFREE during UEFA 2015
Champions League Final.
Even though ESPN is one of the top on-demand streaming services, the HTTP and
the network delays were large. The HTTP and network delays from all MAs were
very similar, and it was around 235 and 112 ms for the HTTP and the network
respectively. Figure 5.24 shows, the HTTP delay from the SICS Ethernet and WiFi
connections were 231 ± 70 ms and 237 ± 180 ms respectively. And in the student
accommodation the HTTP delay was 229 ± 87 ms and 246 ± 101 ms from the
Ethernet and WiFi connections respectively. As one can observe, the deviation was
larger in the wirelessly connected MAs. In the student apartment WiFi there were
continuous periodic spikes. We tried to find out the causes of the periodic spikes,
and we tried to compare the HTTP and network latencies. As it is clearly shown in
Figure 5.25 the periodic latency increase is in both ping and httping. This means,
something was periodically happening in the network. In order to find the cause we
have to look the ping results carefully.
When we looked the ping results the average network delay was around 113 ms
before the periodic increase, but every multiple of 5 minutes the delay increased. For
example, the RTT was going constantly around 113 ms then at 5th minute it became
500 ms and return to its previous state, then again after multiples of 5 minutes it
increases. We tried to examine the ping measurements to other targets from the
student apartment WiFi. And the behaviour is observed in all ping measurement of
different targets. We did not observe such a behaviour in the results from the
Ethernet connection. So this periodical delay increase may be due to the periodic
56
PSM (Power Saving Mode) of the WiFi dongle in the MA. When using PSM the
wireless interface is put in a sleep mode when the device is not transmitting a data. If
the Access Point (WiFi router) receives data of the client while the client was in a
sleep mode, it buffers the data until it receives the next beacon from the client. So
such periodic WiFi interface sleep mode of the MA can be the cause of the periodic
large RTTs. And in a Raspberry Pi the PSM is enabled by a default.
Figure 5.24: HTTP latency of different MAs to ESPN during UEFA 2015 Champions
League Final.
One can only watch the match on ESPN if one is a subscribed customer. We think
the measurements to ESPN may not represent the traffic during the match at that
time. Because, our target was “​www.espnpalyer.com​”, which was located in US, but
for a subscribed customer it would be different server located closely. So we can not
say the performance observed on the measurements to ESPN during the
Champions League Final are the performance experienced by the customers.
57
Figure 5.25: HTTP and network latency to ESPN from student apartment WiFi during
UEFA 2015 Champions League Final.
58
6. Discussion
WiFi experiences medium contention and so can be slower and more variable than
Ethernet. Our measurement results confirmed and quantified customers with an
Ethernet connection experienced lower latency and packet loss compared to WiFi
users. In most of the measurements the SICS lab achieved better performance in
terms of latency compared to the student apartment. SICS uses SUNET, which
enables Swedish universities and institutions to connect with a well-developed and
effective national and international data communication and related services. And
the student accommodation uses Bredbandsbolaget ISP owned by Telenor.
Before the large media event measurement results were analyzed, we expected that
users who connect to a website on a CDN would receive data with a lower delay
than a non-CDN solution. However our experiments showed the effectiveness of
CDN technology on minimizing the end to end latency depends on how close the
server is located to the customer, and the functioning of the Internet service
provider’s routing policy (or possibly the CDNs). If the designated edge server is very
close (in number of hops and geographically), and the path chosen to reach the
desired domain is short, the latency is low. But if the path chosen to reach the target
is long, the latency will be highly dependent on the network status. Such behaviour is
observed during measurement to EurovisionTV from the SICS lab and student
apartment during Eurovision.
During our httping experiments to a CDN host, we observed a situation where
significant packet loss occurred when the IP address of the target domain changed
to a different edge server. This happened during the second Eurovision semi-final,
as part of the measurements from a student apartment to Swedish Television (SVT).
After six contiguous packet losses the edge target changed, which required a more
hops to be reached by the measurement agents (MA). The HTTP delay increased by
more than 15 times, and all the ping packets were lost after the IP address change.
The ping packets lost were due to the hostname only being resolved once at the
beginning of the experiment. In the student accommodation (Kista), this large delay
continued for more than 40 minutes until the target address returned to its previous
(edge) CDN server. This show how changes made in the network have a great
influence on the latency. The target IP address changed and caused communication
to be routed to a different destination IP. ​Following the new path to communicate
with the target may require more time and this affects the end to end delay. Such a
similar case also happened during measurement to the BBC iPlayer, one week after
the Eurovision final. And the average HTTP delay increased from 11.9 ms to 220 ms
59
and this continued for approximately two hours. We can only speculate at this point
on the reasons, possibly technical failure, possibly load balancing, but it would
require further studies.
It is common to observe some patterns in streaming websites with a single server.
For example, if we do an HTTP measurement to a website which streams a popular
TV show (“Breaking Bad”) on Sunday between 21:00 - 21:50 using a single server,
the HTTP delay on Sunday between 20:45 - 21:55 can be larger compared to the
HTTP delays on a different time. The HTTP delay increased because of the greater
load due to a large number of visitors during that time. But most of the popular
streaming websites, which use CDNs, are capable of handling a large number of
requests, and it is very rare to observe such delays. However during our
measurement to EuroVisionTV which uses CDN, during the time between 16:30 and
22:00 in all Eurovision measurements a higher HTTP delay (for example, 143 ± 136
ms during Eurovision Final) was observed compared to the HTTP delay after 22:00
(for example, 64 ± 33 ms during Eurovision Final). The pattern is not reflected in the
ping measurement and it is most likely cause by the server load. Even though It is
difficult to precisely know the source of such a pattern since a measurement is not
conducted in the server-side, but it can be due to the high load on the CDN server
everyday in the specified time or it may give priority to other tasks instead of HTTP
head request packets.
One of the main purposes of CheesePi is to collect data for operators to troubleshoot
the source of problems in a network. During measurement to CRICHD in 2015 UEFA
Champions League Final, there was a latency increase before and during the match.
By tracing back the results from all the tools the source of the problem was identified
as a specific router or the link that connects it to neighboring routers. The delay
increase can be related to the congestion created on the link or other some reasons,
but the job of CheesePi is to collect and quantify the observations of the problem.
It is not easy to measure the exact performance of the Internet due to its complexity,
but by following smart approaches and using the right tools, measurements can
characterise the performance experienced by a user. All the measurements
conducted during this master’s thesis use ping, httping, and traceroute network tools
to measure the performance of users’ Internet quality. However, it is difficult to make
concrete conclusions on some situations based only on the results obtained from
ping, httping, and traceroute measurements. The tools selection and their output
data defines the possible conclusions that can be drawn. More advanced tools will
allow more advanced conclusions. For example, to observe the behaviour of TCP we
could use a tool such as tcpdump. The longer delays experienced by httping, likely
caused by packet loss being fixed by TCP re-transmissions would be exposed by
this tool.
60
In some of the big media event measurements the httping was crashing several
times when an error occurred. During such measurements it became a challenge not
to have a complete results from some MAs.
61
7. Conclusions
We developed CheesePi, a distributed measurement platform which help users to
characterise their Internet experience. The CheesePi measurement agent runs on a
Raspberry Pi connected to a home router. The measurement agent conducts ICMP,
HTTP, traceroute and VoIP measurements periodically. Additionally, by increasing
the number and distribution of the measurement agents the network can be
characterized better. A dashboard which is locally hosted on the Raspberry Pi, is
used to visualise the experienced connection behaviour.
The measurement results show customers with an Ethernet connection experienced
better latency and packet loss compared to WiFi users. For example, the average
network delay for all targets during Eurovision Final from SICS WiFi was 4 ms and
was more than double the average network delay from an Ethernet connection (1.5
ms). And the average network delay for all targets during Eurovision Final from the
student accommodation WiFi was 1.4 times than the average network delay from the
Ethernet connection. In all the Eurovision measurements, the student apartment
experienced significantly (26-57 times) worse performance in terms of latency
compared to the SICS site.
The relatively larger network delays from the student accommodation during
Eurovision were for targets which use CloudFlare CDN service, and this might be the
traffic carrying agreement between the student’s accommodation ISP and other
ISPs. ISPs prefer to send their traffic through ISPs which do not charge them for
carrying their traffic (Peering), and sometimes this might cause relatively larger
delay. It can be also the content owners business relationship with ISPs for their
CDN. In Boxing and Champions League Final both experienced almost comparable
performance, with 1.x times better or worse performance, the .x depending on the
particular site we chose.
Our measurements showed us that the performance of streaming from most of the
CDN enabled websites is not influenced by large number of visitors during big media
events, which shows that CDNs are well provisioned. Another factor was the
distance of the CDN server to the user, despite being on a highly available and well
dimensioned the effect of the network delay, can dominate. As an example, the
average HTTP delay from SICS Ethernet to ATDHE, a streaming website which is
not CDN enabled, (but close in terms of hops) was 96 ms. The average HTTP delay
from SICS Ethernet to CRICHD, which ​was CDN enabled streaming website,
showed an average delay 212 ms, but far in terms of hops. This shows that the
network delay, can dominate. Of course the counter case has been seen to be true,
too, for example, the average HTTP delay from SICS Ethernet to SBS, a streaming
62
website which is not CDN enabled, was 713 ms. Whereas the average HTTP delay
from SICS Ethernet to BBC iPlayer, a CDN enabled streaming website, was 12 ms.
The CDN server was close to user, as we found from traceroute.
We showed, it is possible to characterise the network performance using network
tools such as ping, httping, and traceroute. By comparing ping and httping
measurement results, we are able to distinguish whether performance degradation is
created by the network or by the server. If the performance issue is known due to the
network, by looking the traceroute results, we are able to identify where exactly the
source of the problem is. We also able to identify local performance degradations
because of the WiFi problem by comparing WiFi with Ethernet measurement results.
This project shows that it is possible to characterize most of the user’s Internet
experience using a few network tools. By using more additional network tools, and
increasing the number of measurement agents, this project can help home Internet
users and ISPs on identifying and troubleshooting existing network problems.
63
8. Future work
CheesePi is an on-going project, of which my work was only a part. There are many
ways it can be improved or extended. The most most important one is to conduct
measurements from additional locations in Sweden. As of October, Raspberry PIs,
running CheesePi have been installed at Karlstad, Vasterås and Luleå universities.
We could also add other parts of the Nordic Internet. Deployment of more CheesePi
nodes would enable a more detailed network characterisation. Extending the number
of Pis to further home users is the final goal, of course.
In the experiments we have done mostly landmark measurements. With more
deployed nodes we could expand to intra-Pi measurements. This would help
understand the traffic on more links than the landmarks. And specifically between
home users, rather than from home users to large servers.
We have seen path changes during our measurements and how it affects the user
experience, the reasons are not clear at the end of this thesis. Further studies could
and should look at this phenomenon, particularly considering longer measurement
periods (at least a week) and again during popular events with video traffic.
We have done some analysis on the measurement results, however this could be
improved, for example, looking at the correlation between variables. Also some more
detailed time series work could be done. Is there a trend in the data or are the delay
measurements cyclic.
We have used standard networking tools. More advanced networking tools and
database storage engines could be included. For example tcpdump,
paris-traceroute, one-way ping or a short TCP connection. These are done in the
other works, such as Fathom or ARK, described briefly earlier.
64
9. Lessons learned
● This master thesis has helped me to have a much better understanding of
TCP/IPs behavior. I read some technical reports and articles which helps me
to understand some popular network monitoring systems. I learned how to
conduct experiments better. This master thesis was a good opportunity to
gain experience with lower level network tools.
● Writing this scientific report was the best experience during this master thesis.
The thesis report might not be perfect, but I learned a lot about how to
summarize my ideas, how to convey my thoughts in a scientific and accurate
way. The continuous feedback from my supervisors helps me a lot to improve
my report.
● I improved my programming skills mainly in Python and to some extent in
bash scripting. Before this master thesis my Python knowledge was rather
basic, but since all the measurement modules are implemented using Python,
it helps me to improve my programming skills. It also introduced me to
different libraries for data analysis. I also acquired basic bash scripting skills
during this project.
● The project was implemented with another master student, and it was a good
experience to develop my team working skills. We had to share our
knowledge and convey our thoughts properly to each other. Guidance from
our supervisors was very helpful, they helped us to see things in different
perspectives.
● Holding weekly meeting helped me to plan and present my work. I was
presenting what I did every Tuesday and this help me to plan my weekly tasks
ahead and to present my weekly work. It was a reasonable time frame to
report my work and to consider the feedback I got from my supervisors.
● SICS is a great place, and I had a chance to meet different researchers and
students. It was good experience for me to meet people from different
backgrounds and cultures, and everyone around was very supportive. There
was also a weekly paper review session in the Network Embedded Systems
department, the comments from the researchers who supported and opposed
the paper were a good lesson in understanding research.
● This master thesis helps me on how to take initiative and be alert in order to
accomplish your goals. I took some initiatives to conduct network
measurements even not included in this master thesis.
65
References
[1] Greg Bobolo, “5 digital video trends to watch in 2015”,
http://www.inma.org/blogs/innovative-advertising-solutions/post.cfm/5-digital-video-tr
ends-to-watch-in-2015​, 2015, [Online; accessed 10-August-2015].
[2] James F. Kurose, Keith W. Ross, Computer Networking a Top Down Approach
6th Edition, 2013.
[3]Lydia Parziale, David T. Britt, Chuck Davis, Jason Forrester, Wei Liu, Carolyn
Matthews, Nicolas Rosselot, TCP/IP Tutorial and Technical Overview, IBM, 2006.
[4]NetWare TCP/IP Administration Guide,
http://www.novell.com/documentation/nw6p/?page=/documentation/nw6p/tcpipenu/d
ata/hbnuubtt.html​, 2015, [Online; accessed 12-August-2015].
[5] Andrew S. TanenBaum, David J. Wetherall, Computer Networks 5th Edition,
2011.
[6] Olivier Bonaventure, Computer Networking : Principles, Protocols and Practice
Release 0.25, 2012
[7] Cellular Network, ​https://en.wikipedia.org/wiki/Cellular_network​, [Online;
accessed 17-August-2015].
[8] Content Delivery Network, ​https://en.wikipedia.org/wiki/Content_delivery_network​,
[Online; accessed 17-August-2015].
[9] How does a CDN work ,
http://www.cloudvps.com/community/knowledge-base/how-does-a-cdn-work/​,
[Online; accessed 18-August-2015].
[10] What is a Raspberry Pi? ,
https://www.raspberrypi.org/help/what-is-a-raspberry-pi/​, [Online; accessed
18-August-2015].
[11] About 6 million Raspberry Pis have been sold,
https://blog.adafruit.com/2015/06/08/about-6-million-raspberry-pis-have-been-sold-ra
spberry_pi-raspberrypi-mattrichardson-twit-newscreensavers/​, [Online; accessed
18-August-2015].
[12] Passive Vs. Active Monitoring,
https://www.slac.stanford.edu/comp/net/wan-mon/passive-vs-active.html​, 2001,
[Online; accessed 19-August-2015].
[13] Traceroute(8) Linux Man page, ​http://linux.die.net/man/8/traceroute​, [Online;
accessed 20-August-2015].
[14] A. Håkansson, Portal of Research Methods and Methodologies for Research
Projects and Degree Projects. WORLDCOMP'13 - The 2013 World Congress in
Computer Science, Computer Engineering, and Applied Computing, 22-25 July,
2013 Las Vegas, Nevada; USA.
[15] Carlos Ordonez, Yeol Song, Carlos Garcia-Alvarado ,Relational versus
Non-Relational Database Systems for Data Warehousing, 2010.
66
[16] ​Anthony Riccobono​, Sports TV Ratings: How Many People Watched
Mayweather vs. Pacquiao, The Kentucky Derby And NFL Draft?,
http://www.ibtimes.com/sports-tv-ratings-how-many-people-watched-mayweather-vspacquiao-kentucky-derby-nfl-1907833​, [Online; accessed -9 September-2015].
[17] ​Thomas Myer​s, Here's how you can watch Floyd Mayweather fight Manny
Pacquiao tonight,
http://www.mmamania.com/2015/5/2/8468715/how-to-watch-floyd-mayweather-fightmanny-pacquiao-tonight-ppv-boxing-free-online​, [Online; accessed -9
September-2015].
[18] Nearly 200 million people watch Eurovision 2015,
http://www.eurovision.tv/page/news?id=nearly_200_million_people_watch_eurovisio
n_2015​, [Online; accessed -9 September-2015].
[19] Kevin Ashby, Berlin final captures the world's imagination,
http://www.uefa.com/uefachampionsleague/news/newsid=2255318.html​, [Online;
accessed -21 September-2015]
[20] E​RNESTO, ​CRICFREE bounces back after UK police domain seizure,
https://torrentfreak.com/cricfree-bounces-back-after-uk-police-domain-seizure-14052
4/​, [Online; accessed -21 September-2015]
[21] Streaming 101: ESPN Vs Univision In World Cup,
https://www.bizety.com/2014/06/28/streaming-101-espn-vs-univision-world-cup/​,
[Online; accessed -21 September-2015]
[22] Internet World Stats, ​http://www.internetworldstats.com/stats.htm​, ​[Online;
accessed - 07 October-2015]
[23] S. Shunmuga Krishnan, Ramesh K. Sitaraman, Video Stream Quality Impacts
Viewer Behavior: Inferring Causality Using Quasi-Experimental Designs, 2012
[24] RIPE Atlas, ​https://atlas.ripe.net/about/​, [Online; accessed - 08 October-2015]
[25] The Archipelago Measurement Infrastructure (ARK),
http://www.caida.org/projects/ark/​, [Online; accessed - 08 October-2015]
[26] Samknows, ​https://www.samknows.com/​, [Online; accessed - 08 October-2015]
[27] Test Node Briefing Technical information relating to the SamKnows test nodes,
https://www.samknows.com/broadband/uploads/methodology/SQ302-001-EN-TestNode-Briefing-D01.pdf​, [Online; accessed - 08 October-2015]
[28] NetBeez, ​https://netbeez.net/​, [Online; accessed - 08 October-2015]
[29] SpeedTest, ​http://www.speedtest.net/about​, [Online; accessed - 08
October-2015]
[30] OOKLA, ​https://www.ookla.com/​, [Online; accessed - 08 October-2015]
[31] Christian Kreibich, Nicholas Weaver, Boris Nechaev, Vern Paxson, Netalyzr:
Illuminating The Edge Network, 2010.
[32] Mohan Dhawan, Justin Samuel, Renata Teixeira, Christian Kreibich, Mark
Allman, Nicholas Weaver, Vern Paxson, Fathom: A Browser-based Network
Measurement Platform, 2012.
67
[33] PingER,
https://www.slac.stanford.edu/grp/scs/net/talk09/pern-pinger-initiative.pdf, [Online;
accessed - 08 October-2015].
[34] MLAB, http://www.measurementlab.net/ , [Online; accessed - 08 October-2015].
[35] Facts and figures about Akamai,
https://www.akamai.com/us/en/about/facts-figures.jsp​, [Online; accessed - 15
October-2015]
68