Imperial College London Evaluation of Virtual

Imperial College London
Master’s Thesis
Evaluation of Virtual Machine Performance Scalability
Nicolas Khadivi
<nicolas.khadivi09@imperial.ac.uk>
Supervisor: Dr. Jeremy Bradley
Second Marker: Dr. Giuliano Casale
June 20, 2013
This thesis is dedicated to the memory of my father,
Habib Khadivi (1951–2009),
who inspired me to become an engineer.
Abstract
The concept of virtual machine was invented in the early days of computing to overcome the
shortcomings of the operating systems of the time. In the past few years, virtual machines
have become popular again due to the solution they provide to the problem of sharing the
resources of hardware platforms in a safe and reliable way.
In this thesis, we set out to evaluate the scalability of modern virtualised environments.
We implement our own experimental framework, which is designed to automatically create
instances of Wikipedia, benchmark them and log their performance statistics. We run multiple
experiments with varying parameters (number of instances, resource allocation, request rate)
and analyse the results.
Acknowledgements
First and foremost, I would like to thank my supervisor Dr. Jeremy Bradley, and his Ph.D.
student Anton Stefanek for their constant support, guidance and invaluable advice throughout
the project. I would also like to thank Dr. Giuliano Casale, my second marker, for his
contribution in evaluating my progress in the early stages. I am also grateful to the Computer
Support Group in the Department of Computing for their help with the experimental setup.
Finally, I would like to extend my gratitude to my friends and family, for their unconditional
support throughout my studies.
Contents
1
Introduction
1.1 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2
Background
2.1 Virtual Machines . . . . . . . . .
2.1.1 History . . . . . . . . . . .
2.1.2 Principles . . . . . . . . .
2.1.3 Hypervisor . . . . . . . .
2.2 Resource virtualisation . . . . . .
2.2.1 Processor . . . . . . . . . .
2.2.2 Memory . . . . . . . . . .
2.2.3 Device I/O . . . . . . . .
2.3 Related Work . . . . . . . . . . .
2.3.1 Challenges . . . . . . . . .
2.3.2 Queuing network model .
2.3.3 Regression-based model .
3
4
Setup
3.1 Project Machine . . .
3.2 Xen Cloud Platform
3.3 Virtual Hardware . .
3.4 Network . . . . . . .
3.5 Management . . . . .
3.5.1 xe CLI . . . .
3.5.2 XenCenter . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Implementation
4.1 Early experiments . . . . . .
4.1.1 Virtual machines . .
4.1.2 Experiments . . . . .
4.2 The Wikipedia Framework .
4.2.1 Design . . . . . . . .
4.2.2 Templates . . . . . .
4.2.3 Deployment . . . . .
4.2.4 Load generator . . .
4.2.5 Experiment . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
4
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
6
6
7
7
.
.
.
.
.
.
.
.
.
.
.
.
9
9
9
10
10
11
11
13
15
16
16
17
18
.
.
.
.
.
.
.
20
20
20
21
22
22
22
23
.
.
.
.
.
.
.
.
.
26
26
26
28
30
30
33
36
37
39
4.2.6
5
6
Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Evaluation
5.1 Data Processing . . . . . .
5.2 Experiments . . . . . . . .
5.3 Results . . . . . . . . . . .
5.3.1 Transaction Times
5.3.2 Scalability . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
43
43
44
45
45
46
Conclusion
53
6.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Appendix A PEPA
54
Appendix B Poisson Process
56
Appendix C Experiment statistics
57
Appendix D run.sh
58
Appendix E load.py
60
List of Figures
63
List of Tables
65
Bibliography
66
5
1 Introduction
Virtual machines have existed since the early 1960s and over the years, they have been used
to solve a variety of problems. Initially designed to enable users to run multiple applications
on single mainframes, they have now evolved into an important factor in the development of
“cloud computing”, the idea of computing as a utility [17]. In recent years, their importance in
the server market has sharply increased, and it is estimated that more than “23% of all servers
shipped in 2014 will be actively supporting virtual machine technology” [18]. In this context,
investigating and evaluating the scalability of virtualised systems is particularly interesting in
that it provides information on how these systems will be able to handle growing workloads,
and how efficiently they will take advantage of more and more powerful hardware.
1.1
Motivations
The use of virtualisation techniques in data centres provides a number of advantages, an
important one being the sharing of physical resources [18]. With the sharp drop in hardware
prices, servers are now extremely powerful and most applications do not require or use
the amount of resources that servers offer. Virtualisation offers a convenient solution to the
problem of using these resources in an efficient way, by allowing multiple applications to
be run on a single machine while being isolated from each other. Using these techniques,
data centre administrators can perform server consolidation, that is, reduce the total number
of servers (and by extension, energy and hardware costs) by using each server a lot more
efficiently.
One way of increasing the efficiency of server usage is to run multiple applications
concurrently on individiual machines. Finding out how virtualised environments behave for
different load and resource parameters, software and scenarios helps us better understand
the dynamics of the systems and thus minimise the performance degradation that sharing
resources between applications inherently leads to. If we know what resources each application
will be using and how much of these resources they will require, then we can avoid resource
contention by migrating “conflicting” applications to servers with unused resources. One
could even think of a scenario where each server runs only “complementary” applications,
that is, applications that we know will not have to compete over resources. All of these
solutions require information about the applications and their behaviour when running in
virtual machines.
Another motivation for investigating the scalability of virtualised applications is that it
would make it much easier for service providers to abide by their Service-Level Agreements
(“SLA”). The SLA is part of the contract signed between a user and a provider, and usually
details the minimum level of service that the provider must offer to the customer. For example,
the Amazon EC2 SLA states that the customer can expect an Annual Uptime Percentage of
at least 99.95% [1] (equivalent to a downtime of 4 hours over a 365 day period), otherwise,
6
Amazon will offer free Service Credits. Therefore, abiding by the SLA will not only keep
Amazon’s customers happy, but save the company from having to compensate for violating
the SLA. An accurate understanding of the scalability of the service the user is running would
enable the provider to plan ahead and abide by the agreements more easily.
In essence, knowledge about how a virtualised application scales (to a certain degree of
accuracy) can help drive down energy and hardware costs, increase availability and facilitate
the management of virtualised services. Given the future prevalence of cloud computing that
is expected, even small achievements in this research area could lead to massive improvements
in terms of energy usage, cost, reliability and efficiency.
Finally, knowledge on the extent to which a virtualised system is scalable could help model
and predict the performance of virtualised applications, and thus optimise resource allocation
and the management of virtual services [37].
1.2
Contributions
The objective of this project is to evaluate the scalability of a given virtualised environment.
We first implement a generic experimental framework designed to run simple commandline tools on a single virtual machine.
Later, we implement another framework that automatically deploys instances of Wikipedia
(virtual machines) and logs their performance statistics during experiments.
Within this framework, we implement our own load generator, which we use to send
requests to the instances that we deploy.
We run multiple experiments with varying parameters (rate, number of requests, resources
allocated to the virtual machines) and process the data.
Finally, we analyse our experimental results and draw a number of conclusions.
1.3
Structure
The remainder of this thesis is laid out as follows:
• Chapter 2 provides an overview of virtualisation, a short history, existing techniques
and related work.
• Chapter 3 presents our experimental setup and the software that we use throughout our
project.
• Chapter 4 focuses on the implementation of our main experimental framework, which
enables us to put load on instances of Wikipedia and log performance statistics.
• Chapter 5 presents our results, their interpretation, and our conclusions about the
scalability of our virtualised environment.
• Chapter 6 contains concluding remarks about our project, as well as ideas for future
work.
7
2 Background
2.1
2.1.1
Virtual Machines
History
Half a century ago, the word “computer” referred to bulky, expensive mainframes which
were operated using punched cards and magnetic tapes. At the time, the cost of operating
such machines and their scarcity provided an important incentive to maximise their usage
[29]. A few years later, in the late 1960s, the idea of “virtual machine” was first developed to
compensate for the shortcomings of early operating systems and more specifically, to overcome
the fact that that these systems could only run one application at a time [11].
Virtual machines monitors (VMM, also known as “hypervisor”), provided a simple and
elegant solution to this problem, essentially transforming a single machine into the “illusion of
many” [11], and thus providing an efficient way of running multiple applications at the same
time on a single machine [29]. The technology was hugely successful both in industry and
academia, until the 1980s and the simultaneous advent of cheap hardware and multitasking
in operating systems. As mainframes were being replaced with microcomputers, the idea
gradually became irrelevant, and by the end of the decade, virtual machines were a thing of
the past [29].
Interestingly, the factors that led to the deprecation of virtual machines also played a crucial
part in their resurrection. The sharp drop in hardware costs led to a proliferation of computing
power, often wasted, and innovation in operation systems led to problems with regards to
security and reliability. Just as they had a few decades before, virtual machines provided an
elegant solution to these concerns by making it possible to run multiple applications sideby-side while isolating them from each other. With the widespread development of network
infrastructures and the Internet in the early 2000s, the concept of “cloud computing” emerged
as a new paradigm, in large part thanks to advances in virtualisation technology. “Cloud
computing” refers to the model of computing as a utility (similar to water, electricity etc.), and
it is defined as a “model for enabling ubiquitous, convenient, on-demand network access to a
shared pool of configurable computing resources” [23].
Within cloud computing, virtualisation plays a crucial role in enabling server consolidation, greater resource utilisation and centralised administration [17]. VMware ESX and
Citrix XenServer, two commercial solutions, are now widely used in data centres and major
corporations such as Microsoft (Hyper-V) are entering the competition. Their hypervisors offer
a convenient solution to security and reliability concerns, increase efficiency and reduce space
and management costs [29]. More and more, users are taking advantage of the infrastructures
offered by “cloud providers” for their storage and processing needs, at an ever cheaper cost.
Finally, although they are not as popular as their server counterparts, virtual machines
have also entered the desktop market. On a desktop computer, much like on servers, VMs
enable the user to install multiple operating systems side-by-side. For example, a Mac user
9
will be able to install Windows or Linux to take advantage of software only available on those
platforms. In this case, however, the hypervisor is usually installed on top of the host operating
system. This is an important distinction compared to server solutions, and one that we will
expand on in the next section.
2.1.2
Principles
There exists different types of virtualisations:
• Hardware or platform virtualisation refers to the software simulation of some hardware
platform on top of which one or more virtual machines run. Typically, the virtual machine
monitor is the layer below the guest operating system (the OS running inside a virtual
machine), and sits either directly on top of the hardware or runs from within a host
operating system.
• Application virtualisation refers to the encapsulation of applications from the operating
systems. The Java virtual machine (JVM) [27] is a good example of this type of virtualisation. The Java code is compiled into bytecode, which is then executed inside the
JVM. Since the JVM is available on most major operating systems, this makes most Java
applications easily portable.
• Desktop virtualisation is the concept of separating the desktop environment from the
hardware on which it runs. Within an organisation, a client-server model is used to
provide end-users with a desktop. Each user is given a home folder and a desktop on a
central server, which is then accessed via a “thin client” (a very simple computer, with
limited amounts of CPU, RAM and hard disk space) and a network connection. Users
then interact directly with the central server, which provides the illusion of a regular
desktop.
There are a number of other types of virtualisation, but we can notice a trend in these solutions:
they each provide a number of advantages by simulating some environment. To the end-users,
everything looks (almost) the same, but for system administrators, virtualisation techniques
allow for easier management of computing resources. In essence, virtualisation is all about
illusion.
Throughout this section, we will focus on hardware virtualisation, as it is the only type that
is relevant to our project. Furthermore, we will use the terms virtual machine monitor, VMM
and hypervisor interchangeably. Similarly, guest, virtual machine and VM are equivalent.
2.1.3
Hypervisor
The hypervisor is the most important element of any virtualised environment. Defined
as “a software-abstraction layer that partitions a hardware platform into one or more virtual
machines” [10, 29], it abstracts away the underlying hardware by simulating a uniform platform
across virtual machines. The hypervisor manages the execution and resource usage of the guest
operating systems by virtualising all hardware resources (CPU, Memory, Storage, Network
etc.) and allocating them to each virtual machine. However, for all intents and purposes,
the hypervisor retains ultimate control of hardware resources. For example, hypervisors can
dynamically deallocate memory from a virtual machine and allocate it to a different virtual
machine.
We have seen that virtual machines sit directly on top of the hypervisor, but there are two
different types of hypervisors as defined by R.P. Goldberg [10]:
10
Type 1 (native, bare-metal) hypervisors sit directly on top of the host’s hardware, which they
control directly. They also manage the guest operating systems. Citrix XenServer (which
is based on Xen [2]) and VMware ESX are the most popular commercial solutions, and
they are widely used in data centres. They represent the original and most efficient
implementation of virtualised solutions [see 11], in that performance losses are much
lower than with Type 2 hypervisors.
Type 2 (hosted) run like regular applications within an operating system. They are common
in personal or corporate environments, and provide a convenient way for end-users to
run multiple operating systems side-by-side. Oracle VirtualBox, VMware Fusion and
Parallels Desktop are typical examples.
Figure 2.1: The two types of hypervisors. (Wikimedia Commons)
The major differences between these two types are highlighted in Figure 2.1. Throughout
this report, we will focus on Type 1 hypervisors, as they are the ones that are used in datacentres.
Finally, like any other piece of software, a number of tradeoffs must be made when
designing hypervisors, and the most important goals for virtual machine monitors are:
• Compatibility: one of the main advantages of virtual machines is their ability to run
legacy software and abstract away the specificities of the underlying platform.
• Performance: the ultimate goal is to run the virtual machine at native or near-native
speed, minimising the virtualisation overhead.
• Simplicity: implies reliability and ensures that the hypervisor is free of bugs and rarely
crashes (since a crash would probably cause all virtual machines to fail).
2.2
Resource virtualisation
Throughout this section, we refer extensively to material from [24], [29], [34] and [30].
2.2.1
Processor
Before we describe CPU virtualisation techniques, we need to familiarise ourselves with some
computer architecture principles. Some of these principles are simplified and details are
omitted for convenience. Those readers who wish to broaden their knowledge in this area can
refer to the excellent book by Patterson and Hennessy [28].
11
A processor’s instruction set is usually divided in two (or more) categories (see Figure 2.2):
Unprivileged instructions are those that are not allowed to change the state or allocation
of resources that are shared between various running processes (e.g. processor, main
memory, network connections).
Privileged instructions include all those that change the state or allocation of shared resources
(e.g. set the program counter, shut down the machine, I/O instructions).
Furthermore, processors operate in (at least) two modes (see Figure 2.2):
Supervisor mode (also called Ring 0) is the mode in which the entire instruction set (privileged
and unprivileged) is available. This is the mode in which the operating system’s kernel
runs.
User mode (also called Ring 3) is the mode in which only unprivileged instructions can be
executed. This is the mode in which user processes (e.g. applications) run.
Privileged
User
mode
Supervisor
mode
Unprivileged
Figure 2.2: Types of instructions and processor operating modes
Thus, if a user-mode process needs to execute privileged instructions, it must send a request
to the kernel (via a system call). The kernel will then perform the task (on behalf of the
process) and return control of the CPU back to the process. This works well in non-virtualised
environment – the operating system runs in supervisor mode and manages the hardware
resources, applications run in user mode and perform system calls when required.
Direct execution
In virtualised environments, however, operating systems running in virtual machines cannot be
allowed to manage resources directly as this is precisely the role of the hypervisor. Therefore,
how does one leave resource management to the hypervisor while allowing the operating
system to run privileged instructions? (see Figure 2.3).
The solution to this problem is a technique called direct execution. With direct execution,
the virtual machines (guest operating systems) run in user mode and are only allowed to
run unprivileged instructions. The hypervisor runs in supervisor mode and can execute both
privileged and unprivileged instructions. When virtual machines attempt to execute privileged
instructions, an interrupt is generated (since the virtual machine runs in user mode) and the
instruction is dealt with by the hypervisor.
12
Virtualised
Applications
User mode
Kernels (VMs)
?
Hypervisor
Supervisor mode
Non-virtualised
Applications
User mode
Kernel
Supervisor mode
Figure 2.3: Operating modes in non-virtualised and virtualised environments
Further Techniques
Nonetheless, many modern architectures (most notably x86) were not designed to be virtualisable. Therefore, some instructions cannot be dealt with by the hypervisor and are said to be
non-virtualisable. This issue has a number of solutions:
Full virtualisation (FV) relies on a technique called binary translation, that is, patching nonvirtualisable instructions on-the-fly. Unprivileged instructions are usually virtualisable,
and are executed using direct execution. Only non-virtualisable, privileged instructions
are translated, which means that out-of-the-box OSs can run inside a virtual machine.
However, binary translation is complicated to implement and can result in large overheads.
Paravirtualisation (PV) replaces the non-virtualisable parts of the instruction set with virtualisable equivalents. Instead of trying to run non-virtualisable instructions and have
the hypervisor intercept them, the guest “cooperates” by calling the hypervisor directly.
Since this is done at the interface between the hypervisor and the virtual machine,
operating systems need to be made compatible with the modified instruction set. Thus,
proprietary and legacy operating systems will not work with this technique. On the
other hand, with a ported OS, most applications will run unmodified, and there are
notable performance gains.
Hardware-assisted virtualisation (HVM) is the result of enhancements made to the x86 processors by Intel and AMD. The hypervisor runs at an even lower level (sometimes called
Ring -1), which enables virtual machines to run in supervisor mode (like they would
normally). Privileged instructions executed by guests are automatically trapped to the
hypervisor in hardware. Operating systems can run unmodified but this approach leads
to high CPU overheads (mostly due to I/O instructions, which are privileged). There
exists a hybrid solution called PVHVM, which involves using paravirtualised device
drivers. This approach considerably reduces the performance overhead of “pure” HVM
approaches [26].
2.2.2
Memory
Memory virtualisation involves sharing the physical memory between virtual machines and
dynamically allocating it to each of them. Fortunately, this is a problem that computer scientists
encountered a number of years ago and which was solved using virtual memory. Initially,
this technique was used in the 1950s to extend the main memory (e.g. RAM) by storing
data on the secondary memory (e.g. hard disk drive). But when operating systems started
supporting multitasking, it began being used as a way to efficiently share and allocate the
physical memory between multiple processes.
Virtual memory hides the different types of memory (main, secondary) behind a virtual
address space (VAS). This provides two main advantages:
13
Fragmentation hiding means that an application is given what looks like a virtual, contiguous
and directly addressable chunk of memory (regardless of where the actual data lies
physically). The program uses logical addresses to access data, which are translated into
physical addresses. The operating system keeps a mapping of virtual to physical pages in
the page table, and takes care of the translation. Modern CPUs provide hardware support
for virtual memory – via a memory management unit (MMU) and a translation lookaside
buffer (TLB), which are used by the operating system to speed up the translation.
Process isolation is achieved by providing each process with a separate virtual address space.
Each process only has access to its own VAS and thus, to its own pages (data) stored in
memory. Although this complicates inter-process communication, it ensures that they
cannot interfere with each other’s memory.
The same way an operating system keeps page tables mapping virtual pages to physical
pages, the hypervisor maintains a shadow page table, which lets it monitor the state of machine
memory. This adds another level of abstraction (for more details, see Smith and Nair [30]):
Physical memory is the hardware memory, for example, the RAM sticks in your computer.
Real memory is what the virtual machine thinks is physical memory, managed by the guest
OS. In other words, it is a virtual address space given to the guest OS which provides
the illusion of physical memory.
Virtual memory remains the same – the illusion given to applications that they have a contiguous address space.
Physical
Physical
Real
···
Virtual
Virtual
Figure 2.4: Simple model of memory in virtualised and non-virtualised environments
Given this setting, the obvious thing to do would be to get the hypervisor to establish a realto-physical mapping in its shadow page table. This, however, would require two translations
(virtual-to-real, real-to-physical) for each memory access and would incur non-negligible
overhead (see Figure 2.5).
Virtual
Guest OS
Real
Hypervisor
Physical
Figure 2.5: Virtual-to-Real, Real-to-Physical mapping
Instead, every time a virtual machine establishes a virtual-to-real mapping in its page table,
this action is trapped to the hypervisor, which then updates its own shadow table with the
corresponding virtual-to-physical mapping and the guest’s page table with the virtual-to-real
mapping.
14
This is sometimes called MMU emulation, as the guest thinks that it is using the MMU,
when in fact only the hypervisor is modifying the MMU by loading the shadow page table for
each guest into it when that guest is being used. This allows for direct lookup, and reduces
the overhead considerably (see Figure 2.6).
Virtual
Guest OS
Real
Physical
Hypervisor
Figure 2.6: Virtual-to-Physical direct lookup
2.2.3
Device I/O
Alongside CPU and memory, I/O is the third main resource that needs to be virtualised. Given
the unique characteristics of each device and the large variety of devices that exist, this is a
difficult challenge. Thankfully, similar to what was done with memory management, a number
of techniques have been borrowed from operating systems to deal with these issues.
The general idea is that for each device, a virtual equivalent is constructed (emulated)
by the hypervisor and given to each virtual machine. Since a virtual device usually has an
equivalent underlying physical device, the hypervisor’s role is essentially to intercept I/O
requests emanating from the guests and redirect them to the relevant device.
There are different types of devices, and a number of techniques used to virtualise them
[30]:
Dedicated devices are those that are always used by a single guest at any given time, for example the keyboard and the mouse. In theory, requests relating to these could completely
bypass the hypervisor. In practice, however, the VMM handles these devices because
VMs run in user mode and don’t have access to some instructions (e.g. interrupts).
Partitioned devices, such as the hard disk, are usually split across guests in the form of a
virtual disk. The hypervisor maintains a map of parameters (e.g. sectors), intercepts and
translates requests from the guest and then reissues them to the physical disk. Similarly,
the VMM translates status information from the disk before presenting it to the guests.
Shared devices refer to devices that are being shared “at a fine time granularity” [30]. A good
example is the network adapter, which is constantly being used by all the guests. Each
guest is allocated a virtual network address, which the VMM knows. Outgoing requests
from a guest’s virtual address are translated to requests from the machine’s physical
address on a given network port. Likewise, incoming requests to different network ports
are translated into requests for a specific guest, using the virtual network addresses. In
general, requests to and from guests relating to any shared device are translated by the
hypervisor by means of a virtual device driver.
These techniques usually incur large overheads (due to the device emulation), but have the
advantage of contributing to standardisation as the actual hardware doesn’t matter anymore.
Indeed, all guests are configured to run on the same virtual hardware with the hypervisor
taking care of everything else, using drivers from an existing operating system (e.g. the Linux
kernel) to communicate with the hardware.
15
The virtualisation overhead can be dramatically reduced using paravirtualised device
drivers in place of virtual drivers on guest operating systems. In this case, the guest is aware
that it is running in a virtual machine. These drivers have a front-end (installed in the VM) and
a back-end (at hypervisor level) which “cooperate” together to allow guests to communicate
with I/O devices. The downside is that PV drivers don’t necessarily exist for each operating
system, but when they do, they can be installed on unmodified guests. On the other hand, this
“cooperation” vastly improves I/O performances, since the emulation layer is removed.
2.3
Related Work
The idea behind predicting the performance of a virtualised application is to estimate the
resource utilisation (e.g. CPU) and performance measures (e.g. response time, throughput) of
that application for a given set of parameters (e.g. workload, operating system, hardware) [3].
Predicting the performance of non-virtualised applications is already a difficult task [18], and
as we have seen in Section 2.1, virtualisation introduces a number of overheads which also
affect the performance of applications. With the recent “rebirth” of virtualised solutions, the
literature on this topic has grown extensively, the main challenges have been identified and
discussed, and a number of models have been proposed to predict the performance of virtual
environments.
2.3.1
Challenges
Although the theory behind virtual machines allows us to identify sources of overhead, as
we did in Section 2.1, quantifying this overhead is a difficult task because it involves very
low-level metrics (e.g. TLB misses). Much work has been done to improve the collection of
such metrics.
For example, Menon et al. [25] identify network I/O as one of the main sources of overhead
in virtualised environments and introduce a custom tool called “Xenprof”, which allows
deep profiling of multiple guests running concurrently within a Xen environment. Similarly,
Benevenuto et al. [3] quantify the overhead introduced by the isolated driver domain (the domain
responsible for cooperating with the guest’s device drivers, usually Dom0) and present a
queuing network model using this experimental data as an input (see Section 2.3.2).
These approaches, while beneficial as a starting point for more complicated models, only
focus on I/O overhead and more specifically networking. Extending these approaches, Tickoo
et al. [31] present the three key challenges to overcome in order to produce more accurate
models:
1. Modeling the contention of visible resources (cores, memory, I/O)
2. Modeling the contention of invisible resources (shared caches, shared memory bandwidth)
3. Modeling the overhead introduced by the hypervisor.
These criteria are then used to propose a modeling approach called virtual platform architecture.
Huber et al. [18] complete this work by presenting a generic feature model of the factors
that influence the performance of virtualised environment (see Figure 2.7). Their experiments,
based on a number of different benchmarks, provide interesting insight on performance
degradation due to virtualisation. For example, they show that the CPU performance in a
virtual environment is within 3% (negligible) of native performance, while memory (up to
40%) and network (up to 30%) performances suffer from much higher losses.
16
816
N. Huber et al.
Legend
Virtualization Platform
exclusive OR
inclusive OR
Resource Management
Configuration
Virtualization Type
Full Virtualization
Binary Translation
Workload Profile
I/O
Number of VMs
CPU Scheduling
Memory
Memory Allocation
CPU
Para-Virtualization
CPU Allocation
Disk
CPU Priority
Network
Core Affinity
2. Major
performance-influencing
factors
of virtualization
platforms
Figure 2.7:Fig.
Feature
model
of the factors influencing
performance
in virtualised
environments
(Huber et al. [18])
Several influencing factors are grouped under resource management configuration. First, CPU scheduling has a significant influence on the virtualization
platform’s
turn, (dedicating
it is influenced
several
factors.
The first
Finally, theyperformance.
show that coreInaffinity
a CPUby
core
to a VM)
can counter-balance
CPU allocation
reflects
the
number
of virtual
CPUs
allocated to a VM.
thisfactor
performance
degradation.
Finally,
they
show that
the impact
of overcommitment
(creating
more
VMs
there are physical
performance
is inversely
to the
Most
of than
the performance
loss ofcores)
CPUon
intensive
workloads
comes proportional
from core and
overcommitment
factor.
For example,
consider
a machine
1 core, and
4 VMs, each of
cache inferences
[2]. Hence,
the second
factor
that haswith
a significant
performance
which
is allocated
1 core.
Thespecifying
performance
each virtual
machine
will
be extremely
close to
influence
is core
affinity,
if aofvirtual
CPU of
a VM is
assigned
to a ded1/x,
that is,
25% in our
icated
physical
corescenario.
(core-pinning). The third parameter reflects the capability
of assigning different CPU priorities to the VMs. For example, the Xen hyper2.3.2
Queuing
visor’s
weight network
and cap model
parameters or VMware’s limits and fixed reservations
parameters
are
represented
by these CPU priority configurations. Finally, the
“A model is a representation of a system” [3]. As such, the primary purpose of a model is to
memory
the as
number
of VMs
influence
the the
resource
management
represent
theallocation
underlyingand
system
accurately
as possible.
Given
complexity
of virtualised
configuration,
too.
Managing
memory
requires
an
additional
management
systems, a number of details need to abstracted away, and models must endeavour tolayer
focus on
in
the
hypervisor.
The
number
of
VMs
has
a
direct
effect
on
how
the
available
the most important factors that impact performance.
resources
are shared
among
VMs.
Queuing networks
is one
of theall
first
models that was proposed to model VM performance.
Last
but
not
least,
an
important
influencing
is theoftype
of workload
Benevenuto et al. [3] introduce this simple idea
by usingfactor
three types
benchmarks
to gather
executed on
theinvirtualization
platform.
different types of resources
experimental
data
both non-virtualised
and Virtualizing
virtualised environments:
causes different performance overheads. For example, CPU virtualization is suphttperf
is used
generate
several
HTTP workloads
to suffer
the instances
• Web very
server:
ported
well
whereas
I/Otoand
memory
virtualization
currently
from of
Apache
installed
on
the
non-virtualised
and
virtualised
machines.
significant performance overheads. In our model we distinguish CPU, memory
and
I/Ointensive:
intensivea workloads.
In thebetween
case oftwo
I/Odirectories
workload,
distin• Disk
2GB file is copied
on we
the further
same partition.
guish between disk and network intensive I/O workloads. Of course, one can
• CPU
intensive:
a Linux mixture
kernel is consisting
compiled. of all three workload types.
also
imagine
a workload
The virtual environment consists of a dual-core machine where one core has been dedicated
4 IDD
Automated
Analysis
to the
and the other Experimental
is the guest CPU. The
non-virtual environment is also a dual-core
machine, where both cores work in parallel.
We
a generic
on anisautomated
experimental
The now
mainpresent
observation
that is approach
made frombased
the results
that the kernel
compilationanalresults in
ysis
to
quantify
the
performance-influence
of
the
factors
captured
in
our
feature
0 CPU IDD utilisation, whereas the web server and the file copy result in considerable CPU
usemodel.
due to First,
disk and
This
significantlysetup
impacts
performance
wenetwork
give anactivity.
overview
of activity
the experimental
andthe
describe
the of
thegeneral
system compared
to
the
non-virtualised
environment.
process that is followed when conducting experiments. We then describe
These
results sets
are used
to implementand
a simple
queuing
network
First,
anperforasymptotic
the
different
of experiments
how to
structure
themmodel.
to assess
the
bound
is
calculated
for
request
rates:
mance influence of a given factor. We assume that in each set of experiments a
selected benchmark is executed multiple times in different scenarios to charac1
1
λmax ≤
=TheV process
= 7874
terize the impact of the considered
factor.
is completely automated (2.1)
M
k
maxi=1 Di
17
DCPU
Then, a slowdown factor (the overhead due to virtualisation) is defined as:
Sv =
Bkvirt
Bk
(2.2)
where Bkvirt and Bk are the busy times of resource k on the virtual and non-virtual environments
(respectively).
Furthermore, the overhead introduced by the IDD is defined as:
CostVIDD
Mi =
IDD
BCPU
V Mi
BCPU
(2.3)
IDD is the fraction of CPU time consumed by the IDD and BV Mi the CPU time
where BCPU
CPU
consumed by a guest i.
These two metrics are then used to calculate input parameters to the queuing model such
as service demands, utilisation and residence time at a resource k. Finally, the virtual system’s
response time is defined as the sum of residence times at each of the resources in the system:
M
VM
Rvirt = RV
CPU + R IDD + R Disk
(2.4)
In general, the results are very interesting:
• the response time sub-estimates the measurements by a small margin.
• the response time dramatically increases as the request rate reaches its upper bound
(7800 requests/sec), as expected
The simplicity of the workloads that were applied and of the model itself make it a good
starting point for more accurate and larger models. However, this also means that it is relevant
only to the CPU performance of certain application. Memory usage and Network I/O are
notably absent from the model and only basic workloads were applied.
2.3.3
Regression-based model
Another approach that has been tried is a regression-based model (see Wood et al. [37]). Again,
three types of workloads were used to generate results in order to create native and virtualised
usage profiles:
• Computation intensive calculates a given number of terms of the Fibonacci series
(according to the computation time) when it receives a request.
• Network intensive: has two modes: transmit mode sends a file to the client, whereas in
receive mode, the clients upload files to the server.
• Disk intensive: also has two modes, read and write, whereby a random file is either read
from or written to.
The platform profiles are then generated using no less than eleven different metrics (e.g.
user space and kernel CPU %, received and transmitted packets/sec and disk read/write
requests/sec).
The model is created using these platform profiles and by applying three regression
algorithms:
18
• Robust linear regression to reduce the importance of outliers (sample that deviates
considerably from the rest of the data).
• Stepwise regression to remove the statistically insignificant metrics from the final model.
• Model refinement to detect erroneous benchmark results (due to unexpected background
processes, network problems etc.)
Once again, the results are interesting – a model trained with the Fibonacci benchmark
will accurately model CPU usage within 1%, but applying the same model to another type
of workload yields a median error of 670%. However, a model trained with all three types of
workloads yields an error of 3% for Dom0 (IDD) and 7% for the VM.
The authors also make a point of showing that modeling I/O is very important. To do
so, they compare a model based exclusively on CPU metrics (user space and kernel CPU %),
and another dominated by I/O related metrics (kernel CPU %, I/O wait, Rx Packets/sec, Tx
Packets/sec and Disk Write Req/sec). Unsurprisingly, the multi-resource model outperforms
the CPU-only model significantly.
Although the model predicts the performance of the virtualised environment to an acceptable degree of accuracy, “each microbenchmark was purposely created to capture some unique
performance problem present in one or more important applications”, and thus provides “little
information” on the behaviour of a given application on some specific platform.
19
3 Setup
In this chapter, we present the reader with our experimental setup, the software that we use
throughout our project and some of the constraints that we faced.
3.1
Project Machine
In order to perform experiments on virtual machines, we were provided with a single computer
located in the labs of the Department of Computing. The machine we used was an HP Compaq
8200s, with the following configuration:
•
•
•
•
CPU: Intel Core i7-2600 (64 bit, 4 Physical Cores, 8 Logical Cores) @ 3.40GHz
Memory: 2×4GB (DDR3 1066Mhz)
Hard Disk: 500GB (7200RPM, 16MB cache)
Network: Intel 82579LM Gigabit Ethernet Controller
In the remainder of this report, we will sometimes refer to the project machine as project12,
it’s hostname on the College network.
3.2
Xen Cloud Platform
The hypervisor we chose to use for our project is Xen, a free and open-source1 hypervisor,
initially released in 2003 [2]. The early versions were developed by the University of Cambridge
Computer Laboratory, but the project is now community-managed with the support of Citrix
Systems. Citrix also ships commercial versions of Xen targeted at corporate customers, under
the name XenServer. Since its release, Xen’s market share has continuously increased, and it is
notably used by Amazon and Rackspace as their primary hypervisor [5].
In the hierarchy of hypervisors, Xen is bare-metal (Type 1), which means that it directly
controls the hardware. In Xen terminology, virtual machines are referred to as domains.
Domain0 (the control domain) is the first domain that is started when Xen boots. It runs a
Linux distribution that has direct access to the hardware and provides the tools to manage
the other domains (DomainU for user). The hypervisor part of Xen is responsible for memory
management, CPU scheduling and starting Dom0.
Dom0 is responsible for managing the virtual machines (create, start, shutdown, etc.) and
performing I/O operations on their behalf. Because Xen implements paravirtualisation by
default, Dom0 will run the back-end driver which forwards hardware requests made by the
DomU via their front-end drivers. This means that the guest operating systems must have
been modified to work in paravirtualised mode.
1 licensed
under the GPLv2
20
needs areneeds
goingare
to going
be prior
to be
to prior
transitioning
to transitioning
it to the itvirtual
to theenvironment.
virtual environment.
If these If th
overheads
overheads
are not are
accounted
not accounted
for during
for initial
duringplanning,
initial planning,
an application
an application
could co
be deployed
be deployed
to a server
to awith
server
insufficient
with insufficient
resources,
resources,
resultingresulting
in unacceptable
in unaccepta
application
application
performance.
performance.
Xen andXen
VMware
and VMware
ESX server
ESXdemonstrate
server demonstrate
the two the
popular
two popular
I/O models
I/O for
models
In the original Xen design [2], the device drivers existed at hypervisor level which meant
that if
a driver
crashed,
virtual
bethe
affected.
In
newer
versions
VMs. InVMs.
ESX
In
(and
ESX
Xen
(and
in then
Xen
its all
original
in
its machines
original
designwould
[5]),
design
[5]),
hypervisor
the
hypervisor
itself conitself c
(including the one we use), device drivers can be executed in isolated driver domains [8] (see
ains device
tains
driver
devicecode
driver
andcode
provides
and provides
safe, shared
safe,access
sharedfor
access
I/O for
hardware
I/O hardware
(see
(
Figure 3.1). If a driver crashes, then a simple reboot of the associated domain suffices to solve
Figure 1Figure
a).theLater,
1 a).without
the
Later,
Xen
the
team
Xen
proposed
team
proposed
a new
architecture
a new
architecture
[9] that
[9]
allows
that allo
problem,
affecting
other
virtual
machines.
By default,
the control
domain
Dom0 is
also the driver domain, which results in high CPU usage on the part on Dom0 for I/O intensive
unmodified
unmodified
device drivers
deviceto
drivers
be hosted
to beand
hosted
executed
and executed
in isolated
in isolated
“driver domains”
“driver domai
applications [37].
see Figure
(see1Figure
b). 1 b).
Virtual
Machine
Net Driver
Hypervisor Hypervisor
NIC
Virtual
Machine
Disk
Net Driver
Disk
NIC
Domain-0
Domain-0
Virtual
Machine
Net Driver
Net Driver
Disk Driver
Disk Driver
Hypervisor Hypervisor
Disk Driver
Disk
NIC
Virtual
Machine
Disk
NIC
Disk
(a) VMware
(a)
VMware
I/O Model
I/O Model
(b) Xen I/O
(b) Xen
Model
I/O Model
Original
Improved
Fig. 1. Two
Fig.popular
1. Two I/O
popular
models
I/Ofor
models
VMs.for VMs.
Figure 3.1: Xen I/O Models (Wood et al. [37])
In Xen, the
In Xen,
management
the management
domain Dom-0
domainhosts
Dom-0
unmodified
hosts unmodified
Linux device
Linuxdrivers
device dri
and plays
and
theplays
role of
thethe
role
driver
of the
domain.
driver domain.
This I/OThis
model
I/Oresults
modelinresults
a moreincomplex
a more comp
Installing Xen from scratch is not straightforward. The procedure (roughly) consists in
CPU usage
CPU
model.
usage
For
model.
I/O For
intensive
I/Osupports
intensive
applications,
applications,
CPUsetting
usage
CPU
has
usage
two
has
compotwo com
installing
a Linux
distribution
that
Xen (usually
Debian),
up logical
volumes
and
networking
for
guests
andthe
finally
installing
Xen, drivers
and additional
management
nents: CPU
nents:
consumed
CPU consumed
bythethe
guest
by
virtual
guest
machine
virtual
machine
(VM)
and
(VM)
CPU
and
consumed
CPU consum
software. However, there exist many out-of-the-box virtualisation solutions built around Xen
by Dom-0
by which
Dom-0
performs
which which
performs
I/Oinclude
processing
I/O
processing
onXenbehalf
onofbehalf
thealso
guest
of
the
domain.
guest
domain.
(called
distributions),
not only
the
hypervisor,
but
an optimised
Dom0
kernel
and
large
numbers
andof
tools
towe
manage
Xen
“toolstack”).
In this work,
In this
without
work,
without
lossofofdrivers
generality,
loss
generality,
demonstrate
we(thedemonstrate
our approach
our approach
using us
The Xen distribution that we chose to install on our machine is Xen Cloud Platform (XCP)
Xen running
Xen1.6,running
paravirtualized
paravirtualized
We
VMs.
believe
that
believe
that
approach
our
approach
can
belike
applied
can be app
which comes
in the ofVMs.
a ready-to-use
ISO We
file that
can our
be installed
on our
machine
any other operating system. XCP was chosen both for convenience, and because it is the
open-source equivalent of Citrix XenServer, which is commonly used in datacentres [39]. The
ISO file includes Xen 4.1.3, a pre-configured CentOS 5.7 as the control domain’s operating
system, the XAPI toolstack for management, drivers for a large number of devices, and many
other components that facilitate interaction with Xen [40] (see Figure 3.2).
3.3
Virtual Hardware
Once installed on our machine, XCP dedicated 1 VCPU and 950MB of RAM to the control
domain. In addition, Dom0 used 4.0GB of hard disk space to function. This meant that there
was 7250MB of RAM and our entire hard drive (500GB) left for our virtual machines to use.
As we will see later, each virtual machine is created using three parameters:
• VCPU: Number of virtual CPUs assigned to the VM. Note that the total number of
VCPUs is not limited by the number of logical cores (i.e., we could assign more than 8
VCPUs in total even though we only have 4 physical and 8 logical cores on our machine).
21
XCP
XAPI
Dom0
(CentOS)
DomU
DomU
Xen (Hypervisor)
Hardware
Figure 3.2: Xen Cloud Platform and its components
• Memory: Amount of RAM assigned to the VM. The total amount of RAM assigned to
all virtual machines is limited by the amount of physical RAM that is available.
• Hard Disk: Amount of HDD space assigned to the VM. This is also limited by the
amount of available hard disk space.
3.4
Network
Our project machine automatically had access to the Internet via the College network. In
practice, this meant that Dom0 (CentOS) had access to the outside world, but not the DomU.
In order for the virtual machines to be connected, we needed one MAC address per VM,
and a corresponding reserved IP per MAC address on the DHCP server. The Computer
Support Group graciously allocated us 20 MAC addresses (02:d0:d0:59:61:[50-63]) and 20
corresponding IP addresses (146.169.15.1[50-69]) on the College network, thus allowing our
guests to interact with each other, with other College machines and with the outside world.
As a result, our virtual machines were given the hostnames project12-vm[01-20].
Finally, for security reasons and due to general College restrictions, neither project12, nor
the virtual machines were accessible from outside the College network. When access from
outside was required, we connected to the internal network via VPN or through a two-hop
SSH connection via shell[1-4].
3.5
Management
As we mentioned in 3.2, XCP includes the XAPI toolstack, a set of tools and APIs to manage
Xen and its guests. This enabled us to use xe, a powerful command-line interface available in
Dom0, and XenCenter, a Windows GUI application.
3.5.1
xe CLI
The xe command-line interface facilitates the management of virtual machines. This includes
creating, deleting and copying guests, as well as changing VM parameters such as the amount
of RAM, the number of VCPUs or the MAC address of the virtual network interface attached
to it. The command-line interface proved especially useful when writing scripts to manage all
aspects of the virtual machines, as we will see in later sections.
22
Figure 3.3: Configuration Interface of Xen Cloud Platform
xe <command-name> <argument=value>
# General layout of an xe command
xe vm-list
xe vm-start uuid=xxxx
xe vm-destroy uuid=xxxx
# List all VMs and their state
# Start the VM with the specified UUID
# Destroy the VM with the specified UUID
xe vm-param-get uuid=xxxx param=xxxx # Get a parameters of the VM
xe vm-param-set uuid=xxxx param=xxxx # Set a parameter of the VM
xe template-param-list uuid=xxxx
xe vbd-destroy uuid=xxxx
xe vif-create mac=xxxx
# List all parameters of a template
# Destroy the virtual block device
# Create a virtual network interface
Figure 3.4: Examples of xe commands
3.5.2
XenCenter
When we did not need to use the command-line interface (e.g. for one-off operations), or for a
better overview of the state of our virtual machines, we turned to XenCenter.
XenCenter is actually the graphical management interface that ships with Citrix XenServer
(the commercial version of Xen Cloud Platform). Since XenServer and XCP share the same
codebase and toolstack (XAPI), XenCenter is fully compatible with XCP.
23
Figure 3.5: The XenCenter Interface
24
4 Implementation
Investigating the scalability of virtualised systems requires us to gather data by running a
number of experiments within a virtualised environment and monitoring their performance.
This chapter presents the design and implementation of the programs we used to run our
experiments.
4.1
Early experiments
In this section, we describe the implementation of the framework we used for our early
experiments. The purpose of these experiments was to familiarise ourselves with Xen Cloud
Platform. To this end, we wrote scripts that enabled us to run some short experiments (inspired
by Benevenuto et al. [3]).
4.1.1
Virtual machines
Before we can run programs and measure their performance, we need virtual machines. This
section describes how we create them in an automated fashion, using bash scripts. We followed
the excellent tutorial by Grant McWilliams [12] and used his scripts with some modifications.
Create a template
Create the VM
Install Ubuntu
Post-installation
Figure 4.1: Virtual machine creation process
Templates
With Xen Cloud Platform, virtual machines are created from templates. Templates are preconfigured “images” that can be instantiated into virtual machines in two ways [38]:
• by using a complete template with a pre-installed operating system
• by installing the operating system onto the instantiated template over the network
XCP ships with a range of templates (Debian, Red Hat, Ubuntu, Windows, etc.), which
contain settings relevant to each of the operating systems (e.g. installing the paravirtualised
drivers) and allow virtual machines to be installed simply by providing a network repository
or an ISO image of the OS install disk.
26
Operating System
Throughout our experiments, we used Ubuntu 12.04 Precise Pangolin on our virtual machines.
Ubuntu is one of the most widely used Linux distributions in the world, with more Xenspecific documentation available on the Internet than any other distribution. In addition,
version 12.04 is an LTS (Long-Term Support) version, guaranteeing stability and robustness.
Finally, and most importantly, Ubuntu is one of the distributions that is available in a
Xen-optimised paravirtualised version.
Template creation
The first step consists in creating a template with our chosen hardware configuration. To
achieve this, we clone the existing “Ubuntu Precise Pangolin 12.04 (64-bit)” template and
modify its parameters to specify the amounts of memory and hard disk space we want to
allocate, using the xe template-param-set command.
Once we have our template, we can instantiate a virtual machine from it. This is done
using the xe vm-install command.
Finally, we create a new virtual network interface (xe vif-create) and assign it to our
newly created VM, specifying the MAC address that it will be using. This ensures that our
virtual machine will be able to obtain an IP address from the DHCP server.
These commands are executed via a bash script create_template.sh, which takes as
arguments how much RAM (in MB) and hard disk space (in GB) we want our template to be
allocated.
Unattended installation
We now have a virtual machine, but there are still a few parameters to change before the
operating system can be installed onto it.
Since we need our virtual machines to use paravirtualised drivers, we are required to install
Ubuntu from a network repository1 . For obvious reasons, we choose the UK Ubuntu archive
(http://uk.archive.ubuntu.com/ubuntu/), which we set using the following command:
xe vm-param-set uuid=${VMUUID}
other-config:install-repository="http://uk.archive.ubuntu.com/ubuntu/"
In addition, we would like the installation to run on its own without prompting us for
configuration details. To this end, we use a feature of DebianInstaller2 called preseeding.
Preseeding enables us to specify answers to the questions asked by DebianInstaller in a
configuration file. We upload our configuration file on a web server and pass it to the installer:
xe vm-param-set uuid=${VMUUID}
PV-args="preseed/url=http://www.doc.ic.ac.uk/~nnk09/fyp/ubuntu.cfg"
Our preseed file is based on the standard Ubuntu preseed [32]. The only major difference is
the set of packages to install, which we choose to be the “standard” metapackage. This means
that in addition to the essential packages required to boot, connect to a network and install
other packages, we installed a set of packages that “provide[d] a comfortable command-line
Unix-like environment.” [33]. When additional packages were required, they were installed on
1 standard
2 the
ISO images do not contain paravirtualised drivers
installation system for Debian-like distributions
27
an individual basis after the VM was created, thus keeping our operating system as minimal
as possible.
Finally, we start the virtual machine (xe vm-start) and wait for the installation to complete.
These commands are executed via a bash script install_ubuntu.sh, which takes as arguments an index (01-20 – the suffix of the VM’s hostname, e.g. project12-vm01), and amounts
of RAM (in MB) and hard disk space (in GB). We include the possibility of installing multiple
virtual machines with install_multiple.sh, which simply loops through an array of indices
(e.g. 01 to 10), and calls install_ubuntu.sh.
Post-installation
Once the installation has completed, we run a script post_install.sh to add our SSH keys to
~/.ssh/, so that we can connect directly to the VM without being prompted for a password
every time. Our script also installs XenServer Tools on the virtual machine, a guest agent which
provides XCP with additional information about the VM and the OS that runs in it, such as
Linux distribution name, kernel version, IP address and total & free memory.
4.1.2
Experiments
Now that we have a virtual machine to work with, we would like to run experiments, once
again in an automated fashion. To this end, we design a simple framework that enables us
to run any command-line utility on a single virtual machine, while monitoring the VM’s
processor, memory and network performance. The architecture of our framework can be seen
in Figure 4.2.
Machine
vm02
...
...
pr
e
pa
re
vm01
gi
ng
Hypervisor
ru
n
st
ar
tl
og
(project12)
ging
t log
star
Main computer
plot
Figure 4.2: Overview of the experimental framework
As before, our framework is implemented with a collection of simple bash scripts. The
main script is run.sh (see Appendix D) which takes two arguments: exp_id and vm_id, and
is executed from a “main computer” (personal laptop or any other machine on the College
network).
28
In our implementation, we ran 4 experiments, each characterised by an experiment ID:
• (001) File copy: copies a 2GB file to another directory on the same drive
• (002) Fibonacci: calculates the Fibonacci sequence for n=50 (using the naive algorithm
int fib(int n) { if(n < 2) return n; return fib(n-1) + fib(n-2); })
• (003) OpenSSL Hash Calculation: run openssl speed, a built-in benchmarking suite
that calculates hashes using different algorithms (MD5, SHA-1, RSA-2048 etc.)
• (004) Linux Kernel: compiles the Linux Kernel (version 3.7.6, latest as of 06/02/2013)
Preparation & Running
Each experiment consists of at least two files, pre.sh and exp.sh, along with any other
files relevant to the experiment (e.g. for Fibonacci, a C++ source file). pre.sh contains any
commands that need to be executed for the experiment to run successfully, whereas exp.sh
runs the actual experiment.
As an example, let us have a closer look at the scripts for the Linux kernel experiment. In
the preparation script 004/pre.sh (see Figure 4.3), we connect to our virtual machine over SSH
and download a tarball containing the Linux kernel source code, as well as a configuration file
from a remote server. We also decompress the tarball and move the configuration file where it
belongs.
echo "Linux Kernel Compilation"
cd /root/
wget http://www.doc.ic.ac.uk/~nnk09/fyp/linux-3.7.6.tar.bz2
tar -xvjf linux-3.7.6.tar.bz2
cd linux-3.7.6/
wget http://www.doc.ic.ac.uk/~nnk09/fyp/kernel_config
mv kernel_config .config
Figure 4.3: pre.sh script for the Linux Kernel experiment
Once the preparation is over, we simply need to run the experiment script 004/exp.sh (see
Figure 4.4) from the virtual machine. Since all the relevant files are located where they belong,
the experiment consists in running a simple make.
echo -n "Experiment ${EXP_ID}: Linux Kernel Compilation..."
cd ~/linux-3.7.6/
make
echo "Done"
Figure 4.4: exp.sh script for the Linux Kernel experiment
Logging
The logging of processor, memory and network data is done both at Dom0 and DomU level,
using the excellent resource statistics tool dstat by Dag Wieers [7], which enables us to collect
data for all of our system resources at once, while the experiment is running.
Logging starts after the experiment preparation is done and before the experiment starts
running, using two scripts xcp.sh and vm.sh, which are called from the main script run.sh.
29
The scripts both start in the background, run dstat and save the logs locally (i.e., on the
machine they were logging).
Once the experiment has finished, the logging is stopped remotely by connecting to
the hypervisor and the VM over SSH and killing the dstat process. Finally, the logs are
downloaded from both machines onto our main computer.
Plotting
The final part of our framework consists in plotting the experimental data (logs) that we
obtained from our two machines. This is done using gnuplot [9] scripts that output one pdf
file per plot. We choose to have one plot per resource per machine, which equates to 6 plots in
total per experiment (2 machines, 3 resources), an example of which can be seen in Figure 4.5.
CPU
100
user
system
wait
80
percent
60
40
20
0
18:20
18:30
18:40
18:50
19:00
19:10
19:20
19:30
time
Figure 4.5: Virtual machine CPU usage during the Linux kernel compilation
4.2
The Wikipedia Framework
Instead of benchmarking multiple applications, we decide to focus on one widely used web
application. Inspired by Mazzucco and Mitrani [20], we set our choice on MediaWiki, which
powers Wikipedia, one of the most widely visited website in the world. We implement an
experimental framework to automatically deploy virtual machines, send requests to Wikipedia,
and log performance data.
4.2.1
Design
This section details the major design choices that we made during our implementation.
Framework
The experimental framework for the evaluation of Wikipedia is similar to that of the experiments we presented in section 4.1. Once again, it is centred around a “main computer”, which
30
we will call “client” in this section. The client machine runs scripts and sends requests to the
virtual machines, and processes the logs when the experiments are over.
As we will see in the remainder of this section, some of the challenges that we face are
similar to those faced in our earlier experiments. As a consequence, we draw on our previous
work while heavily extending the experimental framework to fit new requirements.
Instances
The Wikipedia encyclopedia runs on a free web application called MediaWiki, which is
developed by the Wikimedia Foundation. MediaWiki follows a relatively standard design,
with a front-end written in PHP that runs on any standard HTTP server (Apache in our case),
and a back-end that relies on a MySQL database. We use MediaWiki 1.20.3, Apache 2.4.4 and
MySQL 5.5.
back-end
(MySQL)
Images
front-end
(Apache)
Figure 4.6: An instance of Wikipedia, and the image repository
Throughout this section, we will refer to instances of Wikipedia, which run MediaWiki
(version 1.20.3). An instance is composed of a virtual machine for the front-end, and another
for the back-end (see Figure 4.6). We choose to separate the front-end from the back-end to be
able to monitor the performance of each independently.
Additionally, we have a separate image repository, which serves images for all of the
instances. Thus, we have two virtual machines per instance and a global image server, all
of which have Apache and MySQL installed, and run separate installations of MediaWiki to
communicate with each other.
As an example, let us consider the setting of Figure 4.6: we have one instance of Wikipedia
(two virtual machines front-end and back-end), along with the image repository images. A
user visiting our instance will be interacting with front-end, and when she requests a page:
1. front-end fetches the page data (text and formatting of the article) from back-end’s
database and the URLs of the images contained in the article from images’s database
2. front-end then renders the page and sends it back to the user’s browser
3. the browser then proceeds to download the images directly from images using the URLs
that were provided to front-end by images in step 1.
Therefore, in the case of the front-end, the database is unused since the data required by
the rendering process is fetched from the back-end server. For similar reasons, the Apache web
server on the back-end is never used. Finally, for the image repository, the database is used to
provide URLs to the front-end, and Apache is used to serve the images that are requested by
the client.
31
...
db
db
data
db
images
...
http
re
qu
es
t
pa
ge
pa
ge
http
s
age
t im
ues
req
http
Hypervisor
(project12)
ges
ima
Client
Figure 4.7: Overview of the experimental framework for requests sent to a single instance
In the remainder of this section, we will use the terms front-end and Apache server interchangeably. Similarly, back-end, MySQL server and database are equivalent.
Scripting language
Previously, we relied exclusively on simple bash scripts to prepare, run and log the experiments.
For this set of experiments, we decided to switch to the Python programming language.
Python has a number of features that made it particularly attractive for our implementation.
As a simple and concise scripting language, communicating with existing environments (e.g.
the xe command-line interface) was made very easy. Additionally, its rich collection of modules
facilitated our implementation of more complex features, as we will see later. Finally, Python
ships by default with Linux distributions and is available on other platforms, ensuring that
our scripts can be run almost everywhere.
We used Python 2.7.3 and the following modules3 :
•
•
•
•
•
•
•
•
•
argparse: parser for command-line arguments
csv: reading, writing and parsing .csv files
datetime: date/time manipulation
matplotlib 1.2.1: plotting graphs
numpy 1.7.1: extended maths functions
paramiko 1.10.1: easy SSH connection
requests 1.2.3: sending HTTP requests
smtplib: sending emails
threading: high-level threading interface
3 modules
that do no have a version number are part of the standard library
32
4.2.2
Templates
In section 4.1.1, we introduced the notion of templates, that is, “images” that form the basis for
creating virtual machines. Templates can easily be created from existing virtual machines by
changing the is-a-template flag in the VMs parameters using the xe command-line interface.
Thus, we create three templates: vm-apache, vm-mysql and vm-images by instantiating
three VMs, making relevant modifications and converting them to templates. The resulting
templates can be instantiated into guests in a very short amount of time, thus enabling us to
quickly create instances of Wikipedia.
All three templates were created from a base installation of Ubuntu 12.04 Precise Pangolin
as described in section 4.1.1.
As we pointed out in section 3.4, we were allocated 20 hostnames project12-vm[01-20],
along with corresponding MAC and IP addresses. We allocate project20 to the images server,
project12-vm[01-08] to Apache servers and project12-vm[09-16] to MySQL servers.
vm-apache
This template is used to create Apache virtual machines, which contain no data other than
a base installation of MediaWiki and some configuration files. As such, we allocate 10GB of
hard disk space to the template (and thus to each VM that is instantiated from it).
Note that the template is given the MAC address for project12-vm01 (02:d0:c0:59:61:50) –
every time a VM is instantiated, the MAC is modified to match the hostname we want it to
use (e.g. for project12-vm02, it will be changed to 02:d0:c0:59:61:51).
First, we install MediaWiki using a simple bash script:
cd /tmp
wget http://www.doc.ic.ac.uk/~nnk09/fyp/mediawiki-1.20.3.tar.gz
tar xf mediawiki-1.20.3.tar.gz
mv mediawiki-1.20.3/ /var/www/wiki/
chmod a+x /var/www/wiki/mw-config/
Once the installation has completed, MediaWiki is configured with default settings using a
wizard. The resulting default configuration file, /var/www/wiki/LocalSettings.php needs to
be modified for three reasons:
• the database settings point to the local vm-apache database, which is empty. Instead, we
want it to point to a vm-mysql VM.
• we would like to enable PHP caching, a common optimisation [21] which avoids PHP
files being compiled into bytecode everytime they are executed.
• the image server is configured to be Wikimedia Commons. This means that image
URLs will point to a remote location (e.g. http://upload.wikimedia.org/wikipedia/
commons/8/8d/SecondSevernCrossing_Jan2006.jpg), instead of pointing to our own
image repository.
To this end, we create two files LocalSettingsCommons.php and LocalSettingsRepo.php,
based on the default configuration file. In both files, we set the database to be project12-vm09
and enable PHP caching using the Alternative PHP Cache [21] by adding the following
directives:
$wgDBserver
$wgMainCacheType
=
=
"146.169.15.151";
CACHE_ACCEL;
33
# IP address of project12-vm09
In LocalSettingsRepo.php, we also add the following lines to make MediaWiki use our
image repository instead of Wikimedia Commons:
$wgUseInstantCommons
= false;
$wgForeignFileRepos[] =
’class’
’name’
’url’
’directory’
’dbType’
’dbServer’
’dbUser’
’dbPassword’
’dbName’
’fetchDescription’
array(
=> ’ForeignDBRepo’,
=> ’shared’,
=> "http://project12-vm20.doc.ic.ac.uk/wiki/images",
=> ’/var/www/wiki/images/’,
=> "mysql",
=> "project12-vm20.doc.ic.ac.uk",
=> "root",
=> "password",
=> ’wikidb’,
=> true
);
Notice that the class directive is set to ForeignDBRepo. This means that whenever the
Apache server needs information about an image, it will connect directly to the database on
the image server and fetch what it needs.
MediaWiki offers another option, ForeignAPIRepo, where the Apache server fetches information from the database via the MediaWiki API (e.g. by calling http://project12-vm20.
doc.ic.ac.uk/wiki/api.php). We used this option in the early stages of our implementation,
but soon found that calls to api.php were very expensive and considerably slowed down the
image repository, thus affecting experiments results.
The two configuration files are uploaded on the Internet and we create two bash scripts
enableRepo.sh and enableCommons.sh (see Figure 4.8), which we leave in the home directory
of our template. According to the configuration we want our VM to have, one of the scripts is
run immediately after instantiation to downlaod the relevant configuration and replace the
default. This enables us to change settings without having to modify the template every time
(for more detail about deploying VMs, see section 4.2.3).
cd /var/www/wiki/
mv LocalSettings.php LocalSettingsOriginal.php
wget http://www.doc.ic.ac.uk/~nnk09/fyp/LocalSettingsRepo.txt
mv LocalSettingsRepo.txt LocalSettings.php
chown 1001:1001 LocalSettings.php
chmod ug+w LocalSettings.php
~/updateLocalSettings.sh
Figure 4.8: Script to enable the use of the image repository (enableRepo.sh)
When a virtual machine is created from our vm-apache template, its LocalSettings.php
will refer to project12-vm01 and project12-vm09 as the server and database hostnames. This
is an issue when we deploy more than one instance, since different instances will be made of
guests with different hostnames (e.g. if we create two instances, we will have (vm01, vm09)
and (vm02, vm10) pairs).
To solve this problem, we create updateLocalSettings.sh, which updates the configuration
files to refer to the correct hostnames, according to the name given to the VM by the DHCP
server (recall that each guest is given MAC and IP addresses which uniquely define it on the
network).
34
For example, let us consider a setting where two virtual machines vm01 and vm02 are
created from the vm-apache template. The configuration for vm01 will be correct since the
template was created using the MAC address assigned to vm01. For vm02, the MediaWiki
configuration file will have the same settings as for vm01, which is incorrect:
$wgServer
$wgDBserver
= "http://project12-vm01.doc.ic.ac.uk";
= "146.169.15.158";
# refers to vm09
Our script updates this information to the correct settings using sed:
$wgServer
$wgDBserver
= "http://project12-vm02.doc.ic.ac.uk";
= "146.169.15.159";
# vm10
vm-mysql
We create a second template vm-mysql for back-end virtual machines. This template is similar
to vm-apache in that we allocate it 10GB of hard disk space and install MediaWiki.
The difference lies in the fact that we populate the database with the contents of a Wikipedia
backup dump, from which the front-end will fetch the data required to rendered article pages
(text and layout).
The Wikimedia Foundation offers free access to dumps of the Wikipedia database in XML
format [36]. At the time of our implementation, the latest dump was dated from 4 March 2013.
The size of the full dump of all articles, templates and primary meta-pages was approximately
9.1GB in (bzip2) compressed format, with an uncompressed size of ~50GB. Instead, we used a
sample of this dump, namely the first 10,000 articles & pages4 , with an uncompressed XML
size of 146.6MB.
The next step consists in importing this XML file into the MySQL database such that
MediaWiki is able to access the data. We use mwimport.pl[35], a Perl script provided by the
Wikimedia Foundation for this purpose, which we call using the following command in the
vm-mysql template:
bzcat enwiki-20130304-pages-articles1.xml-p000000010p000010000 \
| perl mwimport.pl | mysql -f -uroot -ppassword wikidb
Finally, we modify mysql.conf to allow remote hosts (in particular, our Apache servers) to
connect to the database.
vm-images
The last template we create is vm-images, on which we also install MediaWiki. We initially
allocate this template 50GB of hard disk space.
Having installed MediaWiki, we need to:
1.
2.
3.
4.
Parse the XML database dump for image filenames
Reconstruct the direct link to the image on the Wikimedia Commons servers
Download the images from Wikimedia Commons
Import the images into MediaWiki’s database
We write a Python script, images.py to achieve the first three steps:
images.py enwiki-20130304-pages-articles1.xml-p000000010p000010000 ~/images/
4 http://dumps.wikimedia.org/enwiki/20130304/enwiki-20130304-pages-articles1.
xml-p000000010p000010000.bz2
35
In Wikipedia articles, an image will be referred to by [[File:image_filename.jpg]].
Therefore, we simply search the dump for the [[File: string, get the string located between
that and the next ]] string and check whether if it is a valid filename that ends in .jpg, .png
or .svg5 .
Next, we need to construct the direct link to the image, so that we can easily download
it. Within Commons, images are sorted in subdirectories of the form /x/xy/. These letters
correspond respectively to the first and first two letter of the MD5 hash of the filename, which
we can easily calculate in our script.
Let us consider the following example:
[[File:Building the Second Severn Crossing - 1993.jpg]]
Building the Second Severn Crossing - 1993.jpg
Building_the_Second_Severn_Crossing_-_1993.jpg
6c8681b7c9d2e58e4845c06c6421e692
/6/6c/Building_the_Second_Severn_Crossing_-_1993.jpg
#
#
#
#
#
As referred to in the dump
Parse out the filename
Replace ’ ’ with ’_’
MD5 Hash of the above
Commons subdirectory path
which yields the following direct link http://upload.wikimedia.org/wikipedia/commons/6/
6c/Building_the_Second_Severn_Crossing_-_1993.jpg.
Using this method, we compute direct links to all the images that we parse from the XML
dump. In our case, this amounted to ~15GB of images, which we download to a directory in
the template. After all the images have been downloaded, we use a PHP script provided with
MediaWiki (importImages.php) to import the images into the MySQL database:
php importImages.php ~/images/ svg png jpg jpeg gif SVG PNG JPG JPEG GIF
Once this process has finished, we resize the template’s hard disk to 20GB. This makes it
faster for us to instantiate VMs, as it cuts the time to copy the template’s virtual hard drive by
half.
Finally, as we did for vm-mysql, we modify mysql.conf to allow the front-end servers to
connect to the database.
4.2.3
Deployment
Having created our templates, we write a Python script, deploy.py, to quickly and easily
deploy instances of Wikipedia on our project machine:
deploy.py [-h] [--images] (--repo | --commons) instances ram vcpu
The command-line arguments enable the user to specify whether or not to create an image
repository (–images and whether to use the image repository (–repo) or Wikimedia Commons
as the source for images. Note that the latter two flags basically specify whether to run
enableRepo.sh or enableCommons.sh (which we introduced in section 4.2.2) after the instances
have been created.
Furthermore, and perhaps more importantly, the user can also set:
• instances: number of instances to create
• ram: amount of RAM to allocate to each virtual machine (in MB)
• vcpu: number of virtual CPUs to allocate to each virtual machine
For more details on the limits for RAM and VCPU, we refer the reader to section 3.1.
For example, if the user wants to create 4 instances with 1024MB of RAM and 2 VCPUs for
each VM, an image repository (which is always created with 2048MB of RAM and 1 VCPU)
and to enable the instances to use the images server, she will run the command:
5 in
reality, there are more extensions and a number of edge cases that our script handles
36
deploy.py --images --repo 4 1024 2
In debug mode, the script outputs every xe command that it runs on the hypervisor (see
Figure 4.9)
Figure 4.9: Output of deploy.py in debug mode, creating 2 instances
The script is essentially a wrapper around xe commands. The deployment process is based
on the three templates vm-apache, vm-mysql and vm-images and on the newVMFromTemplate
function which takes two arguments.
This function copies the template specified by the template_uuid argument and converts
the new template into a virtual machine (by setting its is-a-template flag to false). It then
uses the index argument to generate the corresponding MAC address and creates a virtual
interface using that MAC address, which it attaches to the virtual machine. Finally, it modifies
the RAM and VCPU allocations according to the values from the command-line.
4.2.4
Load generator
The purpose of the load generator is to simulate the behaviour of multiple users browsing our
Wikipedia instances, and thus to fetch pages like a regular browser would.
We investigated different existing load generators such as httperf and Apache JMeter, but
found that they were too simple, too complex, or simply did not achieve what we wanted.
Therefore, we decided that writing our own would be the best solution, giving us the most
flexibility.
From the server’s perspective, when loading a page, a browser usually6 [22]:
6 we
omit some steps that are irrelevant to the design of our load generator (e.g. opening a TCP connection)
37
•
•
•
•
Sends a requests to the server for the page
Downloads the HTML response from the server
Parses the HTML response for referenced content (images, videos etc.)
Downloads all the referenced content
Thus, we design and implement a Python module that performs the same actions, which
we call load.py (see Appendix E). Note that load.py is a module, not a script – it is used in a
more general script called experiment.py (see section 4.2.5).
The main part of the code is the Request class, which is derived from threading.Thread.
This enables us to send requests concurrently, that is, new requests can be sent without waiting
for previous requests to return.
The Request class has 3 attributes:
• index: the request number
• delay: the (exponential) delay before the request is sent to the server
• url: the URL to which the request is sent
and a single method run(self), which is executed when the Thread/Request is started.
The run method performs the actions described above (see Figure 4.10). It first sends a
request for the page located at self.url. The page that is returned is then parsed for images
(essentially, by looking for any string that starts with “project12-vm20.doc.ic.ac.uk” and ends
in a valid image extension), which are finally downloaded.
Thus, when we refer to a request being sent and having returned, we mean that we have
attempted to fetch a page like a browser would have, and that the page and all of its referenced
content have been downloaded.
If any errors occur (e.g. if the connection is reset by the peer, or times out), an exception is
thrown, handled gracefully and recorded in the log file. Client-side logging (time and errors)
is also done in the run method, and is discussed in further detail in section 4.2.6.
start_time = datetime.datetime.utcnow()
try:
r = requests.get(self.url)
except (ConnectionError, Timeout):
print "Error: " + str(self.index)
end_time = datetime.datetime.utcnow()
else:
image_time = datetime.datetime.utcnow()
for image_url in getImages(r.text):
requests.get(image_url)
end_time = datetime.datetime.utcnow()
# page request is sent
# error occurs
# page request returns
# images start being downloaded
# images finish downloading
Figure 4.10: The run method of the Request class
The fact that we can send requests concurrently enables us to define the amount of time to
wait between the sending of requests. To determine this delay (for each request), we sample
an exponential distribution whose only parameter (the rate), is defined for each experiment in
experiment.py. As such, the arrivals of our requests at the server define a Poisson Process
(see Appendix B).
Finally, we would like to point out that the load generator was made particularly easy to
implement thanks to the excellent Requests module by Kenneth Reitz [19].
38
4.2.5
Experiment
The main script that we write is called experiment.py. The purpose of this script is to run an
experiment, by performing the following actions:
•
•
•
•
•
•
Start logging on every machine
Generate load on the Wikipedia instances
Stop logging
Retrieve logs from all machines
Perform cleanup
Send an email alert
The script is called in the following way:
experiment.py [-h] instances rate count
The first argument, instances, is the number of instances that we want to put load on
(usually those that we created with deploy.py). We pass two further parameters rate and
count – the former is the mean rate at which request should be sent (i.e., the parameter to
the exponential distribution we sample for our delays), while the latter is the total number of
requests that we send throughoutt the experiment.
Figure 4.11: Output of experiment.py in debug mode, sending 100 requests to 2 instances at a
mean rate of 10 req/s
URLs
Before we can start generating load, we need to know which URLs we will be sending the
requests to.
In the early stages of our implementation, we sent requests to the Special:Random page on
each instance (e.g. project12-vm02.doc.ic.ac.uk/wiki/Special:Random). Special:Random
39
is a special page in MediaWiki which simply redirects to a random article. This solution, while
convenient, turned out to be very expensive and significantly affected our experimental results.
Instead, we decide to write a simple Python script, links.py, which parses the database
dump for all article titles and outputs them to a file. Thus, to generate a list of URLs, the
experiment script will parse the output file generated by links.py, randomly choose n article
titles (where n = the count argument) and reconstruct the links to the articles based on the
number of instances that we will be hitting.
For example, if we are hitting four instances, we will randomly prepend project12-vm[01-04]
to the randomly chosen article titles, thus ensuring that we are hitting random articles on all
instances.
Timing issues
We begin an experiment by starting logging at every level, that is, on the client machine, the
hypervisor, and every virtual machine that make up the instances.
Our main constraints are that logging should (ideally) start at the exact same time on
every machine, and that it should happen in the background. Thankfully, all of our machines
(including the client) reside on the same network and are synchronised with the same time
server. This enables us to use the UNIX command at to schedule the logging instead of
starting it immediately.
The first step consists in determining at what time the logging should start. We need to
give our script enough time to connect to each machine and schedule the logging, which
usually takes less than 30 seconds. We could simply calculate what the time will be in 30
seconds and pass this to at but unfortunately, at only accepts times of the form HHMM, which
means that the logging has to start on the minute.
Therefore, we use the following algorithm to determine what time to pass to at:
def logging_start_time():
now = datetime.now()
if now.second < 30:
return (now + timedelta(minutes = 1, seconds = -now.second))
else:
return (now + timedelta(minutes = 2, seconds = -now.second))
For example, if now = 20:13:15, the algorithm will return 20:14:00, giving the script 45
seconds to connect to the machines and schedule dstat to start. If now = 20:13:45, then the
function will return 20:15:00. Thus, this algorithm ensures that the logging will start at least
30 seconds and at most 90 seconds from now.
Given the start time, the script can now connect to each machine and schedule dstat to
start silently:
echo "dstat -tcmndy -C 0,1,2,3,total --output=some_file_name" | at 2014
For more details on logging, we refer the reader to section 4.2.6
Load
Having scheduled dstat on all machines, we wait until it starts and call the textttputLoad
function of the load module described in section 4.2.4.
The putLoad function has four parameters:
• urls: a list of URLs to hit
40
• count: the number of requests to send
• rate: the mean rate at which to send the requests
• log_file: the path where the client machine’s log is saved
Note that count and rate are the same as the arguments passed to experiment.py via the
command-line.
When called, putLoad instantiates a Request object (and thus, a Thread) for each request to
be sent, with delays defined by the samples from the exponential distribution between each
instantiation.
Finally, it waits for all the requests to return (which can take a long time depending on
how fast the Wikipedia instances process the requests), before returning itself.
Post-experiment
After all the requests have return, we perform some post-experiment tasks which include
connecting to each machines and killing dstat, retrieving all the logs to the client machine,
and restarting Apache on the front-end machines and the image repository.
Because some experiments can run for a long time, we usually run the experiment script
on a College computer in the background. Additionally, we send an email to the user to inform
her that the experiment has completed, saving the user having to periodically check whether it
has finished or not.
4.2.6
Logging
As we have seen in the previous section, logging is started before putting load on the servers,
and all machines start logging at the same time. We record data at all three levels, that is, on
the client machine, on the hypervior (project12) and on every virtual machine.
On the client machine, we record information about the requests that are sent. This is done
in the run method of the Request class in load.py. For each Request, we record three points
in time:
• start: time at which the request for the page is sent
• image: time at which the page is received and the first image starts downloading
• end: time at which all the images have been downloaded
These three points enable us to calculate exactly how long it took for the page to be received,
for the images to be downloaded, and by adding these two values, the time it took to get the
entire page like a browser would have.
Additionally, we log the request index, the delay preceding the request (i.e., the sample
we got for that Request from the exponential distribution), and the URL that was hit by the
request. When an error occurs, we catch the exception (ConnectionError or Timeout) and
record the URL as Error. This data is saved to a CSV file on the client machine. A sample of
experimental data from the client can be seen in Figure 4.12.
Figure 4.12: Sample of data from the client machine as seen in Microsoft Excel
41
With regards to the hypervisor, logging is done in the same way as in our early experiments
(see section 4.1.2). We use dstat to log RAM and CPU usage, as well as Network and Disk
I/O every second. dstat runs in the background and saves to a CSV file on the hypervisor.
We adopt the same strategy for all the virtual machines: we start dstat in the background on
all the VMs, log data about RAM/CPU usage and Network/Disk I/O and save it to a CSV file.
All of these logs are retrieved over SSH from the relevant machines to one of our computers
for processing (see section 5.1) after the experiment has completed. In addition to the logs
mentioned above, we collect logs from the Apache HTTP server for every front-end virtual
machine (those that serve the pages to the client), and for the image server. This is not
necessary for back-end (MySQL database) VMs since Apache is never used on those machines.
42
5 Evaluation
Having implemented our experimental framework, we now focus on running experiments and
gathering results. Our aim is to investigate the scalability of our virtualised environment by
finding out what happens when we change the parameters of our experiments, e.g. when we
•
•
•
•
increase the number of instances
reduce the RAM allocation per VM
gradually increase the rate at which the requests are sent
increase the number of requests that we send
5.1
Data Processing
Every time we run an experiment (i.e., send requests to a number of instances), we log a large
amount of performance statistics for each machine. This means that the amount of data to
process quickly increases. As an example, running an experiment on 4 instances results in
62.5MB of data in 96 different files (16 files per rate × 6 different rates).
Once again, we write a Python script (process.py) to process this data, perform relevant
calculations and output plots and experiment statistics. We use a combination of the numpy (a
scientific computing package) and matplotlib (for plotting) modules to carry out these tasks.
We also make heavy use of Python lists and standard library methods that apply to them.
For each plot, we follow these steps:
•
•
•
•
•
•
parse the CSV file(s) that contain the data we need
perform calculations on the parsed data (e.g. calculate means, add to other data etc.)
define what data is the x-axis (usually time or request rates)
plot the y-axis lines again the x-axis
specify the title of the plot, the axis labels and the contents of the legend
output the plot to a PDF file
Issues
We encountered a number of issues while processing the data and generating our plots.
First, parsing the log files was not as straightforward as it should have been. More
specifically, it turns out that the CSV format is not standardised, and in particular, dstat writes
multiple headers to its output, which broke our CSV parser. We solved this issue by merging
the headers and ensuring they were unique.
Furthermore, we were not familiar with the plotting module beforehand, and had to rely
heavily on trial and error to ensure that our plots represented the data that we wanted to show
in the way we wanted. As a consequence, we used a bottom-up approach to the design of the
script, resulting in sub-par code quality.
43
Extra features
We implemented the ability to create LATEX files that include ‘groups’ of plots per experiment
group or per rate. This enables us to generate a PDF file that contains multiple plots for
comparison, and thus visualise our data more easily.
In addition to this, we added a command-line interface that enables us to generate only
certain plots, instead of generating every plot every time we execute the script.
An interesting point to note is that our process.py script contains more lines of code than
all of the scripts of our experimental framework combined! (681 vs. 522 according to cloc)
5.2
Experiments
As we mentioned in section 4.2.2, we used a database dump from the English version of
Wikipedia, dated from 4 March 2013. The sample we used contains a total of 10,000 pages &
articles which amounts to approximately 6,300 different articles that we can hit.
We decided to run two sets of experiments, which we will refer to as full and single
experiments. As a reminder, each experiment consists of sending a number of requests (with
an exponential delay between each) to some instances of Wikipedia at different mean rates.
In the first set (full experiments), we gradually increase the number of instances of
Wikipedia from 1 to 8 but keep a single image repository that serves all the instances1 .
Every time we add instances, we reduce the RAM and VCPU allocation per instance (i.e., for
each pair of Apache & MySQL VMs), but keep the resources of the image repository constant
at 2048MB of RAM and 1 VCPU. In short, we split the resources of our machine between more
and more virtual machines:
ID
Instances
Apache VM
MySQL VM
RAM/VM
VCPU/VM
Total RAM
Total VCPU
1
2
3
4
1
2
4
8
1
2
4
8
1
2
4
8
2048MB
1024MB
512MB
256MB
3
2
1
1
6144MB
6144MB
6144MB
6144MB
7
9
9
17
Table 5.1: Resource allocation of instances for the first set of experiments. Note that Total
RAM/VCPU include the resources allocated to the image VM
The second set (single) is similar to the full experiments with the notable exception that
each experiment is limited to a single instance. The purpose of these single experiments is to
compare them with the first set and investigate how similar instances conflict with each other
when running at the same time:
ID
Instances
Apache VM
MySQL VM
RAM/VM
VCPU/VM
Total RAM
Total VCPU
5
6
7
1
1
1
1
1
1
1
1
1
1024MB
512MB
256MB
2
1
1
6144MB
6144MB
6144MB
5
3
3
Table 5.2: Resource allocation of instances for the second set of experiments.
For example, consider experiment no. 4 in the first set. This experiment is simply composed
of 8 copies of the instance that makes up experiment no. 7 in the second set.
1 the reason for this is that multiple image repositories would be too expensive in terms of hard disk space – the
image server contains 15GB of images
44
For the full experiments, we send 1000 requests at rates ranging from 1.0 to 32.0 per
second (multiplying the rate by 2 every time). To make the comparison between single and
full experiments fair, we use the decomposition property of the Poisson Process2 (for an
introduction to Poisson Processes, see Appendix B).
When we send k requests at rate λ in an experiment that has n running instances of
Wikipedia, those k requests will each hit a given instance with probability 1/n. Thus, in the
corresponding single experiment, we send k/n requests to one instance at a rate λ/n. For
example, if we send 1000 requests at a rate of 32 req/s in experiment no. 4 (8 instances), the
equivalent single experiment (no. 7) will consist in sending 125 requests at a rate of 4 req/s.
We run all of these experiments from a client machine located on the same local network
as our project machine, and use screen to execute the script in the background.
5.3
Results
As we mentioned earlier, we run a total of 7 experiments, each with 6 different rates, hence a
total of 42 subexperiments.
5.3.1
Transaction Times
Our first area of investigation is transaction times (which we shorten to t.t.). We already know
that the inter-arrival times of requests are exponentially distributed (with a rate that we specify
for each experiment), and want to know how the transactions times are distributed. We define
three transaction times:
• total transaction time: time between when we send the page request and finish downloading the entire page including images
• request transaction time: time to fetch the page alone
• image transaction time: time to download all the images
and measure them from the client’s perspective (i.e., in our load generator, see section 4.2.4).
Note that for each request, we have that
total t.t. = image t.t. + request t.t.
and therefore for a group of requests,
mean total t.t. = mean image t.t. + mean request t.t.
Consider experiment no. 2 (in Figure 5.1). At high rates (in this case, 32.0 req/s), the
transaction times do not seem to be distributed in any meaningful way. This is a consequence
of requests taking longer and longer to complete as the instances try to keep up with the
successive arrivals, which also explains the very high mean transaction times for high rates.
At lower rates, however, the left tails of the distributions are very similar and look exponential. If we discard arrival rates that yield high mean transaction times (say, > 1.0 second),
we can try to approximate the exponential rate parameter that fits the remaining (low rate)
curves best.
To do so, we use a brute force approach. First, we calculate the transaction rate of each
curve (by taking the inverse of the mean transaction time). Then, we take the mean of
2 Recall
that because the delays between requests are all sampled from the same exponential distribution (where
the parameter is the rate at which we send the requests), the arrivals of our requests at the instances define a
Poisson Process.
45
3.00
1.0 req/s
2.0 req/s
4.0 req/s
8.0 req/s
16.0 req/s
32.0 req/s
2.75
2.50
probability density
2.25
2.00
1.75
1.50
1.25
1.00
0.75
0.50
0.25
0.00
0
1
2
3
4
5
6
transaction times
7
8
9
10
Figure 5.1: Distributions of total transaction times for experiment no. 2 (2 instances)
these transaction rates (call it mean_rate) and generate 1000 evenly spaced sample rates in
[0; 4 × mean_rate]. For each of these samples rates, we calculate the error using the following
algorithm:
def error(x, y_list, sample_rate):
exp_y = exp_distribution(x, sample_rate)
for y in y_list:
distances.append(norm(exp_y - y))
error = sum(distances)
return (sample_rate, error)
# for each curve
# |curve - exp. dist|
We calculate this error for the 1000 sample rates that we have generated and return
the sample rate with the minimum error (say, best_rate). Finally, we plot an exponential
distribution with parameter best_rate, as seen in Figure 5.2.
Although our example only applied to experiment no. 2, we find that all of our experiment
exhibit this characteristic, i.e. that the distribution of total transaction times for ‘well-behaved’
subexperiments can be approximated using the exponential distribution.
As a matter of fact, this is true not only for total transaction times in all subexperiments
but also for request and image transaction times, as seen in Table 5.3.
This is a very interesting result because it means that for each instance and for the image
repository, both the inter-arrival times and transaction times can be assumed to be exponentially
distributed (the former actually is, while the latter can be approximated). The corresponding
rates could then be used to parametrise some Markov model (for example, using PEPA – see
Appendix A).
5.3.2
Scalability
Using our processing script, we gather statistics about each experiment (a selection of which
is presented in Table 5.3). For each experiment, we calculate the mean total transaction time
46
3.00
1.0 req/s
2.0 req/s
4.0 req/s
8.0 req/s
exp(3.29)
2.75
2.50
probability density
2.25
2.00
1.75
1.50
1.25
1.00
0.75
0.50
0.25
0.00
0
1
2
3
4
5
6
transaction times
7
8
9
10
Figure 5.2: Distributions of total transaction times for experiment no. 2 with an exponential
distribution (discarding higher rates)
ID
Instances
RAM (MB)
VCPU
Total
Request
Image
1
2
3
4
5
6
7
1
2
4
8
1
1
1
2048
1024
512
256
1024
512
256
3
2
1
1
2
1
1
3.27
3.29
3.05
2.86
3.44
3.05
2.79
4.20
3.92
3.62
3.42
4.19
3.54
3.26
8.43
8.36
8.37
8.32
8.33
8.32
8.27
Table 5.3: Best fit parameters for the approximation of transaction times using the exponential
distribution
(MTT), mean request transaction time and mean image transaction time. We also calculate the
percentage of total time spent fetching the page and fetching the images. Finally, we define
low rates as ranging from 1.0 to 8.0 req/s, and high rates as 16.0 and 32.0 req/s.
At low rates, most MTTs are within the same order of magnitude and very low (< 1
second). In fact, the MTTs for full experiments are only marginallly greater than the MTTs
for their equivalent single experiment (see Table C.1 in Appendix C). The only exception to
this is for 8 instances, where the MTT at 8.0 req/s is 3.78, about 10 times greater than the
corresponding single experiment. Thus, at low rates, splitting resources between more VMs
only marginally impacts performance, but the more VMs we add, the higher the performance
degradation.
In Table 5.3, we focus on statistics for full and single experiments at high rates (i.e., those
rates at which the transaction times increase dramatically).
We notice that single experiments almost always outperform their multi-instance equivalent.
For example, sending 16 req/s to 8 instances has an MTT of ≈ 12,000s, where 2 req/s to a
47
Instances
RAM
VCPU
Requests
Rate
Total
Request
Image
Req. %
Img. %
Errors
1
1
2
2
1
1
4
4
1
1
8
8
1
1
2048
2048
1024
1024
1024
1024
512
512
512
512
256
256
256
256
3
3
2
2
2
2
1
1
1
1
1
1
1
1
1000
1000
1000
1000
500
500
1000
1000
250
250
1000
1000
125
125
16
32
16
32
8
16
16
32
4
8
16
32
2
4
4.59
24.32
5.29
1287.52
0.47
974.38
10.46
12588.82
0.81
6216.31
12113.64
40230.09
0.39
0.60
2.46
21.70
0.58
1286.93
0.32
967.92
1.58
12587.98
0.64
6214.45
12108.13
40229.75
0.30
0.50
2.13
2.62
4.72
0.59
0.15
6.47
8.89
0.84
0.16
1.86
5.51
0.34
0.09
0.10
53.60%
89.25%
10.88%
99.95%
68.76%
99.34%
15.08%
99.99%
79.60%
99.97%
99.95%
100.00%
77.20%
82.67%
46.40%
10.75%
89.12%
0.05%
31.24%
0.66%
84.92%
0.01%
20.40%
0.03%
0.05%
0.00%
22.80%
17.33%
0
5
0
151
0
67
0
26
0
0
0
18
0
0
Table 5.4: Selected statistics for our experimental results. Total, Request & Image refer to mean
transaction times and Errors is the number of requests that encountered a Connection Error.
single instance yields an MTT of 0.39s. Even when single instances yield high MTTs (e.g., line
6, MTT = 974.38), the corresponding full experiment will yield an even higher MTT (line 4,
MTT = 1287.82). Hence, it would seem that more instances yield higher transaction times, a
sign that resource contention among VMs seriously impacts performance.
Finally, we notice that when the total transaction time is extremely high (> 1000 seconds),
most of this time is spent processing the page request, which means that the bottleneck could
be either the Apache or the MySQL servers.
Memory
We now turn to memory usage. For experiment no. 3 (4 instances), we notice that at lower
rates, memory does not seem to be an issue. Indeed, in Figure 5.3, we can see that although
memory usage does increase slightly as rates increase, it never gets close to the amount of
RAM that the VMs are allocated. For the Apache server, the maximum is 200MB, whereas the
MySQL server never uses more than 150MB. Although we have not plotted its memory usage,
the Image server also uses only 150-200MB of RAM when it is given 2048MB.
Higher rates paint a similar picture for the database and image server. No matter how fast
requests arrive at the instances, the MySQL server never goes above 250MB, and the Images
repository remains under 175MB.
The behaviour of the Apache server is more interesting. As we can on Figure 5.4, the
memory usage increases at the higher rates. At 32 req/s (⇔ 8 req/s per instance), the memory
usage reaches 500MB, maxing out the allocation before gradually decreasing to regular levels.
Additionally, comparing the behaviour of single instances at high rates, we can see that the
lower the amount of RAM allocation, the higher the MTT.
A possible interpretation of these results is that as requests arrive at each instance, they
are stored in memory while they wait to be processed. When the arrival rate is much higher
than the service rate, memory fills up quickly and when there is no more memory left, the
guest OSs starts swapping to disk, thus considerably slowing down the entire VM. As soon
as requests stop arriving, memory usage decreases as requests are processed. This is also
consistent with the only exception, namely experiment no. 7, where we only send 125 requests
– the low request count would explain why the MTT is very low despite the fact that each VM
is only allocated 256MB of RAM.
48
350
350
1.0 req/s
2.0 req/s
4.0 req/s
8.0 req/s
325
300
275
275
250
250
225
225
200
175
150
200
175
150
125
125
100
100
75
75
50
50
25
25
0
1.0 req/s
2.0 req/s
4.0 req/s
8.0 req/s
325
memory usage
memory usage
300
0
200
400
600
time
800
1000
0
1200
Apache
0
200
400
600
time
800
1000
1200
MySQL
Figure 5.3: Apache & MySQL servers memory usage at low rates for experiment no. 3 (512MB
of RAM per VM)
500
1.0 req/s
2.0 req/s
4.0 req/s
8.0 req/s
16.0 req/s
32.0 req/s
450
400
memory usage
350
300
250
200
150
100
50
0
0
500
1000
time
1500
2000
Figure 5.4: Apache server memory usage at all rates for experiment no. 3 (4 instances, 512MB
of RAM per VM)
CPU
We now look closer at CPU statistics. For multiple instances, we have CPU usage data for each
virtual machine (i.e., if we have 4 Apache VMs, we will have 4 equal-sized vectors containing
CPU percentages over time). To make comparison easier, we aggregate this data by taking the
mean CPU usage at each point in time. This yields a single vector of mean CPU usage over
time across VMs. We take the mean of this vector to get a single value, which we can use to
49
compare experiments with variable number of instances3 .
Instances
1
1
2
2
1
1
4
4
1
1
8
8
1
1
Requests
1000
1000
1000
1000
500
500
1000
1000
250
250
1000
1000
125
125
Rate
16
32
16
32
8
16
16
32
4
8
16
32
2
4
Total
4.59
24.32
5.29
1287.52
0.47
974.38
10.46
12588.82
0.81
6216.31
12113.64
40230.09
0.39
0.6
Req.
2.46
21.7
0.58
1286.93
0.32
967.92
1.58
12587.98
0.64
6214.45
12108.13
40229.75
0.3
0.5
Img.
2.13
2.62
4.72
0.59
0.15
6.47
8.89
0.84
0.16
1.86
5.51
0.34
0.09
0.1
Req. %
53.60%
89.25%
10.88%
99.95%
68.76%
99.34%
15.08%
99.99%
79.60%
99.97%
99.95%
100.00%
77.20%
82.67%
Img. %
46.40%
10.75%
89.12%
0.05%
31.24%
0.66%
84.92%
0.01%
20.40%
0.03%
0.05%
0.00%
22.80%
17.33%
Errors
0
5
0
151
0
67
0
26
0
0
0
18
0
0
Apache CPU
62.60%
44.41%
41.99%
2.50%
51.36%
3.82%
34.55%
0.60%
50.00%
1.15%
0.59%
0.61%
23.26%
42.48%
Table 5.5: Statistics for selected experiments, including Apache mean CPU usage
For the MySQL and Image servers, the CPU usages vary from 0.1% to 2.5% across experiments, which is negligible. Thus, we discard this data and focus solely on Apache CPU usage,
which we present in Table 5.5
Analysing this table, and in particular the rows in bold, we notice that wherever the MTT is
high (> 1.0s), the corresponding Apache CPU usage is extremely low, especially in comparison
with experiments that have ‘normal’ transaction times.
In fact, in Figure 5.5, we can see that CPU usage is close to 100% at the beginning before
steeply decreasing to less than 10% for the remainder of the experiment. This happens not
only for experiment no. 5, but also for the other experiments that do not perform well under
heavy load.
In comparison, Figure 5.6 tells a different story. Under heavy load (16.0 & 32.0 req/s),
experiment no. 2 performs relatively well. It has a higher total transaction time than at lower
rates, but orders of magnitude lower than the experiments in bold. Looking at the CPU usage
of its Apache server, we can see that it is close to 100% throughout the experiment.
Thus, we can conclude that there is some correlation between high MTT and the sharp
drop in CPU usage. Although we do not have an exact explanation for this result, we would
like to point out that our disk I/O data indicates that low CPU usage also correlates with high
disk I/O. This would tend to validate our interpretation in the previous section with regards
to memory being maxed and the guest OS having to swap to disk.
Unfortunately, due to a bug in processing, we decide not to rely on our I/O statistics, and
leave this to be further investigated.
3 in this situation, taking the mean makes sense because each instance contributes equally to the experiment –
recall that requests are sent uniformly at random to all instances
50
Apache Server CPU Usage
(1 instance(s), 500 requests, 1024MB, 2 VCPU)
100
16.0 req/s
90
80
70
cpu usage
60
50
40
30
20
10
0
0
100
200
300
400
time
500
600
700
800
900
Figure 5.5: Apache CPU usage over time for experiment no. 5 (1 instance, 1024MB RAM,
2VCPUs) under heavy load (line 6 in Table 5.5)
Apache Server CPU Usage
(1 instance(s), 1000 requests, 2048MB, 3 VCPU)
100
16.0 req/s
32.0 req/s
90
80
70
cpu usage
60
50
40
30
20
10
0
0
20
40
60
80
time
100
120
140
160
Figure 5.6: Apache CPU usage over time for experiment no. 2 (2 instances, 1024MB RAM,
2VCPUs) under heavy load (line 2 in Table 5.5)
51
6 Conclusion
When we started this project, we set out to devise a model for the performance of applications
within virtualised environments.
To this end, we implemented an experimental framework that enabled us to completely
automate the running of experiments, i.e., creating and deploying virtual machines, putting
load on them while logging their performance statistics, and processing those logs.
Although we didn’t quite achieve our initial goal, we made some interesting discoveries that
have gotten us closer to our aim. We discovered that transaction times can be approximated
using the exponential distribution, a wonderful result for adepts of Markovian modeling.
Furthermore, we found that virtual machines in Xen Cloud Platform have some scalability
issues when we increase the number of virtual machines or when we gradually increase the
arrival rate of requests beyond a certain point.
Finally, we found that at high arrival rates, high mean transaction times correlate with an
increase in memory usage and a sharp drop in CPU usage.
6.1
Future Work
As we have already mentioned, the initial aim of this project was to come up with a model for
the performance of virtual machines. Our findings make this even more appealing, and future
work should (and probably will) focus on this goal.
From an experimental point of view, the obvious path to follow would be to run more
experiments in different settings and generate more results. There are a number of gaps in our
experiments, either due to time or setup constraints, and these could be filled in the future. In
particular, we could investigate the performance of our system for arrival rates that are not
exponentially distributed (for example, by using real-life load profiles).
With regards to implementation, there are some improvements that could be made to
the experimental framework. First, we could generalise the code to work with any machine
running Xen Cloud Platform or Citrix XenServer (instead of being specific to our experimental
setup). Additionally, our scripts (in particular, deploy.py) implement a number of methods
that are essentially Python wrappers around xe commands. We could complete this with more
xe commands and release it a Python module for the benefit of the wider Xen community.
Finally, earlier in the thesis, we alluded to the fact that our processing script had been
designed from the bottom-up. Given the importance of processing the data correctly and
efficiently, rewriting this script with the added benefit of experience, a better design and the
ability to process even more types of data would be a valuable addition to the experimental
framework.
53
A PEPA
Throughout this section, we refer extensively to material from [6], [14], [15], [16] and [13].
A process algebra is a language used to formally model concurrent systems [15]. Tony
Hoare’s CSP and Robin Milner’s CCS are the most famous examples of process algebras,
but the one we are interested in is called PEPA and was developed by Jane Hillston at the
University of Edinburgh in 1994.
PEPA is an acronym for Performance Evaluation Process Algebra and it is a stochastic process
algebra, used to model concurrent systems. It is stochastic in the sense that it maps to an
underlying continuous-time Markov Chain which describes the system. The next state depends
on a negative exponential distribution associated with each action. If the rate at which an
action occurs is λ, then the probability that this action will occur before time t is given by
F = 1 − e−λt
(A.1)
A PEPA model is described by the interaction between components, through a set of actions
that each of them can perform. In other words, actions are transitions – their rate r is the
parameter that determines the next state in which the component will find itself.
Formally, PEPA can be described by the following grammar:
S ::= (α, r ).S | S + S | Cs
(A.2)
C P | P/L | C
P ::= P B
L
(A.3)
where S is a sequential component and P a parallel component.
The semantics of this grammar are as follows:
Prefix describes a transition from one component state to another and is denoted (α, r ).S. This
means that action α occurs with exponential parameter r (also called rate or duration) and
results in a transition to state S. The process is said to enable the action α.
Choice is denoted by P + S and means that the component can evolve to be P or S. More
precisely, (α, rα ).P + ( β, r β ).S means that the component will produce either an action
α with rate rα or an action β with rate r β . P and S are effectively in a race condition –
whichever performs its action first (α or β) will determine the choice (P or S).
Cooperation expresses the fact that two components must work in parallel (i.e. they must
C Q effectively means that P and Q run in parallel and must cooperate,
synchronise). P B
L
while L is the set of actions over which they must cooperate. L can be the empty set, in
C Q is shortened to P || Q and P and Q are independent from each other.
which case P B
∅
If an action is enabled by P or Q, but is not in the set L, then it behaves in the usual way.
Hiding is denoted by P/L and is used to hide actions in L. Any action not in L will execute as
usual, but actions in L will instead perform a silent action τ. This is used to hide actions
in L from other processes. Cooperation is not allowed on τ.
54
def
Constant A defines a label for a component state, for example A = (α, r ).A will perform α at
rate r forever.
Similar to other process algebras, PEPA provides a number of interesting features:
Parsimony – it is a simple language, which makes it easy to reason about.
Compositionality – complex systems are easily built in terms of simpler, concurrent subsystems using the cooperation operator B
.
C
·
For an extended introduction to PEPA, Grouped PEPA and fluid-flow approximation, we refer the
reader to the excellent article by Hayden and Bradley [13]
55
B Poisson Process
The Poisson Process is a continuous-time counting process (i.e., increasing positive integer)
N (t) with the following properties:
•
•
•
•
•
N (0) = 0
N (t) is positive, integer-value, increasing
Increments are independent (e.g. arrivals at a server)
The probability distribution of N (t) is a Poisson distribution
The probability distribution of the waiting time until the next occurence (increment) is
an exponential distribution (e.g. time between two consecutive arrivals at a server).
In particular, the Poisson Process has a property that is of interest to us:
Property. (Decomposition of a Poisson Process) If a Poisson Process A with rate λ is decomposed
into process B1 , ..., Bn by assigning each arrival of A to the Bi with independent probability qi (where
∑in=1 qi = 1), then B1 , ..., Bn are independent Poisson Processes with rates q1 λ, ..., qn λ.
q1
q2
λ
q3
B1
B2
B3
Figure B.1: Example of Poisson decomposition into 3 streams
For an extended introduction to Poisson Processes, we refer the reader to Boxma and Yechiali [4]
56
Instances
1
1
1
1
1
1
2
2
2
2
2
2
1
1
1
1
1
1
4
4
4
4
4
4
1
1
1
1
1
1
8
8
8
8
8
8
1
1
1
1
1
1
ID
1
1
1
1
1
1
2
2
2
2
2
2
5
5
5
5
5
5
3
3
3
3
3
3
6
6
6
6
6
6
4
4
4
4
4
4
7
7
7
7
7
7
57
3
3
3
3
3
3
2
2
2
2
2
2
2
2
2
2
2
2
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
VCPU
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
500
500
500
500
500
500
1000
1000
1000
1000
1000
1000
250
250
250
250
250
250
1000
1000
1000
1000
1000
1000
125
125
125
125
125
125
Requests
1
2
4
8
16
32
1
2
4
8
16
32
0.5
1
2
4
8
16
1
2
4
8
16
32
0.25
0.5
1
2
4
8
1
2
4
8
16
32
0.125
0.25
0.5
1
2
4
Rate
0.45
0.38
0.44
0.64
4.59
24.32
0.38
0.35
0.40
0.68
5.29
1287.52
0.39
0.30
0.29
0.37
0.47
974.38
0.36
0.37
0.41
0.75
10.46
12588.82
0.32
0.43
0.37
0.34
0.81
6216.31
0.80
0.47
0.41
3.78
12113.64
40230.09
0.44
0.38
0.38
0.32
0.39
0.60
Total
0.28
0.25
0.26
0.30
2.46
21.70
0.27
0.24
0.25
0.33
0.58
1286.93
0.26
0.23
0.22
0.27
0.32
967.92
0.26
0.26
0.29
0.38
1.58
12587.98
0.25
0.29
0.27
0.26
0.64
6214.45
0.67
0.34
0.28
0.41
12108.13
40229.75
0.32
0.29
0.27
0.26
0.30
0.50
Request
0.17
0.13
0.18
0.34
2.13
2.62
0.11
0.11
0.15
0.34
4.72
0.59
0.12
0.07
0.07
0.11
0.15
6.47
0.10
0.11
0.11
0.37
8.89
0.84
0.07
0.14
0.10
0.08
0.16
1.86
0.13
0.12
0.13
3.37
5.51
0.34
0.12
0.09
0.11
0.06
0.09
0.10
Image
61.75%
65.83%
58.79%
46.49%
53.60%
89.25%
70.68%
68.47%
62.59%
49.31%
10.88%
99.95%
68.46%
75.32%
76.35%
71.77%
68.76%
99.34%
72.06%
70.41%
72.22%
50.40%
15.08%
99.99%
78.99%
66.86%
72.03%
75.96%
79.60%
99.97%
84.29%
73.33%
68.71%
10.90%
99.95%
100.00%
72.34%
76.90%
71.29%
79.77%
77.20%
82.67%
Request %
38.25%
34.17%
41.21%
53.51%
46.40%
10.75%
29.32%
31.53%
37.41%
50.69%
89.12%
0.05%
31.54%
24.68%
23.65%
28.23%
31.24%
0.66%
27.94%
29.59%
27.78%
49.60%
84.92%
0.01%
21.01%
33.14%
27.97%
24.04%
20.40%
0.03%
15.71%
26.67%
31.29%
89.10%
0.05%
0.00%
27.66%
23.10%
28.71%
20.23%
22.80%
17.33%
Image %
Table C.1: Complete statistics for our experimental results.
2048
2048
2048
2048
2048
2048
1024
1024
1024
1024
1024
1024
1024
1024
1024
1024
1024
1024
512
512
512
512
512
512
512
512
512
512
512
512
256
256
256
256
256
256
256
256
256
256
256
256
RAM (MB)
0.98
0.51
0.26
0.13
0.06
0.03
0.98
0.50
0.26
0.13
0.06
0.03
1.89
1.06
0.51
0.23
0.13
0.06
1.04
0.50
0.25
0.13
0.06
0.03
4.12
2.22
1.01
0.52
0.26
0.12
0.95
0.48
0.26
0.13
0.06
0.03
8.04
4.16
2.24
0.95
0.64
0.23
Delay
0
0
0
0
0
5
0
0
0
0
0
151
0
0
0
0
0
67
0
0
0
0
0
26
0
0
0
0
0
0
0
0
0
0
0
18
0
0
0
0
0
0
Errors
C Experiment statistics
D run.sh
# !/ bin / bash
# RUN . SH : Run experiments [001 -004] on a given virtual machine
# ( Runs from your laptop or any machine on the DoC network )
EXP_ID = $1
VM_ID = $2
function usage {
echo " usage : run . sh exp_id vm_id "
}
# Check arguments , usage Howto
if [ $ # - ne 2 ]
then
echo " $ # argument ( s ) supplied (2 expected ) . "
usage
exit 2
fi
read -p " Press [ Enter ] to start experiment ’$ { EXP_ID } ’ on
project12 - vm$ { VM_ID } , or CTRL + C to exit . "
# Synchronise files
echo " Synchronising files "
rsync - az -- delete ~/ exp / root@project12 . doc . ic . ac . uk :~/ exp / \
-- exclude ’. git / ’ -- exclude ’. DS_Store ’
rsync - az -- delete ~/ exp / root@project12 - vm$ { VM_ID }. doc . ic . ac . uk
:~/ exp / \
-- exclude ’. git / ’ -- exclude ’. DS_Store ’
# Pre - processing
echo " Pre - processing "
ssh root@project12 - vm$ { VM_ID }. doc . ic . ac . uk " ~/ exp / exps / $ { EXP_ID }/
pre . sh "
echo " Experiment $ { EXP_ID } "
# Start logging
58
echo " project12 : start logging "
ssh root@project12 . doc . ic . ac . uk " echo ’~/ exp / xcp . sh $ { EXP_ID } ’
| at now "
echo " project12 - vm$ { VM_ID }: start logging "
ssh root@project12 - vm$ { VM_ID }. doc . ic . ac . uk " echo ’~/ exp / vm . sh
$ { EXP_ID } $ { VM_ID } ’ | at now "
# Run the experiment
sleep 5
echo " project12 - vm$ { VM_ID }: run the experiment "
ssh root@project12 - vm$ { VM_ID }. doc . ic . ac . uk " ~/ exp / exps / $ {
EXP_ID }/ exp . sh $ { EXP_ID } "
sleep 5
# Stop logging
echo " project12 : stop logging "
ssh root@project12 . doc . ic . ac . uk " pkill dstat "
echo " project12 - vm$ { VM_ID }: stop logging "
ssh root@project12 - vm$ { VM_ID }. doc . ic . ac . uk " pkill python "
Unsafe
#
# Post - processing
echo " Gathering logs "
scp -r root@project12 . doc . ic . ac . uk :~/ logs /* ~/ exp / logs /
scp -r root@project12 - vm$ { VM_ID }. doc . ic . ac . uk :~/ logs /* ~/ exp / logs /
echo " Cleaning "
ssh root@project12 . doc . ic . ac . uk " rm - rf ~/ logs /*; rm - rf ~/ dead .
letter "
ssh root@project12 - vm$ { VM_ID }. doc . ic . ac . uk " rm - rf ~/ logs /*; rm rf / var / mail / root "
echo " Done . "
59
E load.py
# !/ usr / bin / env python
# -* - coding : utf -8 -* import random , requests , sys , datetime , time , threading , traceback
info = []
class Request ( threading . Thread ) :
def __init__ ( self , index , delay , url ) :
threading . Thread . __init__ ( self )
self . index = index
self . delay = delay
self . url = url
def run ( self ) :
start = datetime . datetime . utcnow ()
# Do your job
try :
r = requests . get ( self . url )
except ( requests . exceptions . ConnectionError , requests .
exceptions . Timeout ) :
formatted_lines = traceback . format_exc () . splitlines ()
print " Error : " + str ( self . index )
# print traceback . print_exc ()
end = datetime . datetime . utcnow ()
info . append (( self . index , self . delay , start , end , end ,
formatted_lines [ -1]) )
else :
image = datetime . datetime . utcnow ()
for img in getImages ( r . text ) :
# print img
requests . get ( img )
end = datetime . datetime . utcnow ()
info . append (( self . index , self . delay , start , image , end
, r . url ) )
def putLoad ( urls , count , rate , output ) :
60
delaysList = listOfDelays ( count , rate )
urlList = listOfURLs ( count , urls )
requestsList = listOfRequests ( count , delaysList , urlList )
# Send the requests
sendRequests ( requestsList , delaysList )
# Wait for all requests to get a responses
[ request . join () for request in requestsList ]
# Write data to output
if not ( len ( output ) == 0) :
writeToFile ( output )
summary = " Mean arrival ( sleep ) time : " + str ( mean ( delaysList )
) + " seconds \ n "
summary += " Mean service ( response ) time : " + str ( mean ([
timediff ( req [2] , req [4]) for req in info ]) ) + " seconds "
return summary
def sendRequests ( requestsList_ , delaysList_ ) :
i = 0
print " requests : " ,
for request in requestsList_ :
print i +1 ,
sys . stdout . flush ()
request . start ()
time . sleep ( delaysList_ [ i ])
i += 1
print " "
def listOfRequests ( count , delays , urls ) :
requestsList = []
for index in range ( count ) :
requestsList . append ( Request ( index +1 , delays [ index ] , urls [
index ]) )
return requestsList
def listOfDelays ( count , rate ) :
delaysList = []
for index in range ( count ) :
delaysList . append ( random . expovariate ( rate ) )
return delaysList
def listOfURLs ( count , urls ) :
urlList = []
for index in range ( count ) :
urlList . append ( random . choice ( urls ) )
return urlList
61
def writeToFile ( filename ) :
f = open ( filename , " wb " )
f . write ( " index ; delay ; total_resp_time ; image_resp_time ; start ;
image ; end ; url \ n " )
for request in sorted ( info ) :
f . write ( getInfoLine ( request ) )
f . close ()
def getInfoLine ( req ) :
line = []
line . append ( str ( req [0]) )
# index
line . append ( str ( req [1]) )
# delay
line . append ( str ( timediff ( req [2] , req [4]) ) )
# total
line . append ( str ( timediff ( req [3] , req [4]) ) )
# image
line . append ( str ( timestamp ( req [2]) ) )
# start
line . append ( str ( timestamp ( req [3]) ) )
# image
line . append ( str ( timestamp ( req [4]) ) )
# end
line . append ( str ( req [5]) )
# url
return ’; ’. join ( line ) + ’\ n ’
def getImages ( string ) :
extensions =( " . jpg " , " . JPG " , " . jpeg " , " . JPEG " , " . png " , " . PNG " ,
" . svg " , " . SVG " , " . gif " , " . GIF " , " . tiff " , " . TIFF " , " . xcf " , "
. XCF " )
index = string . find ( " < img " )
images = []
while index > 0:
index = string . find ( " src =\" http :// project12 - vm20 . doc . ic . ac
. uk / " , index )
end = string . find ( " \" width = " , index )
if string [ index : end ]. endswith ( extensions ) :
index += 5
images . append ( string [ index : end ])
index = string . find ( " < img " , index )
return images
def mean ( list ) :
return sum ( list ) / len ( list )
def timestamp ( date ) :
return time . mktime ( date . timetuple () ) + date . microsecond
/1000000.0
def timediff ( date1 , date2 ) :
diff = ( date2 - date1 ) . total_seconds ()
if diff < 0:
return - diff
else :
return diff
62
List of Figures
2.1
2.2
2.3
2.4
2.5
2.6
2.7
3.1
3.2
3.3
3.4
3.5
The two types of hypervisors. (Wikimedia Commons) . . . . . . . . . . . . . . .
Types of instructions and processor operating modes . . . . . . . . . . . . . . . .
Operating modes in non-virtualised and virtualised environments . . . . . . . .
Simple model of memory in virtualised and non-virtualised environments . . .
Virtual-to-Real, Real-to-Physical mapping . . . . . . . . . . . . . . . . . . . . . . .
Virtual-to-Physical direct lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Feature model of the factors influencing performance in virtualised environments (Huber et al. [18]) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
12
13
14
14
15
Xen I/O Models (Wood et al. [37]) . . . . . . .
Xen Cloud Platform and its components . . .
Configuration Interface of Xen Cloud Platform
Examples of xe commands . . . . . . . . . . .
The XenCenter Interface . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
17
.
.
.
.
.
.
.
.
.
.
21
22
23
23
24
Virtual machine creation process . . . . . . . . . . . . . . . . . . . . . . . . . . .
Overview of the experimental framework . . . . . . . . . . . . . . . . . . . . . .
pre.sh script for the Linux Kernel experiment . . . . . . . . . . . . . . . . . . .
exp.sh script for the Linux Kernel experiment . . . . . . . . . . . . . . . . . . .
Virtual machine CPU usage during the Linux kernel compilation . . . . . . . .
An instance of Wikipedia, and the image repository . . . . . . . . . . . . . . . .
Overview of the experimental framework for requests sent to a single instance
Script to enable the use of the image repository (enableRepo.sh) . . . . . . . .
Output of deploy.py in debug mode, creating 2 instances . . . . . . . . . . . .
The run method of the Request class . . . . . . . . . . . . . . . . . . . . . . . . .
Output of experiment.py in debug mode, sending 100 requests to 2 instances
at a mean rate of 10 req/s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.12 Sample of data from the client machine as seen in Microsoft Excel . . . . . . .
.
.
.
.
.
.
.
.
.
.
26
28
29
29
30
31
32
34
37
38
5.1
5.2
. 46
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9
4.10
4.11
5.3
5.4
5.5
Distributions of total transaction times for experiment no. 2 (2 instances) . . .
Distributions of total transaction times for experiment no. 2 with an exponential
distribution (discarding higher rates) . . . . . . . . . . . . . . . . . . . . . . . .
Apache & MySQL servers memory usage at low rates for experiment no. 3
(512MB of RAM per VM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Apache server memory usage at all rates for experiment no. 3 (4 instances,
512MB of RAM per VM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Apache CPU usage over time for experiment no. 5 (1 instance, 1024MB RAM,
2VCPUs) under heavy load (line 6 in Table 5.5) . . . . . . . . . . . . . . . . . . .
63
. 39
. 41
. 47
. 49
. 49
. 51
5.6
Apache CPU usage over time for experiment no. 2 (2 instances, 1024MB RAM,
2VCPUs) under heavy load (line 2 in Table 5.5) . . . . . . . . . . . . . . . . . . . . 51
B.1 Example of Poisson decomposition into 3 streams . . . . . . . . . . . . . . . . . . 56
64
List of Tables
5.1
5.2
5.3
5.4
5.5
Resource allocation of instances for the first set of experiments. Note that Total
RAM/VCPU include the resources allocated to the image VM . . . . . . . . . . .
Resource allocation of instances for the second set of experiments. . . . . . . . .
Best fit parameters for the approximation of transaction times using the exponential distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Selected statistics for our experimental results. Total, Request & Image refer to
mean transaction times and Errors is the number of requests that encountered a
Connection Error. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Statistics for selected experiments, including Apache mean CPU usage . . . . . .
44
44
47
48
50
C.1 Complete statistics for our experimental results. . . . . . . . . . . . . . . . . . . . 57
65
Bibliography
[1] Amazon Web Services. Amazon EC2 Service Level Agreement. https://aws.amazon.
com/ec2-sla/, 2008.
[2] P. Barham, B. Dragovic, and K. Fraser. Xen and the art of virtualization. ACM SIGOPS
Operating Systems Review - SOSP ’03, pages 164–177, 2003.
[3] F. Benevenuto, C. Fernandes, M. Santos, V. Almeida, J. Almeida, G. Janakiraman, and J. R.
Santos. Performance models for virtualized applications. Frontiers of High Performance
Computing and Networking - ISPA 2006 Workshops, pages 427–439, 2006.
[4] O. J. Boxma and U. Yechiali. Definitions of the Poisson process. URL http://www.tau.
ac.il/~uriy/Papers/encyc57.pdf.
XenServer 6.1 âĂŞ Growing Market Share and Leading the Sec[5] Citrix.
ond Source Hypervisor Trend.
http://blogs.citrix.com/2012/10/03/
xenserver-6-1-growing-market-share/, 2012.
[6] A. Clark, S. Gilmore, J. Hillston, and M. Tribastone. Stochastic Process Algebras. SFM’07:
Proceedings of the 7th International Conference on Formal Methods for Performance Evaluation,
pages 132–179, 2007.
[7] Dag Wieers. dstat: Versatile resource statistics tool. http://dag.wieers.com/home-made/
dstat/.
[8] K. Fraser, S. Hand, and R. Neugebauer. Reconstructing I/O. Technical Report. University of
Cambridge, (596), 2004.
[9] gnuplot. gnuplot. http://www.gnuplot.info/, 2013.
[10] R. P. Goldberg. Architectural Principles for Virtual Computer Systems. PhD thesis, Harvard
University, 1973.
[11] R. P. Goldberg. Survey of Virtual Machine Research. IEEE Computer, 1974.
[12] Grant McWilliams. Automated install of Ubuntu 12.04 VM (64-bit) using preseed. http:
//grantmcwilliams.com/item/631-ubuntu1204-on-xcp, 2012.
[13] R. A. Hayden and J. T. Bradley. A fluid analysis framework for a Markovian process
algebra. Theoretical Computer Science, 411(22-24):2260–2297, May 2010. ISSN 03043975.
[14] J. Hillston. Process Algebras for Quantitative Analysis. LICS’ 05: 20th Annual IEEE
Symposium on Logic in Computer Science, pages 239–248, 2005.
66
[15] J. Hillston. A compositional approach to performance modelling. Cambridge University Press,
Cambridge, 2005. ISBN 9780511569951.
[16] J. Hillston. Tuning Systems: From Composition to Performance. The Computer Journal, 48
(4):385–400, May 2005. ISSN 0010-4620.
[17] N. Huber and M. V. Quast. Evaluating and modeling virtualization performance overhead
for cloud environments. CLOSER 2011: Proceedings of the 1st International Conference on
Cloud Computing and Services Science, 2011.
[18] N. Huber, M. V. Quast, F. Brosig, and S. Kounev. Analysis of the performance-influencing
factors of virtualization platforms. OTM’10: Proceedings of the 2010 International Conference
on the Move to Meaningful Internet Systems, pages 811–828, 2010.
[19] Kenneth Reitz. Requests: HTTP for Humans. http://docs.python-requests.org/en/
latest/, 2013.
[20] M. Mazzucco and I. Mitrani. Empirical Evaluation of Power Saving Policies for Data
Centers. ACM Greenmetrics 2012, 2011.
[21] MediaWiki. Cache. http://www.mediawiki.org/wiki/Manual:Cache, 2013.
[22] P. Meenan. How Fast is Your Website? Communications of the ACM, 58(4):49–55, 2013.
[23] P. Mell and T. Grance. The NIST Definition of Cloud Computing. NIST Special Publication,
2011.
[24] D. A. Menascé. Virtualization: Concepts, applications, and performance modeling.
Proceedings of the Computer Measurement Group’s 2005 International Conference, 2005.
[25] A. Menon, J. R. Santos, Y. Turner, G. J. Janakiraman, and W. Zwaenepoel. Diagnosing
Performance Overheads in the Xen Virtual Machine Environment. VEE ’05: Proceedings of
the 1st ACM/USENIX International Conference on Virtual Execution Environments, page 13,
2005.
[26] J. Nakajima and A. Mallick. Hybrid-virtualization: Enhanced Virtualization for Linux.
Proceedings of the Linux Symposium, 2007.
[27] Oracle Corporation. The JavaTM Virtual Machine Specification. http://docs.oracle.
com/javase/specs/jvms/se7/jvms7.pdf, 2012.
[28] D. A. Patterson and J. L. Hennessy. Computer Organization and Design: The Hardware/Software Interface (Fourth Edition). Morgan Kaufmann, 2011. ISBN 978-0123747501.
[29] M. Rosenblum and T. Garfinkel. Virtual Machine Monitors : Current Technology and
Future Trends. Computer, 38(5):39–47, 2005.
[30] J. Smith and R. Nair. Virtual Machines: Versatile Platforms for Systems and Processes. Morgan
Kaufmann, 2005.
[31] O. Tickoo, R. Iyer, R. Illikkal, and D. Newell. Modeling Virtual Machine Performance
- Challenges and Approaches. SIGMETRICS Performance Evaluation Review, 37(3):55–60,
2010.
67
[32] Ubuntu. Preseed File Example. https://help.ubuntu.com/12.04/installation-guide/
example-preseed.txt, 2012.
[33] Ubuntu. MetaPackages. https://help.ubuntu.com/community/MetaPackages, 2013.
[34] VMware. Understanding Full Virtualization, Paravirtualization, and Hardware Assist,
2007. URL https://www.vmware.com/files/pdf/VMware_paravirtualization.pdf.
[35] Wikimedia Foundation. mwimport.pl. http://meta.wikimedia.org/wiki/Data_dumps/
mwimport, 2013.
[36] Wikimedia Foundation. Wikimedia Downloads. http://dumps.wikimedia.org/, 2013.
[37] T. Wood, L. Cherkasova, K. Ozonat, and P. Shenoy. Profiling and modeling resource
usage of virtualized applications. Middleware ’08: Proceedings of the 9th ACM/IFIP/USENIX
International Conference on Middleware, 2008.
[38] Xen Project. Xen Cloud Platform Virtual Machine Installation Guide. http://www.xen.
org/files/XenCloud/guest.pdf, 2009.
[39] Xen Project. XCP Overview. http://wiki.xen.org/wiki/XCP_Overview, 2012.
[40] Xen Project. XAPI. http://www.xenproject.org/developers/teams/xapi.html, 2013.
68