For Impetus’ White Papers archive, visit- http://www.impetus.com/whitepaper
In this white paper we talk about how you can integrate on-premise and cloud-based options to build an effective performance testing strategy.
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Cloud-enabled Performance Testing vis-à-vis On-premise- Impetus White Paper
1. Cloud-Enabled vis-à-vis On-Premise
Performance Testing: A face-off
W H I T E P A P E R
Abstract
In this white paper we talk about how you can integrate on-
premise and cloud-based options to build an effective
performance testing strategy.
We will also discuss the available cloud-based performance
testing solutions and whether they can effectively fill the gap
areas created by the on-premise approach.
Finally, we share our recommended strategy which incorporates
both on-premise and the cloud approach and presents a robust
performance testing plan.
Impetus Technologies, Inc.
www.impetus.com
2. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off
2
Table of Contents
Introduction...........................................................................................................3
Limitations of the on-premise approach ...............................................................3
Benefits of cloud-based load testing .....................................................................4
Challenges facing the cloud-based approach........................................................5
Data security................................................................................5
Testing intranet applications .......................................................5
Repeatability and consistency .....................................................5
Debugging and bottleneck identification ....................................5
Cloud-enabled solutions........................................................................................6
Using on-premise tools in the cloud............................................6
Provisioning the tool from the cloud...........................................7
The Hybrid approach ...................................................................8
Impetus’ recommendations...................................................................................8
Conclusion............................................................................................................11
3. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off
3
Introduction
Performance engineers worldwide are harnessing the infinite computing power
of the cloud to test the performance of their web, mobile and cloud
applications.
The immense popularity of cloud-enabled performance testing is based on the
fact that it helps to overcome the limitations of a test strategy that relies
entirely on on-premise performance testing.
However, before moving to cloud-based performance testing, it is important to
examine the on-premise model, so called, because all its components—
including the tool, recorder, controller, system under test, etc., are deployed
within the local environment.
Limitations of the On-Premise Approach
Administrators need to be hired to maintain and support the environment.
Since setting up this environment is a costly affair, there is a possibility that you
may be requested to share this set up with other teams or for other activities,
such as for a staging environment or for User Acceptance Testing.
Also, it is very difficult to scale the test environment to simulate sporadic peak
loads because it involves procuring additional servers, licenses etc. This might
result in extra expenditure. However, this is mostly about the infrastructure
considerations.
The other significant aspect that we have to consider is that the on-premise
model does not measure up to the realistic simulation of user behavior. Since
your application will get hits from all corners of the globe, it is important that its
performance is measured from multiple geographies.
Also, while conducting the tests from within the local environment, we ignore
aspects such as network latency, firewall behavior, DNS servers and other
important components in the end-to-end delivery chain. By doing this, we might
not be testing the actual user experience.
These are some of the limitations of the traditional on-premise approach. cloud-
enabled performance testing can address these gap areas.
At Impetus, we believe that performance testing with the cloud enables testing
teams to take a big step forward in conducting more efficient and realistic, large
The first and the very basic
step of on-premise
performance testing are
setting up a test environment.
This involves putting together
a suitable infrastructure,
procuring servers, and
software in order to host the
load generation hardware and
system under test. Setting up
this kind of an environment
requires considerable
investment and effort and has
overheads such as license and
maintenance Cost.
4. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off
4
scale tests. Moreover, cloud technology enables organizations to realize
significant savings in cost and time.
Benefits of Cloud-Based Load Testing
The cloud can help set up the test environment and have it running in minutes.
You just do not have to worry about the licensing costs of the software,
Operating System, tools, etc.
Furthermore, it eliminates the need to spend on the purchase and maintenance
of the hardware and load testing tool that you would otherwise have to buy in
the on-premise model.
Using the pay-as-you-go approach, you can rapidly set up the testing
infrastructure that you need, when you need it, and only for as long as you need
it. From a business standpoint, the cloud lowers the Total Cost of Ownership
(TCO), while increasing flexibility.
Often, the web application experiences sporadic surges in traffic. Generating the
load for large scale tests to mimic these unanticipated spikes in the production
traffic, may require tens or even hundreds of machines.
While purchasing and configuring these systems requires a significant
investment in time and money, with the cloud, you can rapidly set up as many
load generating machines as you need, on-demand using automated
provisioning of required hardware. Automated provisioning ensures that the
load generators are consistent and optimized to create the desired load.
Another advantage that the cloud offers is the realistic simulation of test
scenarios. With the cloud, you can execute load tests that access your web
application as your users will—from the outside of your firewall—and validate
all the infrastructure components, including the firewall, DNS, network
equipment, and ISPs.
These tests are significantly accurate and enable you to evaluate the real-world
effects of third-party components, such as content delivery networks, analytics
servers, and ad servers. And since your users will not access your application
from one particular geographical location, a realistic load test will require
simulation from different geographies. This can be achieved via the cloud.
Cloud-based solutions can also be accessed via browsers from multiple devices,
such as your PCs, Smartphones, Tablets, etc., and therefore, you can choose to
run and analyze performance tests from anywhere, anytime.
5. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off
5
While cloud-based solutions offer several benefits for performance testing, the
cloud does have its own set of limitations. Let’s see what they are.
Challenges Facing the Cloud-Based Approach
Data security
We will start with data security, which is a major concern for any performance
testing activity happening via the cloud. This is a critical challenge as the test
scripts, test data, and test results are moved to the cloud. There is a need to
figure out ways by which sensitive data is not accessed via any unauthorized
source.
Testing intranet applications
Another problem area is testing intranet applications on the cloud. Performance
testing service providers actually need to work with the client’s IT department in
order to receive permission to access the application from the cloud. This
involves exposing the intranet servers to the external world and requires
relevant security mechanisms in place, to avoid any unauthorized access.
Repeatability and consistency
The other challenges are repeatability and consistency of the test results. Often
in performance testing, you need to precisely measure the effect of changes
made to the application code or configurations. For example, you may need to
determine any improvement that results from resolving a specific defect, or
evaluate performance for a range of cache sizes and other related settings.
With cloud load testing, such precise measurements are difficult because of the
variations in Internet traffic and bandwidth availability. Such variations can
make it almost impossible to duplicate uniform test conditions.
On the other hand, on-premise testing can achieve consistent and repeatable
results by isolating effects that are due to the application or infrastructure.
Debugging and bottleneck identification
Debugging and bottleneck identification is another major challenge when the
tests are conducted from the cloud. When load testing uncovers a problem, the
next step is identifying which layer in the application chain is causing the
problem. You can use monitors to check the performance metrics such as hits,
average response time per request, and average bandwidth for each layer in the
chain.
6. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off
6
You can also employ application performance management (APM) solutions to
identify bottlenecks in the code. These techniques work reasonably well when
the environment is controlled and free from external factors.
Let us now move on to the next problem area—Test Assets Management.
Since the cloud instances may not be available all the time, it is important to
have a pertinent test data management strategy in place, as well. A proper back
up mechanism for test scripts, test data and test results has to be created, so
that they are available for reference, at a later stage.
After going through the benefits and challenges of using the cloud for
performance testing activities, let us look at the solutions available in the
market today that can help in optimally leveraging the cloud.
Cloud-Enabled Solutions
These solutions can be classified into three major categories.
The first and simplest solution is to move the current on-premise tool to the
cloud, as it is. A few companies also offer automated Ccoud provisioning portals
for load generation, as well as executing and controlling tests.
The first two categories have a few limitations and hence a third category called
the ‘Hybrid approach’ is in place. This involves using on-premise tools to
generate on-demand load, using the cloud. It supports high concurrency testing
while still providing the benefits of a controlled environment and security.
Impetus recommends a two-phase approach, which we believe brings the best
of on-premise and cloud options and can help you to build a successful
performance testing strategy.
Let us examine each of these strategies in detail:
Using on-premise tools in the cloud
As we know, there are some standard components of
performance testing tools such as controllers, load generators,
recorders and analyzers.
The variation in the deployment models depends on where all
these components are placed when we conduct the tests. You
need to choose the appropriate deployment model based on your
needs.
In the case of the first model, all components of the tool are
moved to the cloud. This is the simplest way to leverage the cloud
This model is suitable to test cloud-
based applications or Internet
applications, since these get hits
from all parts of the world. The
scenarios can be designed in a
manner that simulates a realistic
load across multiple geographies.
7. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off
7
for performance testing and is used by tools like JMeter, Grinder and Load
Runner.
This solution helps in generating a realistic simulation, unlike the on-premise
model. You also have the option of instantiating as many load generators as you
require, quickly and on-demand. Since there is no hardware procurement
involved, you are unlike to incur massive expenditure on setting up the
infrastructure.
Another benefit of this approach is that there is no learning curve involved. The
teams are already familiar with the tool and can start testing quickly. Also, there
are no license costs involved. The process becomes simplified and you are only
charged for the period you use the cloud infrastructure.
However, there are several
limitations in this model. It
requires manual configuration of
instances and knowledge about
cloud APIs. The instances can only
be accessed via RDP, Putty or SSH,
which involve physical connection
to the machines. Also, the test
data resides in the cloud and after
the instance is terminated can be
lost. The user has therefore to
come up with an appropriate data
archival strategy before
terminating the instances.
Provisioning the tool from the
cloud
Let’s now talk about another popular approach, which is provisioning the tool
from the cloud. This involves using the tool provisioned on the cloud for load
generation, execution and test management. All the components of a load
testing tool are deployed on the cloud and the user can access the solution from
the browser. This provides an easy interface that can be accessed from
anywhere, at anytime.
Though this approach is very popular, it has a few limitations that you need to
know about. Since the tool is accessed via the Internet, data security is one of
the major concerns. All the test artifacts and results are stored on the cloud,
thereby, presenting a challenge to data confidentiality. Also, since the tests are
run over the Internet, appropriate firewall access needs to be provided for the
intranet applications.
Impetus SandStorm enables you to
provision the tool from the cloud.
One of the major benefits of this
approach is on-demand load
generation, which means the user
can scale its load tests at any given
time. Pre-created instances help in
quick scaling of the load tests. This
process is suitable for mobile and
cloud-based applications that have
a large user base, and scales
quickly.
8. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off
8
Due to varying network conditions, the tests might not be repeatable.
It is also difficult to identify performance bottlenecks in such cases, as the
number of variables is relatively high. This approach is well suited
for measuring the end user experience and exercising the entire
delivery chain including firewalls, ISP, CDN, routers etc. in a
typical application deployment.
Now, let us talk about the Hybrid approach that tries to address
the limitations of the earlier two approaches.
The Hybrid approach
In this case, the components of the performance testing tool are
deployed either on the cloud or on-premise. There are many
combinations possible here, but let us take up the most likely
scenario to understand this better.
This is because regular testing can be done on premise not only to
get the performance numbers, but also to determine the bottlenecks in the
application. In case there is need to test an application with peak volumes, high
concurrencies and more transactions, the Hybrid model can be used.
Impetus’ SandStorm and a few other tools offer this option. The controller is
deployed on-premise in this model and enables you to use the integrated
resource monitoring and diagnostic features of the tool. This also helps address
any security concerns over the crucial data as the controller remains in-house.
Since the data is available locally, it can be archived and referred to at any
suitable time.
This model too suffers from a couple of limitations related to remote access and
parallel test executions. Since, the controller is installed within the local
network; the tests can only be executed via physical access to these machines.
Impetus’ Recommendations
We will now talk about some of the work we have done and the best practices
we have deployed successfully across multiple customer engagements. These
We have the load generators on
the cloud and all the other
components, such as, the
controller, recorder and analyzer
on-premise. This model is more
suitable for organizations which
have already invested in
performance test labs and have a
hardware set up. These
organizations can use the Hybrid
model when they need to scale
quickly.
9. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off
9
best practices can help you maximize the advantages, and minimize the
challenges of load testing with the cloud.
As we discussed earlier, while the on-premise approach has numerous
limitations, it cannot be replaced completely by the cloud. And while cloud
offers scalability along with cost-effectiveness, it also has some pertinent
challenges associated with it. Therefore, let us examine the fourth approach
that integrates the cloud and on-premise solutions for a successful and effective
performance testing strategy.
The on-premise model is suitable for initial performance tests and debugging
and forms the first stage of our process. You can use the local environment to
create test scripts, prepare test data, and validate the test scripts for runtime
data changes. Then, can you conduct internal tests with a medium load to
quickly identify and resolve preliminary issues.
The source of any issue identified in the first stage is clearly within the firewall
(because no other external systems are involved in the test, as of now).
Therefore, it is easier to pinpoint and fix internal problems when they are not
being compounded by other issues that will originate outside the firewall.
With this two-stage process, you do not have to wait for the application to be
deployed and accessible from the Internet to test it. You can test the application
internally, and earlier in the application lifecycle, when the defects are easier
and less expensive to fix. With internal testing, you have much more control
over the environment, so you can precisely measure the effect that a code or a
configuration change might have on the application performance.
The tests are repeatable, and therefore, the tuning/optimization changes can be
quickly validated. This approach also helps in creating scripts locally, and retains
them for scaling at a later stage, using the cloud. After the performance has
been validated internally, you can proceed to the second stage, which is cloud-
based load testing.
The cloud can be used to test large scale, a peak load that validates the entire
delivery chain of the application. You can compare the results of the same test
scenario run internally and from the cloud, to get a better handle on how the
application server and network infrastructure contribute to the overall response
times.
cloud testing is based on pay-per-use model, and since you have already tested
the application internally on your existing hardware setup, you have an option
to reduce the amount of cloud testing and save on costs. You can also detect
potential network issues at the cloud provider level. And having gone through
this model, we will talk about the best practices associated with using cloud-
based performance testing solutions.
10. Cloud-enabled vis-à-vis on-premise Performance Testing: A face-off
10
As discussed, one of the primary concerns of users is data security. We
recommend encrypting the communication between your controller and load
generators. This will help secure the data sent to the load generators during the
test (including account information), as well as the data that is retrieved
(including error messages). If possible, use SSL to secure the communication
between the browser and the test server. Finally, ensure that your load
generators are secured with their own firewalls to protect them from outside
threats.
To ensure that your load generator machines in the cloud are capable of
generating larger loads, we would recommend tuning the system to support the
creation of a high number of sockets and threads per process. Additionally,
allocating an appropriate heap size for Java-based load generators will help in
such scenarios. The default settings for a typical machine allow all programs to
share resources fairly. In the case of load generators, the machine is dedicated
to a single task, so you can improve the performance by allocating a significantly
larger share of the available resources to the load generation tasks.
Automated provisioning can also save time and efforts to scale the test
environment to support high concurrency and transaction volumes. It also
makes sure that the load generators are consistent and optimally tuned as they
are generated from a single image.
We recommend using an integrated solution for on-premise and cloud. If the
solutions are interoperable, it reduces the learning curve for the team. The test
scripts, scenarios can be re-used internally as well as on the cloud.
Having covered the four available cloud-based performance testing solutions,
we would like to point out that Impetus’ two-phased approach appears to meet
all the criteria of a robust performance testing strategy.
Almost all other approaches, other than cloud provisioning, need tool
familiarity. Even the cloud provisioning option does not allow reuse of the
scripts created locally for large scale testing from the cloud.
A major advantage that the cloud provisioning and Impetus approaches offer is
the ability to run multiple tests in parallel. This is very critical for large
organizations where multiple parallel releases are happening for different
applications. Another advantage is that they both provide browser-based
testing to simulate the end user experience. This is very important, owing to the
rapid changes taking place in client side technologies like HTML5, FLEX, etc.
Each of these four solutions that we have spoken about have their own merits
and demerits. The selection of specific tools depends on factors such as cost-
effectiveness, type of application, performance objectives, etc.