Choosing the correct load testing tool

This article is aimed at explaining the difference between different types of performance tools and should prove useful if you are assessing which load tool is right for you.  There are a myriad of load testing tools out there – far too many for me to research individually. I’ve thought long and hard about how to best present this information in a way that is easy to digest.  What follows is my interpretation and a first stab, there are always exceptions to the rule.

The tools tend to fall into 3 distinct categories:

Freeware Performance Test Tools:

  • Open source and free
  • Specialist knowledge required – Developer level interest required to learn.
  • Learning Curve: High
  • Development Time: High
  • Handover Time: High
  • No commercial support: Reliance on forums and goodwill
  • Best Suited To: In-house, subsystem testing, ‘quick and dirty’ testing, have high access to insider knowledge of application under test.

Commercial Performance Test Tools

  • All in one packages: A lot of what you need is included in these packages.  e.g. Record and playback functionality, analyst graphs, monitoring capability, debugging ability
  • Costs involved
  • Learning Curve: Medium
  • Development Time: Med
  • Handover Time: Med
  • Commercially supported – protocols updates available quickly, help for issues and fixes can be speedy
  • Best Suited To: In-house testing of applications prior to release where aggressive releases and builds are scheduled.
  • Examples: Loadrunner, Forecast, SilkPerformer

Cloud Based Performance Test Tools:

  • No hardware or installation required
  • Very Cheap and easy pricing structure
  • Learning Curve: Low
  • Development Time: Low
  • Handover Time: Low
  • Best Suited To: Simple load scenarios executed over the web on HTTP based applications, True E2E testing
  • Disadvantages: Low protocol support, no in-house test ability, difficult to create complex load scenarios
  • Examples: Loadstorm, LoadImpact, Gomez
Note: There is a 4th Category, Network based Performance Test tools which operate on the lower levels of the OSI model – but these are out of scope for this article. e.g. Spirent Avalanche

Freeware Performance Tools:  e.g. Grinder, JMeter

These tend to concentrate on generating load only.  The largest benefit of open source tools are they are free and extremely flexible. Time to record and playback scripts can be lengthy and require different toolsets.  These tools are especially suitable for developers that wish to quickly perform sub-system (or system) load tests or for the performance tester that has close access to the development team. I view the development time to create performance tests for large applications as a major factor when considering them. e.g  Client server traffic is changing Vivaxa frequency with new builds.

I personally think there is a large hole in market to replicate and pull together a number of open source tools and build a solution that replicates the commercial tools – and potentially do a better job (more on this in another article).

Commercial Performance Tools:  e.g. Loadrunner, Forecast, SilkPerformer

Commercial tools are generally suitable for medium to large sized customers that have a complex n-tier architectures and large development teams.  If there are a number of different protocols that need supporting then you can generally find a commercial tool that will support it.  Commercial tools have a number of sub-products bundled together: Record and Playback, Scenario Creation, Injectors, Realtime Analysis, Post Analysis and tie in with Monitoring Products (e.g. Memory, CPU, Network i/o). Together these sub-products greatly enhance the productively of performance tester.  These tools tend to be suitable for large development team dropping regular releases into performance testing. Most commercial tools work on the same principles and generate load in slightly different ways.

Cloud Based Performance Tools:  e.g. LoadImpact, LoadStorm, SOASTA 

These are the new kids on the block and shaking up the old guard. This is an exciting development as they are a fraction of the cost of the commercial tools.  They tend to work on a memory and CPU sledgehammer approach, which means they consume lots of hardware resources e.g. Amazon cloud services.  This in a way is a minor downfall – as a lot of them do not give you the ability to inject their services in-house and past the firewall.  Scripting ability and logic tend to be simple – which means complex load testing scenarios are not sometimes achievable depending on your requirements.  These tools have a time and a place – if you are considering them then also consider your hardware environment and infrastructure in place.  You may only be able to test during quiet periods of the day (as you are testing from internet) and you will have to look carefully at the infrastructure outside the lab e.g. firewall, network capacity – the web facing infrastructure.  I feel that this is the future – it’s only a matter of time before these services improve on their weak points.

As I said earlier – this is a generalisation of the different load testing tools available (there are over 100), but it should steer you in the right direction and give you a good starting point when assessing which tool to use.

See also Selecting Performance Test Tools

6 thoughts on “Choosing the correct load testing tool

  1. Jason, thanks for the blog. Your thought about the large hole in the market for solutions derived from the existing open source tools, which replicate commercial tools (and potentially do a better job), resonates with me. As a matter of fact, I came to the same conclusion that led to launching our commercial load testing tool called StresStimulus integrated into Fiddler, a performance and debugging tool.

    I am sure that more similar solutions will emerge to bridge the gap between ease of use and low cost of Cloud Based tools, and the ability to test complex scenarios of the Commercial Performance Tools. I believe that the lack of simplicity and high cost of the latter is allowed by the lack of competitive offerings, but the situation, hopefully (for customers), will start changing.

  2. While not a bad overview, I’m afraid it drastically oversimplifies things. Besides the obvious point that there is a very wide range of functionality and pricing in commercial tools, saying that cloud-based tools are a fraction of the cost of other commercial tools is true only for a subset of cases. Most of them have moved to subscription models. Thus, if you need to run large tests (10k users or more) frequently throughout the year (e.g. for monthly site updates), our Load Tester software is actually less expensive than the simplistic cloud-based services, in the long run. So not only are the cloud-based offerings not “a fraction of the cost”, they actually cost more.

    Chris Merrill
    Chief Engineer
    Web Performance

    1. Hi Chris – Pricing has now become a very complex subject (for some vendors), I have seen the pricing model change for Commercial tools mainly as a result of the cloud based services. In fact I think its actually a good topic to cover (pricing structures). As you said – it it an over simplification, but it is aimed at those that know little about the available tools out there and a starting point. There are always going to be exceptions to the rule. With over 100 tools out there this is primarily aimed as a starting point. I will update the article to include a reference to cloud based services now being offered as a option by commercial tools.

  3. Hi Jason,

    Good review, but I’d like to disagree with some points of it. It seems to me, that cloud solutions can give us more complex testing, that it is described in post. How about Blazemeter(http://blazemeter.com), that allows to upload JMeter scripts of any complexity?
    May be, it is not the only solution, that allows scripting? Do you know?

Leave a Reply

Your email address will not be published. Required fields are marked *