Selecting Performance Test Tools

Note: This article will work on the assumption that a client has no performance tools in place and urgently requires a load testing solution.

Selecting the correct performance test tool is fraught with difficulties for the novice.  I’ve been into many sites where the client has had an urgent need and they aren’t sure which direction to go.  Quite often these clients have experienced a live performance issue, lost creditability with clients and feel the whole business has been jeopardised.  Major live issues get the attention of CEO’s immediately – and they don’t want this to happen again. So which performance tool to choose and how?

To the uninitiated it is daunting – there are many tools out there, some free, some reasonable and some are very expensive.  So where to start?  These are the questions I ask first:

  1. What caused the issue on live?
  2. What was the business impact?
  3. What is the plan for introducing a performance testing phase into their QA lifecycle?
  4. What is required to be performance tested?
  5. How often do they release new builds into live?
  6. What monitoring capability do they currently have?
  7. Do they have any in-house tools?
  8. What technologies need to be performance tested?
  9. Is there going to be budget for a performance environment?
  10. What is the budget for tools?
  11. What is plan for the handover of the performance process?

I use the above questions to assess what load testing tools are required, it is surprising how many consultants will recommend a tool before they know the answers to these.   The next step is to assess appropriate performance tools, there are many tools (75+) out there it is difficult to assess each individually in the time allotted.  What I then do is narrow the search to those vendors that will best solve the problem and will continue to serve the business going forward.  These are the factors I take into consideration:

  1. Can the performance tool do the job required: Does it support the protocols required now and going forward.
  2. Ability to learn the load testing tool / support: It is of no use using a tool if it requires complex knowledge and cannot be handed over easily to someone else.  For me maintenance and ease of use is as important as doing the job required.  Pick a tool where the code can be easily be peer reviewed and handed over should someone decide to move on.
  3. Ability to re-script: How often are new builds dropped into testing, how often are they expected to change. Does the tool have good record and playback ability.  Client server traffic can change frequency and the performance testers development time is made faster if they can re-record a performance script against a new build.
  4. Support for issues: Freeware tools are great, but if you get an issue the support can debilitatingly timely.  Some vendors have great forums, some vendors have poor support. Some vendors just fire back a generic set of questions. Support is key and can move things along much more quickly.
  5. Monitoring ability:  Generating load is great – but its 100% more useful if you can easily tie in with metrics and counters.  Metrics and diagnostic software are just as important. Visibility is everything.
  6. Cost / Maintenance : What is Electronic Cigarette the cost, maintenance: Amazingly simple and for some reason it is a question some vendors have difficultly with. It took over 4 weeks in one instance for a tool vendor to come back  with a price for a 5k virtual user license, and then it took one day to understand the pricing structure they had put in place.  This was a vendor with a strong lock in.
  7. Price: Don’t be afraid to negotiate on cost – this can save a substantial amount, in one case I managed to negotiate a £400k discount.  The difference between a 5k virtual user license and a 20k virtual user license is a software key – no extra software, hardware or support.  Freeware tools have a price to – One of the most obvious is the length of time to develop Vs Commercial tools.  Another is maintenance – you could think of it as the UNIX Vi editor Vs Word.

Do not do this:

  1. Allow a load testing vendor to come in, Do a Proof Of Concept and then make a decision solely based on this.  Do this at your peril – some tools can be made to do the job but are costly to support going forward and can require a lot of expensive expertise.  In order to avoid lock in, development delays and substantial costs have someone in-house assess the tool.  It is in the vendors interest to sell the tool regardless of it being a good or bad fit.
  2. Write lengthy selection documents first: Get online, download a tool and start scripting against the application. A lot can be leant about a tool by attempting to solve the job in hand – rather that research via a document. You also learn a lot about the application first hand – do a mini POC against the application then write a document. It will be a lot more informed, relevant and useful to everybody involved.  Invited vendors will also find your knowledge a lot more useful and you can ask them more detailed questions.

So the answer here is there is no silver bullet; each client has slightly different needs and each tool has strengths and weaknesses. In-house, freeware or commercial tools really depend on the clients need and circumstances.  If you are faced with assessing performance/load tools but are out of your depth then it really is worth getting in an independent performance consultant.  Selecting the right performance test tool does should not just stop here – A good consultant will also take a look at the live site and recommend tools that will give visibility on live behavior and helping in tuning performance. This is where some of the largest savings in hardware and time can be acquired  and will be covered in a separate post.

Key Takeaways:

  • Don’t be rushed into a quick decision – this can be costly if you do not consider all factors
  • Do not make purchasing decisions based solely on a vendor Proof Of Concept
  • Assess the tool in house and evaluate all costs – the hidden and obvious
  • Monitoring ability should be factored in – this can drastically reduce performance development timelines.
  • Negotiate on cost for commercial vendors – its only a license key.

See also: Load Testing Tool types – advantages/disadvantages 

insert link to [Post Performance Testing – live tools and practices]

3 thoughts on “Selecting Performance Test Tools

  1. Well done Jason. You make some very good points.

    One aspect of performance testing tools that may be overlooked is that not all commercial tools are license key activated. Have you considered cloud (SaaS) tools?

    Cloud based tools (e.g. LoadStorm) are much less expensive than the legacy tools such as LoadRunner, Silk, or Rational. Plus, it they have the advantage of scalability without the need to invest time & money in the test generation hardware or network environment.

    Excellent article.

    Thanks,
    Scott

  2. Great article.

    One thing I would add, PT tools and solutions do fall into two categories single and multi-protocol. What’s good for one may not be good for the other, i.e. outside-in testing of thick client applications can be impossible or far from practical. Choosing a vendor that offers outside-in (SaaS), inside-in and hybrid approaches gives you full flexibility to replicate mixed application client types and from the locations that matter most to you.

Leave a Reply

Your email address will not be published. Required fields are marked *