This article aims to describe the different pricing structures you may face when evaluating commercial load testing tools. I’ll use the market leading performance test tool for a concrete illustration – HP Performance Center also commonly referred to as Loadrunner. I was called in to act as a Performance consultant on behalf of a client that had an urgent need for a load testing solution. We invited an affiliated HP vendor in to give a demo and ask for a quote for 5000 Virtual users, as that was the amount of users we wished to simulate hitting the clients website going forward. After a brief demo of the tool we asked a simple question – how much for a 5k VU license? After quite some time (a few weeks) I was delivered a spreadsheet with a lot of different options.
Now I like to have options but I am naturally a little suspicious when I’m presented with complex pricing structures (a little like mobile phone tariffs). In total I was presented with 8 pricing structures. HP are a good case study, as other vendors are likely to follow the pricing structure. They split the product pricing into broadly two offerings:
- Software as a Service offering – hosted by HP in the cloud. No associated annual license cost.
- In house product install. 24% associated license cost.
The pricing was then split according to:
- Virtual User Days (VUD’s): So you could have 30k VUD’s allocated. This meant that you used up the VUD’s progressively. e.g. If you used 5k virtual users on day one you would have 25k remaining. This is great if you know how exactly how often you are likely to use the load testing software going forward. Most companies I have dealt will not know this.
- Time boxed period in which you could allocate a fixed number of Virtual Users or VUD’s
There was also different pricing options for:
- SaaS Shared: Shared Hardware resources with other customers
- SaaS Dedicated: Dedicated Hardware resource for the client
When I’m presented with too many pricing options I find the best thing to do is take a step back, think about what I know is required and then look for a pricing structure that fits. The client requirements/constraints were:
- Use of an in house-tool, SaaS wasn’t an option as they needed to test software inside the firewall. The client also didn’t have the front facing infrastructure to support dedicated parallel test environment.
- The client wanted a known and fixed cost – Regular drops of the software required performance testing and the use of the tool was likely to evolve. I didn’t want to tell the accountant we had Viagra used up all our VUD’s and needed a top up.
So the most suitable option for the client was an in-house installation with a 5k VU license. I was assured by the sales person that the VUD option would actually work out cheaper – as on most days we wouldn’t be using 5k. This was probably true – but the fact of the matter was we didn’t know. Theory and practice – I asked him if they would refund the difference if this wasn’t the case over the year and that ended that particular conversation rapidly.
I have the spreadsheet – but I think I will get into some sort of trouble if I make it available. I’m sure you’re all interested in the cost, so I’ll give you an idea of the quotes I received:
- In-house Controller and 5K Vu’s = £700k (1 year)
- In-house Controller and 5K Vu’s = £145k (1 month rolling rental)
- In-house Controller and 15K VUD’s = £60k
- In-house Controller and 30K VUD’s = £90k
Divide Cost by roughly two for the HP SaaS options.
I can think of some good cases where the available options would be suitable for certain types of clients. But overall I generally think this is over complicated and expensive – you can argue that it’s a lot cheaper than hiring 5 thousand testers to load test your system for a year. This was for the web protocol, if you required more protocol support then this would incur additional cost according to the number of VU’s required. HP LoadRunner is the premium enterprise solution – it has everything you could conceivably want, so don’t just judge this product on price. Sometimes HP is the best and most effective cost fit as a solution.
There are associated soft costs which have not been discussed in the article – that of time to learn a tool, available resource pool, frequency of load testing re-scripting and development. These should all be factored into the overall costing when evaluating and purchasing a product.
It’s worth mentioning the new cloud based services here – their pricing is transparent and simpler; but their products are simpler and less capable of complicated load testing scenarios. You need to first weigh up your requirements and then find a tool that fits (see Choosing the correct load testing tool here ).
- Analyse what you need and what you know – then find a price plan that fits your needs. Don’t get confused with all the options presented
- Don’t be afraid of asking for a discount – increasing the number of VU’s is a software key and incurs marginal additional cost for the software vendor.