Concurrency of Users Vs Increasing Transaction Rate

Large Number of Users

I’m going to write about increasing the number of transactions to simulate a higher number of Virtual Users.  This is a common technique used by performance testers to ‘cheat’ or rather avoid the high costs associated charged by vendors.  I recently answered a question on Linkedin and suggested increasing the transaction rate. I was sent the following message by Jim :–

“Saw a reply on here where you suggested increasing no of iterations to increase load… I’m not a big fan of this at all, as it really doesn’t do that, it compresses time instead so for example say I have 200 vusers and they are running 4 iterations per hour each to increase the number of concurrent users I need to double the number of vusers if I double the number of iterations I reduce an 8 hour soak test to 4 hours I know it’s a commonly used technique, but it’s total rubbish, usually when I hear people saying this I just mention the word “sessions” and their head explodes with the thought: huge amounts of what I’ve done for years is wrong!!!”

Being called out and made to think is good thing.

Now I’m not going to go on the defensive and pick holes – In essence what Jim is saying is correct*, and Jim’s response got me thinking.   What on earth is the impact of ‘sessions’, what exactly are they? What is the overhead?  Session is an umbrella term people use regularly but I suspect many actually don’t have a firm understanding of it. So I did a little digging..

Its all about the Session Data

Within the context of HTTP, what exactly is a session?  I’m not going to regurgitate formal definitions of it here: But here’s my attempt at a translation: HTTP is stateless, information about the connecting client is retained on the server architecture using a variety of techniques.  The client is given a unique ID when the initial connection is made to the server (session ID).  All associated information for the client (session data) is linked to this identifier.  Now this is where things can get a little murky .. Because ‘session data’ depends on the implementation within the architecture..   So what does this mean in terms of impact – it means that the actual overhead of session ID’s is small … what is important is the implementation of the associated data with session ID’s.  So if a user logs in and has an associated large amount of information this may put strain on the resources handling the information – which in most cases is likely to be either caches or DB’s.     This all leads me to the following observations/conjectures:

Advantage of Increasing Transaction Rate, not Virtual Users: 

Cost: Its cheap, this is the main one. In an ideal world we would flood a system with the actual real world number of users – but quite often our wallets don’t allow us to.

Transactions: I’ve often been on site and heard “we have 50000 people connected”, we need 50000 VU’s.  When the following question is posed “Whats the transaction rate for a key business process X” … and the response is  “Oh …  about 60/min”. I can immediately see that we do not need 50k users to achieve the goals of the customer.  In fact that rate can be hit and exceeded with just 200 virtual users.  Sometimes a customer thinks they need concurrency but they actually only need a transaction rate in order to achieve their goals.

Coverage: You will eke out and find 90%** of the issues VolumePills you would have found using this method Vs increasing the number of users

Hardware: I need a LOT less hardware to generate the load.  Less hardware components in a performance test also means a lot less risk during a test.

Soak Testing: Not greatly relevant to this argument – if you have a long running soak test, upping the transaction rate can decrease the length of the test. I’ve used this to effectively reduce a 3-day soak performance test into 8 hours (Link Here).

When Increasing Transaction Rate will/may not work:

  1. Sizing:  If you are using Caches for session data – then a lower number of users with an increased transaction rate will not fill the allocated space of associated caches as effectively (This is implementation dependent).  This means caches size could actually be too small when you go to live
  2. Thrashing and Time To Live: A large number of users will naturally cause a large breath of caching to be exercised.  I have tested systems where we have ‘come a cropper’ because when the system went live the breath of data searched by a large number of users caused the cache to thrash the DB.  Luckily this was raised as a risk beforehand.
  3. Non Issues: By increasing transaction rates (and not users) you may cause issues you would not experience on live (contention, deadlocking ..).   The upside of this disadvantage is that you build a system which is more robust by solving these non, or yet to be experienced issues. PM’s and the business tend not to be fans of this.
  4. Stateful: If you have a stateful system and the memory required by a connection client is created and retained for the duration of the life of the connecting client.
  5. Large ViewStates: Large viewstate sizes indicates large use of session data. Generally considered to be a warning sign of poor implementation.

So when would it be safe to use a High Transaction Approach?

This depends on the implementation of system you are testing against and your understanding of the underlying architecture.  Here is my initial stab of when it would be safe:

  1. A site is generally static i.e. doesn’t acquire or retrieve a large amount of session data for connected users.  Passive site such as newspapers or blogs are a good example of this.
  2. When caching mechanisms are not heavily utilised by middle tiers
  3. When a client connecting to a server doesn’t enact a large retrieval of data in order to process associated business flows.
  4. Writing calls to atomic webservices.

That’s my initial stab at putting a little more meat around the bones of the Transaction Vs User argument. I think it’s a little strong to come firmly down in one camp or another.  If you have the ability to scale to the actual number of users then great, if not then you should evaluate, assess and report the risk. If anyone has other concise examples I can use then let me know, I’ll update the article and give credit.

Jason B

Related articles:

*Strictly correct, but we don’t live in a perfect world and compromises often have to be made.  I also think the alternative technique is correct within the context of the question originally posted.

**A completely questionable and unscientific figure based on my admittedly bad memory and past experiences

7 thoughts on “Concurrency of Users Vs Increasing Transaction Rate

  1. Jason,

    One more related article is http://www.oracle.com/technetwork/oem/grid-control/overview/wp-rapid-1.pdf

    By the way, the comparison between these two, say, approaches makes sense while the system handles the load well (low response times, etc.). As soon as the system starts to slow down, the load becomes quite different (as far as with a limited number of users you have a limited number of requests sent to the system).

    Increasing transaction rate always leaves some risks and it is always better to test for full concurrency in the end. But it may be a good technique to speedup performance testing and catch some issues earlier.

    Alex

  2. Soak Testing: Not greatly relevant to this argument – if you have a long running soak test, upping the transaction rate can decrease the length of the test. I’ve used this to effectively reduce a 3-day soak performance test into 8 hours (Link Here).

    Seems like definition of Soak tests has changed over time, I was under impression that if we change duration of soak tests by whatever means, it defeats its purpose.Its no longer called soak tests and so it cannot be used to verify the availability of systems under some load for the said durations.

    1. Hi Kiran,
      A formal definition: ”Soak testing involves testing a system with a significant load extended over a significant period of time, to discover how the system behaves under sustained use”. I interpret that to mean Time and Throughput to be the two components. In that particular instance – Running a soak test for three days under normal load was unacceptable and unworkable (cost, length of time, env resources, support etc). Increasing the transaction rate to inject 3 days load into 8 hours was much more practical solution that addressed the risks sufficiently. I would say that this adjusted test would still constitute a soak test by most definitions.

      1. I can read the situation here Jason.It happens quite often that we initially have requirements for soak test with 3 days and then later on for various reasons ,it changes to 8 hrs or maybe in some cases 4 hrs.I have been through all this.I feel achieving 3 days transaction volumes in 8 hrs or 4 hrs or 1hrs defeats the purpose of original requirement which is to make sure that systems should be up and running for 3 business day and should achieve transaction volumes as normally happens in live environments.By definition wise I think it makes sense ,but it defeats the overall purpose of soak tests.Using this technique we might address some risks but often it does not cover the risks which gets associated with soak tests.Just 2 my cents.

  3. Jason,

    Please consider an example. System has one server with 8 CPUs (for simplicity they all are single core). Each transaction can be executed on two threads and each thread is executed by one CPU. There is non-zero probability that four users will max server out because four transactions will use all 8 CPUs. But it never happens if we are running only 2 or 3 users with whatever high transaction rate generated by load testing framework.

    The bottom line – trading a number of users for transaction rate creates workloads incapable to use fundamental quality of today’s software and hardware – ability of transaction parallel processing on different levels (server farms, multiple CPUs per server, multiple cores per CPU, multiple hyperthreads per core).

    Regards,
    Leonid Grinshpan
    Oracle Corporation

Leave a Reply

Your email address will not be published. Required fields are marked *