Load Testing GUIs dont work Dont swap the wheel for a square.

Screen Shot 2012-05-03 at 16.28.17

Performance testing is complex, you can’t use just a screwdriver – you need a swiss army knife.

I’ve seen yet another performance test tool that sells itself as needing ‘no scripting’ ability. A scriptless tool – ‘no coding required’.  I’ve lost count of the number of tools I see selling with these points. If you’re thinking about getting one let me save you the trouble of assessing them – they tend not to work well, are clunky and an incorrect fit for most performance testing jobs, simple as that. They unravel as soon as you need to do something pretty unsophisticated (like calculate a date based on a value coming back).  This means you  have to drop out of the GUI and into scripted view.  Almost instantly, instead of having a single look and feel for solving the load testing issue in hand, you have two very different views (GUI and code).  GUI’s are good for very simple performance testing but incredibly clumsy for serious performance testing.

The Load Testing Scriptless Oxymoron:

GUI’s sound like a nice idea on paper, and generally they are great for when you are creating from scratch.  But here’s why I think they fall down with Load Testing Tools.  Web pages are created by a people that have used a cocktail of programming approaches – css, html, javascript, jquery, ajax.  This allows them to express creativity in their implementation.  This freedom of expression is what makes web pages both beautiful, ugly, powerful and varied.  GUI’s work best when a problem domains behavior is well understood, defined and rigid.  This couldn’t be further from the truth when your problem domain is a webpage. If you are ever going to see a quirky mixed bag of solutions, a webpage is going to be where you’ll find it.  Question: How can we concretely measure the success of a Scriptless and GUI approach? Answer: Measure how frequency and quickly you have to drop into a ‘scripted’ approach – A scriptless approach that requires scripting.  Hmmmmmmmmm.

Creating traditional performance tests is fundamentally a programming exercise; there is no escaping this.  I’ve recently used a tool which made an extensive use of GUI’s, after a short period the GUI just got in the way and then continued to get in the way. Programming is powerful because you are given a small number of constructs and then given freedom to solve.   As soon as you create a GUI you start creating walls, rules and restrictions. Your freedom to express a range of solutions automatically becomes eroded.  Do we ever see programmers implementing code using just GUI’s – no, there is a reason for this.  Its still more productive to to code in an IDE than it is to use a GUI. This hasn’t changed for 20 years.

LoadRunner – The Dinosaur that stands the Performance Test of time.  

Loadrunner – not my favorite tool, but one that has seriously stood the test of time and revered by many performance testers.  Why? Because the scripted approach it uses to solving the problem in hand is fundamentally correct.  It’s an sound approach, and still now, after almost 15 years this approach hasn’t had to change – and its still works with minimum modifications for the new technologies.  Doesn’t that say something profound about a scripted approach? LR does have a GUI view, but no-one I’ve met seriously uses this, its relegated and quickly forgotten.  I’m guessing the Loadrunner team started on this route and realized it created more problems than it solved.  If you need a good LR alternative, look up Facilita – it is LR evolved.

Here’s a summary of the fundamental short coming’s of a Performance GUI approach:

  1. Drop Out:: You need more that one view – as soon as you can’t solve an issue in a GUI then you have to drop out into script.  How quickly and regularly you have to drop into scripting is a measure of how weak the approach is
  2. Rescript: If you have regular drops of code and have to re-script, then attempting to duplicate logic (for loops, conditional logic) created in the GUI for the new drop become cumbersome and time consuming.
  3. Features: Need a new feature in the GUI to cope with the next new thing? I bet it can be done in code and in several different ways.  The GUI is stuck in a never ending game of “catch up” and you could be left high and dry. This isn’t even taking into the consideration the quality,  the lack of clumsiness of  GUI implementation and the time it takes to arrive.
  4. Transfer of Knowledge: A GUI view looks nice, but is impractical when attempting to understand logic. Having to expand tree views, inspect conditional logic and not easily being able to see the control flow at a glance means you end up endlessly clicking, expanding and inspecting.  This isn’t just to understand the code of others its also your own.  (See Clicky Click)
  5. Click flick Click, Expand, Clicky click:  The job becomes impossible without the aid of a mouse and constant flicking between screens.  Make sure you have a robust mouse.
  6. Knowledge not Skill::   With a few basic constructs and a LR scripting approach, a performance tester can pick up and learn a tool quite easily. GUI’s require extensive knowledge and learning.
  7. Fools Gold: By appearing to wrap in a GUI there is a perception that the less skilled can acquire the skills and productively goes up. This couldn’t be further from truth.
  8. Limited Solutions:  In a purely scripted approach I can have 10+ ways of solving a problem, I then have the nice problem of selecting the one of these is best.  In a GUI approach it becomes “how do I solve this within the confins of this framework”. Ironically, the act of implementing a GUI to make life simpler creates restrictions, limits options and makes the task in hand more difficult.  (See also knowledge not skill)

There are many more, but I’ll stop there.  So what are GUI’s good for? ::

  • Creating Performance Tests for simple static and stateless types of websites.
  • Being used as a visual metaphor for recorded, played back traffic and perhaps even a representation of the scripted approach.
  • A starting point to template a creation of a solution to a problem
  • Solving problems that adhere and conform to a rigid set of standards (Most of the web out!)
  • Scenario creation – Thats build performance scripts together to create more complex and realistic scenarios
  • Interpreting Data – Thats the run-time analysis and post analysis data.
  • Selling and Marketing to people that are the purchasers, uninformed and not the Performance Engineers. “We have a GUI solution that increases productively” (It doesn’t!!!).


Using an umbrella for a parachute – seems like a good idea

Traditional Performance testing is a programming activity.  Using a GUI to replace a programming paradigm may seem like a good idea, but its not. Don’t get sucked in, there are no easy ways out.  Assess your problem and chose the most appropriate tool for the job – don’t get drawn into the misguided hype.

My one line Performance Takeaway:

Love & Harry Houdini:  If a tools sells itself with a GUI approach be suspicious.  In my experience these tools actually inhibit productively, require extensive knowledge, unravel quickly and tie the experienced Performance Engineers hands behind their back.  Let your performance engineer try before you decide to buy, they will love you much more you if you do.

– And yes, I have been amazed at how many tools have been bought before in-house engineers have had the chance to try.

May Also Be interested In:

Performance Testing is hitting the wall:  – Introducing a new concept of GUI based performance tools & applies to mainly ‘outside’ the firewall

Selecting the Correct Performance Tool:  – Broad outline of how to go about selecting a tool

The Death of a Giant and birth of New Children: – A new dawn for Performance Test Tools

5 thoughts on “Load Testing GUIs dont work Dont swap the wheel for a square.

  1. “I’ve recently used a tool which made an extensive use of GUI’s, after a short period the GUI just got in the way and then continued to get in the way.”

    Would you care to reveal which tool this was?

    1. Hi Chris. Not really, because this article is mainly about GUI’s being unsuitable for scripting purposes. Programming is a complicated exercise and I’ve seen several instances of GUI’s being used as a main selling point for replacements to this practice. e.g. Stubb Creation tools that claim to be ‘scriptless’. Some tasks just cannot be boiled down to GUI’s. I think it would be inappropriate and unfair to highlight the tool in question because that could be taken out of context. Besides, it applies to every single performance tool I’ve used that claims to be scriptless. What has your experience been?

      1. Jason,

        I’ll start with a disclaimer: our product is one of those scriptless tools you referred to. Our experience has been very positive. Our software customers are very loyal and our services customers tell us that we work faster and more cost effectively than other load testing services. I’d like to give our software all the credit for that, but we have great people working for us…they deserve most of it 😉 Naturally, our tool has limits – it is not a swiss-army knife. There are some web-apps that it just doesn’t work on. For the apps it does work on (perhaps 80%), the productivity increase is substantial. One of our customers published a reduction of their testing cycle by 75-80% when they switched to our product.

        I agree that a tool that claims to be scriptless and then drops you into a script to perform common customizations is a sham. We welcome the challenge of testing the most complex sites without scripting and are continuously improving our ability to do that. That is the point of any tool, after all – making hard things easier.

        Chris Merrill
        Chief Engineer @ Web Performance

        1. This is interesting, I have to agree with Jason about the tools that claim to be scriptless don’t do the job all that well. I have also faced situations where i had to give up on the GUI and start writing code.

          With that said, I have an open mind and i am always willing to try out new tools. I’ll check out Chris’s tool and see its capabilities for myself.

  2. I am not one of the geekiest command-line only, script only zealots, and I do appreciate the effort of the vendors to make certain tasks easier by providing a GUI that could allow one to replace several lines of code with a mouse click. However, I do want it to get out of my way when I need to dig in and write something that is a bit customized or doesn’t fit the normal situation. I generally shy away from anything that tries to dumb down something as complex as performance engineering with “scriptless” technology. It is inevitable as some point, you will need to get your hands dirty and need complete control. This article matches my experience and viewpoint in every way.

Leave a Reply

Your email address will not be published. Required fields are marked *