Just about any developer who is dedicated to Test Driven Development (TDD) as a way of life has had to work with JMeter at some point. Yet, despite the tool’s popularity, from a modern developer’s and DevOps point of view, getting JMeter to do what needs to be done is far from intuitive. In fact, it’s a painful time sink, primarily for 4 reasons. These reasons are:

  • JMeter is all about the GUI. Most developers express themselves via text.
  • JMeter obscures simple tasks.
  • JMeter obscures variable declaration and scope.
  • JMeter’s dependency model is strange, if not magical.

Allow me to elaborate.

JMeter is all about the GUI

JMeter is GUI based. Developers are text based; they type out what they think. Asking a developer to play with a UI is akin to asking a sysadmin who lives by the command line to start messing with dropdowns and checkboxes to get work done. They just don’t work that way and yet JMeter demands that they do.

Consider testing the following use case:

Imagine an API that returns a named list or an object that contains all named lists. The endpoint,

GET /list/

returns all named lists. The JSON response looks like this:

{

 

   “stooges”: [“moe”, “larry”, “curly”],

   “jetsons”: [“george”, “jane”, “judy”, “elroy”],

   “flintstones”: [“fred”, “wilma”, “pebbles”],

   “rubbles”: [“barney”, “betty”, “bambam”],

   “beatles”: [“john”, “paul”, “george”, “ringo”]

}

Listing 1: An object that describes lists by name

The other end point,

GET /list/{name}

returns a specific list according to name. For example, GET /list/stooges returns:

{

 

   “name”: “stooges”,

   “list”: [“moe”, “larry”, “curly”]

}

Listing 2: A specific named list

It’s a simple enough API that has two endpoints. One endpoint returns all resources. The other endpoint returns a particular resource by id. In this case the id is, name. The testing is straightforward, too: write a set of HTTP tests that exercise the /list and list/{name} endpoints.

A developer will think, “OK, let me go get all the list names and then make a call with assertions for each list name retrieved.” The developer will then express his or her thinking in code.  Listing 3 below shows the API tests that a developer might write in NodeJS using the Mocha/Chai test libraries.

‘use strict’;

 

const promise = require(‘bluebird’);

const _ = require(‘lodash’);

const supertest = require(‘supertest’);

const chai = require(‘chai’);

const expect = require(‘chai’).expect;

const describe = require(‘mocha’).describe;

const it = require(‘mocha’).it;

const app = require(‘../app’);

describe(‘HTTP Tests: ‘, () => {

  it(‘Can get lists and details from API’, function (done) {

      //Go get all the lists

      supertest(app)

          .get(‘/lists’)

          .set(‘Accept’, ‘application/json’)

          .then((res) => {

              //inspect the response (res)

              if (res.statusCode !== 200) throw new Error(res.text);

              const listNames = [];

              const obj = JSON.parse(res.text);

              expect(obj).to.be.an(‘object’);

              for (const prop in obj) {

                  expect(obj[prop]).to.be.an(‘array’);

                  listNames.push(prop);

              }

              //return listNames;

              const listCalls = _.map(listNames, name => {

              //Make GET for each list name

              supertest(app)

                  .get(`/lists/${name}`)

                  .set(‘Accept’, ‘application/json’)

                  .end((err, res) => {

                      //Make the expectations

                      const obj = JSON.parse(res.text);

                      expect(obj).to.be.an(‘object’);

                      expect(obj.name).to.equal(name);

                      expect(obj[name].list).to.be.an(‘array’);

                      expect(obj[name].list.length).to.be.gt(0);

                  })

              });

              //go up against all the endpoints

              return promise.all(listCalls).then(() => {

                  done()

              });

          })

          .catch(done);

  });

});

Listing 3: A simple HTTP API test that exercises multiple endpoints

Listing 3 is straightforward, apparent and in the scheme of things, simple. Do the tests need more granularity to the assertions? Maybe. But should another developer come to the test code afterwards to make improvements, everything he or she needs to know to do the refactor is there in writing, no pun intended.

Now, let’s look at the same scope of testing implemented in JMeter, as shown in Figure 1, below.

In JMeter all tests are created through visual composition

Figure 1: In JMeter all tests are created through visual composition

Pretty clear, right? Wrong! Based on the illustration alone, without access to the JMeter IDE, you have no idea what is being tested nor the logic that is being implemented in the tests. All you have are some pictures.  Neither the logic nor the test flow are apparent. If you want to understand what is going on, you need to fiddle around with the JMeter UI. And, even before you start to fiddle around, you need to know a whole lot about JMeter’s plugin framework and the details of each plugin used in order for things to make sense.

If you have experience with JMeter you can point and click your way to some sort of clarity. But, asking a developer with little or no prior experience with the tool to ensure the quality of his or her code using JMeter is the same as asking the developer’s opinion of dental pain. Only a masochistic will answer in the affirmative, the logic nor the test flow are apparent. If you want to understand what is going on you need to fiddle around with the JMeter UI.  And, even before you start to fiddle around, you need to know a whole lot about JMeter’s plugin framework and the details of each plugin used in order for things to make sense.

Get the Code

You can get the SimpleAPI source code, Mocha/Chai tests and JMeter test as JMX file at GitHub, here.

 

JMeter obscures simple tasks

OK, so let’s say the developer does bite the bullet and decides to take the time to get a modicum of competency with JMeter.  The first thing he or she will need to learn is the plugin framework. JMeter plugins are GUI widgets that represent operational and logical tasks. Take, for example, a typical piece of conditional logic such as an if statement. In plain old code, a developer will write

 

if( bob !==  joe) {

 

 //some logic to go to Google.com

}

However, in order to execute the if statement in JMeter, you need to use a plugin called the If Controller as shown below in Figure 2.

When an If Controller evaluates to true, children nodes of the controller execute

Figure 2: When an If Controller evaluates to true, children nodes of the controller execute

The way the If Controller works is you put the if condition in the controller dialog. Then, add a child plugin to the If Controller. When the If Controller evaluates to true, the child is executed.

Yep, you got it. Something as simple as a typical if statement requires at least two plugins to execute. And, in order to figure out the conditions and logic in force between the plugins, you need to go in and view the plugin dialogs. Developers use conditional statements the way bartenders use shot glasses, fast and often. Can you imagine a developer slowing down to use the JMeter equivalent to express conditional thinking? Don’t think so. They’ll either revolt or take hostages.

Let’s move to variables.

 

JMeter obscures variable declaration and scope

Take a look at this dialog from the List API JMeter test in Figure 3 below. Pay particular attention to the JMeter variable, $(list) highlighted with a red border:

A variable in JMeter might be declared anywhere in the project, explicitly or implicitly

Figure 3: A variable in JMeter might be declared anywhere in the project, explicitly or implicitly

Now answer this question, where is the variable $(list) declared? If you are stumped, don’t feel bad. It’s not you. It’s JMeter. The variable $(list) is declared implicitly in the CSV Data Set Config plugin. Let me repeat because this point is particularly painful. The variable $(list) is declared implicitly in the CSV Data Set Config plugin.

The way the CSV Data Set Config plugin works is that you create a CSV file in which a field name for data is declared in the first row of the file. This is typical of the CSV format. Then, when the CSV Data Source plugin reads the CSV file, it creates variables in memory that correspond to the field name(s) declared in the CSV file. Thus, when the JMeter CSV Data Set Config plugin consumes a CSV file like the one shown in Listing 4, the tool implicitly creates a JMeter variable(s) according the the values in the first line of the file, in this case, ${list}.

 

list,

 

stooges,

jetsons,

flintstones,

rubbles,

beatles,

Listing 4:  When consuming a CSV file, the JMeter Data Set Config plugin will create a variable for each fields name, in this case the variable name, list.

Implicit declaration of variables using a CSV file is an extreme example of the way JMeter puts a variable in play. However, JMeter allows variables to be declared in a plugin and have those variables accessible from anywhere in the test. (For example, see the way that the variable, ListName is declared in the JSON Extractor and used in Get ListName Assertion plugin.) The notion of variable scope is a murky concept in JMeter. Given the plugin in which the variable is being declared, you might be able click a checkbox to set the variable’s to global scope or local scope. But it’s by no means declarative, consistent and apparent. In fact, declaring a variable and setting its scope is almost magical.

 

JMeter’s dependency model is strange, if not magical

As mentioned many times in this article, nothing in JMeter is apparent. You have to know a lot to compose even the simplest test, such as the simple API endpoint test scenario described at the beginning of this article. Figure 4 below shows all the plugins with their associated dialog that need to be used to execute the API endpoint test. Previously, I’ve glossed over the particulars of what is going on. But, now it’s time to look at the gory, if not magical, details.

The details of each plugin required to do a simple HTTP test using JMeter

Figure 4: The details of each plugin required to do a simple HTTP test using JMeter

 

The first step in any JMeter is to set up a Thread Group, which we did (1). Notice that the loop count highlighted in red in the Thread Group dialog is set to 3. Why? Because the looping in the Thread Group is the way that the CSV plugin (2) knows to iterate through the values in in the CSV file that contains all the list names. (Remember, the CSV plugin will implicitly assign a value in the file to the variable name, list.)  If the loop count were not set, the only list name that will be tested is the first one. Also, if the loop count is not set to the length of all the names on the list, not all names will be exercised. The implicit relationship between the number of names in the CSV list and value of the loop count in the Thread Group plugin is a significant shortcoming.

(For what it’s worth, using  the CSV file to store list names is a hack. I ran out of time trying to figure out how to iterated over the return of all list names from an HTTP response to the endpoint, /lists. My hack is to make the HTTP server spit out the list names to a CSV file which I copied into the JMeter bin directory. Then JMeter reads the list of list names from the CSV file. Again, it’s a hack, one I am not that proud of. But hey, time is money.)

The actual work of making the HTTP request to the API endpoint is done by HTTP Plugin (3). Notice that the path is set to /lists/${list}. ${list} is the JMeter variable the contains the name of the list to retrieve. Where did ${list} come from? As mentioned earlier, ${list}is implicitly declared within the CSV plugin, which is a branch higher up in the simpleAPIJMeterTest tree. How does JMeter iterate through the values of ${list}? Why, that’s done in the the Loop Count section of the Thread Group. As I said, variable assignment in JMeter is obscure and almost magical.

OK, so we have the ability to traverse a set of list names and make HTTP requests based on the list name. How then do we make an assertion against the HTTP response? This is where we use the JSON Extractor (4). The HTTP response is sent from the HTTP Request plugin to the JSON Extractor plugin provided that the JSON Extractor is a child node of HTTP Request plugin. If the developer places the JSON Extractor as a peer to the HTTP Request plugin, nothing happens. (It only took me twenty minutes to figure this out.) When the JSON Extractor plugin gets the HTTP response, it does a JSON path search for the property, name in the response body object using the nomenclature,  $.name. Then, the JSON Extractor assigns the value associated with the property, $.name. to a variable, ListName. ListName is sort of explicitly declared in the JSON Extractor, in the Variable Names textbox highlighted in red in callout 4 of Figure 5, above. Thus, when the HTTP Response object is:

{

   “name”: “stooges”,

   “list”: [“moe”, “larry”, “curly”]

}

The value of $.name is stooges and the value of ListName is stooges.

Now comes the whole reason to engage in all this agony, the test. Tests are made up of assertions. In JMeter, each assertion is represented by an assertion plugin. In this case we use an Response Assertion plugin. One of the assertions that we want to make is the that the value provided as the path parameter name in the endpoint /lists/{name} returns a list with the corresponding name. In other words, when /lists/stooges is called the response contains a JSON object in which the name property has the value, stooges.

Callouts 5 and 6 shows how the Response Assertion plugin handles the situation. The Response Assertion plugin gets data from the post processed response object emitted from the JSON Extractor. The JSON Extractor implicitly provided the variable, ListName. Also, the Response Assertion plugin has access to the variable, list which was created by the CSV plugin. The Response Assertion asserts that ListName equals list.

We start the HTTP server outside of JMeter on localhost:3000. Then, to run the tests you go the Run menu and select the Start menu item. Or strike the green arrow head in the upper bar. The tests run. How do we know that they passed? As is usual in JMeter, add a plugin. One plugin that displays test results is the View Results Tree plugin as shown below in Figure 5. And should things go wrong and we need to debug, we need to remember to add the Debug Sampler Plugin. So many plugins, so little time…..

Using the JMeter Debug Sampler with View Results Tree plugin allows you see the debug information with test results

Figure 5: Using the JMeter Debug Sampler with View Results Tree plugin allows you see the debug information with test results

 

And this is why JMeter sucks for developers.

How to Make JMeter Not Suck for Developers

The easiest way to make JMeter not suck is to keep it away from developers. JMeter is better suited for systems level, black box testing. For example, testing system behavior that involves the integration of services of APIs from among many domains. Also, the tool allows testers to simulate usage of an application by hundreds, if not thousands of users. This is not a trivial feature.

JMeter has been around for a while. There is a significant plugin ecosystem that allows testers to add plugins to JMeter that meet special testing needs. Some of these plugins are quite powerful. However, no matter what, it takes time to learn how use a plugin. For those indoctrinated into the JMeter way of life, getting up to speed on a particular plugin or solving familiar testing problems may be a quick study. But for those new to the environment, becoming competent in a short period of time is a challenge, no matter how much experience and talent the newbie has. For a developer, twenty minutes spent on StackOverflow trying to figure out how to get JMeter to to work as needed is time that can be used just writing the code for the test. If it weren’t such a chore to get JMeter to do simple things, particularly as a newbie, there might be a good argument for promoting its use to developers. But, it’s a hard value proposition to sell.

Working with JMeter is a particular skill set that does not scale well into the modern CI/CD infrastructure. In a world that is becoming more ephemeral and script oriented, the graphical nature of JMeter requires too many work arounds. Simply deploying the XML based JMX file which represents the particular JMeter test suite is not enough. Managing JMeter deployments in a CI/CD environment comes with it’s own set of challenges, well beyond the scope of this article.

As has been said repeatedly in this piece, developers and DevOps personnel are text based. They are accustomed to writing efficient code, quickly. To their thinking, at some point everything is something to be programmed, whether it’s an application, a computing environment or a testing regimen. Having to fiddle with a UI to get something done does not fit well with the way they think or the way they do things.

No doubt, JMeter has it benefits at the level of black box, system testing. But, as far as being the tool to meet the daily testing needs of developers and DevOps personnel, its day has well passed. Fiddling with a graphical tree full of plugins to implement testing was good in the early days of client-server computing. But that was then and this is now. The industry has moved forward. Applications are more complex, more granular in design and subject to significant variations in scale. JMeter has tried to keep up, but the implicit programming model that is intrinsic to the plugin framework holds it back. Other text based, testing frameworks allow both programmer and testers to ensure code quality in a way that is explicit, apparent and efficient. Sadly, JMeter is none of the above. At one time it was *the* tool to use for testing. However, for modern developers,this is no longer the case. They have moved on. Maybe it’s time for the testing community to move on too.

Perhaps mabl’s testing service will learn from the shortcomings of JMeter.  Sign up to reserve your spot to test it for yourself.