There is a common belief that dogs age around seven times faster than humans. According to the American Kennel Club, the first year of a dog’s life is about 15 human years and then nine years for each human year subsequently after that. I think software ages like dogs. How many times have you heard someone complaining that the software they are using runs like a dog? You can either upgrade your hardware or improve the software by making it more nimble. Before you can do that, you need to do some performance testing.
Load Testing
For a long time, if you wanted to do load or stress testing of your application, your first choice may have been to go to Apache JMeter. JMeter was first released in 1998, making it 25 years old. Happy Birthday, JMeter!
If software ages like dogs, that makes it as old as your Grandpa. He’s no longer flash or nimble but he can still get the job done. However, there is always a young whipper snapper who thinks they can do the job better.
K6* (which comes from Grafana Labs) is this young upstart. It’s a command line tool written in Go and runs tests written in Javascript. So all those young'uns will like it. No XML here, Grandpa! Unlike JMeter, there is no Graphical User Interface; you just write Javascript. This makes it super simple to run as part of a Continuous Delivery pipeline.
This is what a basic test looks like:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
discardResponseBodies: true,
scenarios: {
example: {
executor: 'constant-vus',
vus: 50, // Run 50 Virtual Users
duration: '60s', // for a duration of 60 seconds
},
},
};
export default function () {
const result = http.get(‘insert your URL to test here!');
check(result, {
'http response status code is 200': result.status === 200,
});
}
See the comments inline in the above test, which hopefully makes the test above self-explanatory. You can run this by installing K6 (brew install K6 on a Mac) then running the test:
k6 run test.js
Please remember to only performance test services that you have permission to.
After a minute, you’ll get a test report in your console.
Similar to Junit, you can do setup and teardown as part of your tests. You might use setup to put your system in a certain state before you hammer it, then use teardown to put it back again to clean up after yourself. Setup looks like this and runs once:
export function setup() {
const result = http.post(``);
check(result, {
"Successfully set up test": result.status === 200,
});
}
Teardown is very similar and also runs once:
export function teardown() {
const result = http.post(``);
check(result, {
"Successfully back in original state": result.status === 200,
});
}
There’s a whole bunch of different executors you can use to try to shape the traffic as you wish on K6's own Docs page.
What I’ve found particularly helpful is the ‘ramping-vus’ executor, which allows you to control your test in a fine-grained fashion. Example:
export const options = {
scenarios: {
example: {
executor: 'ramping-vus',
startVUs: 0, // Start off with 0 virtual users
stages: [
{ duration: '20s', target: 1000 }, // Ramp up to 1000 VUs over 20 seconds
{ duration: '10s', target: 10 }, // Back down to 10 VUs over 10 seconds
{ duration: '30s', target: 10 }, // Maintain 10 VUs for 30 seconds
],
}
}
};
Nice and easy. To give this a go with JMeter, you have to use “Thread Groups,” which represent your users. Each thread group has a number of users you can specify and the ramp-up time. This sounds similar to what we are doing in the code above. The screenshot shows the “ramp up."
The problem comes when we want to control it a bit more and reduce our users to 10. You have to start digging around in the documentation. Then you realise you need a plugin to achieve what you want to do. But before you can do that, you need to install a plugin manager by downloading a ‘jar’ file and copying it into the installation! Here, you can install the “Ultimate Thread Group” plugin, which allows you to make finer adjustments to your plan. This plugin shows a graph as you build your test plan, e.g.:
What I found with messing around with the default Thread Group, then installing a plugin, then fiddling around with the above graph to get the test plan to do what I wanted it to do, was that I wanted an easy life and just wanted to write a bit of code. Everything is ‘as-code’ these days; who wants to use a GUI? That’s for old people!
CI/CD Pipeline
As previously mentioned, you can run K6 on the command line so we can put it in a CI/CD pipeline with ease. It may be wise to have it as a manual step as part of a pipeline so you can run it in an ad-hoc fashion or run it when your code is deployed to a performance test environment, for example. GitLab CI and Github actions will let you do this easily. You will, however, need a test report of some form, and you may find the textual test report from K6 a bit lacking. On the other hand, JMeter allows you to get an HTML test report that you can publish as part of your pipeline. One up for JMeter there.
K6 can output the test results to other formats, such as a JSON file, so you can transform it into something you want. It also supports the ability to send the output to Prometheus using ‘Remote Write.’ This then allows you to run a Prometheus query on your test result metrics. Building yourself a Grafana dashboard from these metrics to display test results becomes possible.
Scaling Out
Given the low requirements of K6 in terms of memory compared to other alternatives, you can generate quite a lot of load with it. You will get to the point, though, that your laptop will not be able to generate enough load due to factors such as network bandwidth, CPU load, or memory. You could run it from multiple laptops, but the more sensible way of doing it is running it in the cloud.
One option is Grafana Cloud, where scaling is taken care of for you at a cost. Or running it in Kubernetes. If you go with the Kubernetes option, you can install a “K6 operator” into your cluster that allows you to run your K6 tests. This means you can treat your K6 tests as normal Kubernetes resources, so it works great if you are already on that platform.
Pros and Cons
JMeter requires heavy use of the UI, which still looks like something made in the 1990s and hasn’t embraced ‘as-code.’ In JMeter 5.6, there is an experimental feature to write tests as code, so it’ll be interesting to see how this develops.
K6 is written in Go, so it has a very small executable and starts up quickly. You install it and run it. This makes it very easy to run in a CI/CD pipeline.
JMeter, on the other hand, is significantly larger and has a bigger memory footprint due to the GUI. Also, running on the JVM adds an overhead in terms of startup time and memory. The GUI is meant to be used for designing tests and not running a load test, however. There is a CLI version of JMeter, so you can use it in a CI/CD pipeline.
K6 has a very small learning curve. Even though JMeter has a GUI, it’s not as easy to learn, counterintuitively.
If you want test reports from K6, you’re going to need to do some work to get a nice report (or spend some money). With JMeter, it’s built-in.
Alternatives
Gatling is another tool that allows you to write tests ‘as-code’ in Scala, which is not to everyone’s taste. It now also allows you to write in Kotlin though, which is probably more palatable than Scala. It also runs on the Java Virtual Machine so has some of Java’s baggage. It also allows you to create HTML test reports and provides a paid plan to allow you to scale your tests.
Summary
Along with unit testing and other types of testing commonly found in the test pyramid, performance testing should complement your current testing strategy, and K6 is an easy-to-use tool to achieve this.
As for JMeter, there’s life in the old dog yet!
* Note: If you turn the 6 upside down in K6, you get K9. Coincidence? :-)