What's the trickiest part of DevOps? It's software testing, hands down. Year after year, respondents to our annual DevSecOps surveys have called out testing as the most likely reason for release delays. And that's not all they said: "Testing takes too long," "There are too many tests," "We need to do more testing," “We need more automated testing but don't have time," "Testing happens too late," etc.
Clearly something this fraught needs all the help, so here is our best advice to get testing "just right" in any modern DevOps practice.
Use the right metrics
All of the testing in the world doesn't matter if a DevOps team is measuring the wrong things. At GitLab, we use industry-standard metrics, but we look at them a bit differently. When it comes to S1 and S2 bugs we don’t count the time to close but rather the age of the bugs that remain open. Our reasoning? We want to look forward, but we also don't want to incentivize closing only newer bugs. So it's important to make sure DevOps teams are looking at the right metrics and with shared goals in mind.
Forget flaky
Tests are noisy, and they can be flaky, setting off alarms and disrupting developer flow, often for no reason. That's at the heart of developer frustration with testing, and one of the biggest problems DevOps teams need to solve. GitLab's Vice President of Quality Mek Stittri suggests re-thinking how automated tests are created. Tests need to be validating the right things, but that must include looking at how all of the code components work together and not just at pieces of code. Finally, it doesn't hurt to develop a manual testing mindset.
Make it modern
In fact, a manual testing mindset, where test designers create tests that actually mimic what real users do, is a key underpinning of modern software testing in DevOps. Testers need to consider getting certified, embracing new technologies like AI, and, perhaps most importantly, be evangelists for quality on a DevOps team.
Make automation work harder
Software testing may be the most annoying DevOps step, but there's no doubt that automating the process makes everything work more smoothly. Teams with test automation have fewer complaints about release delays. And teams that have taken it up a notch and added AI/ML into their test automation process are even more upbeat about testing. After all, bots don't need to take a lunch break or a vacation. Finally, if automation is well thought out, QA and developers can actually work together to get code out the door.
Test for everything
For all the developer finger-pointing around software testing, it's also clear from our surveys that more testing – of everything – has to happen. When considering how to modernize a software testing strategy, don't forget that "nice to haves" like accessibility testing aren't actually optional but critical for success.
And also don't overlook the potential of newer test techniques like fuzzing, which can work with Go, Rust, and other languages, and take testing into places other methodologies cannot.
The bottom line
Testing doesn't have to be the enemy of speedy releases or the object of so much frustration. Start fresh with a modern software testing approach and and make it easy for teams to get the most out of QA.
