The Case for Codeless Testing

A couple of years ago, I gave several talks at software conferences about testing microservices. While most engineers who attended the talks agreed that testing was important, only about half wrote automated tests regularly.

Some of these engineers had dedicated testing teams, but the majority cited “lack of time” as the reason they weren’t able to write automated tests. I’m not the only software engineer who has observed this problem:

“Test automation is essentially another software development project. Many teams aren’t in the position to take on this responsibility. Either they’ll need to allot time for their existing developers to also contribute to and maintain this, or hire additional coders. It’s a huge investment.” - Angie Jones, Developer Relations at Applitools

There’s data that backs this up as well. A 2020 Forrester report on testing tools points to the same issue:

“Many people think that developers are doing more testing or will do so. However, Forrester survey data suggests that this is not happening fast enough and that developers spend insufficient time testing.”

Automated tests are almost always a good investment. According to a 2008 IBM study, bugs are 15 to 100 times more expensive to fix when they get to user testing or production. Despite all evidence pointing toward the value of testing, developers still aren’t writing as many as they’d like.

If you’re a technology team leader, the obvious next question is, “What can you do about it?”

The Testing Pyramid

Assuming your goal is to increase software quality and you believe that tests can help, your next step is to decide which kinds of tests you need to adopt. A helpful mental model for this is called the testing pyramid.

Typically, developers have taken the testing pyramid to mean they should focus on unit tests, but unit tests have limited value on their own. “Unit tests are quick,” Iliyan Panchev, Product Manager at Test Studio told me, “so they can be the base of your testing. But if all business transactions happen through the UI, visual testing is top priority.”

Your testing strategy also depends on the type of application you’re building. As “no-code” software tools become more popular, it’s essential to ask how the traditional view of the testing pyramid holds up.

“If you’re building a complete web application from scratch, then I think the Test Pyramid is still a solid scheme,” Justin Klemm, CEO of Ghost Inspector said. “But maybe you’ve just got a WordPress website with a custom theme and some ecommerce plugins … Are you going to write unit tests? Is that even possible?”

What is Codeless Testing?

When you look at the testing pyramid, it can be daunting. As an engineering manager, I am always looking for ways to use technology to solve problems like this, and one area I’m especially excited about is “low-code” or “codeless” testing.

Similar to no-code software, codeless testing tools offer higher-level abstractions for writing tests. Most codeless testing platforms offer visual test recorders for end-to-end tests, but many also include lower-level interfaces for engineers to write custom tests or selectors as well.

For example, tools like Ghost Inspector and Telerik Test Studio offer browser-based test recorders and editors. This gives test engineers the ability to create tests by clicking through their apps in a browser and then tweaking the automatically generated tests before running them.

Other codeless testing tools like Perfecto rely on artificial intelligence to automate the test writing and maintenance process. While there’s certainly some hype around the field of AI, these tools will continue to improve in the next decade.

“With the ever-increasing user demand for quality and pressure from the highly competitive market, the future of testing is achieving and maintaining extremely high efficiency of test engineers and developers.” - Iliyan Panchev, Product Manager, Progress Telerik Test Studio

Reasons to Adopt Codeless Testing

Codeless testing platforms can remove some of the barriers that stand in the way of test automation today. As the number of codeless testing tools grows, software engineering teams have ample reasons to supplement or replace their existing end-to-end testing solutions.

Here are five reasons to consider codeless testing in your QA lifecycle today:

1. Improve Code Coverage

If you’re currently struggling to improve test coverage across your application, end-to-end tests can be a good place to start. When I inherited a legacy application a few years ago, I started the refactoring process by adding end-to-end tests. We started the process with Mink—a code-based BDD testing framework—but quickly moved to a codeless testing tool for some of the reasons here.

This no-code testing strategy allowed us to write tests ensuring critical pieces of the application worked (like the login, checkout and user registration workflows) while we refactored the internals. Eventually, we broke the application into microservices and a modern frontend framework, all the while maintaining our codeless tests outside the application.

2. Lower Learning Curve

As software developers, we have a lot to keep up with. New frameworks, languages and best practices emerge all the time—especially in web development—so taking the time to learn a low-level browser automation tool is often hard to prioritize.

“The biggest barrier we see that stops organizations from adopting automated testing is a steep learning curve that puts the responsibility on an engineering team that’s often overburdened already.” - Justin Klemm, CEO of Ghost Inspector

Codeless testing platforms are typically GUI-based, and software developers can probably pick most of them up in an afternoon. While many of the codeless testing tools I’ve encountered are designed to be managed by QA engineers or developers, some are straightforward enough for product owners or business analysts to use effectively. “The majority of Telerik Test Studio users are non-technical or somewhat technical,” Panchev told me.

3. Minimize Test Maintenance

End-to-end tests tend to be the hardest to keep up to date. If you’re using low-level CSS selectors to look for specific elements on the page, every little change to the DOM will result in dozens of broken tests.

While some codeless testing tools suffer from this same problem, others do not. For example, Telerik Test Studio includes a hybrid element detection, that can find elements based on complex selector queries and, when needed, images. That way it is less likely to break your tests when you update your page’s hierarchy. This takes a considerable burden off your team when maintaining end-to-end tests.

4. Faster Feedback Loops

End-to-end tests are often the slowest to run because they require you to build, compile and run your entire application. In some cases, it’s not even possible to run the entire suite locally, so developers must push their build to a CI server and wait for results before they proceed.

Many codeless testing solutions are externally hosted and allow you to run a tunnel to test your application locally. For example, Ghost Inspector can use ngrok to access your local build and run your end-to-end tests quickly using their workers. They also allow you to run multiple instances of their test workers to speed up the testing process.

5. Tests Can be Maintained Independent of Code

Finally, many codeless testing tools can be used outside your existing codebase. This means that they aren’t adding maintenance overhead, libraries to upgrade or merge conflicts to rebase. “I think we all can agree that code is a liability,” Jones noted. “The less code that we have to write and maintain, the better.”

The Limitations of Codeless Testing

Skeptics will be quick to point out that codeless testing is rarely completely free of code. In fact, many “codeless” testing platforms allow engineers to use advanced features by writing custom code. It might be more accurate to call most of these tools “low-code” testing tools, but “codeless” tends to be a more common moniker.

The reality is that a hybrid approach to testing—some tests running in code and others in codeless tools—is probably the most useful. “[Codeless testing] tools are just a way to improve test coverage and ensure superb user experience,” Panchev asserted. “You need to be very clear on the main business goal of your application and define testing scope accordingly.” A hybrid approach to testing will give your engineering team the power of platform integrations and customization with the ease of recording tests in their browsers.

Another limitation to codeless testing today is that most of them (with the exception of Telerik JustMock) focus on end-to-end tests. These don’t preclude the need for lower-level unit or integration tests, and they still require maintenance, especially if your application’s UI changes dramatically.

Finally, like the use of any third-party tool, developers should consider the risk of vendor lock-in. Relying on a third-party vendor to manage your tests will give you a lot of functionality for a low cost, but it might not be possible to migrate off some codeless testing platforms.

Codeless Testing and the Future

Despite these limitations, the future of codeless testing looks bright.

The tools available today are useful for end-to-end testing, but we could see more codeless unit testing tools emerge in the future. By allowing non-technical team members to contribute tests, these tools will continue to shorten feedback cycles and improve cooperation between developers and product managers throughout the software development process.

Finally, if you assume that artificial intelligence will continue to improve in the coming years, it’s pretty safe to bet that more of it will show up in codeless testing tools. “We’ve been hearing a lot about [AI and ML] technology steadily getting into the testing world,” Panchev said, “But I believe that the real value and innovation here is yet to be seen.”

We’re still in the early days of codeless testing, so it’s exciting to imagine what this trend will hold for the future of our industry. I’d love to hear your thoughts, so feel free to leave a comment below.