Manual Testing Vs Automation Testing: How Much Does It Really Cost?
We compared hours and hourly rates and the outcome might surprise you
Automation testing or manual testing? That is the question! Which one is better? This is something that gives many in the tester community sleepless nights. Especially during a project when we have to determine the testing strategy, even at its later stages. This is also, unfortunately, a question with no clear answer. Why? Because the reality is that these two ways of testing are inseparably connected with each other.
If you’re still wondering which is more profitable, effective, or better suited to your or your client’s project, let’s dive right in. Here we’ll be revealing the true value of automation vs manual testing, based on one of our clients’ applications – CloudAdmin.
Why CloudAdmin?
Together with Pawel Gieniec (CEO & Founder CloudAdmin), we are creating and developing a web application – CloudAdmin. Cloud Admin is a cost-optimization platform, which helps save thousands of dollars by eliminating hidden cloud instance buying costs.
The project turned out to be perfect for introducing an automation architecture for modules that are always tested during regression testing (the process of running older tests to ensure that new updates to a piece of software haven’t introduced or re-introduced previously eradicated bugs). Our goal is to save time, which we will then be able to spend on manually testing new functionalities. The application is developing dynamically, so we’ll be needing more and more time to check all of the fresh new features.
One of the core functionalities is logging. Nothing surprising, this is probably the module most often automated.
We decided to use automated testing with Python and pytest.
And this is how it went:
We based our login tests on these 7 test cases below.
The first video is from PyCharm (the Python IDE). It shows how long it takes to run and finish all of the tests for the login module. As you can see, all of them passed. This means all of the cases have been tested correctly and have achieved the expected results, i.e. the functionality works properly.
Now take a look at the video below. It shows how long it takes to carry out manual testing for the same test cases as for the automated tests.
How test automation reduces the time
At first, you might think “Hmm, this is no big deal. There’s only an 18-second difference in favor of automation.”. Okay, you’re right, but let’s think more in the long-term and globally.
First of all, it’s not just an 18-second gain. It’s an almost 2-minute yield for one run! This will ultimately give us more time to spend on the project. How?
If you decide to implement automation architecture for sections that always needs to be covered by manual regression testing, you will gain all of this time for doing other things (manual testing for new features, application development, automation architecture development).
The login module is just a drop in the ocean. But let’s take a look further. Let’s compare working times by adding the next 3 modules: AWS Ri Planner, AWS RI Management, AWS Spend.
As you can see, the automation tests give us a 35% time yield relative to the manual tests (9:16 min vs 14:14 min). And again, this is a 14:14-minute gain, we have saved on regression tests – for only 4 modules.
How test automation increases productivity
As mentioned before, by dint of test automation, the testing team gains more time they can spend on doing other tasks. But this is not the only thing they get. Regression testing can be tiring. Imagine how tedious it would be to repeat the same action over and over again. This might sound trivial, but the time involved in repetitive, manual testing is wasted, especially over a longer course of time. With this in mind, think about how much more productive testers could be if they were able to devote more time to performance testing, security testing, or testing of new features.
Moreover, testers have to quickly and continuously execute a lot of time-consuming tests simultaneously, just to be sure an application is performing as expected at all times.
How long will setting up an automation architecture take?
It’s rare that everyone on your team has automation experience, so when it comes to resources, you will really have to gauge how much time it will take for your tool to make an impact. You will either need to invest in expanding your team’s automation knowledge or hire automation specialists.
Let’s go back to our CloudAdmin tests.
Considering the 4 sections previously mentioned (Login, RI Planner, RI Management, Spend), our QA Engineer (the automation tests were written by one person) devoted 80 hours to:
- choosing and configurating tools – 16 hours
- writing automation tests for every test case – 16 hours per module / 64 hours per 4 modules
It might seem like this is a long period of time, but don’t be fooled. Automation architecture is reusable. We can use it limitlessly. Maintenance and development of automated test cases is an ongoing process. Unfortunately, there’s no guarantee that spending eighty hours automating a set of tests means you won’t have to deal with them again. But it will still take definitely less time in general. Take a look at the graph below:
The turnaround time for automatic testing is definitively longer than for manual. However, the benefits of automation outweigh the time benchmark in terms of:
- Reduction in execution time
- Reduction in effort
These two metrics will provide an incredible boost towards higher efficiency and a better rate of reaction to potential bugs.
What about costs?
Last but not least, is something most crucial when establishing a contract with a client – costs. Test automation might be a catchy slogan during negotiations but it’s worth checking the hidden costs. You don’t want the client to think this is just another fashionable balloon, ready to burst at any time.
If you intend to adopt an automated testing process to meet the rising demand for bug-free releases and faster delivery cycles, it’s vital to assess whether the return on investment (ROI) is worth the changes. Before executing, or even considering building an automation strategy, you will want to calculate the net gain you will achieve from transitioning. Divide this by the net investment needed to transition (i.e., the tools and resources you use), and you will see your ROI for automated testing.
The maintenance cost of test automation is not usually trivial but if you keep enhancing the test automation framework, improve the application testability and overall testing processes, a high ROI is very much achievable.
To get the total cost, you will also want to take into account the hourly cost of the number of team members executing the tests.
Based on the average earnings for an Automation QA Engineer and a Manual QA Engineer (I rely on data from Glassdoor), we can make a labor cost comparison simulation based on the CloudAdmin case study (hours needed to perform each type of tests for this application).
As I mentioned before, it took 80 hours for our QA Engineer to develop and execute the automated tests for the 4 CloudAdmin sections. How many times might we manually test the same part of the app at the same time? 337 times (for the record: manual testing time -> total time from the module times table). So what’s better:
- spending $2,100 on performing the same set of regression, manual tests 337 times
or
- spending $2,700 on building an automation architecture – which once created is reusable?
We’ll let you answer that question ;)
Final battle: Manual Testing vs Automation Testing
To wrap up, is an automation testing a cure-all for time-consuming manual testing? Not really. Manual testing will always be important. There are scenarios that will always require manually executed test cases. This means you will still have to create, run, and maintain the tests. But there are still a lot of situations when you can successfully implement automation. This can be a cost-effective method for regression testing of software products that have a long maintenance life.
With a balanced approach to the testing process and a smart and effective amalgamation of both automation and manual testing, you can spend more time focusing on building new features to enhance your product quality – ultimately giving yourself the opportunity to stay ahead and be more competitive in the market.
Check out other testing-related articles
- REST ASSURED API TESTING TUTORIAL. HOW TO USE REST ASSURED WITH JAVA AND SCALA
- HOW WE MADE MANUAL TESTING OF 590 TIMEZONES IN 2 DAYS POSSIBLE
- TESTING FOR MOBILE APPS
- PERFORMANCE TESTING
Check out other business-related articles
Authors
QA Engineer for two years, Economist by profession, Traveler by passion, typical Aquarius. I love my job because of the many self-growth capabilities and the great people I have the opportunity to meet. I never say no to board games, cycling, traveling and fashion ;)
Popular Posts in category
Looking into Scala.js
Your first microservices using Scala and Lagom
Handling Split Brain scenarios with Akka
