DevOps and Agile primarily focus on how to add value to both the product and the business, and test automation is considered an important activity in the delivery pipeline. In the past, test automation metrics and ROI were used to determine the benefits of automation testing. Let’s say an automated test suite runs perfectly for 6 months without any changes and refer to the metrics from the executions, it will show positive outcomes. But the underlying problem here is that the automated test scripts which are untouched and not improved don’t provide any value at all. So in this blog, we will be going through how to effectively measure the value of automation testing.
Measuring the value of test automation in DevOps & Agile is totally different. Let’s take a look at some of the more traditional value measuring techniques and explore why they are no longer effective.
Let’s start off with one of the most basic metrics, the time is taken to perform the tests using automated test suites. In most cases, automation test execution will be faster than manual test execution. But isn’t it more important to see what is being tested instead of how quickly it is tested in the initial phase? We understand that providing quick QA feedbacks to a team is a paramount task, and are not denying it. You can employ multiple Docker containers and run the scripts in parallel to achieve speed. But make sure you don’t compare the execution time between manual and automation testing for each test case. It does not help you to measure the value of your automation tests.
Percentage of Pass & Fail
False Positives & Negatives is one of the major productivity killers for automation testing that requires you to script well to avoid them. Just because all your tests have passed, you cannot assume that your scripts are doing well. You need to ensure that your automated test suites are constantly improved by adding new test scenarios, updating the existing scenarios when functionalities change, and also fixing the false positives & negatives. There is no doubt that the percentage of pass & fail is a good metric, but you need additional supporting data to measure the value of your automation testing more precisely.
In DevOps, you need to improve your automated test suites continuously. Only stable test scripts are qualified to fit in the delivery pipeline. As a leading QA Company, we use automated testing reporting dashboards to view all the collected metrics to measure their value. So based on our vast experience, we recommend you collect the below metrics to measure the value of automation testing.
- Keep track of the number of scripts that are up and running in the pipeline.
- Collect data on the number of scripts that are getting added.
- Monitor how many scripts are updated & removed.
- Check how many new test cases are identified for automation testing from your Exploratory Testing Sessions.
- We know that unstable scripts have to be quarantined, so find out how many scripts have been quarantined. Make sure that you bring them into the delivery pipeline only after stabilizing them.
- Keep track of the frequently failing tests.
- Your tests don’t just have to work, they have to be fast as well. So, collect data on the slowest tests as this metric will expose if any page is loading slowly or if it has any scripting issues.
- Monitor common exception messages. Since Selenium and Appium have an exception list, you can improve your automated tests by fixing the common exceptions.
- Track the Success Rate when you retry. Sometimes, running the failures again might get them to pass. But if you notice any script which is passing only after retry, it needs your attention.
Number of defects and other issues
- Keep track of the number of defects you find in testing
- Monitor the number of script issues
- Check the number of issues in the test data
- Collect data on the number of infrastructure-related issues
Being the best company for automation testing, we at Codoid have switched our test automation value measuring strategy from the traditional approach. We have seen great results and you will also be to witness the same when you try our approach. You will be able to capture the above-mentioned metrics during test execution, and from the framework & the test management tool to measure test automation value.