Automation Testing success can be measured using its metrics. Analyzing only the last execution results will not help you to determine test automation success completely. In order to achieve better results, you need to save all the historical execution results to enable automation testing metrics. For example: if you want to know how many times a feature is passed and failed, then you will have to ingest each execution’s test result in a database and create a web dash board. So that your team can see what benefits we have reaped so far from automated testing.
As a leading automation testing company, while formulating test automation metrics we consider the following two crucial aspects.
- Automation Testing Benefits perspective.
- Script Quality Perspective. if an automated script keeps on providing false-positives, then it will be an indication for automation testing team to make the script robust enough to stop false positives. Let’s see all the required automation testing metrics one by one.
Pass and Fail Count
Pass and Fail count from last execution helps to conclude the testing. However, if you get more failures, then automation testers should troubleshoot the failures. If a failure is a false positive, then check how many times the script failed in the past and mark it for the update. When sharing the historical failures count, your team only focuses is to stop false positives.
Automation Testing Coverage
Automation testing starts from the senior management. They are the one who approve the budget to kick-off automation testing project to reap benefits from it. As soon as the project is started, sharing how many scripts are converted from manual to automation on weekly basis is a vital information for the management team. Consider, 30% of test cases are not automated due to various reasons. your team can plan how much effort is required for manual regression testing for the non automated test cases.
No. of executions
If you have automated test suite, then it should be utilized as much as possible for various testing activities. No. of executions is a great metric for your team and management. Once the regression testing is taken care by automation, the testers from your team will get more time to conduct exploratory testing and to test new features, stories, & other impacted areas manually. No. of times the automated test suite executed metric is an input to your management team to measure the success of test automation.
Failure Count
This metric is useful to enhance scripts quality. How? If you are capturing failure count for Feature, Scenario, and Step level, then you can easily identify automated scripts which are failing often. When your test automation suite is robust enough, then it will add more value to your testing.
Execution Duration
Execution duration metric is an input for your team and management. Let’s see how is it valuable to the management team? If everyone in your team is aware of how quickly your automated smoke and regression test suites complete the execution, then your team can take important decision on how many hours/minutes do we need to show up maintenance page during product release deployment? And how many hours do we need to wait to complete the regression testing on lower environments? Even your team can revisit the scripts to fine tune to reduce the execution time.
Conclusion
We recommend you not to execute your automation testing scripts without collecting metrics. Doing so will be a great value add to your project. You can capture many metrics. However, as a first step, capturing the test automation metrics which are mandatory and useful is important. As a test automation services company, we collect metrics to make the effort visible to clients, and the management so as to improve the scripts’ quality.
Comments(0)