Feasibility Analysis in QA Automation

In this blog we are going to discuss the software test automation lifecycle followed by the feasibility analysis that is required. The feasibility study is a must before taking any action with the automation since it is the deciding factor for automation sign-off based on the value that is going to add to the business.

Life cycle of Software Test Automation

Life Cycle in layman definition is the series of steps/changes that happens in a process to get the desired result.

Life cycle denotes the subsequent actions in a timely fashion. Below are the lists of things to be considered for test automation life cycle.

Feasibility study

Tool identification

Framework identification

Proof of concept

Test scripts design

Test scripts execution

Test scripts review

Feasibility analysis & its significance

As discussed at the start, the feasibility study is something without which we can’t judge the outcome of automation efforts also can’t identify the right automation candidate. This study helps the test team in the forecast on the following activities.

Can the application be automated or not?

What type of tool/ test automation framework can be used?

How much automation is possible?

In spite of high efforts, what’s the value add in automating?

Below are some key parameters while performing the feasibility study, let’s look into each one of them in brief

Feasibility study on the application and understand the functional flow

Before we proceed with automating any application, the automation tester must possess the knowledge of the application. Without having this understanding when any test scenario is considered for automation and we may get into several problems and the task either would be left incomplete or likely delayed.

As an automation tester the focus would be to gain sufficient functional knowledge, get the relevant test cases from the functional (manual) team then conduct feasibility analysis to know how many scenarios can be automated. If at all some cases can’t be automated there must be a valid rationale and the agreement with the stakeholders must be taken.

The analysis helps the test team to showcase their deliverables because after thorough examination team will be in a position to understand what will be covered in automation and how long they require to complete them. Failing to conduct this study could lead to irksome state.

Feasibility study in tool selection

We must conduct a careful study before finalizing the tool. The key points to be considered are

Whether or not the tool can detect all the objects of an application?

The tool that we are going to use for automation must be able to locate all the elements of the application as the locators are the key for the script to interact with the application elements

In case of selenium, we rely on Xpath, CSS selectors, name, Id..etc

If the case is to automate the windows application then selection of a tool that can interact with windows must be chosen

Eg: Coded UI – can locate the windows much better.

Whether or not the tool is best compatible with the application technologies?

There should be a sample POC conducted with all the list of proposed tools to automate the applications then decide which one should be of the good match by considering the application development technologies and its compatibility with the tool.

Eg: while selenium is used to automate all the java, .net, PHP oriented applications, it’s essential to opt for protractor when the application front end is mainly designed with Angular JS. Because angular elements can’t be caught with selenium

Whilst choosing the tool, the cost that would be required for investment should be taken care of. If at all there is a tool exists which does the job w/o any degradation of quality we can fairly go with that as that’s going to add a lot of savings, which will eventually prefer to get better ROI.

Community that’s available for that tool, should be given the importance to an extent as that helps in troubleshooting some of the issues we see in our daily routines.

More the community more preferable the tool is

Feasibility study on framework development

Once the tool choice has come to a conclusion, the next step is to perform the study on framework piece. It should be chosen in a way that the tool supports also it should be quite generic. It should give us better reports to analyze the results

A POC has to be conducted, then after observed the results, we must think of coming up with more reusable components. We might take a lot of hours of time than what is really required for manual validation. But in the long run, as we pass by different phases, the suite adds a lot of value since the execution time will come down to a great level.

A proper check on the regression scripts and baselining them is needed as well because that helps to add more coverage.

Feasibility study on test strategical approach

When it comes to the test approach, the business should take a call whether what between the below strategies should be followed.

Test driven approach

This approach is first to write the tests and then try to accomplish the fix for it. This would be a sophisticated approach and it’s not that easy for non-technical stakeholders to decipher properly. This is completely a call taken by the business based on their budget and other parameters that make sense for a project.

Behavior driven approach

This is buzzing in recent times, as the progress is made by documenting the task in plain English then try to map them with the technical actions. This gives more insight into what a particular test case was written for, what validations are performed. This business model delivers a better report than the above model.

Eg: Cucumber, Specflow, Gauge..etc

Feasibility study on infrastructure availability

In order to proceed with hassle-free automation the test team must not have any issues at infra level, the possible issues that might fall under this category are

Not having access to create project/ branch in a chosen configuration management tool

Not having access to the application/ DB servers to work independently

Software license procurement by the business

Network availability and kind of access to Application, which will delay the execution and cause failures at times

Not holding admin rights to perform certain tasks

The above-listed problems will prohibit automation activities from point to point. Test team must raise the access requests to all the needed things in the project and gain access.

Stable environment and a major part of development should be completed

As a test team we should know when to automate a particular application. If the build quality is at an infancy stage, we see the basic smoke tests are failing then there is no point in automating the test cases at that stage.

When we go for automation when the major development is in progress, we end up adding a lot of maintenance to the test suite as the UI locator elements might get changed often or at times there are some changes expected at functional level, then we may need to re-write the scripts as per new design which is tedious.

Assessing how much automation can be achieved

100% automation is fallacy! Given the complexities of whether or not they are technical or no-technical, that may occur during automation development.

As an automation tester doing this exercise helps the business understand the index of cost and effort saving. Ideally, this is achieved by comparing the functional test cases with the automation test cases. This metric defines what percentage of the tests has been automated and accordingly, the savings were understood.

Calculating the ROI metrics

Metrics are key numbers, which will denote the status, progress, outcome in various ways. As a test engineer/ development engineer when we bother about the accomplishment of tasks, but the management always looks at the numbers extracted as metrics. This is one short and explainable way of presenting data to understand where we are? What have we achieved?

In the same way, in automation world we have certain metrics to be calculated to project the savings to the business, following are a couple of quick examples

Automation savings

This is to understand how much time in man-hours and effort saved because of bringing in automation

Savings = Manual test efforts – Automation test efforts

Return on investments

Certainly we are investing during automation in the means of script development, managing talents and the cost of procuring the infrastructure that’s needed.

Cost of automation investments = Cost of talents + Cost of script development and maintenance + Cost of infrastructure.

Conclusion

More the ROI, the more the profits and that is what business needed as well. In order to achieve better ROI, we should focus on below

Try to bring in more automation

Focus of in-sprint automation development

Right choice of tools, which will incur no or as minimal budget as possible

Develop a generic framework with more reusable components, to avoid extra and often maintenance

Frequently Asked Questions