What is Continuous Testing? - Codoid
Select Page
Software Testing

What is Continuous Testing?

Irrespective of the business that your client is in, marketing is an integral part of their operations. In fact, in this day and age of social.

What is Continuous Testing

Continuous Testing is the method of providing feedback to the project stakeholders on quality of build through automated test execution as soon as the build is deployed with the help of QA tool stacks.

Since in the Traditional Software world, there is a waiting period of the development team to receive the feedback from the testing team post-deployment of code, which consecutively delayed the delivery. Continuous Testing came up with the approach of Test Quickly, Test Often and Test everything which helped to improve the code quality and quality of delivery by shortening the delivery cycle through the automated way.

By building the delivery pipeline where automated builds, test, and deployment are Orchestrated in one release workflow, there is seamless feedback on the each of the pipelines and based on the success of each of the pipeline process continues which helps to assess the quality of the release.

Why Continuous Testing (CT)?

It helps tightly connect Testing with Development and Operations team to ensure quicker time to market with better quality and saves cost

Instant feedback to developers and stakeholders on code quality and functional coverage to course-correct

Releasing with speed without quality is compromising Customer objective and hence CT should be plugged into CI and CD to deliver speed with quality to meet Client objective

Agile is shortening the release cycle with maximizing delivery throughput. 90% of delivery happens through Agile practices across Organization and hence DevOps (CI/CD/CT) is the key to meet this objective.

Key of Digital Transformation (which is swiftly happening in later 2018) for any customer is quality and the way to achieve with speed is through Continuous Testing

Popular Tool Stacks

Continuous Testing

Steps to achieve Continuous Testing:

Assess the number of Applications in-scope

Assess the Application category-UI (Web / Windows), API, Database, and The infrastructure needed for Application to be executed.

Criticality of Application

Number of Test cases / Application

Tooling Strategy

Validate and Assess CI / CD methodology

Framework Requirement Assessment for CI / CD Integration

Build Framework

Validate CT pipeline

Build sanity automation scripts

Integrate with CI / CD pipeline

Fine-tune framework and Test Automation approach

Start building Automation suites to achieve CI/CD/CT

Plan for N and N-1 Automation based on Organization goal

Assess the number of Applications in scope:

1) Before going to Continuous Testing, we need to assess the total number of Applications which need to be moved to CT pipeline needs to be assessed

2) This can be decided based on understanding the lifespan of Application, number of releases (major / minor / Fix release or Trivial) / year, stakeholder understanding and operational users type of this application

Assess the Application Category

1) Based on the number of Applications and its lifespan, understand the technology in which it is built with. Classify the Application type (UI, API, DB or any of 2 or 3 combinations)

2) Classify the Application based on the infrastructure as well. Some Application will work in Linux, some may support Windows as well as Linux etc. Also classify based on Mobile or other technology

Criticality of Application:

Below are the key factors that determine the criticality of Application:

Reputation damage if implemented wrongly or of any quality issue

Financial loss to company

Operational risk

Sensitive information leakage

Legal violation

Number of Test cases per Application

This is as good as understanding the size of Application

Regression Test cases created till date

We also need to categorize this based on UI, DB, API, etc

Tooling Strategy

Below are the pointers that will help to decide the tooling strategy

Organization policy

Organization support to Commercial tools / Open source tool

Organization willingness to invest in tools

Organization willingness to invest on tool training

Feasibility approach outcome / Application

Application classification and the tools support

Level of 3rd party customization involved from the development aspect

Tools support for CT (Integration support with other tools like Test Management, Reporting, Build Management, configuration management, etc)

Validate and assess CI / CD methodology

In order to achieve success in CT, Automation tool and CT requirement should meet the CI / CD requirement as well so that all 3 can communicate to each other successfully and deliver the desired result.

Example If the development team is working on Orchestration on a particular tool like Jenkins and if a Testing team is working on a tool which does not support Jenkins or which require additional effort to bring in this integration, then it is not advisable to use that tool for CT requirement. We should opt for a different tool which has ready-built support for Jenkins. Hence understanding CI / CD requirement beforehand is essential to reduce the effort from CT perspective in the later stage.

Framework Requirement Assessment for CI / CD Integration

Once tooling strategy is complete and CI / CD methodology also assessed, we can understand the Framework requirement to meet CT requirement.

Below are the additional feature expected in Framework for CT:

Re-usable libraries to support for multiple technologies and different object classification

Incorporation of BDD. BDD is a process where Business Analyst, Development team and Automation Testers come closer to work together to extend the functionality based on each release. Features are added by Business Analyst, The development team shares the wireframe + Object information to the Testing team and Automation team adds the code required to add this additional feature.

Connectivity with Test Management tool for reporting purposes. Good Automation framework should have automated Test set creation based on the build configuration, pull the scripts required for execution, execute them and update the result back in Test Management tool based on execution outcome.

Ability to report outcome based on where the trigger has been generated. Adhoc / Standalone execution or execution through CI or trigger from the Test Management tool

Define Dashboard requirement for Management and Key stakeholders

Build Framework

Based on the framework assessment and technologies, build a robust framework to meet the project requirements. It takes approximately 3-6 months to customize and build framework.

Validate CT pipeline

Post building the framework, validate CT pipeline is meeting the project requirement. Here we validate various touchpoints like how automated build definition works, whether Framework is able to integrate with Test Management tool and CI tools and dashboard result is working as expected.

Build Sanity Automation Scripts

Focus on automating sanity Test suites for each application which helps to detect the code stability to proceed for further Testing.

Integrate with CI / CD pipeline

Once the framework is built, Sanity scripts are developed and the CT pipeline is validated. It is time to integrate with the actual CI/ CD pipeline to evaluate CT process integration with CI/CD. Here again, validation is done to check whether Integrated portions provide detailed information to Management and key stake holders of the project.

Fine-tune framework and Test Automation approach

There might be a specific requirements from the project to project certain additional information to Management and that needs to be enhanced either in the framework level or in the CI/CD/CT definition and that needs to be worked out.

Start building Automation suites to achieve CI/CD/CT

Once the current sanity integration is done and tested, now is time to develop Automation scripts for Regression Test cases and develop different Automation suites based on need and integrate with CI/CD.

After adopting all these steps we have achieved CT goal for an Organization.

Plan for N and N-1 Automation based on Organization goal

Once base Regression suite is prepared, next activity would be to increase the coverage through N and N-1 Automation based on Organization requirement which would really bring in CT benefits.

Key Automation Metrics for Continuous Testing

Test Pass Percentage

Quality of Build

Automation Coverage for Sprint / Release

Requirement Coverage for Sprint / Release

Defect Resolution Time

Defect Rejection Ratio

Build success Rate

Application Crash Rate

Since in CT everything is Automated. The above metrics are also captured in an automated way in the dashboard to Management and Key stakeholders.

Schematic View

What is Continuous Testing?

Key Factors that decide the success rate of Continuous Testing

Willingness of Management to undertake this digital strategy

Willingness of Management to invest time and money towards this initiative

Hiring right resource to perform this Transformation

Proper Analysis on Tool Strategy

Effort spent in Robust Framework creation

Training to key teams on Framework (Key to success is to have separate team for building framework and its customization with the delivery team focusing on building Automation suites)

Framework customization and Maintenance

Dedicated effort to track all these activity

Team should revisit their metrics for every quarter to see if that meets the project requirement and consider revising it to meet project goal

Written By

Submit a Comment

Your email address will not be published. Required fields are marked *


Continuous Testing is the method of providing feedback to the project stakeholders on quality of build through automated test execution as soon as the build is deployed with the help of QA tool stacks.

Since in the Traditional Software world, there is a waiting period of the development team to receive the feedback from the testing team post-deployment of code, which consecutively delayed the delivery. Continuous Testing came up with the approach of Test Quickly, Test Often and Test everything which helped to improve the code quality and quality of delivery by shortening the delivery cycle through the automated way.

By building the delivery pipeline where automated builds, test, and deployment are Orchestrated in one release workflow, there is seamless feedback on the each of the pipelines and based on the success of each of the pipeline process continues which helps to assess the quality of the release.

Why Continuous Testing (CT)?

It helps tightly connect Testing with Development and Operations team to ensure quicker time to market with better quality and saves cost

Instant feedback to developers and stakeholders on code quality and functional coverage to course-correct

Releasing with speed without quality is compromising Customer objective and hence CT should be plugged into CI and CD to deliver speed with quality to meet Client objective

Agile is shortening the release cycle with maximizing delivery throughput. 90% of delivery happens through Agile practices across Organization and hence DevOps (CI/CD/CT) is the key to meet this objective.

Key of Digital Transformation (which is swiftly happening in later 2018) for any customer is quality and the way to achieve with speed is through Continuous Testing

Popular Tool Stacks

Continuous Testing

Steps to achieve Continuous Testing:

Assess the number of Applications in-scope

Assess the Application category-UI (Web / Windows), API, Database, and The infrastructure needed for Application to be executed.

Criticality of Application

Number of Test cases / Application

Tooling Strategy

Validate and Assess CI / CD methodology

Framework Requirement Assessment for CI / CD Integration

Build Framework

Validate CT pipeline

Build sanity automation scripts

Integrate with CI / CD pipeline

Fine-tune framework and Test Automation approach

Start building Automation suites to achieve CI/CD/CT

Plan for N and N-1 Automation based on Organization goal

Assess the number of Applications in scope:

1) Before going to Continuous Testing, we need to assess the total number of Applications which need to be moved to CT pipeline needs to be assessed

2) This can be decided based on understanding the lifespan of Application, number of releases (major / minor / Fix release or Trivial) / year, stakeholder understanding and operational users type of this application

Assess the Application Category

1) Based on the number of Applications and its lifespan, understand the technology in which it is built with. Classify the Application type (UI, API, DB or any of 2 or 3 combinations)

2) Classify the Application based on the infrastructure as well. Some Application will work in Linux, some may support Windows as well as Linux etc. Also classify based on Mobile or other technology

Criticality of Application:

Below are the key factors that determine the criticality of Application:

Reputation damage if implemented wrongly or of any quality issue

Financial loss to company

Operational risk

Sensitive information leakage

Legal violation

Number of Test cases per Application

This is as good as understanding the size of Application

Regression Test cases created till date

We also need to categorize this based on UI, DB, API, etc

Tooling Strategy

Below are the pointers that will help to decide the tooling strategy

Organization policy

Organization support to Commercial tools / Open source tool

Organization willingness to invest in tools

Organization willingness to invest on tool training

Feasibility approach outcome / Application

Application classification and the tools support

Level of 3rd party customization involved from the development aspect

Tools support for CT (Integration support with other tools like Test Management, Reporting, Build Management, configuration management, etc)

Validate and assess CI / CD methodology

In order to achieve success in CT, Automation tool and CT requirement should meet the CI / CD requirement as well so that all 3 can communicate to each other successfully and deliver the desired result.

Example If the development team is working on Orchestration on a particular tool like Jenkins and if a Testing team is working on a tool which does not support Jenkins or which require additional effort to bring in this integration, then it is not advisable to use that tool for CT requirement. We should opt for a different tool which has ready-built support for Jenkins. Hence understanding CI / CD requirement beforehand is essential to reduce the effort from CT perspective in the later stage.

Framework Requirement Assessment for CI / CD Integration

Once tooling strategy is complete and CI / CD methodology also assessed, we can understand the Framework requirement to meet CT requirement.

Below are the additional feature expected in Framework for CT:

Re-usable libraries to support for multiple technologies and different object classification

Incorporation of BDD. BDD is a process where Business Analyst, Development team and Automation Testers come closer to work together to extend the functionality based on each release. Features are added by Business Analyst, The development team shares the wireframe + Object information to the Testing team and Automation team adds the code required to add this additional feature.

Connectivity with Test Management tool for reporting purposes. Good Automation framework should have automated Test set creation based on the build configuration, pull the scripts required for execution, execute them and update the result back in Test Management tool based on execution outcome.

Ability to report outcome based on where the trigger has been generated. Adhoc / Standalone execution or execution through CI or trigger from the Test Management tool

Define Dashboard requirement for Management and Key stakeholders

Build Framework

Based on the framework assessment and technologies, build a robust framework to meet the project requirements. It takes approximately 3-6 months to customize and build framework.

Validate CT pipeline

Post building the framework, validate CT pipeline is meeting the project requirement. Here we validate various touchpoints like how automated build definition works, whether Framework is able to integrate with Test Management tool and CI tools and dashboard result is working as expected.

Build Sanity Automation Scripts

Focus on automating sanity Test suites for each application which helps to detect the code stability to proceed for further Testing.

Integrate with CI / CD pipeline

Once the framework is built, Sanity scripts are developed and the CT pipeline is validated. It is time to integrate with the actual CI/ CD pipeline to evaluate CT process integration with CI/CD. Here again, validation is done to check whether Integrated portions provide detailed information to Management and key stake holders of the project.

Fine-tune framework and Test Automation approach

There might be a specific requirements from the project to project certain additional information to Management and that needs to be enhanced either in the framework level or in the CI/CD/CT definition and that needs to be worked out.

Start building Automation suites to achieve CI/CD/CT

Once the current sanity integration is done and tested, now is time to develop Automation scripts for Regression Test cases and develop different Automation suites based on need and integrate with CI/CD.

After adopting all these steps we have achieved CT goal for an Organization.

Plan for N and N-1 Automation based on Organization goal

Once base Regression suite is prepared, next activity would be to increase the coverage through N and N-1 Automation based on Organization requirement which would really bring in CT benefits.

Key Automation Metrics for Continuous Testing

Test Pass Percentage

Quality of Build

Automation Coverage for Sprint / Release

Requirement Coverage for Sprint / Release

Defect Resolution Time

Defect Rejection Ratio

Build success Rate

Application Crash Rate

Since in CT everything is Automated. The above metrics are also captured in an automated way in the dashboard to Management and Key stakeholders.

Schematic View

What is Continuous Testing?

Key Factors that decide the success rate of Continuous Testing

Willingness of Management to undertake this digital strategy

Willingness of Management to invest time and money towards this initiative

Hiring right resource to perform this Transformation

Proper Analysis on Tool Strategy

Effort spent in Robust Framework creation

Training to key teams on Framework (Key to success is to have separate team for building framework and its customization with the delivery team focusing on building Automation suites)

Framework customization and Maintenance

Dedicated effort to track all these activity

Team should revisit their metrics for every quarter to see if that meets the project requirement and consider revising it to meet project goal