by admin | Feb 21, 2020 | Software Testing, Fixed, Blog |
Continuous Testing is the method of providing feedback to the project stakeholders on quality of build through automated test execution as soon as the build is deployed with the help of QA tool stacks.
Since in the Traditional Software world, there is a waiting period of the development team to receive the feedback from the testing team post-deployment of code, which consecutively delayed the delivery. Continuous Testing came up with the approach of Test Quickly, Test Often and Test everything which helped to improve the code quality and quality of delivery by shortening the delivery cycle through the automated way.
By building the delivery pipeline where automated builds, test, and deployment are Orchestrated in one release workflow, there is seamless feedback on the each of the pipelines and based on the success of each of the pipeline process continues which helps to assess the quality of the release.
Why Continuous Testing (CT)?
It helps tightly connect Testing with Development and Operations team to ensure quicker time to market with better quality and saves cost
Instant feedback to developers and stakeholders on code quality and functional coverage to course-correct
Releasing with speed without quality is compromising Customer objective and hence CT should be plugged into CI and CD to deliver speed with quality to meet Client objective
Agile is shortening the release cycle with maximizing delivery throughput. 90% of delivery happens through Agile practices across Organization and hence DevOps (CI/CD/CT) is the key to meet this objective.
Key of Digital Transformation (which is swiftly happening in later 2018) for any customer is quality and the way to achieve with speed is through Continuous Testing
Popular Tool Stacks
Steps to achieve Continuous Testing:
Assess the number of Applications in-scope
Assess the Application category-UI (Web / Windows), API, Database, and The infrastructure needed for Application to be executed.
Criticality of Application
Number of Test cases / Application
Tooling Strategy
Validate and Assess CI / CD methodology
Framework Requirement Assessment for CI / CD Integration
Build Framework
Validate CT pipeline
Build sanity automation scripts
Integrate with CI / CD pipeline
Fine-tune framework and Test Automation approach
Start building Automation suites to achieve CI/CD/CT
Plan for N and N-1 Automation based on Organization goal
Assess the number of Applications in scope:
1) Before going to Continuous Testing, we need to assess the total number of Applications which need to be moved to CT pipeline needs to be assessed
2) This can be decided based on understanding the lifespan of Application, number of releases (major / minor / Fix release or Trivial) / year, stakeholder understanding and operational users type of this application
Assess the Application Category
1) Based on the number of Applications and its lifespan, understand the technology in which it is built with. Classify the Application type (UI, API, DB or any of 2 or 3 combinations)
2) Classify the Application based on the infrastructure as well. Some Application will work in Linux, some may support Windows as well as Linux etc. Also classify based on Mobile or other technology
Criticality of Application:
Below are the key factors that determine the criticality of Application:
Reputation damage if implemented wrongly or of any quality issue
Financial loss to company
Operational risk
Sensitive information leakage
Legal violation
Number of Test cases per Application
This is as good as understanding the size of Application
Regression Test cases created till date
We also need to categorize this based on UI, DB, API, etc
Tooling Strategy
Below are the pointers that will help to decide the tooling strategy
Organization policy
Organization support to Commercial tools / Open source tool
Organization willingness to invest in tools
Organization willingness to invest on tool training
Feasibility approach outcome / Application
Application classification and the tools support
Level of 3rd party customization involved from the development aspect
Tools support for CT (Integration support with other tools like Test Management, Reporting, Build Management, configuration management, etc)
Validate and assess CI / CD methodology
In order to achieve success in CT, Automation tool and CT requirement should meet the CI / CD requirement as well so that all 3 can communicate to each other successfully and deliver the desired result.
Example If the development team is working on Orchestration on a particular tool like Jenkins and if a Testing team is working on a tool which does not support Jenkins or which require additional effort to bring in this integration, then it is not advisable to use that tool for CT requirement. We should opt for a different tool which has ready-built support for Jenkins. Hence understanding CI / CD requirement beforehand is essential to reduce the effort from CT perspective in the later stage.
Framework Requirement Assessment for CI / CD Integration
Once tooling strategy is complete and CI / CD methodology also assessed, we can understand the Framework requirement to meet CT requirement.
Below are the additional feature expected in Framework for CT:
Re-usable libraries to support for multiple technologies and different object classification
Incorporation of BDD. BDD is a process where Business Analyst, Development team and Automation Testers come closer to work together to extend the functionality based on each release. Features are added by Business Analyst, The development team shares the wireframe + Object information to the Testing team and Automation team adds the code required to add this additional feature.
Connectivity with Test Management tool for reporting purposes. Good Automation framework should have automated Test set creation based on the build configuration, pull the scripts required for execution, execute them and update the result back in Test Management tool based on execution outcome.
Ability to report outcome based on where the trigger has been generated. Adhoc / Standalone execution or execution through CI or trigger from the Test Management tool
Define Dashboard requirement for Management and Key stakeholders
Build Framework
Based on the framework assessment and technologies, build a robust framework to meet the project requirements. It takes approximately 3-6 months to customize and build framework.
Validate CT pipeline
Post building the framework, validate CT pipeline is meeting the project requirement. Here we validate various touchpoints like how automated build definition works, whether Framework is able to integrate with Test Management tool and CI tools and dashboard result is working as expected.
Build Sanity Automation Scripts
Focus on automating sanity Test suites for each application which helps to detect the code stability to proceed for further Testing.
Integrate with CI / CD pipeline
Once the framework is built, Sanity scripts are developed and the CT pipeline is validated. It is time to integrate with the actual CI/ CD pipeline to evaluate CT process integration with CI/CD. Here again, validation is done to check whether Integrated portions provide detailed information to Management and key stake holders of the project.
Fine-tune framework and Test Automation approach
There might be a specific requirements from the project to project certain additional information to Management and that needs to be enhanced either in the framework level or in the CI/CD/CT definition and that needs to be worked out.
Start building Automation suites to achieve CI/CD/CT
Once the current sanity integration is done and tested, now is time to develop Automation scripts for Regression Test cases and develop different Automation suites based on need and integrate with CI/CD.
After adopting all these steps we have achieved CT goal for an Organization.
Plan for N and N-1 Automation based on Organization goal
Once base Regression suite is prepared, next activity would be to increase the coverage through N and N-1 Automation based on Organization requirement which would really bring in CT benefits.
Key Automation Metrics for Continuous Testing
Test Pass Percentage
Quality of Build
Automation Coverage for Sprint / Release
Requirement Coverage for Sprint / Release
Defect Resolution Time
Defect Rejection Ratio
Build success Rate
Application Crash Rate
Since in CT everything is Automated. The above metrics are also captured in an automated way in the dashboard to Management and Key stakeholders.
Schematic View
Key Factors that decide the success rate of Continuous Testing
Willingness of Management to undertake this digital strategy
Willingness of Management to invest time and money towards this initiative
Hiring right resource to perform this Transformation
Proper Analysis on Tool Strategy
Effort spent in Robust Framework creation
Training to key teams on Framework (Key to success is to have separate team for building framework and its customization with the delivery team focusing on building Automation suites)
Framework customization and Maintenance
Dedicated effort to track all these activity
Team should revisit their metrics for every quarter to see if that meets the project requirement and consider revising it to meet project goal
by admin | Jan 4, 2020 | Software Testing, Fixed, Blog |
The business of the telecom operators is an evolving landscape with data and non-voice services having surpassed voice services. Telecom companies therefore have the responsibility and must deliver top quality, innovative, and reliable services if they are to retain existing clients and attract new ones for the long term. A wide range of top quality mobile applications is part of the expectations that customers have from telecom companies as part of the evolving landscape. As a leading Software Testing Company our work within the telecom sector enables us to understand the challenges this sector faces with regard to the new trends and technologies. Augmented reality, the Internet of Things (IoT), virtual reality (VR), and many more are all technologies that customers expect, which forces telecom companies to restructure and readjust their traditional business strategies – posing a significant challenge for many operators.
The telecom industry is comprised of sections such as internet service providing, satellite and wireless communication, and more, but to gain and sustain a competitive edge each company must deliver customized and value added solutions to their customers. Innovative solutions for this industry would include software based updates, customer segmentation, customized marketing campaigns, and top quality customer service.
1. The significant and consistent elevation in new technologies, rise of customer expectations, and the increasing number of internet service providers and other competing telecom companies have led to a sharp decline in the profit margins for telecom companies overall. Such companies need to turn their attention toward ICT innovation, and ensure that their organizations remain flexible enough to adapt to the ever evolving digital transformation. As experts in the realm of QA services, we know that QA testing is essential to ensure optimal working of software and the systems, such that they deliver the expected output.
2. As the number of subscribers and a range of products increase, there is a need for tailored solutions. All this in turn makes customer care, operations, configuration of service, billing cycles and other such aspects, a lot more complicated. Telecom companies therefore, have increased spends on getting more resources and customized tools to ensure that customers get what they want and to gain an edge over competitors. As a result the financial overheads have increased significantly. With so much at stake, these companies must ensure that their products not only function as per customer requirements, but that the products are efficient, convenient, and secure. QA testing is therefore essential – not just to ensure proper functioning of the products, but also to mitigate the challenges and risks that telecom companies could face.
3. Meticulous and regular QA testing would ensure the proper implementation of the solutions, while addressing all the existing and possible issues. Telecom companies must ensure that their end to end solutions are properly supported such that their business remains profitable.
4. As mentioned, the challenges that telecom companies face, is the constant restructuring of infrastructure, the IT connectivity, and maintaining focus on offering reliable and affordable services. The most important is network security given that threats to technology are becoming more potent and frequent. The QA testing experts would be able set up a proper process and provide solutions customized to fit the business and customer base of the companies.
5. No business today can succeed without retaining existing customers and attracting a larger customer base. This can happen only when consistently high quality service is provided, along with effective back-end software, and a user-friendly and highly responsive customer use application. QA testing will ensure that telecom services are elevated and there is a significant reduction in the time to market and customer complaints.
As leaders in the realm of QA services for the telecom industry (and more), we have consistently helped our clients with all the features mentioned above. QA testing analyses all the major operating systems, while optimizing their testing along with hardware devices. Our in-depth expertise enables us to ensure the security, functionality, reliability, and compatibility of the applications used by the telecom sector through wide and rigorous software testing. Connect with us for specialized software testing services and a personalized software testing strategy for your business within the telecom sector (and many more).
