Blog

The 6 Step Solution for achieving Continuous Testing with Automated Testing

by admin | Oct 26, 2021 | Automation Testing, Blog, Latest Post | 1 comment

Releasing software multiple times into production in a day to deploy the new features/fixes with high quality will need an automation testing setup in all the stages of your delivery pipeline. In DevOps, you have to automate as much as possible to enable continuous testing. So when a product is getting deployed into production more frequently, you have to test the product continuously. For you to proactively search for quality issues, continuous testing has to be enabled across all stages of your delivery pipeline. Since software Test Automation is the key enabler of Continuous Testing, we will be taking a look at the 6 steps that can help you achieve Continuous Testing.

Treating Test Code as Production Code

Whenever a change happens in the product, you have to update your test code to accommodate the change. Let’s say you have to make a change on a common page/screen, your test automation framework should enable you to update the change in that one place instead of updating all the scripts.

Implementing the best practices and design patterns in test automation ease script maintenance. So you have to design your framework in such a way that any script change can be added to the test suite quickly without any hassle.

Kick-off Test Execution for Every Code Commit

Every code commit to version control should kick off automated unit & acceptance tests. When you deliver fast, you will also be in need of quick feedback for the changes or fixes that you have made. Lack of code commit validations will lead to eventual quality issues and regression defects.

Test Automation Framework

You can’t just build the entire framework and immediately start the script creation for Continuous Testing. We have to use the Acceptance Test-Driven Automation (ATDA) approach to develop an automation testing framework. ATDA enables you to write automated test scripts from the very first day instead of waiting for the framework development phase to be completed.

In the past, automation testers used to spend at least two weeks, or even a month in certain cases to develop the test automation framework. The script development itself would begin only after that. So in order to achieve Continuous Testing, you have to start the script development from day one, and then go forward with the product development.

How can we develop scripts without a framework?

Let’s say your team is comfortable with Java. You can use JVM-Cucumber, Selenium, Maven, Appium, and IntelliJ Community edition to start writing automated test scripts from day one. If a script needs a new framework feature, you first have to develop the feature and complete the script. That is how you can evolve the framework and not hold up the script development during the framework development.

Avoid & remove Flaky Tests from CI

If an issue messes with the DevOps pipeline, the entire team should focus on the issue and fix it immediately. Similarly, if a script is flaky, it will make the pipeline unstable. Automation Testing Services is our core service, and we know for a fact that you’ll need a highly skilled team that follows the best practices and uses proper object locators to avoid flaky tests and achieve Continuous Testing.

So if some of your tests are flaky, you must quarantine, schedule, and run them separately, and bring them into the delivery pipeline only when it is fixed and stable.

Test Data Generation

Test data plays an important role in UI, Functional, Non-functional, and Integration Testing. So you have to park adequate test data for test execution. Make sure to avoid failures that can be caused by invalid or missing test data. You should also have an automated system that allocates the required test data and cleans up the consumed data during execution.

Setup the Tests for all the phases

Continuous Testing should be set up in all phases of your delivery pipeline starting from development to production. For example, if a feature is being actively used by the end-users on a daily basis for a particular period of time. You can write an automated test script to check the feature’s usage from the production monitoring data to proactively check for any quality issues. So make sure you don’t just focus on smoke and regression testing as you have to set up tests for different stages in the pipeline.

Conclusion

Being one of the best automation testing service providers, we, at Codoid, follow strict scripting practices and design patterns to avoid flaky tests and have helped many clients to enable Continuous Testing in their DevOps pipeline. Fast feedback is critical for DevOps. You also shouldn’t delay test code development. You have to add the test code as and when new features are deployed into the pipeline.

The 3 Primary Objectives of Automation Testing to Focus On

by admin | Oct 19, 2021 | Automation Testing, Blog, Latest Post | 0 comments

Automation testing is not your departmental goal. Period. So when software testers or automation testers create automated test scripts, they should focus on how the tests will add value to the business or product instead of focusing on increasing the script count. Since analyzing the goals and objectives of automation testing before starting any project is very important, let’s learn about it in this blog.

A Confidence Booster

Automation test executions boost the confidence of your team to release a product. Testing a system manually on multiple browsers & devices is a tiring effort for a tester. Moreover, it is very much prone to errors. Whereas, if rightly done, automation scripts can execute your tests without any deviation. So this ultimately frees up the testers and lets them focus on the problem-solving activities.

In DevOps, there is always a need to automate as much as possible so that the testers can concentrate on Exploratory Testing to unearth bugs, identify new features, and capture edge cases for automation testing. So if there are no application issues, robust and reliable automated tests will run without any hassle, and the passing automated tests will boost the confidence of your team.

Avoid False Positives & Negatives

An automation test script should fail only when there is an application issue. But if your scripts are flaky, you will not be able to reap the benefits of automation testing, and your team will eventually lose confidence in the automated test scripts. That is why we have it in our top 3 objectives of the automation testing list. As a leading automated software testing company, we have seen success in our test automation by following the points.

How to avoid false positives & negatives:

• Recruit talented test automation engineers.
• Train your team to follow the best practices and coding patterns.
• Strictly avoid boilerplate codes.
• Enable the peer-review process.
• Quarantine flaky tests and bring them back to the execution pipeline only if & when they are stable & robust.
• Park the adequate test data required for automated test execution to avoid failures due to invalid data.

You need a highly skilled team to develop robust automated test scripts. We have been providing automation testing services to many of our clients over the years. During this period, we have revived many of the projects that had failed to reap the benefits of effective automation testing. We achieved it by trying to reuse the existing scripts and libraries. But at times, we have thrown away the poorly written automation test scripts into bins. So it is important to keep in mind that if the best practices and coding designs patterns are not followed when creating automation test scripts, you will have to suffer from such scripts forever.

Scripts fail due to invalid test data as well. But feeding the test data before starting the execution is a cumbersome task. In DevOps, automated test executions are kicked off as soon as product changes are pushed into master. So you cannot hold the execution for feeding the test data. Moreover, you will not be able to feed the data a day before the execution if your team is part of the continuous delivery pipeline as they will deliver changes into production multiple times a day.

So make sure to use a test data management tool to upload adequate test data for script executions. Your test automation scripts should be able to pick up the required data from the test data management tool.

Maintenance

When there is a change in the application, you have to accommodate the change in your script as well. If an automated test suite is not changed for a longer period, then it is going to perform the same testing again and again. You should have a process to obtain the below details constantly to keep your scripts up-to-date.

• You should know when a change is introduced in existing functionality.
• Note when a new feature is getting adding to the product.
• Note when an existing feature is removed from the system.

Conclusion

Adding more automated tests to improve coverage is a primary goal of an automation tester. However, if you do it without thinking about the system, goals, and objectives of automation testing, you will be missing out on the many benefits. Automation testing is not an isolated activity and it is pivotal for you to know how your scripting efforts are adding value to the system.

Frequently Asked Questions

An Overview of Gecko Driver & Marionette Driver

by admin | Oct 14, 2021 | Automation Testing, Blog, Latest Post | 0 comments

Gecko is a browser engine for Firefox that has been in usage since the year 1998. But Gecko was initially developed and maintained by a company called ‘America Online’. In the year 2003, Mozilla Corporation took control of Gecko development. Since then, the Firefox browser has improved a lot over the years. Firefox Quantum was released back in October of 2016 with multiple performance and rendering-related improvements. Quantum is an improvement project for the Gecko engine. Being a leading software testing company, we maintain automated test suites for multiple web products. You should be familiar with how Selenium WebDriver works for different browsers in order to achieve multi-browser execution. So in this blog article, you will learn a few important details about the Gecko Driver. (A proxy between WebDriver & Marionette Driver)

pip install marionette_driver

./mach run -marionette

from marionette_driver.marionette import Marionette
 
client = Marionette('127.0.0.1', port=2828)
 
client.start_session()

System.setProperty("webdriver.gecko.driver", "drivers/geckodriver");
     
WebDriver driver = new FirefoxDriver();
     
driver.get("https://google.com");

System.setProperty("webdriver.firefox.marionette", "false");

Conclusion

We, as an automation testing company, run automation test scripts across multiple browsers. As stated earlier, Selenium WebDriver is an important tool that will help you achieve multi-browser testing. To achieve that, you should know how each implementation of the WebDriver works on different browsers. When you know the enhancement details of a browser and its driver, you will be in a position to plan and accommodate the changes in the test automation framework well ahead of time without any hassle.

The 7 Best Open Source Test Automation Frameworks

by admin | Oct 12, 2021 | Automation Testing, Blog, Latest Post | 0 comments

Nowadays, we have many open-source test automation frameworks to choose from that we have created this list of the top 7 open-source test automation frameworks to use. But back in the day, test automation frameworks had to be developed using commercial test automation tools. Before the emergence of such open-source automation testing tools, many proprietary tools had the below-listed limitations.

• Only a few limited programming/scripting languages were supported.
• The scripts could be executed only on IE browsers.
• Separate licenses had to be purchased for script executions.
• Only limited Reporting Features were available.
• Third-party test management tools integration was not possible.
• There was a lack of documentation.
• The annual tool support fee was huge.

Even now, some of the commercial’s test automation tools have these limitations. In the earlier days, the Excel-driven test automation framework was very popular. The claim was that testers didn’t have to open the tool to write the automated test scripts. Stakeholders thought that filling the excel sheet to create an automated script was simple and a game-changer. So if any change was required in the framework, it would be done by the test automation architect. That is when the open-source automation frameworks slowly started to emerge in the market.

Selenium brought the capability to automate both Firefox and IE. This was in a time period when even many of the popular commercials automation testing tools didn’t have the feature to automate any other browser apart from IE. It is important to remember that Selenium is a browser automation library; it is not a test automation framework. You have to use Selenium to build the framework.

Why do we need a test automation framework?

With the help of a test automation framework, we will be able to

• Organize automated test scripts
• Avoid duplication of scripts
• Reduce the effort required for maintenance
• Troubleshoot failures quickly with the help of test reports
• Integrate with test management tools
• Manage test data that are required for test execution
• Follow common scripting guidelines
• Ease portability & adaptability

Now that we have seen why we need an automation framework, let’s move over to our top 7 list of open-source test automation frameworks which can be used right away instead of having to build an in-house framework. As a leading QA company, we get projects requests from our clients to work with ready to use framework. Given our expertise in the domain, we believe that this blog article will help you to choose a suitable open-source automation testing framework to kick start script development from day one.

Serenity

First up, we have Serenity BDD which is a ready-made automation testing framework. It has many built-in methods for both Selenium and API automation which will ease the script development process. The Serenity test reporting feature will be very helpful in understanding what tests have been executed, and what requirements have been tested.

We know for a fact that Serenity is a popular open-source test automation framework. Let’s take a look at what makes it so popular.

• Serenity introduces one more layer between the Step Definitions and page objects. ‘Steps’ is the new layer that represents actors or personas who interact with the application. This layer makes your test scripts more readable and maintainable.
• You can mention environment-specific configurations in serenity.conf file.
• Built-in methods and test reporting system for REST API automation testing using REST Assured is a great value-add.
• Serenity supports the Screenplay pattern which helps to produce readable test code.
• It introduces a new Ensure class to write readable assertions.
• You can also run Serenity tests on a Zalenium server to achieve parallel execution.

Robot Framework

Robot Framework is an open-source automation testing framework that has many built-in keywords and libraries. The robot framework’s core is developed using Python. It was developed by Nokia Networks and open-sourced in 2008.

The various standout Robot framework features are:

• It is a keyword-driven framework.
• You can write test scenarios in the Gherkin format.
• It produces both HTML and XML reports.
• You can tag tests to categorize them.
• Robot Framework has standard and external libraries. Standard libraries come along with the installation. Whereas, you can install the external libraries which are required for the project.
• It has plugins for all popular IDEs like (Eclipse, Intellij, PyCharm, SubLime, TextMate, Vim, Emacs, Brackets, Atom, etc…).
• Execution can be scheduled in Jenkins.
• You can use Robot Framework for Acceptance Test-Driven Development (ATDD).
• Desktop App test cases can be automated using WhiteLibrary.
• You can also automate JavaSwing GUI.
• Video recording and screenshots can be captured during execution.
• It supports IBM Mainframe automation testing.

Robot Framework is very vast, and so you will be able to automate any application or API. So even if you don’t find any library for your requirements, you won’t be in trouble as Robot Framework lets you write your own library.

Gauge

Gauge is an open-source test automation framework developed by the Thoughtworks team that eases automated acceptance tests. You can write the test scenarios in the .spec or .md file format. The implementation codes can be written in JavaScript, TypeScript, Java, C#, & Python languages. Gauge allows you to focus only on tests instead of spending time on utilities & keywords development.

If you want to share data from one test scenario to another, you can use the Gauge data store factory. There are three types of data stores (ScenarioStore, SpecStore, & SuiteStore).

If a test execution has any failures, you will be able to rerun only the failed scripts using the below command.

gauge run --failed

You can run these specs in parallel, but you must be able to write the thread-safe test code. Gauge is a simple and flexible test automation framework that avoids complex test code layers.

Atata

Atata is an open-source C# based Web App Automation Testing framework. It provides a rich set of libraries to automate web-related test cases and uses Page Object Pattern to define the UI elements & actions to be performed on web pages. Atata is supported in all major C# based unit & BDD testing frameworks (NUnit, Xunit, MSTest, & Specflow). Atata even integrates ExtentReports for test report generation.

Carina

Carina is a Java-based open-source automation testing framework that can be used to automate Web Apps, Mobile Apps, REST Services, Windows apps, and even DB-related test cases. Carina has one very unique and resourceful feature for mobile app automation testing. It allows you to have common automated snippets for both iOS and Android platforms.

WebDriver.IO

WebDriver.io uses the Selenium, Appium, and DevTools protocol to automate Web apps, mobile apps, and Electron-based Desktop Apps. The DevTools capability was introduced in Selenium 4. However, the WebDriver.io framework has already implemented DevTools without relying on the WebDriver protocol.

Some notable features of WebDriver.io are:

• It is suitable for automating all modern web, mobile, and desktop apps.
• It supports all major reporting libraries (Allure, Concise, Dot, Junit, Spec, Sumologic, etc).
• You can embed video recording in test reports.
• Execution can be triggered based on GitHub actions.
• WebDriver.io has built-in commands. You can also add custom methods if you need any.
• You can schedule executions in Jenkins and Bamboo CI tools.
• You can also migrate the existing Protractor automated scripts to WebDriver.io.

Citrus

Citrus is an open-source test automation framework best suited for integration testing. You can send, receive messages, and also specify control messages for validation. Citrus allows you to test complex messaging scenarios with a step-by-step sequence of operations.

Conclusion

You need experts such as us by your side to provide the best automation testing services. Yes, choosing the right framework is a critical choice. But your team should be capable to create a custom test automation framework, explore, and even suggest the ready-to-use framework for a project.

Measuring the Value of Automation Testing : How to Get it Right?

by admin | Oct 11, 2021 | Automation Testing, Blog, Latest Post | 0 comments

DevOps and Agile primarily focus on how to add value to both the product and the business, and test automation is considered an important activity in the delivery pipeline. In the past, test automation metrics and ROI were used to determine the benefits of automation testing. Let’s say an automated test suite runs perfectly for 6 months without any changes and refer to the metrics from the executions, it will show positive outcomes. But the underlying problem here is that the automated test scripts which are untouched and not improved don’t provide any value at all. So in this blog, we will be going through how to effectively measure the value of automation testing.

Measuring the value of test automation in DevOps & Agile is totally different. Let’s take a look at some of the more traditional value measuring techniques and explore why they are no longer effective.

Total Duration

Let’s start off with one of the most basic metrics, the time is taken to perform the tests using automated test suites. In most cases, automation test execution will be faster than manual test execution. But isn’t it more important to see what is being tested instead of how quickly it is tested in the initial phase? We understand that providing quick QA feedbacks to a team is a paramount task, and are not denying it. You can employ multiple Docker containers and run the scripts in parallel to achieve speed. But make sure you don’t compare the execution time between manual and automation testing for each test case. It does not help you to measure the value of your automation tests.

Percentage of Pass & Fail

False Positives & Negatives is one of the major productivity killers for automation testing that requires you to script well to avoid them. Just because all your tests have passed, you cannot assume that your scripts are doing well. You need to ensure that your automated test suites are constantly improved by adding new test scenarios, updating the existing scenarios when functionalities change, and also fixing the false positives & negatives. There is no doubt that the percentage of pass & fail is a good metric, but you need additional supporting data to measure the value of your automation testing more precisely.

In DevOps, you need to improve your automated test suites continuously. Only stable test scripts are qualified to fit in the delivery pipeline. As a leading QA Company, we use automated testing reporting dashboards to view all the collected metrics to measure their value. So based on our vast experience, we recommend you collect the below metrics to measure the value of automation testing.

Scalability

• Keep track of the number of scripts that are up and running in the pipeline.
• Collect data on the number of scripts that are getting added.
• Monitor how many scripts are updated & removed.
• Check how many new test cases are identified for automation testing from your Exploratory Testing Sessions.

Stability

• We know that unstable scripts have to be quarantined, so find out how many scripts have been quarantined. Make sure that you bring them into the delivery pipeline only after stabilizing them.
• Keep track of the frequently failing tests.
• Your tests don’t just have to work, they have to be fast as well. So, collect data on the slowest tests as this metric will expose if any page is loading slowly or if it has any scripting issues.
• Monitor common exception messages. Since Selenium and Appium have an exception list, you can improve your automated tests by fixing the common exceptions.
• Track the Success Rate when you retry. Sometimes, running the failures again might get them to pass. But if you notice any script which is passing only after retry, it needs your attention.

Number of defects and other issues

• Keep track of the number of defects you find in testing
• Monitor the number of script issues
• Check the number of issues in the test data
• Collect data on the number of infrastructure-related issues

Conclusion

Being the best company for automation testing, we at Codoid have switched our test automation value measuring strategy from the traditional approach. We have seen great results and you will also be to witness the same when you try our approach. You will be able to capture the above-mentioned metrics during test execution, and from the framework & the test management tool to measure test automation value.