by admin | Oct 11, 2021 | Automation Testing, Blog, Latest Post |
DevOps and Agile primarily focus on how to add value to both the product and the business, and test automation is considered an important activity in the delivery pipeline. In the past, test automation metrics and ROI were used to determine the benefits of automation testing. Let’s say an automated test suite runs perfectly for 6 months without any changes and refer to the metrics from the executions, it will show positive outcomes. But the underlying problem here is that the automated test scripts which are untouched and not improved don’t provide any value at all. So in this blog, we will be going through how to effectively measure the value of automation testing.
Measuring the value of test automation in DevOps & Agile is totally different. Let’s take a look at some of the more traditional value measuring techniques and explore why they are no longer effective.
Total Duration
Let’s start off with one of the most basic metrics, the time is taken to perform the tests using automated test suites. In most cases, automation test execution will be faster than manual test execution. But isn’t it more important to see what is being tested instead of how quickly it is tested in the initial phase? We understand that providing quick QA feedbacks to a team is a paramount task, and are not denying it. You can employ multiple Docker containers and run the scripts in parallel to achieve speed. But make sure you don’t compare the execution time between manual and automation testing for each test case. It does not help you to measure the value of your automation tests.
Percentage of Pass & Fail
False Positives & Negatives is one of the major productivity killers for automation testing that requires you to script well to avoid them. Just because all your tests have passed, you cannot assume that your scripts are doing well. You need to ensure that your automated test suites are constantly improved by adding new test scenarios, updating the existing scenarios when functionalities change, and also fixing the false positives & negatives. There is no doubt that the percentage of pass & fail is a good metric, but you need additional supporting data to measure the value of your automation testing more precisely.
In DevOps, you need to improve your automated test suites continuously. Only stable test scripts are qualified to fit in the delivery pipeline. As a leading QA Company, we use automated testing reporting dashboards to view all the collected metrics to measure their value. So based on our vast experience, we recommend you collect the below metrics to measure the value of automation testing.
Scalability
- Keep track of the number of scripts that are up and running in the pipeline.
- Collect data on the number of scripts that are getting added.
- Monitor how many scripts are updated & removed.
- Check how many new test cases are identified for automation testing from your Exploratory Testing Sessions.
Stability
- We know that unstable scripts have to be quarantined, so find out how many scripts have been quarantined. Make sure that you bring them into the delivery pipeline only after stabilizing them.
- Keep track of the frequently failing tests.
- Your tests don’t just have to work, they have to be fast as well. So, collect data on the slowest tests as this metric will expose if any page is loading slowly or if it has any scripting issues.
- Monitor common exception messages. Since Selenium and Appium have an exception list, you can improve your automated tests by fixing the common exceptions.
- Track the Success Rate when you retry. Sometimes, running the failures again might get them to pass. But if you notice any script which is passing only after retry, it needs your attention.
Number of defects and other issues
- Keep track of the number of defects you find in testing
- Monitor the number of script issues
- Check the number of issues in the test data
- Collect data on the number of infrastructure-related issues
Conclusion
Being the best company for automation testing, we at Codoid have switched our test automation value measuring strategy from the traditional approach. We have seen great results and you will also be to witness the same when you try our approach. You will be able to capture the above-mentioned metrics during test execution, and from the framework & the test management tool to measure test automation value.
by admin | Oct 8, 2021 | Selenium Testing, Blog, Latest Post |
Have you ever noticed the inconsistencies between different Selenium browser drivers before Selenium’s W3C recommendation? If so, you would have a clear idea of what we are going to explore in this blog. But, if you are a novice automation tester, then the chances of you experiencing that are very low. So to clear things up, before Selenium WebDriver’s W3C recommendation, it was tough to bring uniform implementation. You will get a better picture once we go through an example.
Let’s say you want to run your scripts on Chrome, IE, Safari, and Firefox. Executing the scripts on IE and Safari browsers was a challenge as the IEDriver and Safari Driver are maintained by Microsoft & Apple. This meant that if any changes were needed in these drivers, then it should be done only by the browser vendors. So the issues would be raised and fixed only after the Selenium contributors notify the browser vendors.
Most of the modern browsers are W3C compliant. This means that HTML, CSS, and JavaScript should be interpreted as recommended by W3C across all the browsers. So there should not be any case where a particular HTML tag is rendered differently in a browser for all the websites.
Selenium WebDriver is a browser controlling library that is used globally and it is also supported by the major browsers. That is why Selenium WebDriver becomes a valid candidate to become W3C compliant. So the browser vendors adhere to the WebDriver’s W3C recommendations when developing the browser drivers.
The W3C
If you are not that aware of W3C, then you need to know what W3C is and what it does before we could proceed any further. Tim Berners-Lee and Michael Dertouzos met in Zurich and discussed forming the W3C, and eventually formed the World Wide Web Consortium (W3C) in October 1994.
- W3C’s Mission – Web for all, and Web on Everything.
- W3C’s Vision – Web for Rich Interaction, Web of Data and Services, & Web of Trust
W3C Browser Testing and Tools Working Group
W3C defines the international web standards and helps to make sure that the web is accessible from any hardware, browser, and geo-location. There are 42 open working groups to get the job done, and Browser Testing and Tools Working Group is one among the 42. The mission of the group is to produce technologies that can be used in testing, debugging, and troubleshooting Web applications running in Web browsers. So naturally, this is also the group that takes care of the Selenium WebDriver’s W3C standards.
A working group is approved and formed once the recommendations, sample code, and technical reports are submitted. WebDriver’s W3C recommendation was submitted on 5th June 2018 to inform the world to follow the recommended standards when developing Selenium drivers.
There are 38 participants from 9 organizations in the Browser Testing and Tools Working Group.
S. No |
Name |
Company |
1 |
David Burns |
W3C Invited Experts |
2 |
Michael[Tm] Smith |
W3C |
3 |
Patrick Angle |
Apple, Inc. |
4 |
Christian Bromann |
Sauce Labs |
5 |
Brian Burg |
Apple, Inc. |
6 |
Rick Byers |
Google LLC |
7 |
John Chen |
Google LLC |
8 |
Karl Dubost |
Mozilla Foundation |
9 |
Jim Evans |
W3C Invited Experts |
10 |
Jim Evans |
Salesforce |
11 |
Titus Fortner |
Sauce Labs |
12 |
Maja Frydrychowicz |
Mozilla Foundation |
13 |
Shuotao Gao |
Google LLC |
14 |
Zoher Ghadyali |
Microsoft Corporation |
15 |
James Graham |
James Graham |
16 |
Peter Hedenskog |
Wikimedia Foundation |
17 |
Philip Jägenstedt |
Google LLC |
18 |
John Jansen |
Microsoft Corporation |
19 |
Wilhelm Joys Andersen |
W3C Invited Experts |
20 |
Joon Lee |
Google LLC |
21 |
Jason Leyba |
Google LLC |
22 |
Shengfa Lin |
Google LLC |
23 |
Karin Lundberg |
Google LLC |
24 |
Clayton Martin |
Salesforce |
25 |
Marcus Merrell |
Sauce Labs |
26 |
Diego Molina |
Sauce Labs |
27 |
Theresa O’Connor |
Apple, Inc. |
28 |
Jan Odvarko |
Mozilla Foundation |
29 |
Michael Pennisi |
Bocoup |
30 |
Devin Rousso |
Apple, Inc. |
31 |
Maksim Sadym |
Google LLC |
32 |
David Singer |
Apple, Inc. |
33 |
Henrik Skupin |
Mozilla Foundation |
34 |
Sam Sneddon |
Apple, Inc. |
35 |
Andy Sterland |
Microsoft Corporation |
36 |
Simon Stewart |
Apple, Inc. |
37 |
Brandon Walderman |
Microsoft Corporation |
38 |
Luke Zielinski |
Google LLC |
Browser Drivers
The below-listed browser drivers are W3C compliant.
- Mozilla Firefox
- Microsoft Edge
- Apple Safari
-
- WebKit GTK Port
- Selenium IEDriverServer
- Chrome
If any browser produces the driver for Selenium WebDriver, it needs to follow the W3C specifications.
Conclusion
There is speculation that only Selenium 4 is W3C compliant, but that is incorrect. The older version of the drivers supports W3C, and even if any of the old drivers don’t support W3C, it will use the JSON wire protocol. We, as one of the automated software testing companies, have faced many driver-related issues before the W3C WebDriver protocol. After embracing the uniformity, the Selenium Browser Drivers have now been stabilized. So there are no longer any issues where an automated test suite runs fine on one browser and fails to do so on another.
by admin | Oct 7, 2021 | Software Testing, Blog, Latest Post |
Since software testing is an integral part of the software development lifecycle, the need for talented QAs is constantly on the rise. There are also many out there who aspire to become a software tester by setting them apart from the overwhelming competition that has a bachelor’s degree in computer science. Certifications are a great way to accomplish just that, but there is a catch as not all certifications will be worth the effort and resources. So in this blog, we will be taking a look at the best QA testing certifications that will help a QA either kickstart their career or move higher in their career.
ISTQB – Foundation Level
The International Software Testing Qualifications Board is one of the most renowned and globally acclaimed QA testing certifications in the industry. The primary reason we are starting with this certification is that it can help you get a perfect start to your software testing career. It will help you understand all the fundamentals of software testing that are needed to fit into the various software testing roles such as test automation engineer, mobile-app tester, user-acceptance tester, and so on. The syllabi that they follow are constantly improved to keep up with the growing business needs. But the primary advantage that their syllabi have is that they are open to the public and you can go through their material before deciding to attend the exam that will cost you $229.
The foundation level certification isn’t the only one offered by ISTQB and we will be exploring the rest as we progress.
CAST – Certified Associate In Software Testing
CAST is yet another apt choice for the ones looking to start their software testing career. It is a foundation-level QA testing certification like the ISTQB certification we saw earlier and it will help you learn the important principles and best practices you would need to know. Let’s take a look at the prerequisites for this certification,
- Have a 3 or 4-year degree from an accredited college-level institution.
- If you have a 2-year degree at the same level, then a year of experience in testing will make you eligible.
- You could do the certification without a relevant degree if you have at least 3 years of experience in testing.
So it is evident here that you have the option to get this high-value QA testing certification just with your degree. So like ISTQB, it is a great option for freshers, and also a viable option for people with experience to certify themselves. CAST would cost you $100 to attempt and you should make sure that you attend the examination within 1 year of payment and get 70% to clear it.
CSTE — Certified Software Test Engineer
Now that we have seen two entry-level QA testing certifications to get started with, let’s move on to the next level. CSTE is more of a professional-level certification that helps you establish your competence and achieve better visibility in your team. So you will be able to present yourself as someone who can take up some additional responsibility and take a step forward in their career. The prerequisites also change accordingly,
- Must have a 4-year degree from an accredited college-level institution and 2 years of experience in testing.
So if you have a 3-year degree, you will need 3 years of experience in testing.
- Similarly, if you have a 2-year degree, then you will need 4 years of experience in testing.
- If you don’t have an appropriate degree, then you could compensate for that with 6 years of experience in testing.
Similar to CAST, you would have to score 70% to pass the test and it would cost you around $350 to $420.
CTM — Certified Test Manager
If CSTE was the first step towards a management role, CTM is the one that helps you get better at it. As the name suggests, this QA testing certification is targeted towards improving your management skills. It is a great option if you are currently in a leadership role as it can help you better your skills. You will be learning about various skills like how to manage test processes & test projects, how to measure a test process so that you can work on improving it, risk management, and so much more. Since it is management-level certification, you would also require experience in any leadership role.
The prerequisite is that you should have 3 years of experience in testing that includes 1 year of experience in a leadership role that is relevant to software testing. This has to be met with by the time the CTM will be granted. You would require a lot of supporting documents for this one. So make sure to check over the complete procedure on their website. Make sure you familiarise yourself with the Test Management Body of Knowledge book as it is the key to this certification.
CSSBB — Six Sigma Black Belt Certification
Another option to sharpen up your management skills is the National Commission for Certifying Agencies (NCCA) certified CSSBB. Like ISTQB, there are other belts available as well, but the black belt certification helps you understand the team dynamics and assign roles & responsibilities accordingly. You will be able to use the Six Sigma principles to optimize the DMAIC (Design, Measure, Analyze, Improve, and Control) model. In addition to that, you will also be equipped with the basics of lean management and how to identify non-value elements. The prerequisites are,
- You would require a minimum of 3 years of experience in one or more areas of the CSSBB Body of Knowledge.
- You would require a minimum of 1 or 2 completed projects with signed affidavits.
The exam would cost you $538 and would be an open-book test. If you are an ASQ member, then you would get a discount of $100 in the cost.
ISTQB – Advanced & Expert Level
The Advanced level of ISTQB is on par with the CTM certification that we saw earlier and the expert level is the one with the highest recognition. So starting with the foundation level certification, you should work yourself up the ladder and complete the advanced level first and then follow it up with the expert level. The expert level is the epitome of all these QA testing certifications as in addition to the in-depth concepts you will be learning about, you will also be handling practical exercises and discussions for almost half of the entire time. The prerequisites would also be very high for this certification.
- One must have at least 5 years of practical testing experience in testing.
- In that 5 years, at least 2 years should be industry experience in the specific topic that you are looking to become an expert in.
Conclusion
As one of the top software testing companies in Chennai, we always concentrate on getting our employees certified with ISTQB. We are currently a Silver-level partner with ISTQB and on a path to become a Platinum-level partner in the near future. We are aware of how valuable these ISTQB certifications are, and that is why we have started and ended the list with them. So we hope you have found this list of our top QA testing certifications to be useful. Make sure to pick the courses best for you and also remember that even though certifications are important, it is the knowledge you build that matters. So pour your heart and soul into it and be on a path of continuous learning.
by admin | Oct 6, 2021 | Manual Testing, Blog, Latest Post |
If you have a basic understanding of what Software Testing is, you will know that it can be done either by using automation or by manual efforts. Testing is commonly misunderstood and assumed to be just checking. But in reality, testing is so much more than what it seems on paper. So be it manual testing or automation testing, one should be equipped with the appropriate skills and should use the best practices to achieve the goal. With the growth of automation in the domain, it is very common to feel that manual testing is in the past and that it is no longer necessary. As a leading manual QA testing company, we understand that manual testing is still a vital part of software testing. So in this manual testing guide, we will be going through the basics you have to know to get started with manual testing.
An Introduction to Manual Testing
The ultimate goal of software testing is to verify the actual behavior of the software against a predefined set of expectations. People always tend to look at manual testing as an option only when they are unable to automate the scenario. Though automation plays an integral part in helping us meet the software needs of today, it shouldn’t always be the first choice for all scenarios. We have covered this exact same topic in another one of our blogs, so make sure to read that to find out how to strike the perfect balance between manual testing and automation testing.
To give you a gist of it, manual testing is usually preferred to perform exploratory testing and usability testing. The reason here is that manual testing helps us get deeper insights about the product from the end user’s perspective. This is a vital aspect that we simply cannot achieve through automation.
Types of Manual Testing
We will not be exploring all the types of manual testing that there is in this manual testing guide. Instead, we would briefly take a look at the 3 basic types of testing. They are namely white box testing, black-box testing, and grey-box testing. If the QA team is aware of the internal code that is used, then it is white box testing. But if the team is unaware of the internal code and performs the tests purely based on the interaction of the software, then it is black-box testing. A combination of both of these types is what makes grey-box testing.
Stages of Manual Testing
Let’s now focus on the different stages of manual testing in the software development lifecycle. We know that the software is built from the ground up. So each block that makes the software work will be developed one by one. These blocks or modules should work together as a system to be used by the end-user. So let’s take a look at the different types of testing that will happen in all these stages.
Integration Testing
In this stage, we would integrate the multiple units that we have already tested to see how well they work with each other. If we take an e-commerce app as an example, then a good way to explain would be clubbing the ‘Add the Cart’ portion and the payment gateway. Both these modules have to work well together for a customer to complete the purchase without any hassle. So it would spell trouble for your business even if one of these modules has any issues.
We have to consider the functional aspects and non-functional aspects like performance, reliability, and stability here. So it is not just about verifying if these modules work, we have to make sure they are on par with the defined expectations in terms of performance and so on.
System Testing
As the name suggests, in this stage of testing we would be testing the system on the whole as one product. Eventually, the software will be used by end-users and so at this stage, we will be performing end-to-end testing that covers everything from functional aspects & non-functional aspects to the expectations & experience of an end-user. So there will be a lot of regression tests performed to verify that the recent changes made to the code have no adverse effects on the parts of the software that were working fine. Stress testing should also be a part as the software could be used by power users as well. By performing stress testing we can ensure that the software can handle usage that is more than expected. On the whole, all critical business requirements will be tested in this stage.
Acceptance Testing
The final leg of testing is verifying how well the software performs in real-world conditions. Acceptance testing is actually performed both internally and externally. The people who are testing the application from within the organization are the alpha testers. The external testing will be done through beta testing programs that involve having actual end-users use the product to see how well it works.
Conclusion
This manual testing guide is more of a walk-through of the basics you should know. There are so many more aspects of manual testing that you should focus on to get better at it. As one of the best manual software testing companies, we have years of experience in the software testing arena that have helped us master both manual and automation testing in an optimal manner. Once you have a better understanding of software testing, you will definitely understand the importance of manual testing as well.
by admin | Oct 5, 2021 | Software Testing, Blog, Latest Post |
Achieving Continuous Delivery is no child’s play as your team should be equipped with certain technical capabilities because deploying the code into production on demand throughout the software delivery cycle is not easy in any way. It is also important to acknowledge that it’s not just the technical practices that will help you achieve Continuous Delivery. It has to be coupled along with good management practices as well. So as a first step, we will be focusing on the key technical capabilities that drive Continuous Delivery in this blog.
Version Control
Keeping the application code in version control alone will not help you achieve Continuous Delivery as you would have to do all the deployments manually when both the system and application configurations stay out of version control. Since manual tasks stall continuous delivery (CD), you have to automate the delivery pipeline to achieve CD.
Test Automation
Automated unit & acceptance tests should be kicked off as soon as developers commit the code in version control. The simple underlying logic here is that if the developers are provided with faster feedback, they will be able to work on the changes and fix the issues instantaneously. This advantage alone doesn’t make test automation a vital component in the Continuous Delivery pipeline. It is also very reliable in ensuring that only high-quality software gets delivered to the end-users.
But that doesn’t mean you can just use flaky test automation scripts as they would only produce false positives and negatives and not add any value to the Continuous Delivery pipeline. So if you want your software to be more reliable and resilient, writing and adding robust automation scripts to the pipeline is the best way to go about it. If at all you find any of the scripts to be unstable, you can move them to a quarantine suite and run them independently until it becomes stable
Test Data Management (TDM)
As mentioned earlier, manual works delay continuous delivery. But at the same time, if your automated test scripts have to be manually fed with test data at frequent intervals, it will cause delays in the delivery. This is a clear sign that your automated test suites are not fulfilling their purpose. You can overcome this roadblock by creating a test data management (TDM) tool to which you can upload all the adequate data required for automated script execution. So, your scripts will call the TDM API to get the required data instead of getting it from an Excel or a YAML file.
In addition to that, once the TDM tool reaches its threshold, it should be able to pull and allocate more data automatically from the system DB. As an automation testing services company, we have a TDM for automation testing alone. We feed the data to it once every month and as a result, the automation test scripts no longer have to wait for the test data to be fed since the adequate data has already been allocated.
Trunk-Based Development
A long-lived branch leads your team towards reworking and merging issues that can’t be fixed immediately. So developers should make sure to create a feature branch, commit the changes within 24 hours into the trunk, and kill the feature branch. If a commit makes a build unstable, everyone on the team should jump in to resolve the issue and not push any changes into the trunk until the merge is fixed.
When a developer pushes the changes from a short-lived branch, your team will resolve the merge issues quickly as the push will be considered a small batch. Trunk-based development is a key enabler for Continuous Integration and Continuous Delivery. So, if the developers commit the changes into the trunk multiple times a day, and fix the merge issues then & there, then the trunk will be release-able.
Information Security
The information security process should not be carried out after releasing the software. Rather, it should be incorporated from the very beginning. Since fixing security-related issues is a time-consuming effort, integrating information security-related practices into the daily work plan of your teams will be instrumental in achieving Continuous Delivery.
Conclusion
Technical and Management practices are crucial for enabling Continuous Delivery. As a software testing company, we use these technical practices with our development team that works on developing automation testing tools. Following such practices alone is not a game-changer. Your team needs to learn and improve the process continuously. Continuous Improvement is pivotal in building a solid development pipeline for Continuous Delivery.
by Anika Chakraborty | Oct 4, 2021 | Software Testing, Blog, Latest Post |
Some still believe that the need for manual testing can be completely eliminated by automating all the test cases. However, in reality, you simply can’t eradicate manual testing as it plays an extremely important role in covering some edge cases from the user’s perspective. Since it is apparent that there is a need for both automation and manual testing, it is up to us to choose between the two and strike a perfect balance. So let’s pit Automation Testing vs Manual Testing and explore how it is possible to balance them both in the software delivery life cycle.
Stable Automated Tests
Stable test automation scripts help your team provide quick feedback at regular intervals. Automated tests boost your team’s confidence levels and help certify if the build is release-able or not. If you have a good talent pool to create robust automated scripts, then your testers can concentrate on other problem-solving activities (Exploratory & Usability Testing) which add value to the software.
Quarantined Automation Test Suites
Flaky automated tests kill the productivity of QA testers with their false positives and negatives. But you can’t just ignore the unstable automated test scripts as your team would have invested a lot of time & effort in developing the scripts. The workaround here is to quarantine the unstable automated test scripts instead of executing both the stable and unstable scripts in the delivery pipeline. So stabilizing & moving the quarantined test scripts to the delivery pipeline is also a problem-solving activity. As a leading automation testing company, we strongly recommend avoiding flaky automation test scripts from the very beginning. You can make that possible by following the best practices to create robust automated test scripts.
Tips to stabilize the unstable scripts
- Trouble-shoot and fix the script failures immediately instead of delaying them.
- Train the team to easily identify the web elements and mobile elements.
- Avoid boilerplate codes.
- Implement design patterns to minimize script maintenance.
- Execute the scripts as much as possible to understand the repetitive failures.
- Run batch by batch instead of running everything in one test suite.
- Make the automated test report visible to everyone in the team.
- If your team is not able to understand the test results, then you need to make the reports more readable.
Exploratory Testing
Test automation frees testers from repetitive scripted testing. Scripted automated testing does enable fast feedback loops, but your team will be able to unearth new product features & bugs, and identity additional test scenarios when performing exploratory testing.
Benefits of Exploratory Testing (ET)
- Scripted testing provides confidence, but ET helps your team to understand whether the new features make sense from the end user’s standpoint.
- ET is usually performed by testers who have good product knowledge. Every feedback from ET is also an input for your automated test suites.
- Some test cases cannot be automated. So you can allocate those test cases in ET sessions.
- ET helps you to create additional tests which are not covered in automated test suites.
Usability Testing
Usability testing is an experiment to identify and rectify usability issues that exist in the product. Let’s say a functional tester is testing the product check-out scenario of an e-commerce app. Here, the tester has good knowledge about the product and knows where the important buttons (Add to Cart & Proceed to Checkout) are placed on the app. However, in real-time, if the end-users struggle to find these buttons, they will definitely have second thoughts about using your app again. So it is clearly evident that functional UI tests can’t help you identify usability issues.
Usability Testing Steps
1. Prepare the research questions and test objectives
2. Identity participants who represent the end-users
3. Setup an actual work environment
4. Interview the participants about the app experience
5. Prepare test reports with recommendations
Conclusion
We hope you’re able to understand that using any one of the two types of testing will only lead to the creation of surface-level solutions. For robust solutions, you should use both manual and automation testing to test the features which are released continuously in the delivery pipeline. So compare Automation Testing vs Manual Testing and use the one that is more apt for the requirement. Testers should be performing exploratory testing continuously against the latest builds that come out of Continuous Integration. Whereas, scripted testing should be taken care of by automation.