by admin | Jun 30, 2021 | Agile Testing, Blog, Latest Post |
Testing is essential to produce quality software. Agile approaches aim to integrate quality assurance into product development from the ground level up by having techies heavily involved with testing right from the initial stages. The idea behind this process is that if problems can be identified and corrected earlier in the design process, then the companies will be able to create defect-free products at quicker rates. Most of the agile teams are flying blind without having any proper strategies or measurements to follow. But as one of the best software testing service providers, we have been able to put the agile approach to good effect. So in this Agile Testing Best Practices blog, we will be sharing some important aspects that will help improve your performance in agile testing.
In addition to that, we will also be focusing on the inherent challenges and the many advantages that the agile method brings to the table.
How Agile testing is different from traditional testing:
Agile testing is a core part of the agile software development process. It is also very different from the conventional testing methods. Traditional testing is done in a phased manner so that the end-product doesn’t get released until all the defects have been fixed. But on the contrary, agile testing follows an iterative approach. So defects are fixed in each sprint and released right away. Testers define the success or failure of the team’s attempts to be agile as they are a part of the delivery team, making them an integral part of any agile team. So the adaptability of testers in SDLC is very important in the agile approach.
Five things make agile testing different from traditional testing, and they are
- Continuous Involvement
- Essential Tools
- Multidimensional Skills
- Effective Communication
- Quick Feedback from Testing
Agile Testing life cycle:
1. User Story Analysis,
It is important to understand the release plan and plan the user stories across the sprints in a release.
2. Test plan / Estimate,
It gives the ability to have all the high-level scenarios, requirements, and estimates in one plan. It is created for each release, and it should also be updated for every release.
3. Sprint Planning,
The purpose of sprint planning is to define what should be delivered in a sprint and how that can that be made possible. The team assembles together to achieve the required velocity of the sprint.
4. Environment Setup,
Agile is a process that requires consistent teamwork and the work environment plays a huge factor in encouraging the team to work towards a common goal. So nurturing such an environment is one of the vital agile testing best practices that can make a difference. This can be accomplished by valuing each individual & their ideas and by healthy collaboration as the process demands frequent changes.
5. Implementation and Execution,
The creation of test cases across the assigned user stories and peer review is performed in this stage. For better quality deliverables, developers can take a look at these test cases and ensure that the overall scenarios have been enlisted. The distribution of various tests based on risk analysis is vital. So test cases must comprise of all test types like Functional, UI, Usability, cross-browser, etc. So once the test cases have been created, we have to ensure if the stories are ready for testing and execute the created test cases.
6. Defect Reporting,
The ultimate point of any form of software testing is to identify the bugs and report them so that they can be fixed. So it is important to devise an efficient method to report all the encountered bugs in an effective bug management tool. Healthy real-time collaboration with the developers can be achieved only when the defect log contains information like summary, description, priority, date of identification, steps to reproduce, name of the tester, unique identifier, etc. Such an in-depth report will enable us to conduct a triage meeting to plan and fix the bugs that have been logged.
7. Release Activity,
Once we are past all the above stages, the build can be deployed for UAT where testing can be performed with a few users. If the build clears the smoke tests that will be performed by the alpha testers, then the release can be given to the beta testers who will test the build before signing it off from their end and ship it to production.
8. Getting ready for the next release,
After the release, the team should prepare for their next release. Any and all business clarifications or issues should be raised and cleared during this phase.
The Quadrants for Agile Testing Best Practices
Agile testing can be simplified by using a system of quadrants that provides necessary classifications that determine which type of test should be run, how often it should be run when it should be run, and who it should be run by. Since there are so many types of testing like acceptance testing, regression testing, unit testing, and so on. Quadrants help reduce the difficulty in deciding which test has to be used where. It will also answer the question of whether manual or automated testing is better suited for the current iteration of the product.
So, now let’s find out how quadrants work their magic. Quadrants divide the whole testing methodology into four grids. These 4 grids help the whole team to communicate and deliver the quality product on time.
Quadrant Q1 – The tests in the first quadrant are unit level, technology facing, and also supportive to the developers. Unit tests and other component level tests that test the code belong to this quadrant. In general, the tests in this quadrant are automated.
Quadrant Q2 – The second quadrant consists of tests that are system level, business-facing, and help conform product behavior. Functional tests belong to this quadrant. These tests can either be manual or automated.
Quadrant Q3 – The tests in this quadrant are system or user acceptance level, business-facing, and focused on real-time scenarios. User Acceptance Tests belong to this quadrant. These tests are manual.
Quadrant Q4 – The tests in the final quadrant are system or operational acceptance level, technology facing, and focus on performance, load, stress, maintainability, and scalability. Special tools can be used for these tests, along with automation testing.
Agile Testing Best Practices to Overcome Challenges:
Let’s take a look at some of the most common challenges that agile teams will face during testing and see how we will be able to overcome them as well.
Last-minute changes:
Changing requirements is an inherited challenge of the agile methodology. This means that even if a work is already completed or half done, it might have to go through some modifications or worst-case scenario, get scrapped as well. So this will most definitely will have an unexpected change in the scope of testing.
How to master?
Testers must be ready to react to the changes or modify the processes in agile methodology as the process is itself prone to changes. Whenever there is a change in the requirements, testers should share the information of what tests they have actioned and which part of the application they have not yet tested. This can help the team understand how to make the required changes.
Not Enough Information (one-line stories):
Sometimes product owners will just have an idea of what features are required. So business owners may not even be aware of the specific requirements. In such situations, testers will not have a good set of details about the user stories at their disposal. All they might have could be just single-line information which is insufficient to build comprehensive test cases. This causes a significant amount of hindrance to the testers.
How to master?
In this situation, the testers can collect all the minutes of each requirement before they start testing and they can even come up with high-level scenarios for the stories and confirm them with the author or product owner and then approach the test cases.
Lack of Communication:
Without effective communication between developers, testers, and the product owner no process can work. So effective communication is also one of the most important agile testing best practices. Lack of communication can result in different teams flying blind with no proper goal as to what should be accomplished.
How to master?
Daily standup meetings with the team are strongly encouraged. It is also important to make sure that the meetings are not too long. Daily standups help identify roadblocks and solve them on a day-to-day basis. It also helps to keep the developers and product owners on the same page as the rest of the team.
Frequent regression testing:
As features get pushed by developers continuously, testers would have to run regression tests very frequently as the chances of the new features breaking the previous code are much higher. The other major challenge is that testers would have to test the application’s functionality correctly for all the users. Testers would also have to check the application’s behavior on different devices.
How to master?
Automation skills can be a great help for this situation in agile projects. Automating the regression test can save you a lot of time, and Selenium is the most popular browser automation tool that might solve all your needs. Apart from that, testers can use tools like Docker or VisGrid to run their automated test scripts in parallel across various browsers and devices.
Technical Skills:
As software becomes more mature, the complexity increases along with it as well. Testers should be in a position to help developers with API testing, integration testing, and more. So having strong technical skills is one of the basic requisites of a tester who is working in agile.
How to master?
Being in the process of continuous learning always helps people excel in their respective fields. Likewise, testers can learn programming languages like Java, Python, C#, etc., and learn automated testing tools like Selenium, JMeter, and so on. It is also crucial for the team to comprise dedicated testers with professional experience. If you are looking for the best agile testing tools, then you could read our blog and explore your options.
Conclusion:
We hope you have enjoyed reading this blog and that it has been informative as well. These agile testing best practices have helped us deliver the best software testing services to our clients, and hope it will enhance your testing methods as well. Obviously, these tips alone will not equip you to face any challenge that comes your way. But these general tips will definitely help you overcome the regular challenges and also help you find the right solutions for every new challenge you would face.
by admin | Jun 21, 2021 | Automation Testing, Blog, Latest Post |
TestNG is not the only option available in the market. But most of the automation testers prefer TestNG over other options like JUnit because TestNG allows parallel execution of test cases that saves loads of time possible with the help of the ‘parallel’ keyword in the XML file, TestNG’s seamless integration with tools like Jenkins, Maven, etc., and grouping of test cases also becomes easy with the use TestNG. These features have been instrumental in us providing the best software testing services to our clients. So in this TestNG tutorial, we will be focusing on how to make use of all these advantages and also explore all the advantages that TestNG brings to the table.
TestNG is an open-source testing framework where NG stands for Next Generation. It provides many features to execute the test methods by using various annotations like @Test @BeforeTest, @BeforeClass, etc. TestNG allows its users to generate robust test results that include the number of test cases that have Run, Passed, Failed, and Skipped. It can also be used to implement the logging feature using TestNG Listeners. On the whole, TestNG can also be used across various testing types like Unit, Regression, Functional, and API Testing. But before we get started with the TestNG Tutorial, let’s see how to set it up.
Installation and Setup of TestNG in IntelliJ:
IntelliJ requires a JAR file to add dependencies for TestNG script development and for running the test suites. So if it is a Maven Project, we would have to download the TestNG Jar File. Since we will be running tests on Java, we need to download an external TestNG Jar File onto our system by using the provided link.
Direct download link: TestNG/7.4.0/testng-7.4.0.jar
Reference: org.testng/TestNG
Please refer to the below image and instructions for a step-by-step guide,
i. Open IntelliJ,
ii. Create a new JAVA Project,
iii. Navigate to ‘File’ >> ‘Project Structure’ in IntelliJ,
iv. Click on ‘Libraries’ from Project Setting,
v. Add it to your ‘TestNG’ jar file, and then click on the ‘Apply and OK’ button.
If we want to work on a maven project, we have to mention the dependency in the POM.xml file
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>7.4.0</version>
<scope>test</scope>
</dependency>
TestNG Annotations:
TestNG annotations that are used to execute the test scripts can be defined by ‘@Test’ annotations. Before we head over to other aspects of the TestNG Tutorial, let’s take a look at the various annotations that are available in TestNG along with a short description of them.
Types of TestNG Annotations:
@Test Annotations:
We use this annotation to mark a method as a test method. The methods marked with @Test annotation are automatically invoked by the TestNG engine in default order (i.e.) alphabetically.
Syntax:
@Test
public void TestMethod1() {
System.out.println("\n Test-Method-001");
}
So in this part of the TestNG Tutorial, we will be focusing on the most important attributes of @Test annotations, starting with the description
1. ‘Description’ attribute of @Test Annotation:
The ‘Description’ keyword is used to provide a description of the test case. It can also be termed as a one-line summary, and it has been illustrated in the below example that describes the action of the user in a single line.
@Test(description = "User visits Codoid Home page")
public void launchCodoidURL(){
System.out.println("\n >>> User visits Codoid Home page...");
}
2. Test Methods with Priority:
As mentioned earlier, we are aware that the methods will execute multiple test annotations alphabetically by default. But what if there is a need to run the methods or the test annotations based on defined priorities? Here is where the ‘Priority’ attribute comes into play, as it can be used to assign priorities to each test method. The highest priority would be given when a test method is marked with 1 and the priority decreases inversely as the number increases.
We have mentioned syntax below to denote how the priority attribute would function.
Syntax:
@Test(priority = 1)
public void Test1() {
System.out.println("\n Test-1");
}
@Test(priority = 2)
public void Test2() {
System.out.println("\n Test-2");
}
3. Test Methods with Priority & without priority:
So in the above-mentioned syntax code, we saw 2 sets of programs that had defined priorities. But what if there is a need to define priorities only for a certain number of methods? It is very easy to accomplish as you can simply leave the methods that don’t need defined priorities and mark the other methods alone in the order that you wish. It is worth noting that the methods that had no defined priority will be run first on an alphabetical basis and then followed by the defined order.
Syntax:
@Test(priority = 1) //with Priority
public void Test1() {
System.out.println("\n Test-1");
}
@Test //without priority
public void TestMethod1() {
System.out.println("\n Test-Method-001");
}
@Test //without priority
public void TestMethod2() {
System.out.println("\n Test-Method-001");
}
@Test(priority = 2) //with Priority
public void Test2() {
System.out.println("\n Test-2");
}
Code Explanation:
– Non-Prioritized methods ‘TestMethod1() and TestMethod2()’ have been executed in alphabetical order. (Line No: 60 and 65)
– Prioritized methods ‘Test1() and Test2()’ have been executed based on the priority level. (Line No: 55 and 69)
4. Depends On Methods in TestNG:
In our test scripts, we might have some methods that depend on the execution of another method before it is executed. For example, if a user wishes to send an email using the email application, then the user has to be logged in to the application first. So there will be no point in trying to send the mail if the sign-in is incomplete. So we can make use of this attribute that will help methods from getting executed if their dependent method doesn’t get a ‘Pass’. The same scenario is shown below as an example.
Syntax:
@Test
public void UserLoginIntoEmailApplication(){
System.out.println("\n >> User Login :: ");
}
@Test(dependsOnMethods = "UserLoginIntoEmailApplication")
public void UserSendAnNewEmail(){
System.out.println(">> User send an eMail");
}
Sometimes, there might be scenarios where a test case depends on more than one test method. So we can pass the parameters in annotations as shown below.
Syntax for dependsOnMethods: (Multiple Dependent Test methods)
@Test(dependsOnMethods = { "UserLoginIntoEmailApplication", "UserSendAnNewEmail" })
public void SignOut() {
System.out.println(">>> Sign Out");
}
5. TestNG Groups:
The grouping concept is one of the best features of TestNG and one of the important parts of this TestNG Tutorial. So using this concept we can create a group of test methods based on modules/test cases. For example, the client requirement might demand you to execute a large number of smoke or regression tests. So you can simply pass the parameter in group attributes as shown below.
Syntax:
@Test(groups = { "Regression" })
public void userLogin() {
....
}
@Test(groups = { "Regression" })
public void userSendEmail() {
....
}
So the groups are specified in the ‘.xml’ file and can either be used with the “groups” tag name or under the ‘test’ tag.
Syntax:
<groups>
<run>
<include name = "Regression"></include>
</run>
</groups>
DataProvider in TestNG:
In this method, a user will be able to pass the test data in test methods and also iterate the test methods using different data from the @DataProvider annotation, and it will return the 2D array objects.
Let’s see a simple example for this annotation,
Code:
@DataProvider(name = "data-provider")
public Object[][] dataProviderMethod() {
return new Object[][]{{"Codoid-001"}, {"Codoid-002"}};
}
@Test(dataProvider = "data-provider")
public void testDataProviderConcept(String strData) {
System.out.println("\n >> Data: " + strData);
}
Code Explanation:
We have maintained the array object and Test methods in the same class. In this method, the “testDataProviderConcept(String strData)” will iterate based on the array size. It will also retrieve the values “Codoid-001 and Codoid-002” from the ‘dataProviderMethod()’ with the help of the @DataProvider annotation.
Multiple Parameters in TestNG DataProvider:
It is even possible to pass multiple data through a single parameter. In this test method, we will be able to retrieve the data using multiple arguments as shown in the example,
For example,
@DataProvider(name = "data-provider")
public Object[][] dataProviderMethod(){
return new Object[][] {{123,456 ,789}, {001, 002, 003}};
}
@Test(dataProvider = "data-provider")
public void testMethod(int value1, int value2, int value3) {
System.out.println("\n >> value1: "+value1+"; Value2: "+value2+"; Value3: "+value3);
}
DataProvider In TestNG Using Inheritance concept:
If you are looking for a way to maintain the code based on the project structure, then the DataProvider methods can maintain one class and the test methods can maintain another class. Let’s take a look at the example to find out how.
For example, we can access the dataprovider class using another attribute like “dataProviderClass”
Syntax :
@Test(dataProvider = “data-provider”, dataProviderClass = DP.class)
Note:
i. dataProviderClass // attribute Name
ii. DP.class // class Name; To inherit the other class.
Code:
//Class-1
@Test(dataProvider = "data-provider", dataProviderClass = DataProviderUtil.class)
public void testDataProviderConcept(String strData) {
System.out.println("\n >> Data: " + strData);
}
//Class-2 "Class Name: DataProviderUtil"
@DataProvider(name = "data-provider")
public Object[][] dataProviderMethod() {
return new Object[][]{{"Codoid-001"}, {"Codoid-002"}};
TestNG Tutorial for using Excel Workbook as a Data Provider:
Now that we have seen how to pass the parameter with single or multiple data through an array object, but what if there is a need that demands a large number of data be used? Adding everything to the code will not be an optimal choice at all. We have a solution to overcome this in our TestNG Tutorial and so let’s focus on how we can retrieve data from an Excel workbook. Knowing this will come in very handy while executing Functional tests, and when there is a need to create a large amount of test data for performing Load tests.
We are going to implement the Excel workbook in the TestNG-DataProvider annotation to fetch the data from the excel sheet and search the products in Amazon. In the example, we have listed a few mobile phones like iPhone 12, Pixel 4a, and so on in the excel sheet.
The below-mentioned Excel util class is used to retrieve the data from the Excel sheet.
ExcelUtil.JAVA
Code:
/*Description: We can retrieve the data from the Excel workbook
* parameter-1: Excel workbook directory path,
* parameter-2: Sheet name*/
public static Iterator<Object[]> getTestData(String strWorkbookPath, String strWorksheetName) {
List<Object[]> data = new ArrayList<Object[]>();
int inRowCounter = 0;
try {
FileInputStream file = new FileInputStream(new File(strWorkbookPath));
//Get the workbook instance for the xlsx file.
HSSFWorkbook workbook = new HSSFWorkbook(file);
//Get the first sheet from the workbook
HSSFSheet sheet = workbook.getSheet(strWorksheetName);
//Get an iterator to all the rows in the current sheet
Iterator<Row> rowIterator = sheet.rowIterator();
Row firstRow = rowIterator.next();
Map<String, String> columnNamesMap = getColumnNames(firstRow);
while (rowIterator.hasNext()) {
Iterator<Cell> cellIterator = rowIterator.next().cellIterator();
Map<String, String> rowMap = new LinkedHashMap();
for (Entry entry : columnNamesMap.entrySet()) {
String strColumnName = entry.getKey().toString();
String strValue = cellIterator.next().toString();
rowMap.put(strColumnName, strValue);
}
}
file.close();
} catch (Exception e) {
e.printStackTrace();
}
return data.iterator();
}
private static Map<String, String> getColumnNames(Row row) {
Map<String, String> columnNamesMap = new LinkedHashMap();
Iterator<Cell> cells = row.cellIterator();
while (cells.hasNext()) {
String strColumnName = cells.next().toString();
columnNamesMap.put(strColumnName, strColumnName);
}
return columnNamesMap;
}
DataProvider.Class
@DataProvider(name = "AmazonProduct", parallel = false)
public static Iterator<Object[]> getDeviceName() {
String strWorkBookPath = new StringBuilder().append(System.getProperty("user.dir")).append(File.separator)
.append("resources\\testdata").append(File.separator).append("testdata").append(".xlsx").toString();
Iterator<Object[]> testData = ExcelUtil.getTestData(strWorkBookPath, "Mobiles");
return testData;
}
TestScripts.java
@Test(dataProvider = "AmazonProduct", dataProviderClass = DataProviderUtil.staticProviderClass.class)
public void searchAmazonProducts(Map<String, String> dataMap) {
System.setProperty("webdriver.chrome.driver", "./resources/drivers/chromedriver.exe");
driver = new ChromeDriver();
driver.get("https://www.amazon.in/");
wait = new WebDriverWait(driver, 25);
WebElement txtSearchEngine = wait.until(ExpectedConditions.visibilityOfElementLocated(
By.cssSelector("#twotabsearchtextbox")));
System.out.println("\n Devices : " + dataMap.get("Devices"));
txtSearchEngine.sendKeys(dataMap.get("Devices"));
txtSearchEngine.submit();
driver.quit();
}
Please refer to the below image,
Parameterization in TestNG:
When there is a need to perform tests by using different test data, or when you are looking to test by logging in with a different user that has distinct usernames and passwords then Parameterization will come in handy as the “@Parameters” annotation in TestNG allows argument values to pass to Test methods via the testng.xml file. (Parameter values are declared in testng.xml) We have explained the different user scenarios using an example.
Syntax:
@Parameters({ “param-name” })
Let’s write a simple program for the @Parameters annotation,
Code:
@Test
@Parameters({"Username"})
public void TestMethod1(String strUsername) {
//We can retrieve the data using this argument. (It’ll retrieve from testng.xml file)
System.out.println("\n Username : " + strUsername);
}
TestNG Tutorial for Parallel Execution:
Saving time during the testing process is always a priority, and one of the best possible ways to do it would be by executing tests in parallel rather than sequential execution. So as promised at the beginning of this TestNG Tutorial, let’s see how we can have test suites executed in parallel on different browsers, and how it can be even defined in the Testing.xml file.
TestNG provides different levels of execution that we can achieve by using the Testing.xml file. The attributes that we can use to achieve them are,
i. Methods,
ii. Classes, and
iii. Tests.
i. parallel=”methods”:
We had already seen about methods that are dependent on other methods for their execution, so accordingly if you are looking to execute many independent methods in parallel, then this attribute can be used. Using this attribute, all test methods will run on a separate thread. In the below example, we have executed two test methods in two threads.
Note: Thread count is the specified number of browser instances that should be assigned for this test execution.
Syntax:
<suite name="Test-method Suite" parallel="methods" thread-count="2" >
Code :
TestParallelForMethodsExecution.java
public class TestParallelForMethodsExecution {
@Test
public void visitCodoidPage() {
String baseUrl = "https://codoid.com/";
System.setProperty("webdriver.chrome.driver", "./resources/drivers/chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.get(baseUrl);
driver.manage().window().maximize();
WebDriverWait wait = new WebDriverWait(driver, 25);
String testTitle = "QA Testing Company - Software Testing Services";
boolean isVerified = wait.until(ExpectedConditions.titleContains(testTitle));
Assert.assertTrue(isVerified);
WebElement elmntAboutUsTab = driver.findElement(By.xpath("//ul[@id='top-menu']/li/a[text()='About Us']"));
elmntAboutUsTab.click();
driver.close();
}
@Test
public void visitAmazonPage() {
String baseUrl = "https://www.amazon.in/";
System.setProperty("webdriver.chrome.driver", "./resources/drivers/chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.get(baseUrl);
driver.manage().window().maximize();
WebDriverWait wait = new WebDriverWait(driver, 25);
String testTitle = "Online Shopping site in India";
boolean isVerified;
isVerified = wait.until(ExpectedConditions.titleContains(testTitle));
Assert.assertTrue(isVerified);
driver.close();
}
}
TestNG.xml
<suite thread-count="2" name="Parallel-Test-Suite" parallel="methods">
<test name="To Run TestNG :: Parallel methods concept">
<classes>
<class name="scripts.TestParallelForMethodsExecution"/>
</classes>
</test>
</suite>
ii. parallel=”tests”:
But the problem is that the above type will not work if we have a lot of test methods that are dependent on each other like how we saw earlier with the email example. So this attribute can be used to overcome this issue by adding all the dependent tests to a suite file (Testing.xml) and executing it separately in a thread as each tag will be considered a test suite and it’ll be executed in a separate thread.
Syntax:
<suite name="Test-Suite" parallel="tests" thread-count="2">
Code
MultipleSession1.java
public class MultipleSession1 {
public static WebDriver driver = null;
public static WebDriverWait wait = null;
@Test
public void visitAmazonPage() {
String baseUrl = "https://www.amazon.in/";
System.setProperty("webdriver.chrome.driver", "./resources/drivers/chromedriver.exe");
driver = new ChromeDriver();
driver.get(baseUrl);
driver.manage().window().maximize();
wait = new WebDriverWait(driver, 25);
String testTitle = "Online Shopping site in India";
boolean isVerified;
isVerified = wait.until(ExpectedConditions.titleContains(testTitle));
Assert.assertTrue(isVerified);
}
@BeforeMethod
public void beforeMethod() {
System.out.println("\n Starting Test On Chrome Browser");
}
@AfterTest
public void terminateBrowser() {
driver.close();
}
}
MultipleSession2.java
public class MultipleSession2 {
public static WebDriver driver = null;
public static WebDriverWait wait = null;
@Test
public void visitCodoidAboutUsPage() {
String baseUrl = "https://codoid.com/";
System.setProperty("webdriver.chrome.driver", "./resources/drivers/chromedriver.exe");
driver = new ChromeDriver();
driver.get(baseUrl);
driver.manage().window().maximize();
wait = new WebDriverWait(driver, 25);
String testTitle = "QA Testing Company - Software Testing Services";
boolean isVerified = wait.until(ExpectedConditions.titleContains(testTitle));
Assert.assertTrue(isVerified);
WebElement elmntAboutUsTab = driver.findElement(By.xpath("//ul[@id='top-menu']/li/a[text()='About Us']"));
elmntAboutUsTab.click();
}
@BeforeMethod
public void beforeMethod() {
System.out.println("\n Starting Test On Chrome Browser");
}
@AfterTest
public void terminateBrowser() {
driver.close();
}
}
TestNG.xml
<suite thread-count="2" name="Parallel-Test-Suite" parallel="tests">
<test name="To Run TestNG :: Parallel Tests concept - 1">
<classes>
<class name="scripts.MultipleSession1"/>
</classes>
</test>
<test name="To Run TestNG :: Parallel Tests concept - 2">
<classes>
<class name="scripts.MultipleSession2"/>
</classes>
</test>
</suite>
iii. parallel=”classes”:
This type has similar usage scope as the previous one with one major difference. Using this attribute, we can execute tests under each class in an individual thread, and the same has been explained in the below example.
Syntax:
<suite thread-count=”2” name =”Parallel-Test-Suite” parallel=”classes”>
Code:
Test1.java
public class Test1 {
public static WebDriver driver = null;
public static WebDriverWait wait = null;
@Test
public void Test1() {
String baseUrl = "https://codoid.com/";
System.setProperty("webdriver.chrome.driver", "./resources/drivers/chromedriver.exe");
driver = new ChromeDriver();
driver.get(baseUrl);
driver.manage().window().maximize();
wait = new WebDriverWait(driver, 25);
}
@Test
public void Test2() {
String testTitle = "QA Testing Company - Software Testing Services";
boolean isVerified;
isVerified = wait.until(ExpectedConditions.titleContains(testTitle));
Assert.assertTrue(isVerified);
}
@Test
public void Test3() throws InterruptedException {
WebElement elmntAboutUsTab = driver.findElement(By.xpath("//ul[@id='top-menu']/li/a[text()='About Us']"));
elmntAboutUsTab.click();
}
@AfterTest
public void terminateBrowser() {
driver.close();
}
}
Test2.java
public class Test2 {
public static WebDriver driver = null;
public static WebDriverWait wait = null;
@Test(priority = 1)
public void launchAmazonURL() {
String baseUrl = "https://www.amazon.in/";
System.setProperty("webdriver.chrome.driver", "./resources/drivers/chromedriver.exe");
driver = new ChromeDriver();
driver.get(baseUrl);
driver.manage().window().maximize();
wait = new WebDriverWait(driver, 25);
}
@Test(priority = 2)
public void verifyAmazonTitle() {
String testTitle = "Online Shopping site in India";
boolean isVerified;
isVerified = wait.until(ExpectedConditions.titleContains(testTitle));
Assert.assertTrue(isVerified);
}
@Test(priority = 3)
public void navigatesToAboutUsPage() throws InterruptedException {
WebElement elmntAmazonPayTab = driver.findElement(By.xpath("//a[text()='Amazon Pay']"));
elmntAmazonPayTab.click();
}
@BeforeMethod
public void beforeMethod() {
System.out.println("\n Starting Test On Chrome Browser");
}
@AfterTest
public void terminateBrowser() {
driver.close();
}
}
TestNG.xml
<suite thread-count="2" name="Parallel-Test-Suite" parallel="classes">
<test name="Using Classes attributes for parallel execution.">
<classes>
<class name="scripts.Test1"/>
<class name="scripts.Test2"/>
</classes>
</test>
</suite>
In the above 3 points, it can be seen that we have defined the keywords for parallel (classes, tests, and methods) attributes on the Testing.xml file. Now, we’re going to explain the sample scripts for classes and methods.
Conclusion:
As promised at the beginning of this TestNG tutorial, we have explored all the plus points of TestNG. So we hope this information would be useful and that you have enjoyed reading this blog. As one of the leading automation testing companies, we have always found TestNG beneficial in crucial circumstances, and hope it will come in handy for you as well.
by admin | Apr 13, 2021 | Selenium Testing, Fixed, Blog, Latest Post |
In every interview, there is that one common question that an automation tester would face, and that question is, ‘What is the use of Selenium WebDriver Event Listener?’. So in this blog article, we are going to precisely answer that question for you. In addition to that, once you have read this article, you will be in a position to use Selenium WebDriver Event Listener effectively to its full potential.
Selenium WebDriver is a Web Browser automation framework, and everyone is aware of that. Let’s say you want to capture a screenshot after a test case execution, you can write Selenium’s screen capturing snippet inside the testing framework’s ‘afterEach’ hook method. For example, if you want to capture a screenshot before/after every element click. You would have to write a screen capture snippet right after every click action which is not only a cumbersome task to do, but it is also a waste of valuable time. But you can make this process so much easier than this by making use of Selenium Event Listener. By using Selenium Event Listener, you can write the screenshot snippet inside the ‘beforeClickOn’ or ‘afterClickOn’ method.
Selenium has an interface called the ‘WebDriverEventListener’, which has 27 methods. All these methods are event listeners for Selenium WebDriver’s important actions. Let’s not leave any stone unturned and explore them one by one.
Implement WebDriver Event Listener Methods
First and foremost, you would have to create a class that implements the listener methods. Once that is done, you can create a concrete class for the WebDriverEventListener interface as shown below.
import org.openqa.selenium.support.events.WebDriverEventListener;
public class MyListener implements WebDriverEventListener {
}
As soon as you have created the class, your IDE will ask you to implement the listener methods. Once you accept the implementation, your class will have all the 27 methods.
import org.openqa.selenium.By;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.events.WebDriverEventListener;
public class MyListener implements WebDriverEventListener {
@Override
public void beforeAlertAccept(WebDriver driver) {
}
@Override
public void afterAlertAccept(WebDriver driver) {
}
@Override
public void afterAlertDismiss(WebDriver driver) {
}
@Override
public void beforeAlertDismiss(WebDriver driver) {
}
@Override
public void beforeNavigateTo(String url, WebDriver driver) {
}
@Override
public void afterNavigateTo(String url, WebDriver driver) {
}
@Override
public void beforeNavigateBack(WebDriver driver) {
}
@Override
public void afterNavigateBack(WebDriver driver) {
}
@Override
public void beforeNavigateForward(WebDriver driver) {
}
@Override
public void afterNavigateForward(WebDriver driver) {
}
@Override
public void beforeNavigateRefresh(WebDriver driver) {
}
@Override
public void afterNavigateRefresh(WebDriver driver) {
}
@Override
public void beforeFindBy(By by, WebElement element, WebDriver driver) {
}
@Override
public void afterFindBy(By by, WebElement element, WebDriver driver) {
}
@Override
public void beforeClickOn(WebElement element, WebDriver driver) {
}
@Override
public void afterClickOn(WebElement element, WebDriver driver) {
}
@Override
public void beforeChangeValueOf(WebElement element, WebDriver driver, CharSequence[] keysToSend) {
}
@Override
public void afterChangeValueOf(WebElement element, WebDriver driver, CharSequence[] keysToSend) {
}
@Override
public void beforeScript(String script, WebDriver driver) {
}
@Override
public void afterScript(String script, WebDriver driver) {
}
@Override
public void beforeSwitchToWindow(String windowName, WebDriver driver) {
}
@Override
public void afterSwitchToWindow(String windowName, WebDriver driver) {
}
@Override
public void onException(Throwable throwable, WebDriver driver) {
}
@Override
public <X> void beforeGetScreenshotAs(OutputType<X> target) {
}
@Override
public <X> void afterGetScreenshotAs(OutputType<X> target, X screenshot) {
}
@Override
public void beforeGetText(WebElement element, WebDriver driver) {
}
@Override
public void afterGetText(WebElement element, WebDriver driver, String text) {
}
}
Attaching EventListener with WebDriver
Now that the event listener class has been created, your next task would be to attach the web driver instance with the event listener, and that can be easily done by following the method shown below.
WebDriver driver = new ChromeDriver();
driver.get(“http”://codoid.com);
EventFiringWebDriver eventHandler = new EventFiringWebDriver(driver);
MyListener listener = new MyListener();
eventHandler.register(listener);
The real value of the WebDriver Event Listener comes from your idea and purpose. If you don’t have a purpose to use it, then implementing Event Listener is simply a waste of time. So, make sure to put in some time and thought before you go straight ahead and start things up. In the upcoming sections, as a test automation company, we have shared how we have effectively used Selenium WebDriver Event Listener for our automation testing projects. Let’s take a look at it beginning from ‘onException’.
onException
When you are in a situation that requires you to run a test suite multiple times, there are chances of getting some failures because of the script or application issues. Selenium has 33 common exceptions. If you manage to capture all these exceptions which are thrown during automated test suite execution, it will be extremely useful for auditing purpose. Let’s say you want to know which Selenium exception was triggered the most often in the past three weeks. When you have the exception history & metrics in hand, you can understand whether the automated test suite is stable enough or if it needs any minor tweaks.
Capturing Selenium Exceptions in the ‘onException’ method is pretty easy. But the question here is, ‘How can you store the exceptions for future reference?’. Well, the solution is easy too, as you can use Sentry to store the exceptions. If you are fairly new to Sentry or unaware of how to configure it in your framework, don’t be worried, we’ve got you covered. All you have to do is follow the steps that are listed below to easily configure Sentry in your test automation framework.
Step 1
Add the Sentry maven dependency in POM.xml
<dependency>
<groupId>io.sentry</groupId>
<artifactId>sentry</artifactId>
<version>4.3.0</version>
</dependency>
Step 2
Sign-up in Sentry and create a project.
Step 3
Once the Sentry project is created, you will get the below code which needs to be integrated with your test suite.
Sentry.init(options -> {
options.setDsn("https://2323.ingest.sentry.io/2323);
});
Note: Each project has its own DSN. So make sure to copy the above snippet from your project and not from here.
Step 4
Create a constructer in the Listener class and add Sentry config inside the constructer.
public class MyListener implements WebDriverEventListener {
public MyListener(){
Sentry.init(options -> {
options.setDsn("https://2323.ingest.sentry.io/2323);
});
}
}
Step 5
Now ingest the exceptions in the Sentry from the ‘onException’ method.
@Override
public void onException(Throwable throwable, WebDriver driver) {
Sentry.captureException(throwable);
}
afterFindBy
Nowadays, setting up a test automation framework is fairly simple. If you are using JVM-Cucumber & Selenium WebDriver, then the framework setup is pretty much straightforward in all aspects. You need to write Scenarios in the Gherkin format, implement a step definition for each Gherkin step using Page Object, and finally integrate the other required components.
In order to produce a robust test automation suite, you would need the following:
- A simple and concrete framework
- A Stable Environment
- A Skilled Team
- A team that follows good Object Locating Techniques
The first three requirements are default elements that every test automation suite must-have. When you have a truly skilled Automation Testing Team, then it is pretty much a given fact that they will follow good object locating techniques. However, you just can’t have the same team members for a longer period. So it is vital to ensure the scripting standards and object locating strategies even with the new team members to maintain the robustness of your test automation suite.
As a QA company, we use peer review and pair scripting sessions to ensure the test automation’s scripting quality and its robustness. However, it is a tedious task to review locators for each object. What we do in the afterFindBy method is, use regular expression and search sub-string logics to verify if the XPaths & CSS Selectors have met the standards that have been set.
beforeClickOn
The ‘beforeClickOn’ method comes in handy when you need to take screenshots of every click action for a workflow. Let’s say someone from your team wants to know what and all buttons/links/objects were clicked in an E2E test. Inside the ‘beforeClickOn’ method, you can access the driver instance and save the screenshot in a PDF file.
When you implement the ‘beforeClickOn’ method, make sure to create a config variable to decide whether you would need a screenshot before every click action or not. Another useful aspect is that, instead of saving the page screenshots before clicking, you can store WebElement screenshots and compare them with the excepted images. When you perform visual testing, you can catch if the CSS Styles are not applied for the Web Elements which is also a major advantage.
Conclusion
We hope you have enjoyed reading this blog article and that it will positively impact your efficiency. Now it is time for you to try and implement Selenium WebDriver Event Listener for any of these ideas in your project and see the results for yourself. If you would like to recommend any other ideas that will help improve the existing methods, please feel free to head over to the comments section share your valuable thoughts. Test Automation Services is one of the core services of Codoid. Our team is always ready to explore new implementations using Selenium & Appium. Subscribe to our blog so that you never miss out on our highly useful automation testing blog articles that will help you be at the top of your game.
by admin | Apr 6, 2021 | Automation Testing, Fixed, Blog, Latest Post |
Automation testing using JavaScript has been gaining popularity in recent days making a blog about the best JavaScript Testing Frameworks available the need of the hour. A conversation between business analysts, developers, and testers about a feature will become specifications. After that, the developers & testers convert the specs into automated acceptance tests. Automated tests help the team to drive the feature development as passing acceptance tests prove the fact that the agreed feature has been developed as expected.
To convert specifications into automated acceptance tests, devs & QAs should use the same programming language. Let’s say your team is using JavaScript for development and Java for automation testing. In this case, you won’t be able to enable effective collaboration between the developers & testers which is a big concern. Analysts & developers can capture acceptance criteria effectively. However, testers are good at adding edge cases making them invaluable as well. That is why everyone’s involvement right from the beginning is vital with common tools, frameworks, and language.
In this blog article, we have listed the best JavaScript Testing Frameworks which are widely used for writing unit tests and acceptance tests.
Jasmine
When developers focus more on components than features, they don’t add value to the business. Writing tests in plain English helps your team focus on the business goals and Jasmine is one of the best testing frameworks to make it possible. Using Jasmine you can write meaningful tests which can be understood by everyone in the team and help the developers to write higher quality code more efficiently.
Jasmine has support for Node.JS, RUBY, and Python. If you want to run the tests on a Web Browser instead of running them in interpreter/programming engine, then you can go for Jasmine Standalone. Now let’s find out how to use Jasmine in Node.JS beginning right from its installation.
Installation:
To install Jasmine, run the below npm command.
npm install jasmine —save-dev
Initialization
To set up Jasmine in a Node.JS project, you need spec files and jasmine.json (Configuration file). You can add these files manually. However, we have an npx command to initialize Jasmine in your project and it has been listed below
After running the above command, you can find the ‘spec’ folder and ‘jasmine.json’ file under the spec->support folder.
Add Jasmine in package.json
Now, you need to inform your node project that you are using Jasmine for testing. Open package.json and update the ‘test’ value as ‘jasmine’ as shown below.
{
"name": "jasmine-sample",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "jasmine"
},
"author": "",
"license": "ISC",
"devDependencies": {
"jasmine": "^3.7.0"
}
}
Describe
In order to group related test cases like how we group scenarios in a feature file in Cucumber, you need to create a JavaScript file and write a ‘describe’ function as shown below.
describe("User Management", ()=>{
})
In the above function, it is a placeholder to add ‘User Management’ related test cases.
Specs
Specs are your test cases. You need to write specs inside the ‘describe’ function as shown below.
describe("User Management", ()=>{
it("should signup",()=>{
})
it("should login", ()=>{
})
})
As we mentioned earlier, everyone in the team should understand what is the purpose of each test. When naming a test case for unit & acceptance tests, you need to describe what you are expecting. However, it is a cumbersome task for a developer to focus on names. If there are a rule and a standard which come from the testing framework, then the team members will be able to stick to the business goals and focus on achieving them instead of concentrating only on the code components.
That is why the Jasmine framework has the ‘it’ function and the first argument starts with ‘should’. When a test case says ‘It should do something’, the developer will focus on that something throughout the development which is exactly what we want.
How is a spec pass or fail determined? You can have multiple assertion lines inside a spec and even if one of those assertions fails, the spec will be marked as ‘Fail’.
Assertions
Jasmine has an in-built assertion function which is called ‘expect’. Expect function has one argument which takes the actual value. You can also chain the assertion using Matcher functions.
expect(“my string”).toMatch(/string$/) – Expect the actual value to match a regular expression.
expect(bigObject).toEqual( “foo”: ‘bar’, ‘baz’ ) – Expect the actual value to be equal to the expected value using deep equality comparison.
expect(thing).toBe(realThing) – Expect the actual value to be === to the expected value.
Configurations
Jasmine has a total of five configurations in the jasmine.json file. Let’s take a look at the purpose of each of these configurations one by one.
“specdir” – A directory path where you have the specs in the current project directory.
“specfiles” – You can include & exclude spec file paths with regular expression.
“helpers” – Before the test execution commences, you may need some supporting files that should be executed. The helpers config is the placeholder to mention the supporting JS files.
“stopSpecOnExpectationFailure” – This is a boolean config. If the value is true, then the spec execution will be stopped when the first assertion is failed.
“random” – Run the specs in random order.
Advantages of Jasmine
- You can customize the Jasmine report.
- It has a very strong community that supports the users.
- You can even create your own Matchers.
- It is extremely simple and easy to use making it the choice of many developers and testers mainly for its simplicity.
Jest
Jest is also a popular JavaScript Testing framework that works with Node.JS, React, Angular, Vue, Babel, TypeScript, and more. Jest was developed by Facebook and so it runs thousands of tests for Facebook on a daily basis. Apart from Facebook, there are many other prominent companies that also use Jest. So let’s see what makes Jest one of the best JavaScript Testing Frameworks and how you can implement it in your project. Let’s begin with its installation process.
Installation
You can easily install Jest by making use of the below code,
npm install --save-dev jest
Add Jasmine in package.json
Update scripts>test node as ‘jest’ in Package.json as shown below.
{
"name": "jest-sample",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "jest"
},
"keywords": [],
"author": "",
"license": "ISC",
"devDependencies": {
"jest": "^26.6.3"
}
}
Configuration
No configuration is required to write or run Jest tests. However, every testing framework has its own configuration file. To create a config file, you can run the below command.
Once you run the above command, Jest will ask you a few questions to create the config file.
Migrating from Jasmine to Jest
If you are planning to migrate from Jasmine to Jest, then we’ve got some good news for you as Jest is compatible with Jasmine tests. This makes the migration far less complicated. However, if you want to do a code-level migration, you can use third-party jest-codemods for the best results.
Assertions
In order to write assertions elegantly in a test, you would definitely need sophisticated Matchers. Jest has a lot of compressive matchers that you could use. However, we have listed some of the commonly used Matchers below.
test('two plus two is four', () => {
expect(2 + 2).toBe(4);
});
test('object assignment', () => {
const data = {one: 1};
data['two'] = 2;
expect(data).toEqual({one: 1, two: 2});
});
test('null', () => {
const n = null;
expect(n).toBeNull();
expect(n).toBeDefined();
expect(n).not.toBeUndefined();
expect(n).not.toBeTruthy();
expect(n).toBeFalsy();
});
Snapshot Testing
Your Acceptance Tests test the application under test by performing actions on the UI. However, it never validates whether the UI is rendered as same as before and if the CSS styles are applied correctly or not. Jest has a feature to address this issue (i.e. Visual Regression Testing). Using Jest, you can capture snapshots and compare them with the expected snapshot thereby creating a huge advantage if that is your primary need.
Advantages
If you want to run your tests in parallel, then Jest is your go-to testing framework. Jest has numerous assertion functions. In terms of ease of adoption, simplicity, support, and documentation, Jest is hands down the clear winner.
Mocha
Mocha is a JavaScript Testing framework only for Nod.JS. Mocha doesn’t have an in-built assertion library. However, it allows you to use the following libraries such as should.js, chai, expect.js, better-assert, and unexpected for assertions. You can execute tests in two modes (Serial & Parallel).
Conclusion
Jest and Jasmine are the best JavaScript Testing frameworks. However, most of the engineers are comfortable with Jasmine over Jest. As an automation testing company, we use Jasmine for Protractor automation testing. Jasmine is the default testing framework for Protractor. Our suggestion is not final and it shouldn’t influence your framework decision.
Jest has many sophisticated features such as Configuring Visual Testing (i.e. Snapshot Testing) which is not as easy to do in Jasmine. However, in Jest, it is pretty straightforward. Apart from that, test creation and assertions are similar in both these frameworks. If you feel Jest is suitable for your project, please go ahead.
by admin | Mar 29, 2021 | Automation Testing, Fixed, Blog, Latest Post |
Even the smallest hiccups like package installation errors can be a bane to your productivity. In this blog article, we have listed some of the basic npm commands which are used by developers and software testers regularly to resolve package installation errors and to even upgrade existing npm packages. As a software testing company, we are experts in automation testing services using Protractor & Jasmine frameworks.
For one to manage npm packages effectively, one should be aware of the most common npm commands. So let’s get the ball rolling with a few basic commands that have been categorized by their purpose.
Package Installation
To install a package in your current project using npm commands, you can make use of either of the codes given below:
npm install package-name -s
or
Use the code given below to install a package under Development dependencies in your current project by using npm commands:
npm install package-name —save-dev
You can install a package globally using npm commands if a package dependency is not for the project by using the below code:
npm install package-name -g
Install a specific package version
Sometimes you might find yourself to be in a situation where you need to install a specific package version rather than the most latest version. But when you run the npm install command, it will only install the latest version of a package. The solution is easy, as you can install a specific package version by simply using the version number in the installation command as shown below.
npm install package-name@version-number
Finding Outdated Packages
When the dependency list grows, it is a cumbersome job to find the outdated packages in a project. However, npm can make your job easy as it has a command to find the packages which can be upgraded.
When you run the above command, it will list the outdated packages from which you can find the version that you are looking for.
Clear npm Cache using npm commands
When a package installation doesn’t go as expected, it may be because of an npm cache issue. One best option would be to clear the cache and then execute the installation command again.
Before clearing the npm cache, make sure to check how many cache entries are available by using the below code.
Now you can clear the npm cache by using the below command.
Audit Packages
Using vulnerable npm packages can make your project less secure. So it is crucial to perform a security check for all the available project packages. You can do so by running the below command.
Once the audit has run, it will list the vulnerable packages which are supposed to be removed to make your packages more secure.
Installing Minor & Patch Releases
When you run the npm install command, it will install the packages which are mentioned in the package.json file. The npm install command can be used to install only the missing packages. But if you are looking to install recent minor versions using the npm install command, you have to prefix the caret (^) symbol before the package version in package.json. To install patch releases, you have to prefix the Tilde (~) symbol.
Conclusion
Auditing packages & removing unused packages periodically will help avoid catastrophic failures when deploying code in a production environment. Node.js is a memory-efficient environment. When an automation test suite runs for a longer duration, you should optimize it by improving the script and using good packages.
QA automation testers who are starting automation test scripting using JavaScript & Node.js will find these basic npm commands very helpful. Being an automation testing company, we use Protractor, Selenium WebDriver, and Jasmine frameworks for web apps automation testing, and therefore in the subsequent blog articles, we will be publishing more on Protractor and automation test scripting using JavaScript & Node.js. So make sure to keep an eye on our website so that you don’t miss out on the upcoming blogs.
by admin | Jan 13, 2022 | OTT Testing, Blog, Latest Post |
Even before the pandemic hit, OTT platforms have been on a rampage in terms of growth and popularity. Netflix has multiplied its user base 10 times over the last 10 years. There is heavy competition amidst the existing big players and the number of OTT platforms entering the market is also on the rise. Even if you have the best content in the market and fail to provide a good user interface and user experience, users will move away. So no OTT platform can afford to have mediocre testing that lets bugs reach the user. As a leading QA company, we deliver uncompromised quality to all our clients and have created this conclusive list of test cases for OTT Platform Testing that will help you achieve maximum coverage.
OTT is an abbreviation for “over-the-top,” which refers to any streaming media service that delivers content via the internet. The term “over-the-top” refers to the fact that the service is delivered on top of another platform. In other words, it is any service that allows you to deliver video and/or live stream feeds to any internet-connected device.
OTT Platform Testing
Robust and exhaustive OTT platform testing is required to successfully deliver video content. There are numerous things to test when it comes to OTT platform testing, but as stated earlier, they boil down to these two major aspects.
- The user interface (UI) allows users to interact with the OTT platform and view the video content.
- The user experience (UX) the person takes away after interacting with the UI.
Let’s dig deeper to get a better understanding of why these two factors are important. Let’s take Netflix as an example as we’ve already discussed how it has grown over the years. In an age that had episodes airing every week, Netflix introduced the binge-watching culture by dropping all of the episodes in one go. They did not want you to leave their platform once you entered.
So instead of just changing the release pattern, they built an intuitive UI that keeps recommending content based on your preferences. But if in case Netflix kept suggesting shows we weren’t interested in, we would just leave. At the same time, even if Netflix was able to provide good recommendations after making the user wait for 2 minutes, nobody is going to wait. That is why it is vital to test how people interact with the product and analyze their experience as well.
As one of the best OTT testing companies, we have created a list of test cases under each category to obtain maximum coverage and quality. We have also mentioned a list of the best testing tools that can also be used as OTT Testing tools. But before we proceed to that, let’s start with the test cases for User Interface and functionality.
UI and Functionality
Playback: Check to see if the video plays/pauses when you click on the video play icon. Also, see if the volume controls, and mute functionality works. Check for audio-video sync as well.
Video Quality: Make sure that the user is able to choose the video playback quality based on their network conditions.
Replay: Check if that the user can replay the video if they wish to see it again once it is finished.
Search: Verify that the video can be found when searched by the movie title, artist names, or relevant keywords that can be found in the description.
Search Results: Check the video’s thumbnail, title, description, length, age rating, etc. and see if the search has relevant autosuggestions. Also, make sure to see if clicking on the video plays the video or opens the content to show additional information.
Voice Commands: Since a major share of the OTT content is consumed via smart televisions, make sure that the voice commands by various assistants also work correctly.
Browse Features: You must be able to filter your search results and content in general so that you can browse through the content without any hassle. Check if the suggested content is based on the user’s viewing history.
History: Once a user starts watching a show or a movie, they must be able to pick up from where they left off and not have to start over again. If it is an OTT platform like YouTube, make sure the history function works well and see if the user is able to remove a video from the history or clear the history on the whole.
Offline Mode: Verify if the user is able to download the content they wish to keep offline and watch it without any internet connection. Also, make sure the downloaded files are not available locally to the user outside the platform.
Bandwidth Capacity: Check how well the content is streamed under different network conditions. If there is limited connectivity, see how well the video is played with lower bitrates.
Direct Access Link: Make sure that the video page can be accessed via a direct link to the video. See if the link can be shared easily using a third-party app.
Browser Compatibility: Verify that both the video and audio play in all flash-enabled browsers. And if it doesn’t, ensure that a meaningful message is shown to the users using a browser without flash or the necessary codec.
Watch list: Add content to a watch later or watch list to see if it gets updated properly. If there are any new additions like a new season to a show in the watch list, make sure it is notified to the user.
Notifications: Other notifications should include the addition of popular and relevant content, release of any content that had a countdown timer or reminder.
Language & Captions: Check if the mentioned captions are available in the content and also see if it in sync with the audio and visuals. If there are other language captions or audios available, then it must be possible to switch easily.
Audio output: Users might be connected to a different audio device via wires or Bluetooth when watching content on a television or mobile. Make sure that the audio is routed to that device properly without any sync issues.
Multitasking
Most of these test cases will be the ones you’d have to cover when testing on mobile platforms. Since OTT platforms employ mobile-only plans, it is an important part of OTT platform testing that can’t be neglected.
Call: Make sure that the video pauses when the user is watching using a mobile device. Also, check if the video resumes from the exact same point once the call is disconnected.
Notifications: Make sure that pop-up notifications don’t pause the playback or disturb it in any way unless the user clicks on it.
Mini Player: If the user does click on the notification, check if the player gets minimized and continues to play. Make sure to drag the mini-player around to see if it works properly. If the mobile or tablet supports split-screen viewing, then make sure the mini player is compatible with that too.
Account
Subscription: Ensure that only users who have paid the subscription fee are able to access the content and stream the video. If there are any ‘Pay per View’ or ‘Video on Demand’ content, test to see if it can be accessed by only those who have the extra payment.
User Profiles: The user must be able to create multiple user profiles and even have separate user histories. Verify if the user is able to select the profile through which he/she wants to watch as soon as the app is launched. There must also be kids or age-restricted profiles which can be accessed by the account with parental rights.
User Count: Make sure the platform is able to detect an account being used in more than the number of allotted screens for the package. Few plans allow different user profiles to access the platform at the same time and not allow the same content to be seen across multiple devices. The same applies to the number of devices that can have offline content. So keep an eye on such factors as well.
Login: Proper account recovery options must be available. But make sure that nobody else is able to access the account without the owner’s knowledge. If the account is being accessed from a new device, make sure that an OTP is sent for verification and enable the option to mark the device as a trusted device. Check if 2-factor authentication works if enabled.
Password: Make sure to suggest the user set a strong password. Ensure that older passwords cannot be reused. If the user wants to change the password, ensure that it is done with an OTP verification. There should also be an option to log out of all the devices.
Information: All the relevant account information like the plan name and so on should be easily accessible in one place for the user to review.
Payment and Security
Since there are both annual and monthly billing options, security features in any OTT platform can’t be taken lightly. User data has to be kept safe and a seamless payment process is very important in providing the best user experience. A user shouldn’t feel burdened or afraid while making the payment.
Successful Payment: Once the payment has been made, ensure that the changes reflect immediately in the account to prevent the user from having any doubts. Also, make sure the user is redirected back to the platform after a successful transaction.
Failure Scenarios: Make sure to think of various failure scenarios and test them out.
- Check what happens if the session ends during the payment process.
- Determine what happens if the payment gateway stops responding during the payment process.
- Make sure the account doesn’t get activated despite having a failed transaction.
- Observe what happens if the payment or transaction goes on hold.
- Check if the session expires within the time limit.
Payment Options: Try making a payment using all the various options like a credit card, debit card, and so on.
Localization: Make sure the amount is displayed in the current currency and the plans for that region are only being purchased. Also, try to change the payment gateway language during the payment process.
Pop-up Blocker: Examine the pop-up blocker settings and observe what happens when the pop-up blocker is turned on and off. At times few payment pages open only in a new window and the blocker might affect that.
Backend: Go beyond the front-end OTT platform testing and analyze what happens in the backend during the payment process.
Buffer Pages: Examine the buffer pages that exist between the payment gateway and the application.
Saved Payment options: If the user wishes to save the payment option, examine the database entries to see if they store credit card information.
Double Checks:
- During the payment process, double-check the error and security pages.
- If you are sending any email for the transaction, ensure that the content is encrypted.
- Verify if all the sensitive information is masked.
- Check that the payment is being made through a secure channel. In other words, the link must begin with HTTPS and not HTTP.
Gateway: Check if the payment is processed with an integrated page or if it is navigated to the bank’s payment page. If it is a page within the platform, make sure the transaction is processed only with the right OTP and card information.
OTT Testing Tools
Though there are no dedicated OTT Testing tools that we use for OTT automation testing, we can still use standard automation testing tools such as Appium and BrowserStack. Now let’s see how these tools will be useful when performing OTT testing.
Appium
Since a major share of OTT devices is powered by Android, Appium can be used to perform OTT Automation Testing. We are not just talking about the wide variety of Android Smart TVs in the market, even devices such as Amazon Firestick run on their own Custom OS (FireOS) that is built with Android Nougat as its base. Since Appium is an already popular testing tool, it can serve as the go-to OTT Testing Tool when it comes to automation.
BrowserStack
We have already seen how Appium can be used to achieve OTT Automation Testing on select devices. Another point of concern when it comes to OTT testing is numerous variations in the OTT devices in terms of an operating system, form factor, and so on. So a cloud testing platform such as BrowserStack is essential in assisting us in testing across a wide range of devices on the market. As an OTT Testing tool, BrowserStack currently supports both Android TVs and FireStick. But according to their website, other popular options like Roku and Apple Tv are on their way to the platform.
Conclusion:
We hope you have found these test cases for OTT Platform Testing to be informative. Though there are many more avenues to explore in OTT platform testing, these are the fundamentals you have to follow. The OTT platform is constantly being updated so that it doesn’t get outdated. Seeing the rise of popularity for TikTok, YouTube has introduced the Shorts feature. Understanding the content for short content, Netflix has created the ‘Fast Laughs’ feature. Netflix is also working on integrating games along with their subscription. So the landscape will always be changing and you should be on a journey of constant learning to keep up with all the new challenges.
Frequently Asked Questions
-
What is OTT testing?
OTT testing can be termed as the process of determining the capability of the platform to stream video and audio content over the internet by testing the platform’s UI, functionality, performance, and security.
-
What is OTT Automation?
OTT Automation is similar to any type of automation testing as it is the process of using automation tools to maintain test data, execute the tests, and analyze the results to increase the OTT platform’s overall quality.
-
What is an OTT Device?
Any device other than a desktop, laptop, tablet, or smartphone that is used to consume OTT content is called an OTT device. Some familiar examples are Smart TVs, Amazon FireStick, Roku TV, Apple TV, etc.
-
What is Streaming Testing?
Streaming Testing is used to test if the videos stream with the expected quality & functionality over the internet under different network bandwidths & loads to deliver a great user experience.
-
Are OTT and Connected TV the same?
No. OTT is the process of streaming content directly on a computer or mobile device and Connected TVs require the TV to be connected to a device with an active internet connection such as an Xbox, Apple TV, or Roku TV for streaming content.
