by Mollie Brown | Nov 13, 2022 | OTT Testing, Blog |
The OTT market has been growing rapidly over the last few years, and the pandemic has only seemed to accelerate the growth even further. With the competition in the OTT market heating up with more new players entering the market, the need for great user experience is also increasing. And OTT testing is one of the best ways to ensure that the users have a great time once they enter the platform. Being a leading OTT Testing service provider, we are aware that just like every type of testing, OTT testing also requires its own approach, strategy, tools, etc. And that is what you are going to learn about in this OTT Testing Tutorial.
What is OTT Testing?
OTT is expanded as Over The Top and could refer to the delivery of any service such as messaging, audio, video, and so on over the web. When it comes to streaming, OTT testing can be termed as the process of determining the capability of the platform to stream video and audio content over the internet by testing the platform’s UI, functionality, performance, and security. As stated above, one of the core objectives of OTT testing is to deliver the best user experience to the user and make sure they keep using the platform for as long as possible. But before we proceed to the OTT testing tutorial part of the blog, let’s take a deeper look at why OTT testing is so important.
Why is OTT Testing Important?
According to a report by Statista, earnings in the OTT video streaming industry are projected to reach a whopping $275 Billion Dollars in 2022. The sheer large number of users is one of the main reasons OTT is so successful. But with a large user base, there are a few obvious concerns that are hard to solve.
- Devices – One can stream OTT content using so many different devices such as smartphones, smart TVs, laptops, desktops, and other OTT devices such as Apple TV, Roku, Amazon FireStick, and so on.
- Platforms – These devices have further variations in terms of the operating system and browser that is used. (Android, iOS, Windows, macOS, WebOS, TizenOS, Google TV for smartphones, computers, and smart TVs) (Google Chrome, Safari, Firefox, Edge, etc in Browsers).
- Bandwidth – In addition to that, network conditions & bandwidth availability might also drastically vary from user to user.
But such conditions should never impact the video quality or viewing experience in general as it will severely affect the user experience. That is why OTT testing is the need of the hour. Now let’s learn how to perform OTT testing in our OTT testing tutorial.
OTT Testing Tutorial
Though different testing approaches are needed to test on the different types of devices on which the OTT content is consumed on, what has to be tested will be a standard. In addition to the defined standards, there will also be specific OTT testing scenarios from device to device. That is why we have created a categorized OTT testing checklist for our OTT testing tutorial. So let’s get started.
General OTT Testing Checklist
If we were to create a general testing checklist for mobile apps, it would be very hard as each mobile app would have a different purpose. But even though there are variations in how OTT platforms work, they do share the same purpose. So let’s now see the common functionality & usability aspects that should be the focus points of OTT testing.
UI, UX & Functionality
Having the best content is one of the best ways to provide a great user experience. But it would all go to waste if the viewing experience isn’t good or if the platform isn’t even functional. So let’s start our OTT testing tutorial from here.
- It is important to test if the content on the platform has been indexed properly which enables the content to be found when the user searches with the title, names of the technicians, genre, and so on.
- Netflix has been known for creating custom thumbnails for its content based on the viewing habits of different users. Even if the OTT platform you are testing may not have such a feature, make sure that the thumbnails help the users easily identify the content.
- Test the video recommendation algorithm by watching content like a real user.
- Check if the features like autoplay, skip the intro, etc appear at the right time and if they work properly.
- Add videos to the watchlist and test if the user is able to view them at a later time.
- Test if the users are able to resume playback of a video from where they left off and if the users are able to differentiate a video they have already seen from a fresh video they haven’t yet seen.
- If there is a trailer autoplay feature, make sure the user is able to pause the video or mute/unmute the video based on their needs.
- Check if the detailed view of a video has relevant information such as the details of the cast & crew, an option to view the trailer, the maturity rating of the film, its genre, and so on.
- Test if the users are able to share a link to individual videos with their friends with ease.
Next up in our OTT testing tutorial, we will be focusing on what aspects have to be covered with regard to the cores of any OTT platform, streaming & performance. It is what users spend most of their time doing once they decide what to watch.
- Make sure the audio and video are in sync and that the available subtitles are easy to switch and in sync with the content as well.
- Change the language of the stream and test if it affects the playback in any way.
- A wide range of devices could be connected to the internet via WiFi, LAN, or Mobile Data. So make sure the streaming quality doesn’t suffer when using either of these technologies.
- Test by switching between these different network options and also at different bandwidths to ensure seamless streaming on all conditions.
- If the user has downloaded offline content, make sure it is possible to playback the video even without any internet connection.
- Make sure the video starts streaming as quickly as possible once the user decides to play it.
- Test the OTT platform’s stability by trying to access the content using a large number of concurrent users.
Account
Since most OTT platforms have different tiers of memberships, it is important to test using different user roles and ensure everything is in order. Also, other account-related features should be tested as mentioned below.
- The users must be able to access the content they have paid for only and should not get anything less or more.
- Beyond their plans, make sure PPV (Pay Per View) & VoD (Video on Demand) content is accessible only to those users who have made the extra payment.
- Test if the correct number of parallel users are able to view the content based on their plan.
- If there is an option to create multiple user profiles, make sure the watchlist, viewing history, downloads, and recommendations don’t overlap.
- Since plan pricing, content rights & availability all vary across different parts of the world, make sure to perform Geolocation based OTT testing with reference to the account’s location.
- Test if the user is able to view all the information about their subscription plan, logged-in devices, billing info, and so on.
Security & Billing
Since most OTT platforms work on a subscription model, the need to store the user’s bank information is there. So testing the OTT platform’s security and payment process is an important aspect that we’ll be covering next in our OTT Testing tutorial.
- Check if it is possible to make the payment using different payment options such as debit card, credit card, internet banking, and so on.
- Test the various payment failure scenarios due to different reasons to make sure the user is made to feel secure with proper guidance and support.
- If a user opts to save their billing information, make sure it is saved properly, and that the users can delete the information if they wish to.
- Make sure that the auto-renewal feature works as expected and that the user is informed about the status.
- Check if the user is navigated to a secure channel to complete the payment process.
- Test if the user is able to download a receipt for the payment made. If the receipt is sent via email, make sure the critical information is either encrypted or masked.
- Make sure that the 2 Factor Authentication process is enabled and that it works as expected.
Accessibility
The need for accessibility is being understood and recognized by many across various industries. Likewise, leading OTT service providers such as Netflix and Amazon Prime Video are taking a step forward in ensuring even people with disabilities are able to access their content.
- Make sure every content on the OTT platform has subtitles that convey more than just dialogues as it should also describe important audio & visual references.
- Test if in-depth audio descriptions that describe the scene as they are happening are available.
- Since flashing lights could trigger seizures for a few people, ensure that such warnings are mentioned for the content that might have such triggers.
- Based on the device and platform they are using, make sure the OTT platform is usable with the help of alternative input or output devices such as screen readers.
As established already, OTT testing has to be performed on a lot of devices and platforms to ensure they can be consumed across all options. So in addition to the general tests we have already seen in our OTT testing tutorial, make sure to cover these device-specific checklists as well.
OTT Testing on Smart TVs & OTT Devices
- Unlike smartphones and computers, Smart TV users will not have easy access to keyboards for them to perform typed searches. So make sure the search functionality works well with various voice commands.
- Since Smart TV users would have to use a remote to control everything, ensure you include test scenarios that test the platform’s usability with a remote.
OTT Testing on Mobile Devices
Unlike Smart TVs and OTT Devices, the primary objective of smartphones is not viewing OTT content. So performing OTT testing by including different interruptions is the key to success.
- Check if the playback is paused when the user receives a call and if the playback resumes from where they left off once the call ends.
- Ensure the playback is not affected by notifications from other apps such as new messages, updates, and so on unless the user clicks on the notification to take them to the app.
- If there is a mini-player feature available, make sure to test it by dragging it to different places on the screen and if the controls work properly.
- Test if the video streams as expected in both portrait and landscape modes.
OTT Testing on Computers
- Test the platform by accessing it using both a browser and the web application if it is available. Ensure that the user experience is not compromised on both platforms.
- Check if the playback is impacted when switching between different tabs or windows.
Conclusion
We hope you now have a clear understanding of how to perform OTT testing after reading our OTT Testing tutorial. Apart from this blog, we have linked other blogs from our website that you will find useful in learning how to automate OTT testing on specific Smart TVs and OTT devices. So make sure to read those blogs as well to become an expert when it comes to OTT testing. Being a leading OTT testing company, we will be sharing more informative content as we continue to break new ground in OTT testing. So make sure to subscribe to our newsletter to not miss out on any of our latest updates.
by Arthur Williams | Jun 20, 2022 | OTT Testing, Blog, Top Picks |
Content is King is a very famous quote that we all would have heard on so many occasions. Though it applies to OTT platforms, there is no denying that user experience is also an integral part of an OTT platform’s success. It is important to note that the technical aspects of the OTT platform also have to be strong to ensure the best user experience as nobody will be willing to watch even the greatest content if it is constantly buffering and taking forever to respond to your action. The only way to ensure that the platform delivers the best user experience is by performing OTT testing. So in this blog, we will be seeing how to perform OTT Automation testing across major devices and see a conclusive checklist as well.
What is OTT Testing?
An OTT (Over the Top) service is any type of streaming service that delivers content over the internet. Though there is a wide range of OTT services such as WhatsApp, Skype, Zoom, and so on, our focus will be entirely on OTT media services such as Netflix, Amazon Prime, etc. So OTT testing can be defined as the process of performing robust tests to ensure the successful delivery of video content such as films or tv shows with the best user experience.
Did You Know?
A very fascinating aspect of OTT testing and Netflix is that Netflix developed its own testing tool called Chaos Monkey to test the resilience of its Amazon Web Services. The concept behind Chaos monkey was that it would intentionally disrupt the service by terminating key instances in production to see how well the other systems managed.
But why is the OTT giant so much focused on testing? To answer this question, we would have to take a look back in time to find out how Netflix and other OTT services emerged.
The Emergence of OTT
The popularity of OTT services has skyrocketed over the roof this past decade. The pandemic only added more fuel to the flame as more and more people started consuming content on OTT platforms. This type of media consumption is actually very different from its predecessors and competition. For example, VCRs, CDs, and DVDs did not need any software or platform, or infrastructure from the vendor’s side. So the end-user experience was majorly dependent on the consumer side infrastructure. Likewise, the end-user experience varied from theatre to theatre. But OTT is very different from all these options as apart from providing the content, the user experience is purely dependent on their services as well.
Netflix is so popular that the name has become synonymous with the term OTT itself. But how was Netflix able to topple Blockbuster video, a company that was still generating $5.9 Billion annually in 2004? The answer is user experience. Blockbuster was making around 16% of its yearly income through late fees. No customer was happy with this working model. In fact, one of the co-founders of Netflix, Reed Hastings had decided to start Netflix after receiving a $40 fine from Blockbuster. So Netflix was born as a company that did not have any late fees.
The User-Experience Factor
You might be wondering how is this story relevant to OTT testing. To answer that question, let’s take a look at the very first Netflix original series, House of Cards. Netflix revolutionized how content will be consumed by releasing the entire series in one shot instead of following the regular weekly one-episode plan. This meant that when the end-user wanted to watch the next episode, they could. The users also had the liberty to watch the show they wanted when they wanted instead of depending on the network’s schedule. So ultimately Netflix made sure that the end-user never left their platform using the user experience they had crafted. People started binge-watching content and Netflix’s strong user experience ensured that it happened.
Now that the infinity scrolling feature for video content has become widely popular, Netflix has introduced a feature called ‘Short Laughs’ that showcases short clips from their best shows. Netflix takes their user experience very seriously as they even developed a Chaos monkey to test their platform’s resilience against vulnerabilities. So if Netflix, the most successful OTT service provider is so focused on testing its platform to ensure the best user experience, the importance of OTT testing can clearly be understood here. So it goes without saying that every OTT platform should be tested for its functionality, performance, etc, and that OTT Automation Testing is an integral part of achieving these goals.
Most Popular OTT Platforms Across the World
After the success of Netflix, many wanted to jump on the bandwagon. And after a flurry of various attempts, these are the top OTT platforms that are popular as of writing this blog.
Most Popular OTT Devices Across the World
One of the major advantages of OTT services is that they are not limited to just the TVs we use like the old-fashioned cable connection. Instead, we can view and enjoy the content on a wide range of devices such as mobile phones, tablets, computers, smart televisions, and so on. Even if your television is not a smart TV, you can still use plug-and-play add-ons such as Apple TV, Amazon Firestick TV, Roku, etc. to access all your favorite OTT platforms.
How to do OTT Automation Testing?
Performing manual testing on platforms with such a huge scale can be a cumbersome task. That is why it is important to implement automation in your OTT testing. As a pioneer in the OTT testing arena, our R&D team had worked hands-on with all the above-mentioned devices to help us render high-quality OTT Automation testing services from the get-go.
First up, we have to connect our computers to the smart tv that we are looking to automate. But since smart TVs are not the only way the OTT content can be consumed, we would be in a position to connect our computers to the various OTT devices such as Apple TV, Amazon FireStick, Roku, and so on.
Android TV Automation
On the surface level, automating an Android app on mobile and on an Android TV might seem to be the same. Though there are a lot of similarities, Android TV doesn’t have a touch input system and has to be operated with a remote. So we would have to keep this in mind when writing the automation scripts.
Required Prerequisites
Step 1: Enable Developer Options by going to the Settings.
Step 2: Obtain the Device Name, Platform Name, Platform Version, and IP address of the TV.
Step 3: Configure your computer with all the required Appium setups.
Step 4: Open Command Prompt and enter the command ‘adb connect ’ to see the list of the connected devices.
Step 5: Run the automation script.
Automation Script
We are now going to see a sample automation script that can be used to launch the application and play a video.
public class LionsGateInstaller {
public static AppiumDriver<MobileElement> driver;
public static WebDriverWait wait;
public static void main(String[] args) throws IOException, InterruptedException {
try {
//Set the Desired Capabilities
DesiredCapabilities caps = new DesiredCapabilities();
//Set the Smart TV Name
caps.setCapability("deviceName", "My Redmi TV");
//Set the Smart TV Platform
caps.setCapability("platformName", "Android");
//Set the Smart TV platform version
caps.setCapability("platformVersion", "11");
//Set the App package and app activity details
caps.setCapability("appPackage", "com.xiaomi.lionsgateplay.videoapp");
caps.setCapability("appActivity", "com.parsifal.starz.ui.features.splash.SplashActivity");
caps.setCapability("unicodeKeyboard", true);
caps.setCapability("resetKeyboard", true);
caps.setCapability("noReset", "true");
driver = new AndroidDriver<MobileElement>(new URL("http://127.0.0.1:4723/wd/hub"), caps);
driver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS);
System.out.println(">>>> APP Launched Successfully::::");
wait = new WebDriverWait(driver, 30);
If you would like to see more sample scripts for various other actions such as performing a search and remote actions, you could visit our related blogs.
Amazon FireStick Automation
If your TV is not a smart TV running on Android or any other equivalent operating system but has an HDMI port, you can still use devices such as Amazon FireStick to consume the OTT content. So based on the device that you use, you would have to follow different steps to automate the OTT testing process.
Step 1: Enable Developer Options by going to the Settings.
Step 2: Obtain the Device Name, Platform Name, Platform Version, and IP address of the Amazon FireStick in use.
Step 3: Configure your computer with all the required Appium setups.
Step 4: Open Command Prompt and enter the command ‘adb connect’ to see the list of the connected devices.
Step 5: Run the automation script.
Automation Script:
public class Aha_OTT {
public static AppiumDriver driver;
public static void main(String[] args) throws MalformedURLException, InterruptedException {
//Set the Desired Capabilities for connect my smart to the Laptop.
DesiredCapabilities caps = new DesiredCapabilities();
//Set the Smart TV Name
caps.setCapability("deviceName", "My Fire TV");
//Set the Smart TV Platform
caps.setCapability("platformName", "Android");
//Set the Smart TV platform version
caps.setCapability("platformVersion", "7");
caps.setCapability("appPackage", "ahaflix.tv");
caps.setCapability("appActivity", "ahaflix.tv.MainActivity");
caps.setCapability("noReset", "true");
driver = new AndroidDriver(new URL("http://127.0.0.1:4723/wd/hub"), caps);
driver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS);
WebDriverWait wait = new WebDriverWait(driver, 30);
System.out.println(">>>>App Launched Successfully::::");
Thread.sleep(25000);
//Find the Top Picks Content of the Platform and Print that Name
WebElement Movie Title = driver.findElement(By.xpath("//android.widget.TextView[@resource-id='Top Picks']"));
if (Movie Title.isDisplayed()) {
System.out.println("Top Pick Successfully Verified");
}
// To Press the Left button by using Remote Action
((PressesKey) driver).pressKey(new KeyEvent(AndroidKey.DPAD_LEFT));
Thread.sleep(10000);
// Finding & Click the Video Icon
WebElement videoIcon = driver.findElement(By.xpath("//android.widget.FrameLayout[@resource-id='android:id/content']//android.widget.ScrollView/android.view.ViewGroup/android.view.ViewGroup[5]"));
if(videoIcon.isEnabled()) {
videoIcon.click();
System.out.println("videoIcon Clicked Successfully");
}
OTT Testing Checklist
Performing OTT testing is not that easy as it is a multifaceted platform that requires testing beyond the core functionality. So we have curated this checklist with the vast experience we have gained by being an OTT testing service provider. We have categorized all the checkpoints in an easy-to-follow way.
Core Functionality Testing Checklist:
- Search for a movie by using the title, genre, name of the actors, and so on.
- Filter the results by year, genre, language, age ratings, and so on.
- Check if the thumbnails of the movies or shows are proper.
- Since most users consume such content on their smart televisions, make sure to test the search using voice commands.
- Open one of the movies and shows to see if all the required assets & information such as trailers, crew information, short descriptions, age rating, length, and so on are available.
- If it is a show that has more than one season, ensure that the user has the option to choose the season and that it works properly.
- Play the content and check if the playback functionalities are working as expected.
- Also make sure that additional features like subtitles, language preferences, and so on also work.
- Test to see if the video is playing without any issues in all available quality levels.
- Close the video and reopen it to see if the playback resumes from where you left off.
- Add one or more movies to the watch list and see if it works correctly.
- Check the pay-per-view feature by seeing if the video is available for the user to watch even after the prescribed limit.
- Test if the downloads or offline functionality works as intended.
User Profile & Account Management Checklist:
- Test if user profiles can be created.
- Check if the history, watchlist, and preferences do not interfere between those profiles.
- Ensure the user is able to switch between the profiles whenever necessary.
- Make sure the parental controls cannot be overridden.
- Test if the appropriate content is only available in the kids or restricted mode.
- Access the platform from more than the allowed number of devices concurrently for different plans.
- Test if the limit on the number of offline or downloaded files functionality is working properly.
- Make sure the downloaded content can’t be accessed by the user locally on the device.
- Check if it is possible for a user to switch between the different subscription plans.
Performance Testing
- Check the startup and rebuffer times to see if they are acceptable.
- Test the platform by connecting to networks under different conditions such as lossy networks, low bandwidth networks, and high latency networks.
- Perform load tests for the web page with normal and peak usage.
- Measure and Graph various parameters like throughout, connect latency, request latency, and lost connections.
- Check if there are any video startup or playback failures.
- Make sure to objectively measure the performance by deriving the Quality of Experience (QoE) correctly.
- Identify the breakpoint of the platform by pushing it beyond the limit.
- Check how much time it takes to recover after their crash.
General Checklist:
- Test the installation, launch, and uninstallation of the application.
- Validate the Login and Sign up process.
- Test if the user is able to change or reset the password in a secure way.
- Check if the user is able to make the payment for the subscription using all the possible payment methods.
- Make sure that the banking information of all the users is kept safely by performing security testing.
- Check if the platform works across various browser and operating system combinations.
- Test if the application or platforms works smoothly across different devices such as desktop, laptop, TV, tablets, and mobiles.
OTT Testing Tools
Apart from the regular tools we use across all automation projects, there are no specific or dedicated tools for OTT automation testing. Tools such as Appium and BrowserStack that we use for other projects are the most important tools we use for OTT testing as well.
Appium
As seen in the blog, Appium is a very important tool when it comes to performing OTT Automation testing for Android-powered OTT devices. Though you might be new to OTT automation testing on Android TVs, if you are equipped with the required knowledge on using Appium for Android Mobile app automation, you will get used to it in no time.
BrowserStack
Since there are numerous variations in the OTT devices that are being used in terms of the operating system, form factor, and so on. BrowserStack plays an important role in helping us test across a wide range of devices available on the market. As of writing this blog, BrowserStack has both Android TVs and FireStick available. And based on their website, other popular options such as Roku and Apple TV are on their way to the platform.
Conclusion
We did see how Netflix climbed up the ladder by keeping the user experience at the forefront. That is why it is always important to provide the best user experience along with the best content. OTT testing is the only way to ensure that the best user experience is delivered at all times. So we hope you now have a clear understanding of the importance of OTT testing, how to perform it, and specifically know how to perform OTT Automation Testing as well. Being a top OTT testing company, we will be publishing more informative content and strongly recommend you to subscribe to our newsletter to never miss out on any of our content.
by Anika Chakraborty | May 13, 2022 | OTT Testing, Blog |
Automating an Android smart TV using Appium is not as same as automating an Android smartphone. For starters, the lack of touch input means that you should know how to automate remote control functions. And with the rise of OTT in recent years, knowing how to perform Android TV Automation Testing using Appium will be an important skill every automation tester must possess. Being a leading OTT testing company, we have written this blog that covers how to connect your computer to the smart TV and shows how to automate important actions such as search, video, and so on.
Enable Developer Options in Android TV:
Enabling Developer options is the first step in the Android TV Automation Testing process and it will make it possible to connect your computer with the TV. Being an Android device, enabling this mode is the same as doing it on an Android smartphone.
- Click the ‘Settings’ Icon.
- Go to ‘Device Preference’.
- Go to ‘About’.
- Find the ‘Build Option’ inside the About.
- Click the build option until the “you are now a developer” message will appear in the display.
- Go back to ‘Device Preference’ and scroll to the bottom.
- Select the ‘Developer Options’ that you find there.
- Find the USB Debugging Option and enable it.
Obtain the required information from the Smart TV
Though we need just the IP address of the smart TV to connect it with the computer, getting the Device Name, Platform Name, and Platform Version will come in handy when writing your automation script. So follow the below-mentioned steps
- Click the ‘Settings’ Icon.
- Choose Additional Settings.
- Go to Device Preference.
- Click the About option.
- Get the Device Name, Platform Name, and Platform Version as you will need them for the script.
- You will also see the Status option (Network) in About.
- Click on it and obtain the IP Address of the Android TV.
Connect your Computer with the Smart TV
Now that everything has been set up, you can perform the below-mentioned steps to connect your computer with the Android smart TV.
- Open Command Prompt.
- Enter the Command “adb connect ”
- Now, you can see the list of connected devices in Command Prompt.
Note: You should have configured all the Appium setups on your laptop before initiating the adb command.
Android TV Automation Testing
Testing an OTT platform is no simple task as there are so many aspects such as Core Functionality, UI & UX, Subscriptions, and so on a QA team would have to cover. In this automation guide of ours, we will be exploring how you can automate your core functionality tests such as
Launching the App
Now that we have all the prerequisites ready, let’s find out how to launch the app you wish to automate on your Android Smart TV, and then we will move on to the other actions later in the blog
public class LionsGateDemo {
public static AppiumDriver<MobileElement> driver;
public static WebDriverWait wait;
public static void main(String[] args) throws IOException, InterruptedException {
try {
//Set the Desired Capabilities
DesiredCapabilities caps = new DesiredCapabilities();
//Set the Smart TV Name
caps.setCapability("deviceName", "Redmi TV");
//Set the Smart TV Platform
caps.setCapability("platformName", "Android");
//Set the Smart TV platform version
caps.setCapability("platformVersion", "11");
//Set the App package and app activity details
caps.setCapability("appPackage", "com.xiaomi.lionsgateplay.videoapp");
caps.setCapability("appActivity", "com.parsifal.starz.ui.features.splash.SplashActivity");
caps.setCapability("unicodeKeyboard", true);
caps.setCapability("resetKeyboard", true);
caps.setCapability("noReset", "true");
driver = new AndroidDriver<MobileElement>(new URL("http://127.0.0.1:4723/wd/hub"), caps);
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
System.out.println(">>>> APP launched::::");
wait = new WebDriverWait(driver, 30);
Note: Ensure that you use the exact name of your Smart TV as it is shown in the menu to avoid any errors.
Using Search Functionality
Since we have chosen Lionsgate for our Android TV Automation Testing explanation, we will be automating the user journey to reach the search bar. Kindly note that this user journey might vary depending on the platform or app you wish to automate.
You will also be able to verify if your automation script is working as intended by using our script. We have used the same input text used in the script to search and pick the first movie title manually. We then validate if both the manual and automation results are the same.
// Finding & Clicking button One
WebElement buttonOne = driver.findElement(By.xpath("/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.RelativeLayout/android.widget.FrameLayout/android.widget.RelativeLayout/android.widget.LinearLayout[1]/android.widget.LinearLayout[2]/android.widget.LinearLayout/android.widget.LinearLayout/android.widget.LinearLayout/android.widget.ImageView"));
wait.until(ExpectedConditions.elementToBeClickable(buttonOne));
buttonOne.click();
// Find & Click the bIcon
WebElement bIcon = driver.findElement(By.xpath("/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.RelativeLayout/android.widget.FrameLayout/android.widget.RelativeLayout/android.widget.LinearLayout[1]/android.widget.LinearLayout[2]/android.widget.LinearLayout/android.widget.LinearLayout/android.widget.TextView"));
wait.until(ExpectedConditions.elementToBeClickable(bIcon));
bIcon.click();
wait.until(ExpectedConditions.elementToBeClickable(bIcon));
bIcon.click();
// Locate the Search Bar
WebElement searchBox = driver.findElement(By.id("com.xiaomi.lionsgateplay.videoapp:id/lb_search_text_editor"));
// Enter the values in the Search input textbox
searchBox.sendKeys("a");
// Store the Movie name in a webelement
WebElement firstMovieName = driver.findElement(By.xpath("//androidx.recyclerview.widget.RecyclerView[@content-desc='Top results for: a']/android.widget.FrameLayout[1]/android.widget.ImageView"));
// Wait for the movie to become clickable and then click to open it
wait.until(ExpectedConditions.elementToBeClickable(firstMovieName));
firstMovieName.click();
// Store our expected Movie name in the STRING
String movieName = "Amores Perros";
// Finding Actual Movie name
WebElement OutputMovieName = driver.findElement(By.id("com.xiaomi.lionsgateplay.videoapp:id/titleText"));
// Get and Store our Actual Movie name in String
String OutComeMovie = OutputMovieName.getText();
System.out.println(OutComeMovie);
//Verify Actual and expected Movie name by using if condition
if (movieName.equals(OutComeMovie)) {
System.out.println("Both are same name");
}
} catch (Exception e) {
e.printStackTrace();
}
Playing a Movie
Now that we have successfully completed the search operation, the next course of action would be to automate a movie from the search results. We have given a sample code to help you to do it. Please note that you can use the same concepts we have seen now to play a video not just from search, but also from the home screen and so on. Just make sure to update the XPath as per your need.
// Finding Play button
WebElement playButton = driver.findElement(By.xpath("/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.RelativeLayout/android.widget.LinearLayout[2]/android.widget.LinearLayout/android.widget.LinearLayout/android.widget.LinearLayout/android.widget.FrameLayout[1]/android.widget.FrameLayout/android.widget.TextView"));
playButton.isDisplayed();
// Wait for the Playbutton to be clickable and Click the Playbutton
wait.until(ExpectedConditions.elementToBeClickable(playButton));
playButton.click();
System.out.println("Play button is selected ");
Android TV Test Automation for Remote actions
Since you will be simulating button clicks from a remote control, you should know what element is currently in focus for ease of use. Once you have figured out how to automate the TV Remote actions, you will use them as a foundation to automate more actions like browsing the platform, turning on subtitles, and so on.
// To press the Up button
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_UP));
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_UP));
// To Press the Center button
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_CENTER));
// To Press the Down button
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_DOWN));
// To Press the Center button
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_CENTER));
System.out.println("None option inside the subtitle is selected");
Thread.sleep(5000);
// To Press the Up button
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_UP));
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_UP));
// To Press the Center button
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_CENTER));
// To Press the Up button
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_UP));
// To Press the Center button
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_CENTER));
System.out.println("English option inside the subtitle is selected");
Thread.sleep(5000);
// To Press the Right button for Forward the video
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_RIGHT));
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_RIGHT));
((PressesKey) driver).longPressKey(new KeyEvent(AndroidKey.DPAD_RIGHT));
System.out.println("Forwarding is done successfully");
System.out.println("Remote action done");
}
}
Conclusion
We hope you found this blog useful in helping you perform Android TV Automation Testing for OTT platforms. The above-discussed actions are the foundational and must-know basics in OTT automation testing. You can easily expand your OTT automation by using them effectively. Being a leading QA company in all domains, we will be publishing more OTT testing content that you will find very useful. So make sure to stay subscribed to our newsletter.
by admin | Jan 13, 2022 | OTT Testing, Blog, Latest Post |
Even before the pandemic hit, OTT platforms have been on a rampage in terms of growth and popularity. Netflix has multiplied its user base 10 times over the last 10 years. There is heavy competition amidst the existing big players and the number of OTT platforms entering the market is also on the rise. Even if you have the best content in the market and fail to provide a good user interface and user experience, users will move away. So no OTT platform can afford to have mediocre testing that lets bugs reach the user. As a leading QA company, we deliver uncompromised quality to all our clients and have created this conclusive list of test cases for OTT Platform Testing that will help you achieve maximum coverage.
OTT is an abbreviation for “over-the-top,” which refers to any streaming media service that delivers content via the internet. The term “over-the-top” refers to the fact that the service is delivered on top of another platform. In other words, it is any service that allows you to deliver video and/or live stream feeds to any internet-connected device.
OTT Platform Testing
Robust and exhaustive OTT platform testing is required to successfully deliver video content. There are numerous things to test when it comes to OTT platform testing, but as stated earlier, they boil down to these two major aspects.
- The user interface (UI) allows users to interact with the OTT platform and view the video content.
- The user experience (UX) the person takes away after interacting with the UI.
Let’s dig deeper to get a better understanding of why these two factors are important. Let’s take Netflix as an example as we’ve already discussed how it has grown over the years. In an age that had episodes airing every week, Netflix introduced the binge-watching culture by dropping all of the episodes in one go. They did not want you to leave their platform once you entered.
So instead of just changing the release pattern, they built an intuitive UI that keeps recommending content based on your preferences. But if in case Netflix kept suggesting shows we weren’t interested in, we would just leave. At the same time, even if Netflix was able to provide good recommendations after making the user wait for 2 minutes, nobody is going to wait. That is why it is vital to test how people interact with the product and analyze their experience as well.
As one of the best OTT testing companies, we have created a list of test cases under each category to obtain maximum coverage and quality. We have also mentioned a list of the best testing tools that can also be used as OTT Testing tools. But before we proceed to that, let’s start with the test cases for User Interface and functionality.
UI and Functionality
Playback: Check to see if the video plays/pauses when you click on the video play icon. Also, see if the volume controls, and mute functionality works. Check for audio-video sync as well.
Video Quality: Make sure that the user is able to choose the video playback quality based on their network conditions.
Replay: Check if that the user can replay the video if they wish to see it again once it is finished.
Search: Verify that the video can be found when searched by the movie title, artist names, or relevant keywords that can be found in the description.
Search Results: Check the video’s thumbnail, title, description, length, age rating, etc. and see if the search has relevant autosuggestions. Also, make sure to see if clicking on the video plays the video or opens the content to show additional information.
Voice Commands: Since a major share of the OTT content is consumed via smart televisions, make sure that the voice commands by various assistants also work correctly.
Browse Features: You must be able to filter your search results and content in general so that you can browse through the content without any hassle. Check if the suggested content is based on the user’s viewing history.
History: Once a user starts watching a show or a movie, they must be able to pick up from where they left off and not have to start over again. If it is an OTT platform like YouTube, make sure the history function works well and see if the user is able to remove a video from the history or clear the history on the whole.
Offline Mode: Verify if the user is able to download the content they wish to keep offline and watch it without any internet connection. Also, make sure the downloaded files are not available locally to the user outside the platform.
Bandwidth Capacity: Check how well the content is streamed under different network conditions. If there is limited connectivity, see how well the video is played with lower bitrates.
Direct Access Link: Make sure that the video page can be accessed via a direct link to the video. See if the link can be shared easily using a third-party app.
Browser Compatibility: Verify that both the video and audio play in all flash-enabled browsers. And if it doesn’t, ensure that a meaningful message is shown to the users using a browser without flash or the necessary codec.
Watch list: Add content to a watch later or watch list to see if it gets updated properly. If there are any new additions like a new season to a show in the watch list, make sure it is notified to the user.
Notifications: Other notifications should include the addition of popular and relevant content, release of any content that had a countdown timer or reminder.
Language & Captions: Check if the mentioned captions are available in the content and also see if it in sync with the audio and visuals. If there are other language captions or audios available, then it must be possible to switch easily.
Audio output: Users might be connected to a different audio device via wires or Bluetooth when watching content on a television or mobile. Make sure that the audio is routed to that device properly without any sync issues.
Multitasking
Most of these test cases will be the ones you’d have to cover when testing on mobile platforms. Since OTT platforms employ mobile-only plans, it is an important part of OTT platform testing that can’t be neglected.
Call: Make sure that the video pauses when the user is watching using a mobile device. Also, check if the video resumes from the exact same point once the call is disconnected.
Notifications: Make sure that pop-up notifications don’t pause the playback or disturb it in any way unless the user clicks on it.
Mini Player: If the user does click on the notification, check if the player gets minimized and continues to play. Make sure to drag the mini-player around to see if it works properly. If the mobile or tablet supports split-screen viewing, then make sure the mini player is compatible with that too.
Account
Subscription: Ensure that only users who have paid the subscription fee are able to access the content and stream the video. If there are any ‘Pay per View’ or ‘Video on Demand’ content, test to see if it can be accessed by only those who have the extra payment.
User Profiles: The user must be able to create multiple user profiles and even have separate user histories. Verify if the user is able to select the profile through which he/she wants to watch as soon as the app is launched. There must also be kids or age-restricted profiles which can be accessed by the account with parental rights.
User Count: Make sure the platform is able to detect an account being used in more than the number of allotted screens for the package. Few plans allow different user profiles to access the platform at the same time and not allow the same content to be seen across multiple devices. The same applies to the number of devices that can have offline content. So keep an eye on such factors as well.
Login: Proper account recovery options must be available. But make sure that nobody else is able to access the account without the owner’s knowledge. If the account is being accessed from a new device, make sure that an OTP is sent for verification and enable the option to mark the device as a trusted device. Check if 2-factor authentication works if enabled.
Password: Make sure to suggest the user set a strong password. Ensure that older passwords cannot be reused. If the user wants to change the password, ensure that it is done with an OTP verification. There should also be an option to log out of all the devices.
Information: All the relevant account information like the plan name and so on should be easily accessible in one place for the user to review.
Payment and Security
Since there are both annual and monthly billing options, security features in any OTT platform can’t be taken lightly. User data has to be kept safe and a seamless payment process is very important in providing the best user experience. A user shouldn’t feel burdened or afraid while making the payment.
Successful Payment: Once the payment has been made, ensure that the changes reflect immediately in the account to prevent the user from having any doubts. Also, make sure the user is redirected back to the platform after a successful transaction.
Failure Scenarios: Make sure to think of various failure scenarios and test them out.
- Check what happens if the session ends during the payment process.
- Determine what happens if the payment gateway stops responding during the payment process.
- Make sure the account doesn’t get activated despite having a failed transaction.
- Observe what happens if the payment or transaction goes on hold.
- Check if the session expires within the time limit.
Payment Options: Try making a payment using all the various options like a credit card, debit card, and so on.
Localization: Make sure the amount is displayed in the current currency and the plans for that region are only being purchased. Also, try to change the payment gateway language during the payment process.
Pop-up Blocker: Examine the pop-up blocker settings and observe what happens when the pop-up blocker is turned on and off. At times few payment pages open only in a new window and the blocker might affect that.
Backend: Go beyond the front-end OTT platform testing and analyze what happens in the backend during the payment process.
Buffer Pages: Examine the buffer pages that exist between the payment gateway and the application.
Saved Payment options: If the user wishes to save the payment option, examine the database entries to see if they store credit card information.
Double Checks:
- During the payment process, double-check the error and security pages.
- If you are sending any email for the transaction, ensure that the content is encrypted.
- Verify if all the sensitive information is masked.
- Check that the payment is being made through a secure channel. In other words, the link must begin with HTTPS and not HTTP.
Gateway: Check if the payment is processed with an integrated page or if it is navigated to the bank’s payment page. If it is a page within the platform, make sure the transaction is processed only with the right OTP and card information.
OTT Testing Tools
Though there are no dedicated OTT Testing tools that we use for OTT automation testing, we can still use standard automation testing tools such as Appium and BrowserStack. Now let’s see how these tools will be useful when performing OTT testing.
Appium
Since a major share of OTT devices is powered by Android, Appium can be used to perform OTT Automation Testing. We are not just talking about the wide variety of Android Smart TVs in the market, even devices such as Amazon Firestick run on their own Custom OS (FireOS) that is built with Android Nougat as its base. Since Appium is an already popular testing tool, it can serve as the go-to OTT Testing Tool when it comes to automation.
BrowserStack
We have already seen how Appium can be used to achieve OTT Automation Testing on select devices. Another point of concern when it comes to OTT testing is numerous variations in the OTT devices in terms of an operating system, form factor, and so on. So a cloud testing platform such as BrowserStack is essential in assisting us in testing across a wide range of devices on the market. As an OTT Testing tool, BrowserStack currently supports both Android TVs and FireStick. But according to their website, other popular options like Roku and Apple Tv are on their way to the platform.
Conclusion:
We hope you have found these test cases for OTT Platform Testing to be informative. Though there are many more avenues to explore in OTT platform testing, these are the fundamentals you have to follow. The OTT platform is constantly being updated so that it doesn’t get outdated. Seeing the rise of popularity for TikTok, YouTube has introduced the Shorts feature. Understanding the content for short content, Netflix has created the ‘Fast Laughs’ feature. Netflix is also working on integrating games along with their subscription. So the landscape will always be changing and you should be on a journey of constant learning to keep up with all the new challenges.
Frequently Asked Questions
- What is OTT testing?
OTT testing can be termed as the process of determining the capability of the platform to stream video and audio content over the internet by testing the platform’s UI, functionality, performance, and security.
- What is OTT Automation?
OTT Automation is similar to any type of automation testing as it is the process of using automation tools to maintain test data, execute the tests, and analyze the results to increase the OTT platform’s overall quality.
- What is an OTT Device?
Any device other than a desktop, laptop, tablet, or smartphone that is used to consume OTT content is called an OTT device. Some familiar examples are Smart TVs, Amazon FireStick, Roku TV, Apple TV, etc.
- What is Streaming Testing?
Streaming Testing is used to test if the videos stream with the expected quality & functionality over the internet under different network bandwidths & loads to deliver a great user experience.
- Are OTT and Connected TV the same?
No. OTT is the process of streaming content directly on a computer or mobile device and Connected TVs require the TV to be connected to a device with an active internet connection such as an Xbox, Apple TV, or Roku TV for streaming content.
by admin | Oct 22, 2020 | OTT Testing, Fixed, Blog |
In choosing a form of communication that combines different digital content forms, popularly known as multimedia, streaming media is indisputable. It delivers multimedia directly to end-users from a service provider using basic HTTP, TCP/IP, and HTML protocols. Now, content suppliers invest a lot into testing over-the-top (OTT) media service applications and other segments related to it by using OTT testing.
OTT testing is a streaming media service method that requires the avoidance of traditional telecommunication infrastructures. Some examples of such infrastructure are cable, broadband, and satellite. This has led to a constant increase in the number of people ditching traditional cable providers for online options, such as Netflix and Hulu. Smart televisions are fast becoming the order of the day. Hence, OTT has become the fastest growing media and entertainment segment. Follow through to learn how OTT testing has become very vital to streaming media.
Seven Key Importance of OTT Testing in Streaming Media
Below are seven major parts of OTT testing that makes it an important facility in streaming media.
Stream Testing Stream media companies expect the content delivered through their platform to be as robust as possible. The stream testing process is conducted to include various traffic patterns and loads of data to match this robust expectation. There is the Adaptive Bit Rate (ABR), used as a method by broadcasters to offer multiple streams of different bitrates to the users. ABR is then varied to test the performance of the stream’s selection algorithm.
UI Testing Stream media requires QA (Quality Assurance), that their services are the highest possible quality and fit for users. In ensuring that QA is achieved, thorough User Interface (UI) testing must be conducted. This is very important to stream media because UI is the aspect of OTT testing services that determines how its user perceives the Stream media company. UI is the space where end-users get served. UI testing includes checking how the buttons are set, the audio mixing console, and other digital interface components, including colors and shapes.
User Experience Similar to the UI is the UX, User Experience. While the interface is what is seen, the experience is what is felt. For instance, the practical, experiential, affective aspects of a user’s interaction with a stream media service. All stream media service providers are focused on providing users with amazing user experience. Hence, an OTT testing service geared at the Quality of Experience (QoE) is very important to streaming media.
Interoperability Streaming media are meant for human comfort. This means it should be functional on as many platforms as possible, fitting its user’s pleasure. An OTT test for interoperability is a check for the character of how well the streaming system can be understood. Its understanding of how it works on other products and systems both at present and in the future is specifically what it means that it can interoperate. Hence, a test for Cross-Browser, Cross-Platform, and Cross-Device must be conducted.
Data Security OTT services include account-related data. This account-related data includes payment facilities, what content is being viewed, and other personal user data. This data is sort for by unauthorized personnel, often seeking it for harmful use, cyber theft. Cyber theft could be identity theft, password theft, theft of information, internet time theft, etc. Hence, the security of these details is fundamental while launching an OTT service. Data security is not limited to preventing personal data from cyber theft. However, it includes preventing account-related data from destructive forces such as a virus.
Network Testing As the digital ecosystem is expanding, especially in media streaming, it will not be a surprise to see network issues. By using a tool to simulate network conditions, Network Condition Simulator, service behavior can be mapped. Mimicking the real environments in Streaming Media is a handy tool for identifying low bandwidth, high bandwidth, and other network conditions. This testing is also key to identifying the Internet connection through locations worldwide. Network tests can be used to achieve Ethernet speed, higher port densities, network switches, and routers.
Performance Testing Since there will be no fewer than millions of stream requests nationwide, it is essential to carry out the Load Testing over one’s OTT network server. This will help determine the streaming performance in terms of processing speed, data transfer rate, scalability, reliability, and stability. Under the expected workload, which is also estimated to vary, the streaming media’s proper performance can be ensured or updated if need be.
Conclusion
OTT testing can never be undermined in the exponential growth being recorded in the media sector. For software test engineers and software QA managers, OTT testing services could include OTT plans or subscription tests. Because providing customized entitlements to users must always be intact. Giving a user less or more than their entitlement could spell revenue loss for the OTT service provider. Therefore, there must be OTT testing and how its subscription login can be supported on one or more platforms. The test must also include channels for which the subscription can be either HD, UHD, or SD. With these advantages, stream media as an industry has its eyes set for the stars.
Call to Duty
Having known the importance of OTT testing to streaming media, it is high time you joined us at Codoid to enjoy our OTT testing services. We have a very passionate and well-experienced team that is ever ready to deliver top-notch QA testing. Our services include Automation Testing, Mobile App Testing, Performance Testing, Data Analysis Testing, API & Webservice Testing, VR & AR App Testing. While we look forward to you being part of our intercontinental community, feel free to ask us for help. We are at your service every time because we believe that “A growing product needs a passionate, dedicated QA team to ensure quality.”
