How to Perform OTT Automation Testing on All Devices?

Content is King is a very famous quote that we all would have heard on so many occasions. Though it applies to OTT platforms, there is no denying that user experience is also an integral part of an OTT platform’s success. It is important to note that the technical aspects of the OTT platform also have to be strong to ensure the best user experience as nobody will be willing to watch even the greatest content if it is constantly buffering and taking forever to respond to your action. The only way to ensure that the platform delivers the best user experience is by performing OTT testing. So in this blog, we will be seeing how to perform OTT Automation testing across major devices and see a conclusive checklist as well.

What is OTT Testing?

An OTT (Over the Top) service is any type of streaming service that delivers content over the internet. Though there is a wide range of OTT services such as WhatsApp, Skype, Zoom, and so on, our focus will be entirely on OTT media services such as Netflix, Amazon Prime, etc. So OTT testing can be defined as the process of performing robust tests to ensure the successful delivery of video content such as films or tv shows with the best user experience.

Did You Know?

A very fascinating aspect of OTT testing and Netflix is that Netflix developed its own testing tool called Chaos Monkey to test the resilience of its Amazon Web Services. The concept behind Chaos monkey was that it would intentionally disrupt the service by terminating key instances in production to see how well the other systems managed.

But why is the OTT giant so much focused on testing? To answer this question, we would have to take a look back in time to find out how Netflix and other OTT services emerged.

The Emergence of OTT

The popularity of OTT services has skyrocketed over the roof this past decade. The pandemic only added more fuel to the flame as more and more people started consuming content on OTT platforms. This type of media consumption is actually very different from its predecessors and competition. For example, VCRs, CDs, and DVDs did not need any software or platform, or infrastructure from the vendor’s side. So the end-user experience was majorly dependent on the consumer side infrastructure. Likewise, the end-user experience varied from theatre to theatre. But OTT is very different from all these options as apart from providing the content, the user experience is purely dependent on their services as well.

Netflix is so popular that the name has become synonymous with the term OTT itself. But how was Netflix able to topple Blockbuster video, a company that was still generating $5.9 Billion annually in 2004? The answer is user experience. Blockbuster was making around 16% of its yearly income through late fees. No customer was happy with this working model. In fact, one of the co-founders of Netflix, Reed Hastings had decided to start Netflix after receiving a $40 fine from Blockbuster. So Netflix was born as a company that did not have any late fees.

The User-Experience Factor

You might be wondering how is this story relevant to OTT testing. To answer that question, let’s take a look at the very first Netflix original series, House of Cards. Netflix revolutionized how content will be consumed by releasing the entire series in one shot instead of following the regular weekly one-episode plan. This meant that when the end-user wanted to watch the next episode, they could. The users also had the liberty to watch the show they wanted when they wanted instead of depending on the network’s schedule. So ultimately Netflix made sure that the end-user never left their platform using the user experience they had crafted. People started binge-watching content and Netflix’s strong user experience ensured that it happened.

Now that the infinity scrolling feature for video content has become widely popular, Netflix has introduced a feature called ‘Short Laughs’ that showcases short clips from their best shows. Netflix takes their user experience very seriously as they even developed a Chaos monkey to test their platform’s resilience against vulnerabilities. So if Netflix, the most successful OTT service provider is so focused on testing its platform to ensure the best user experience, the importance of OTT testing can clearly be understood here. So it goes without saying that every OTT platform should be tested for its functionality, performance, etc, and that OTT Automation Testing is an integral part of achieving these goals.

Most Popular OTT Platforms Across the World

After the success of Netflix, many wanted to jump on the bandwagon. And after a flurry of various attempts, these are the top OTT platforms that are popular as of writing this blog.

Most Popular OTT Devices Across the World

One of the major advantages of OTT services is that they are not limited to just the TVs we use like the old-fashioned cable connection. Instead, we can view and enjoy the content on a wide range of devices such as mobile phones, tablets, computers, smart televisions, and so on. Even if your television is not a smart TV, you can still use plug-and-play add-ons such as Apple TV, Amazon Firestick TV, Roku, etc. to access all your favorite OTT platforms.

How to do OTT Automation Testing?

Performing manual testing on platforms with such a huge scale can be a cumbersome task. That is why it is important to implement automation in your OTT testing. As a pioneer in the OTT testing arena, our R&D team had worked hands-on with all the above-mentioned devices to help us render high-quality OTT Automation testing services from the get-go.

First up, we have to connect our computers to the smart tv that we are looking to automate. But since smart TVs are not the only way the OTT content can be consumed, we would be in a position to connect our computers to the various OTT devices such as Apple TV, Amazon FireStick, Roku, and so on.

Android TV Automation

On the surface level, automating an Android app on mobile and on an Android TV might seem to be the same. Though there are a lot of similarities, Android TV doesn’t have a touch input system and has to be operated with a remote. So we would have to keep this in mind when writing the automation scripts.

Required Prerequisites

Step 1: Enable Developer Options by going to the Settings.

Step 2: Obtain the Device Name, Platform Name, Platform Version, and IP address of the TV.

Step 3: Configure your computer with all the required Appium setups.

Step 4: Open Command Prompt and enter the command ‘adb connect ’ to see the list of the connected devices.

Step 5: Run the automation script.

Automation Script

We are now going to see a sample automation script that can be used to launch the application and play a video.

public class LionsGateInstaller {
    public static AppiumDriver<MobileElement> driver;
    public static WebDriverWait wait;
   public static void main(String[] args) throws IOException, InterruptedException {
        try {
            //Set the Desired Capabilities
            DesiredCapabilities caps = new DesiredCapabilities();
       //Set the Smart TV Name
            caps.setCapability("deviceName", "My Redmi TV");
       //Set the Smart TV Platform
            caps.setCapability("platformName", "Android");
       //Set the Smart TV platform version
            caps.setCapability("platformVersion", "11");
       //Set the App package and app activity details
            caps.setCapability("appPackage", "com.xiaomi.lionsgateplay.videoapp");
            caps.setCapability("appActivity", "com.parsifal.starz.ui.features.splash.SplashActivity");
            caps.setCapability("unicodeKeyboard", true);
            caps.setCapability("resetKeyboard", true);
            caps.setCapability("noReset", "true");
            driver = new AndroidDriver<MobileElement>(new URL("http://127.0.0.1:4723/wd/hub"), caps);
            driver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS);
            System.out.println(">>>>  APP Launched Successfully::::");
            wait = new WebDriverWait(driver, 30);

If you would like to see more sample scripts for various other actions such as performing a search and remote actions, you could visit our related blogs.

Amazon FireStick Automation

If your TV is not a smart TV running on Android or any other equivalent operating system but has an HDMI port, you can still use devices such as Amazon FireStick to consume the OTT content. So based on the device that you use, you would have to follow different steps to automate the OTT testing process.

Step 1: Enable Developer Options by going to the Settings.

Step 2: Obtain the Device Name, Platform Name, Platform Version, and IP address of the Amazon FireStick in use.

Step 3: Configure your computer with all the required Appium setups.

Step 4: Open Command Prompt and enter the command ‘adb connect’ to see the list of the connected devices.

Step 5: Run the automation script.

Automation Script:

public class Aha_OTT {
    public static AppiumDriver driver;
 
    public static void main(String[] args) throws MalformedURLException, InterruptedException {
 
        //Set the Desired Capabilities for connect my smart to the Laptop.
        DesiredCapabilities caps = new DesiredCapabilities();
      //Set the Smart TV Name
        caps.setCapability("deviceName", "My Fire TV");
        //Set the Smart TV Platform
        caps.setCapability("platformName", "Android");
 
       //Set the Smart TV platform version
        caps.setCapability("platformVersion", "7");
        caps.setCapability("appPackage", "ahaflix.tv");
        caps.setCapability("appActivity", "ahaflix.tv.MainActivity");
        caps.setCapability("noReset", "true");
        driver = new AndroidDriver(new URL("http://127.0.0.1:4723/wd/hub"), caps);
        driver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS);
        WebDriverWait wait = new WebDriverWait(driver, 30);
        System.out.println(">>>>App Launched Successfully::::");
        Thread.sleep(25000);
//Find the Top Picks Content of the Platform and Print that Name 
        WebElement Movie Title = driver.findElement(By.xpath("//android.widget.TextView[@resource-id='Top Picks']"));
        if (Movie Title.isDisplayed()) {
            System.out.println("Top Pick Successfully Verified");
        }
       // To Press the Left button by using Remote Action
        ((PressesKey) driver).pressKey(new KeyEvent(AndroidKey.DPAD_LEFT));
        Thread.sleep(10000);
// Finding & Click the Video Icon
        WebElement videoIcon = driver.findElement(By.xpath("//android.widget.FrameLayout[@resource-id='android:id/content']//android.widget.ScrollView/android.view.ViewGroup/android.view.ViewGroup[5]"));
        if(videoIcon.isEnabled()) {
          videoIcon.click();
            System.out.println("videoIcon Clicked Successfully");
        }

OTT Testing Checklist

Performing OTT testing is not that easy as it is a multifaceted platform that requires testing beyond the core functionality. So we have curated this checklist with the vast experience we have gained by being an OTT testing service provider. We have categorized all the checkpoints in an easy-to-follow way.

Core Functionality Testing Checklist:

• Search for a movie by using the title, genre, name of the actors, and so on.
• Filter the results by year, genre, language, age ratings, and so on.
• Check if the thumbnails of the movies or shows are proper.
• Since most users consume such content on their smart televisions, make sure to test the search using voice commands.
• Open one of the movies and shows to see if all the required assets & information such as trailers, crew information, short descriptions, age rating, length, and so on are available.
• If it is a show that has more than one season, ensure that the user has the option to choose the season and that it works properly.
• Play the content and check if the playback functionalities are working as expected.
• Also make sure that additional features like subtitles, language preferences, and so on also work.
• Test to see if the video is playing without any issues in all available quality levels.
• Close the video and reopen it to see if the playback resumes from where you left off.
• Add one or more movies to the watch list and see if it works correctly.
• Check the pay-per-view feature by seeing if the video is available for the user to watch even after the prescribed limit.
• Test if the downloads or offline functionality works as intended.

User Profile & Account Management Checklist:

• Test if user profiles can be created.
• Check if the history, watchlist, and preferences do not interfere between those profiles.
• Ensure the user is able to switch between the profiles whenever necessary.
• Make sure the parental controls cannot be overridden.
• Test if the appropriate content is only available in the kids or restricted mode.
• Access the platform from more than the allowed number of devices concurrently for different plans.
• Test if the limit on the number of offline or downloaded files functionality is working properly.
• Make sure the downloaded content can’t be accessed by the user locally on the device.
• Check if it is possible for a user to switch between the different subscription plans.

Performance Testing

• Check the startup and rebuffer times to see if they are acceptable.
• Test the platform by connecting to networks under different conditions such as lossy networks, low bandwidth networks, and high latency networks.
• Perform load tests for the web page with normal and peak usage.
• Measure and Graph various parameters like throughout, connect latency, request latency, and lost connections.
• Check if there are any video startup or playback failures.
• Make sure to objectively measure the performance by deriving the Quality of Experience (QoE) correctly.
• Identify the breakpoint of the platform by pushing it beyond the limit.
• Check how much time it takes to recover after their crash.

General Checklist:

• Test the installation, launch, and uninstallation of the application.
• Validate the Login and Sign up process.
• Test if the user is able to change or reset the password in a secure way.
• Check if the user is able to make the payment for the subscription using all the possible payment methods.
• Make sure that the banking information of all the users is kept safely by performing security testing.
• Check if the platform works across various browser and operating system combinations.
• Test if the application or platforms works smoothly across different devices such as desktop, laptop, TV, tablets, and mobiles.

OTT Testing Tools

Apart from the regular tools we use across all automation projects, there are no specific or dedicated tools for OTT automation testing. Tools such as Appium and BrowserStack that we use for other projects are the most important tools we use for OTT testing as well.

Appium

As seen in the blog, Appium is a very important tool when it comes to performing OTT Automation testing for Android-powered OTT devices. Though you might be new to OTT automation testing on Android TVs, if you are equipped with the required knowledge on using Appium for Android Mobile app automation, you will get used to it in no time.

BrowserStack

Since there are numerous variations in the OTT devices that are being used in terms of the operating system, form factor, and so on. BrowserStack plays an important role in helping us test across a wide range of devices available on the market. As of writing this blog, BrowserStack has both Android TVs and FireStick available. And based on their website, other popular options such as Roku and Apple TV are on their way to the platform.

Conclusion

We did see how Netflix climbed up the ladder by keeping the user experience at the forefront. That is why it is always important to provide the best user experience along with the best content. OTT testing is the only way to ensure that the best user experience is delivered at all times. So we hope you now have a clear understanding of the importance of OTT testing, how to perform it, and specifically know how to perform OTT Automation Testing as well. Being a top OTT testing company, we will be publishing more informative content and strongly recommend you to subscribe to our newsletter to never miss out on any of our content.