Select Page

Automated ETL Testing that Eased Decision Making

Our incredible success story of how we went from being hired to perform only manual testing to implementing truly effective Automated ETL testing that enabled the stakeholders to identify key problems and make educated decisions.
Automated ETL Testing Case Study
Highlights
  • 4

    Automation Test Engineers

  • 500+

    Tables Tested with Automation

  • 50+

    Databases tested

  • 7+

    Years Partnership

Tools Used
  • Snowflake Icon

    Snowflake for Data Management

  • LemonCheeseCake Icon

    LemonCheesecake for Python Automation Framework

  • Looker Icon

    Looker for Data Profiling

  • Splunk Icon

    Splunk for Logs Monitoring

  • DBT

    DBT for Testing Models

  • Zephyr Icon

    Zephyr for Test case preparation

  • Jenkins Continuous Integration Tool

    Jenkins for CI/CD Integration

  • Airflow Logo

    Apache Airflow for CI/CD Integration

  • JIRA Logo

    JIRA for Project Management in Agile

The Challenge

Cengage, a US-based educational content, technology, and services company that operates in more than 20 countries across the world was in need of an offshore software testing team to perform manual ETL testing. Since they provided services such as online classes, assignments, exams, and so on, there were individual reports for every book, student, instructor, course, platform, etc. So our primary goal was to identify issues such as source duplication, null values, and target duplication that were impacting the quality of their data.

 

Our Solutions

  • Introduced automation testing to save countless man-hours.
  • Set up a Robust Reporting framework using Lemon Cheesecake & Snowflake.
  • Achieved CI/CD Integration by using Jenkins.
  • Improved Data Quality by introducing Data Profiling with the help of Looker.
  • Followed Agile methodology & supported every release without any delay.
Introducing Automation

Though we were tasked with performing only manual testing, we identified a scope for automation testing that could significantly streamline the entire process and save a lot of time and cost. So we wrote our automation script in LemonCheesecake framework using Python and demoed it to our clients. They were immediately impressed with the results and engaged us for automation testing as well.

As firm believers in continuous improvement, we never stopped improving our testing process. With the rise of DBT’s popularity, we identified its true potential and successfully implemented it in our project.

CI/CD Integration in an Agile environment

As our client followed Agile methodology, we had a release every sprint that would last 2 weeks. Over the course of our 7-year partnership, we are happy to claim that we have never been late on our deliverables. We never missed a deadline even if when we had to test a single database that was more than 3 TB in size. We achieved CI/CD Integration in this project by using tools such as Jenkins and Apache Airflow.

Our Reporting Pattern

Tracking the history of failures was an important necessity. So we opted to use the Lemon Cheesecake tool to achieve full integration with Snowflake as it helped us report the number of test cases passed and failed on a daily basis. In addition to that, we also ensured the test results were directly ingested into the database to keep track of the results with ease.

Data Profiling with Looker

Data was flowing to their databases from different sources such as the web, mobile, and so on. So with the implementation of data profiling, we were able to identify quality issues that would usually go undetected if we tested only the source and target databases. So we used Looker to create dedicated dashboards that would display the automation testing results for aspects such as level of ebook usage, number of subscriptions, activations, and so on.

Talk to our Experts

    Amazing clients who trust us
    https://codoid.com/wp-content/uploads/2022/07/pan-logo-light-1.webp
    abb Codoid client
    https://codoid.com/wp-content/uploads/2022/07/logo-polaris-blue-1.webp
    https://codoid.com/wp-content/uploads/2022/07/ooredoo_logo-1.webp
    https://codoid.com/wp-content/uploads/2022/07/Stryker_Logo.webp
    https://codoid.com/wp-content/uploads/2022/07/Mobily_logo-1.webp