Automated ETL Testing that Eased Decision Making
![Automated ETL Testing Case Study Automated ETL Testing Case Study](https://codoid.com/wp-content/uploads/2022/11/Automated-ETL-Testing-Case-Study.jpg)
Highlights
-
4
Automation Test Engineers
-
500+
Tables Tested with Automation
-
50+
Databases tested
-
7+
Years Partnership
Tools Used
-
Snowflake for Data Management
-
LemonCheesecake for Python Automation Framework
-
Looker for Data Profiling
-
Splunk for Logs Monitoring
-
DBT for Testing Models
-
Zephyr for Test case preparation
-
Jenkins for CI/CD Integration
-
Apache Airflow for CI/CD Integration
-
JIRA for Project Management in Agile
The Challenge
Cengage, a US-based educational content, technology, and services company that operates in more than 20 countries across the world was in need of an offshore software testing team to perform manual ETL testing. Since they provided services such as online classes, assignments, exams, and so on, there were individual reports for every book, student, instructor, course, platform, etc. So our primary goal was to identify issues such as source duplication, null values, and target duplication that were impacting the quality of their data.
Our Solutions
- Introduced automation testing to save countless man-hours.
- Set up a Robust Reporting framework using Lemon Cheesecake & Snowflake.
- Achieved CI/CD Integration by using Jenkins.
- Improved Data Quality by introducing Data Profiling with the help of Looker.
- Followed Agile methodology & supported every release without any delay.
Introducing Automation
Though we were tasked with performing only manual testing, we identified a scope for automation testing that could significantly streamline the entire process and save a lot of time and cost. So we wrote our automation script in LemonCheesecake framework using Python and demoed it to our clients. They were immediately impressed with the results and engaged us for automation testing as well.
As firm believers in continuous improvement, we never stopped improving our testing process. With the rise of DBT’s popularity, we identified its true potential and successfully implemented it in our project.
CI/CD Integration in an Agile environment
As our client followed Agile methodology, we had a release every sprint that would last 2 weeks. Over the course of our 7-year partnership, we are happy to claim that we have never been late on our deliverables. We never missed a deadline even if when we had to test a single database that was more than 3 TB in size. We achieved CI/CD Integration in this project by using tools such as Jenkins and Apache Airflow.
Our Reporting Pattern
Tracking the history of failures was an important necessity. So we opted to use the Lemon Cheesecake tool to achieve full integration with Snowflake as it helped us report the number of test cases passed and failed on a daily basis. In addition to that, we also ensured the test results were directly ingested into the database to keep track of the results with ease.
Data Profiling with Looker
Data was flowing to their databases from different sources such as the web, mobile, and so on. So with the implementation of data profiling, we were able to identify quality issues that would usually go undetected if we tested only the source and target databases. So we used Looker to create dedicated dashboards that would display the automation testing results for aspects such as level of ebook usage, number of subscriptions, activations, and so on.
Talk to our Experts
trust us
![Palo Alto Logo Palo Alto Logo](https://codoid.com/wp-content/uploads/2023/03/Palo-Alto-Logo.webp)
![Abb Logo Abb Logo](https://codoid.com/wp-content/uploads/2023/03/ABB-Logo.webp)
![Polaris Logo Polaris Logo](https://codoid.com/wp-content/uploads/2023/03/Polaris-Logo.webp)
![Ooredoo Logo Ooredoo Logo](https://codoid.com/wp-content/uploads/2023/03/Ooredoo-Logo.webp)
![Stryker Logo Stryker Logo](https://codoid.com/wp-content/uploads/2023/03/Stryker-Logo.webp)
![Mobily Logo Mobily Logo](https://codoid.com/wp-content/uploads/2023/03/Mobily-Logo.webp)