ETL Data Quality Testing Best Practices

Why do my ETL data quality testing techniques fail to identify bad data?

Bad data costs organizations dearly in correction activities, lost customers, missed opportunities, and incorrect decisions.

How to perform ETL Data Quality Testing to find bad data?

Impact of poor data

• Poor data can lead to bad business decisions
• Delays in delivering data to decision makers
• Lost customers through poor service

ETL Testing Techniques

• Number of Records Validation
• Data Completeness
• Not Null Validation
• Validate valid values
• Frequency Distribution
• Min and Max Validations
• Duplicate Records
• Pattern Check
• Consistency
• Precision
• Timeliness
• Business Rules
• Data Type Check
• Size & Length Validation

A firm’s basis for competition . . . has changed from tangible products to intangible information. A firm’s information represents the firm’s collective knowledge used to produce and deliver products and services to consumers. Quality information is increasingly recognized as the most valuable asset of the firm. Firms are grappling with how to capitalize on information and knowledge. Companies are striving, more often sliently, to remedy business impacts rooted in poor quality information and knowledge.

– Kuan-Tsae Huang, Yang W. Lee and Richard Y. Wang