A small amount of high-quality data yields much better results than does a large amount of low-quality or outright wrong data. When people hear “data quality,” they usually think of information that’s right and true. But in the world of data analytics and governance, it’s a bit more complicated. Just being right isn’t good enough if we don’t have all the details. In the world of handling lots of data and using computers to do tasks, it’s crucial to grasp the finer points of making sure the data is of good quality.

Understanding Data Quality: The ABCs

Let’s delve into the banking industry to illustrate the importance of accuracy, completeness, and timeliness in ensuring data quality.

Accuracy: In banking, accuracy is crucial for financial integrity. A misplaced decimal point during interest rate entry can cause miscalculations, impacting interest payments and financial statements. System errors leading to duplicate transactions can create confusion, affecting the accuracy of account balances

Completeness: For banks, completeness is akin to having a full financial profile for each customer. Capturing all details, from personal information to transaction history, is crucial. Incomplete records can hinder risk assessment, personalized services, and regulatory compliance.

Timeliness: Timeliness is a key factor in the banking world, especially concerning transaction processing and reporting. Recording loan payments promptly ensures accurate customer statements. Delays in updating account information can lead to discrepancies, affecting financial planning. Recognizing the time-sensitive nature of financial data is essential for trust and regulatory compliance in banking.

In addition, the data quality can be affected by outlier values.

Outlier Values: In the vast landscape of retail transactions, identifying outliers is crucial. Big purchases that might signal data mistakes, such as missing decimal points, not real profit boosts. We need to look at all kinds of values, even negative ones showing returns, to understand things better.

Finally, there is the trustworthiness of the source of data.

Source Trustworthiness: Not all data sources are born equal. Data from machines, precise and in sync with the global clock, is different from handwritten notes, which can have mistaken. Mixing data from these different sources needs careful thinking to avoid treating them as if they’re all equal.

Costly Toll of Bad Data

The yearly costly toll of bad data is estimated to be $3.1 trillion, according to IBM. Studies estimate that the cost of unmanaged data quality can be a staggering 15-20% of revenue for most companies. From an astonishing 50% error rate in sampled records to the annual cost of “bad data” reaching trillions, the financial implications of poor data quality are profound.

Good data quality is really important for many organizations. It helps them make decisions, like when a banker decides whether to approve a mortgage based on a person’s credit score from their transaction history. Similarly, a company’s share price is figured out quickly based on the amount offered by multiple buyers and sellers. These kinds of decisions are very often regulated —clear evidence must be collected on credit-related decisions. It is important to the customer and the lender that mortgage decisions are made according to high-quality data. Lack of data quality is the source of lack of trust and of biased, unethical automated decisions being made.

Challenges

When you gather information from different places or areas, keeping the data accurate and understanding its context becomes tricky. It’s not just about the main storage place understanding the data in the same way as where it came from like how they define unusual values or treat incomplete data. There’s also the chance that the sources of data don’t agree on what certain values mean (like what to do with negative numbers or how to fill in missing ones). To solve this, you can make sure that when you add a new data source, you check if the data is accurate, complete, and timely. Sometimes you might need to do this manually, describing it in a way that other analysts can use, or you can directly adjust it according to the rules in the main storage place.

When an error or unexpected data is introduced into the system, there are usually no human curators who will detect and react to it. In a data processing pipeline, each step can introduce and amplify errors in the next steps, until presented to the business user. In the intricate data acquisition chain, errors at various stages can amplify, resulting in inaccurate data presented to business users. From gathering data from low-quality sources to aggregating wrong data and joining tables of different qualities, each step requires careful attention to prevent skewed outcome.

As organizations delve into the domain of big data analytics, data warehouses become pivotal. However, these massive databases, handling petabytes of information, are not immune to data quality challenges. Extracting, transforming, and loading data from diverse sources demands careful consideration of data quality at every step.

Significance Beyond Technical Boundaries

Proper data quality management isn’t just a necessity for running extensive data analyses; it’s a cornerstone for cost savings and productivity preservation. The repercussions of poorly managed data quality extend far beyond the technical realm, influencing business decisions and operational outcomes. In the complex world of data, making sure it’s good quality is like finding the key to unlock all the amazing possibilities of information in today’s digital age.

Ensuring data quality means verifying that the data is accurate, complete, and timely according to the specific business use case. Various business needs require different standards for accuracy, completeness, and timeliness. It is crucial to maintain a scorecard for each data source, especially when building an analytics workload that relies on data from these sources and their descendants. This approach ensures that the data aligns with the specific requirements of the intended business use, allowing for more effective decision-making and analysis.