Just as living organisms grow, evolve, and adapt to their environments, data undergoes continual transformation and refinement to suit changing needs and demands. This evolution extends from structuring raw data into a standardized table format, to generating interactive visualizations for exploring complex datasets, and beyond. As data undergoes transformations, it is crucial not only to keep context as expressed by the origin but also to maintain consistency and completeness. This emphasizes the need for a structured approach to enable seamless data governance through analysis of data in flight.
There are different ways to transform data, usually referred to as extract-transform-load (ETL) all of which impact governance. Extracting data involves retrieving data from its source system, like a legacy database or a file. Data extraction is a time-consuming process, and it’s important to validate the extracted data to ensure accuracy and completeness. Performing validation while operating within the source system is helpful because it helps you avoid potential confusion caused by later computed results. Additionally, as you progress, you may lose the context of the source data. Next stage is transforming data which involves normalizing the data, which includes tasks like eliminating outliers, joining data from multiple sources, aggregating, or splitting columns. However, early normalization may remove valuable information, so it’s crucial to consider business context and potential future use cases. The Final step is loading the transformed data into its destination, typically a data analytics warehouse. Throughout these processes, maintaining the context, consistency, and completeness and trustworthiness of the data is crucial.
It is important to understand the role of lineage throughout the aforesaid processes. Lineage is the recording of the path that data travels through extraction, transformation, loading, and other movements, as new datasets and tables are created, discarded, restored, and generally used throughout the data lifecycle. As data travel within the data lake, they intermingle and interact with other data sources to generate insights. However, the metadata—information about data sources, quality, and whether the data contain Personally Identifiable Information (PII) is at risk of being lost as data move. Ideally, data sensitivity, quality, and other information available from the origin should filter down to the final data product. This metadata supports decisions about whether to allow certain data products to mix, grant access to the data, and to whom, and so on. When mixing data products, it’s crucial to keep track of their origins. Lineage is therefore essential for an effective data governance strategy.
Governance of Data in Flight
If an organisation doesn’t know where data comes from and how it has changed as it moves between systems, then the organisation cannot prove that the data represent what it claims it represent Using lineage effectively empowers organizations to govern data in flight, addressing issues ranging from debugging to compliance. Let’s delve into how lineage facilitates key aspects of data governance and management.
Understanding Data Changes
Lineage captures the sudden changes in data behavior. Picture a situation where a dashboard unexpectedly displays inaccuracies. By tracing the lineage, stakeholders can pinpoint the source of the issue, whether it’s a data transformation error or missing fields. Understanding the path and transformations of the data can help troubleshoot data transformation errors.
Inferring Data Policies
Field-level lineage enables the inference of data-class-level policies, particularly crucial for managing sensitive information like Personally Identifiable Information (PII). With reliable lineage, organizations can automatically propagate policies from source columns to their derivatives. Whether it’s enforcing access controls or data masking, lineage ensures consistent policy enforcement across data transformations, mitigating risks associated with data misuse or exposure.
Compliance Requirements
In a post-GDPR world, if you mark all sources of PII, you can leverage the lineage graph to identify where PII gets processed, allowing a new level of control. Lineage empowers organizations to identify systems processing sensitive data, facilitating audits and compliance reporting. By tracing the lineage of PII, organizations can ensure adherence to regulatory mandates, establishing an unbroken “chain of trust” from data acquisition to decision-making processes.
Policy Management and Change
Efficient data governance relies on flexible policy management and smooth adjustment to change. Lineage plays a crucial role in policy formulation, allowing for automatic identification of data types and related regulations. Additionally, when making changes like data deletion or updates to access policies, lineage offers valuable understanding of how these changes affect various data products and systems. This proactive management of change reduces disruptions and keeps stakeholders well-informed about upcoming adjustments.
Enabling Auditability and Transparency
In audits and compliance checks, this lineage information is crucial for verifying the legitimacy of decision-making processes and confirming the origin of data. It helps auditors and regulators ensure that organizations are following the rules and maintaining ethical standards in handling data. Having a transparent lineage trail enhances an organization’s credibility and simplifies the audit process, showcasing their commitment to regulatory compliance and ethical data management.
Conclusion
Lineage empowers organizations to govern data effectively throughout its lifecycle. From debugging anomalies to ensuring regulatory compliance. With a lineage, organizations can trace data changes and life cycle, enhancing control and allowing a complete picture of the various systems involved in data collection and manipulation.
While lineage is essentially a technical construct, it’s important to always consider the ultimate business goal. This goal could involve ensuring that decisions are made with high-quality data. For business users, governing data while it is “in flight” through lineage allows a measure of trustworthiness to be inherited from trusted sources or processors along the data path. This enriched context provided by lineage facilitates better decision-making.
Lineage across SCIKIQ is split into a Technical and Business Lineage. The technical lineage covers off the relationships and transformations between various entities within SCIKIQ, whereas the Business Lineage captures the hops of data from record to report. It not only links all the physical components like dashboards, reports, table name but also logical constructs like dataset, catalogues, domains associated together to describe a complete lifecycle of information. SCIKIQ intelligently captures the journey data undergoes from its creation to its transformation over time. It identifies and highlights points of change, illustrating how data adapts and evolves into a diagram that is easy to understand, interpret, and track. This functionality aids users in making informed decisions.
