Data Quality Lifecycle

We define the Data Quality Life cycle in these simple six steps –

• Connect to Multiple Sources – Ability to connect to a wide variety of data sources with multiple options e.g., scan, pull data with or without metadata etc., This can also be extended with the ability to interpret semantics or business context by leveraging your existing data catalog or governance systems’ glossary.
• Discover Semantics – Understand and classify data from a business context. Is the data a phone number, aSSN or a loan origination number? This identification is critical not only for business validation but also for detecting any false positives during forecasting/benchmarking/outlier or anomaly detection. This also enables auto discovery data quality checks allowing all stakeholders to manage expectations and strive for consensus within organizations.
• Measure Data Quality – Measure, score and identify bad data using auto discovered rules across various attributes. Our platform boasts the ability to measure at the attribute level which provides the flexibility to cumulatively measure at a data set, data source, department/function or organizational level. Our platform provides a score that can be understood across all stakeholders and can be used for search,relevance, as well as for discovery of assets.
• Monitor and Alert using Adaptive Thresholds – Ability to set adaptive thresholds without the need for manual rules using benchmarking or forecasting trends. Cover a wide variety of DataOps, Data Observability use cases such as data pipeline monitoring, source to target checks, schema or data level deviations or abnormalities.
• Remediate to Improve Data Quality – Use a set of curation libraries to clean as much as possible automatically. This is also extended with remediation workflows, issue management with third party productivity and collaboration platforms such as Jira, ServiceNow and many more.
• Derive Insights and Recommendations – Ability for both business and technical stakeholders to slice and dice to make sense of the bad data in their own ways. This is particularly useful to generate next best actions both strategic and tactically.

Without a focus on the entire data quality life cycle, organizations will never succeed in siloed or secondary data quality initiatives or outlier detection-based monitoring.