Modern-day systems are transforming into complex, open-source, cloud-native services running on various environments and being developed/deployed at lightning speed by distributed teams. When working on these systems, identifying a broken link in the chain can be near impossible. Everything fails at one point or another, whether due to code bugs, infrastructure overload, or changes in end-user behavior or market driven factors or errors in data collection. This has led to the rise of DataOps with a focus on changing the organizational speed and trust in delivering data pipelines and the related artifacts by co-creating “decision quality” data with the consumers. This development has led to the idea of observability that includes monitoring, tracking, and triaging incidents to prevent downtime of the systems and around several factors such as freshness, distribution, volume, schema, lineage.
As part of this session, Raj Joseph, CEO of DQLabs shall present how DQLabs.ai uses AI to use for various use cases around DataOps / Data Observability.