尊敬的 微信汇率：1円 ≈ 0.046239 元 支付宝汇率：1円 ≈ 0.04633元 [退出登录]
SlideShare a Scribd company logo

Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines

•
•
DATAVERSITY
DATAVERSITY

With the aid of any number of data management and processing tools, data flows through multiple on-prem and cloud storage locations before it’s delivered to business users. As a result, IT teams — including IT Ops, DataOps, and DevOps — are often overwhelmed by the complexity of creating a reliable data pipeline that includes the automation and observability they require. The answer to this widespread problem is a centralized data pipeline orchestration solution. Join Stonebranch’s Scott Davis, Global Vice President and Ravi Murugesan, Sr. Solution Engineer to learn how DataOps teams orchestrate their end-to-end data pipelines with a platform approach to managing automation. Key Learnings: - Discover how to orchestrate data pipelines across a hybrid IT environment (on-prem and cloud) - Find out how DataOps teams are empowered with event-based triggers for real-time data flow - See examples of reports, dashboards, and proactive alerts designed to help you reliably keep data flowing through your business — with the observability you require - Discover how to replace clunky legacy approaches to streaming data in a multi-cloud environment - See what’s possible with the Stonebranch Universal Automation Center (UAC)

Data & Analytics
1 of 30
Download to read offline
© Stonebranch 2022. All rights reserved. ORCHESTRATE the Flow of Data Across Data Pipelines May 3, 2022 Ravi Murugesan Sr. Solution Engineer Scott Davis Global Vice President
2 © Stonebranch 2022. All rights reserved. DevOps Orchestration Layer 01 What is a Data Pipeline 02 How to Orchestrate a Data Pipeline 03 Data Pipeline Orchestration Demo 04 Questions and Answers 05 Agenda
© Stonebranch 2022. All rights reserved. About Data Pipelines 3 Scott Davis Global Vice President
4 © Stonebranch 2022. All rights reserved. Vendor Landscape for DataOps – From Gartner Orchestrators Specialists Portfolio Cloud Service Providers Servware (Services & Software) System Integrators Integration Cataloging Governance MDM Analytics-Ready Enterprise Data Management Industrial Data Data Quality Observability Continuous Delivery Accelerators Privacy & Access Control * Based on “Gartner Data and Analytics Essentials: DataOps,” by Robert Robert Thanaraj
5 © Stonebranch 2022. All rights reserved. Data Pipeline: Simple View
6 © Stonebranch 2022. All rights reserved. Software & Tools By Stage Dashboards Looker, Tableau, Qlik, Power BI, SAP BusinessObjects Embedded Analytics Sisense, Looker, Cube.js Augmented Analytics Throughspot, Outlier, Anodot, Sisu App Frameworks Plotly Dash, Streamlit Custom Apps SMS Messages / Emails Data Science & Machine Learning Databricks, SAS, MathWork, Domino, Dataiku, DataRobot, TIBCO Software, Spark, RapidMiner, H2O.AI, AWS, GCP AI, Azure ML, IBM Watson Studio, Cloudera, Alteryx, TensorFlow, Anaconda Data Lake Databricks Delta Lake, Iceberg, Hudi, Hive Acid Data Lake within Cloud Storage AWS S3, Google Cloud Storage, HDFS, Azure Data Lake Store Data Warehouse Snowflake, BigQuery, Spark, AWS Redshift, Qubole, SAP BW, SAP DWC, Oracle ADW, Hive, Cloudera (for Hadoop) ETL (Extract, Transform, Load) Informatica, IBM, SAP Data Services, Oracle OWB, SAS, Talend, AWS Glue, Azure Data Factory, Pentaho, GCP Data Fusion Stream Data Processing ELT Kafka, Flink, Storm, GCP Pub/Sub Applications / ERP Oracle, Salesforce, SAP, ServiceNow IoT Devices / Sensors Stream Data Website & Mobile Apps Stream Data, Online Transaction Cloud Storage AWS S3, Google Cloud Storage, Azure Data Sources Data Integration & Ingestion Data Store Analyze / Computation Delivery
How Do Enterprises Orchestrate Today? 7 © Stonebranch 2022. All rights reserved. Common Ways to Connect Data Tools Within the Pipeline Point-to-Point Integrations Custom Scripts Don’t Connect (Manual Movement)
How Do Enterprises Orchestrate Today? 8 © Stonebranch 2022. All rights reserved. Common Ways to Connect Data Tools Within the Pipeline Point-to-Point Integrations Custom Scripts Don’t Connect (Manual Movement) Benefits of Proper Orchestration Solutions Centralized View Root-Cause Issues Proactive Support Achieve Scale
Automation Pain Points Common Ways to Connect Data Tools Within the Pipeline Point-to-Point Integrations Custom Scripts Don’t Connect (Manual Movement) How Do Enterprises Orchestrate Today? 9 © Stonebranch 2022. All rights reserved. Benefits of Proper Orchestration Solutions Centralized View Root-Cause Issues Proactive Support Achieve Scale In-Built Schedulers Open-Source Schedulers Cloud Schedulers Legacy On-Prem Focused Schedulers Can’t schedule jobs in other tools Often batch- or time- based automation Focus on their own ecosystems Can’t automate jobs in both on-prem and cloud systems, i.e., no hybrid IT automation
Data Pipeline Orchestration © Stonebranch 2022. All rights reserved. 10
11 © Stonebranch 2022. All rights reserved. Data Pipeline Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline.
Data Pipeline Orchestration Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline. • Centrally schedule and orchestrate automated processes within each tool along the entire data pipeline • Use APIs or Agents to control the various tools used within each stage 12 © Stonebranch 2022. All rights reserved.
Data Pipeline Orchestration Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline. • Centrally schedule and orchestrate automated processes within each tool along the entire data pipeline • Use APIs or Agents to control the various tools used within each stage What you achieve with this approach: • Observability of the logs and data for governance and security • DataOps lifecycle management (Dev-Test- Prod) - including simulations • Centralized control and visibility with visual workflows • Quickly root-cause issues with proactive alerts when something fails 13 © Stonebranch 2022. All rights reserved.
Data Pipeline Orchestration Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline. • Centrally schedule and orchestrate automated processes within each tool along the entire data pipeline • Use APIs or Agents to control the various tools used within each stage What you achieve with this approach: • Observability of the logs and data for governance and security • DataOps lifecycle management (Dev-Test- Prod) - including simulations • Centralized control and visibility with visual workflows • Quickly root-cause issues with proactive alerts when something fails 14 © Stonebranch 2022. All rights reserved.
Orchestration 15 © Stonebranch 2022. All rights reserved. Driven by SOAP Cloud DevOps ERP/Apps IaaS UAC SOAP META-ORCHESTRATION
16 © Stonebranch 2022. All rights reserved. Self-Service Automation Centralized collaboration platform for data, developers, and operations IT ops teams gain operational visibility Data teams approve and trigger automated workflows & pipelines from common business applications Data Pipeline
Putting the Ops in DataOps 17 © Stonebranch 2022. All rights reserved. For Enterprises Ready for the Next Level of Maturity Develop/ Orchestrate Test / Simulate Production / Deploy Continuous Improvement Continuous Deployment Development Controller Production Controller
Develop/ Orchestrate Test / Simulate Production / Deploy Continuous Improvement Continuous Deployment Development Controller Production Controller Putting the Ops in DataOps 18 © Stonebranch 2022. All rights reserved. For Enterprises Ready for the Next Level of Maturity Web GUI As Code Via in-built capabilities Promotion Options Via third-party repositories like GitHub
Data Pipeline Orchestration Demo Ravi Murugesan Sr. Solution Engineer © Stonebranch 2022. All rights reserved. 19
© Stonebranch 2022. All rights reserved. 20 Demonstration Update Visual Dashboard from Multiple Data Sources (both on-prem and cloud-based) Live orchestration of a data pipeline, including • Sources (cloud, on-prem, apps) • Ingestion, transformation (Informatica) • Stores (Azure blob, Snowflake) • Delivery (Tableau)
One of the Largest Global Food & Beverage Manufacturers in the World Customer Use Case 21
Customer Use Case: Overview One of the Largest Global Food & Beverage Manufacturers in the World Evolution & Goal • Goal: Orchestrate the full pipeline end-to-end • Objective: Identify a platform that could connect all their critical data tools Overall Strategy • On-prem to cloud digital transformation • Implemented an enterprise analytics data management environment • Hub-and-spoke model to help keep regional resource groups and services segregated • Approved services are first developed and deployed at the hub level, with further spoke deployment via containers Original Approach • Their data pipeline for the enterprise data management environment with Azure Data Factory • Azure Data Factory worked well in an Azure environment • It served as an entry point for the project • The Challenge: Data Factory did not integrate with their full stack of solutions used along the data pipeline 22 © Stonebranch 2022. All rights reserved.
Data Pipeline Orchestration One of the Largest Global Food & Beverage Manufacturers in the World Achieving Their Goal • Secure and robust file transfer • DataOps: define pipelines as code and gain lifecycle management (test/dev/prod) capabilities • Integrate diverse data pipelines that are built using various cloud-based and on-prem services and tools • For operations: visibility into the process, improve SLAs, real-time monitoring, alerting • Unified view to design and orchestrate workflows across multiple cloud and on-prem applications Orchestration Databases 23 © Stonebranch 2022. All rights reserved.
© Stonebranch 2022. All rights reserved. Data Pipeline Orchestration Solution Universal Automation Center 24
Real Time Hybrid IT Automation 25 © Stonebranch 2022. All rights reserved. Universal Automation Center Platform A Platform Approach Orchestrating IT processes from on-prem, to cloud, to containerized microservices
Find. Deploy. Extend. • Download extensions • Share extensions • Community driven • Constant additions (monthly) • Large Data Pipeline Focus • Rapid creation of new integrations Orchestration = Integration 26 © Stonebranch 2022. All rights reserved.
What to Look for in a Data Pipeline Orchestration Solution 27 © Stonebranch 2022. All rights reserved.
Summary Who is this for? • Want to keep using existing data tools, but are ready to graduate from opensource schedulers to enterprise grade platforms • Would like a single platform to connect Data Teams, Developers, IT Ops, and Cloud Ops teams – to help scale their data program • Need to operationalize DataOps methodologies to gain speed and improve data quality • Want to gain full visibility across the entire pipeline – to move quickly when issue arise • Have a growing or changing data tool landscape, and need the ability to rapidly build new integrations (or download pre-existing integrations) • Need to enable data scientists or business users with simple self-service capabilities via the platform or third-party tools like ServiceNow, Microsoft Teams, or Slack • Bonus: Want a central IT automation and orchestration platform (beyond data pipeline orchestration) to support cloud automation, on-prem automation, traditional job scheduling, and DevOps orchestration © Stonebranch 2022. All rights reserved. 28
© Stonebranch 2022. All rights reserved. 29 Q & A Scott Davis Global Vice President scott.davis@stonebranch.com Stonebranch - Atlanta, USA Ravi Murugesan Sr. Solution Engineer ravi.murugesan@stonebranch.com Stonebranch – Frankfurt, Germany
Thank You © Stonebranch 2022. All rights reserved.

Recommended

Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data PipelinesBest Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Eric Kavanagh
 
The ABCs of Treating Data as Product
The ABCs of Treating Data as ProductThe ABCs of Treating Data as Product
The ABCs of Treating Data as Product
DATAVERSITY
 
[XConf Brasil 2020] Data mesh
[XConf Brasil 2020] Data mesh[XConf Brasil 2020] Data mesh
[XConf Brasil 2020] Data mesh
ThoughtWorks Brasil
 
Modern Data architecture Design
Modern Data architecture DesignModern Data architecture Design
Modern Data architecture Design
Kujambu Murugesan
 
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
DATAVERSITY
 
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsBuilding the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Khalid Salama
 
How a Semantic Layer Makes Data Mesh Work at Scale
How a Semantic Layer Makes Data Mesh Work at ScaleHow a Semantic Layer Makes Data Mesh Work at Scale
How a Semantic Layer Makes Data Mesh Work at Scale
DATAVERSITY
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Tristan Baker
 

Recommended

Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data PipelinesBest Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Eric Kavanagh
 
The ABCs of Treating Data as Product
The ABCs of Treating Data as ProductThe ABCs of Treating Data as Product
The ABCs of Treating Data as Product
DATAVERSITY
 
[XConf Brasil 2020] Data mesh
[XConf Brasil 2020] Data mesh[XConf Brasil 2020] Data mesh
[XConf Brasil 2020] Data mesh
ThoughtWorks Brasil
 
Modern Data architecture Design
Modern Data architecture DesignModern Data architecture Design
Modern Data architecture Design
Kujambu Murugesan
 
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
DATAVERSITY
 
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsBuilding the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Khalid Salama
 
How a Semantic Layer Makes Data Mesh Work at Scale
How a Semantic Layer Makes Data Mesh Work at ScaleHow a Semantic Layer Makes Data Mesh Work at Scale
How a Semantic Layer Makes Data Mesh Work at Scale
DATAVERSITY
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Tristan Baker
 
Intro to Data Vault 2.0 on Snowflake
Intro to Data Vault 2.0 on SnowflakeIntro to Data Vault 2.0 on Snowflake
Intro to Data Vault 2.0 on Snowflake
Kent Graziano
 
Data Mesh
Data MeshData Mesh
Data Mesh
Piethein Strengholt
 
Emerging Trends in Data Architecture – What’s the Next Big Thing
Emerging Trends in Data Architecture – What’s the Next Big ThingEmerging Trends in Data Architecture – What’s the Next Big Thing
Emerging Trends in Data Architecture – What’s the Next Big Thing
DATAVERSITY
 
Improving Data Literacy Around Data Architecture
Improving Data Literacy Around Data ArchitectureImproving Data Literacy Around Data Architecture
Improving Data Literacy Around Data Architecture
DATAVERSITY
 
Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?
DATAVERSITY
 
Lakehouse in Azure
Lakehouse in AzureLakehouse in Azure
Lakehouse in Azure
Sergio Zenatti Filho
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
Databricks
 
Building a Data Governance Strategy
Building a Data Governance StrategyBuilding a Data Governance Strategy
Building a Data Governance Strategy
Analytics8
 
Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data Catalog
DATAVERSITY
 
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
HostedbyConfluent
 
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
DataScienceConferenc1
 
Enterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data ArchitectureEnterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data Architecture
DATAVERSITY
 
Glossaries, Dictionaries, and Catalogs Result in Data Governance
Glossaries, Dictionaries, and Catalogs Result in Data GovernanceGlossaries, Dictionaries, and Catalogs Result in Data Governance
Glossaries, Dictionaries, and Catalogs Result in Data Governance
DATAVERSITY
 
DataOps introduction : DataOps is not only DevOps applied to data!
DataOps introduction : DataOps is not only DevOps applied to data!DataOps introduction : DataOps is not only DevOps applied to data!
DataOps introduction : DataOps is not only DevOps applied to data!
Adrien Blind
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to Mesh
Jeffrey T. Pollock
 
Data Quality Best Practices
Data Quality Best PracticesData Quality Best Practices
Data Quality Best Practices
DATAVERSITY
 
DI&A Slides: Data Lake vs. Data Warehouse
DI&A Slides: Data Lake vs. Data WarehouseDI&A Slides: Data Lake vs. Data Warehouse
DI&A Slides: Data Lake vs. Data Warehouse
DATAVERSITY
 
Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)
Nathan Bijnens
 
Data Architecture for Data Governance
Data Architecture for Data GovernanceData Architecture for Data Governance
Data Architecture for Data Governance
DATAVERSITY
 
Data Quality & Data Governance
Data Quality & Data GovernanceData Quality & Data Governance
Data Quality & Data Governance
Tuba Yaman Him
 
Cloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native appsCloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native apps
VMware Tanzu
 
InfoSphere BigInsights
InfoSphere BigInsightsInfoSphere BigInsights
InfoSphere BigInsights
Wilfried Hoge
 

More Related Content

What's hot

Intro to Data Vault 2.0 on Snowflake
Intro to Data Vault 2.0 on SnowflakeIntro to Data Vault 2.0 on Snowflake
Intro to Data Vault 2.0 on Snowflake
Kent Graziano
 
Data Mesh
Data MeshData Mesh
Data Mesh
Piethein Strengholt
 
Emerging Trends in Data Architecture – What’s the Next Big Thing
Emerging Trends in Data Architecture – What’s the Next Big ThingEmerging Trends in Data Architecture – What’s the Next Big Thing
Emerging Trends in Data Architecture – What’s the Next Big Thing
DATAVERSITY
 
Improving Data Literacy Around Data Architecture
Improving Data Literacy Around Data ArchitectureImproving Data Literacy Around Data Architecture
Improving Data Literacy Around Data Architecture
DATAVERSITY
 
Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?
DATAVERSITY
 
Lakehouse in Azure
Lakehouse in AzureLakehouse in Azure
Lakehouse in Azure
Sergio Zenatti Filho
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
Databricks
 
Building a Data Governance Strategy
Building a Data Governance StrategyBuilding a Data Governance Strategy
Building a Data Governance Strategy
Analytics8
 
Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data Catalog
DATAVERSITY
 
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
HostedbyConfluent
 
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
DataScienceConferenc1
 
Enterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data ArchitectureEnterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data Architecture
DATAVERSITY
 
Glossaries, Dictionaries, and Catalogs Result in Data Governance
Glossaries, Dictionaries, and Catalogs Result in Data GovernanceGlossaries, Dictionaries, and Catalogs Result in Data Governance
Glossaries, Dictionaries, and Catalogs Result in Data Governance
DATAVERSITY
 
DataOps introduction : DataOps is not only DevOps applied to data!
DataOps introduction : DataOps is not only DevOps applied to data!DataOps introduction : DataOps is not only DevOps applied to data!
DataOps introduction : DataOps is not only DevOps applied to data!
Adrien Blind
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to Mesh
Jeffrey T. Pollock
 
Data Quality Best Practices
Data Quality Best PracticesData Quality Best Practices
Data Quality Best Practices
DATAVERSITY
 
DI&A Slides: Data Lake vs. Data Warehouse
DI&A Slides: Data Lake vs. Data WarehouseDI&A Slides: Data Lake vs. Data Warehouse
DI&A Slides: Data Lake vs. Data Warehouse
DATAVERSITY
 
Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)
Nathan Bijnens
 
Data Architecture for Data Governance
Data Architecture for Data GovernanceData Architecture for Data Governance
Data Architecture for Data Governance
DATAVERSITY
 
Data Quality & Data Governance
Data Quality & Data GovernanceData Quality & Data Governance
Data Quality & Data Governance
Tuba Yaman Him
 

What's hot (20)

Intro to Data Vault 2.0 on Snowflake
Intro to Data Vault 2.0 on SnowflakeIntro to Data Vault 2.0 on Snowflake
Intro to Data Vault 2.0 on Snowflake
 
Data Mesh
Data MeshData Mesh
Data Mesh
 
Emerging Trends in Data Architecture – What’s the Next Big Thing
Emerging Trends in Data Architecture – What’s the Next Big ThingEmerging Trends in Data Architecture – What’s the Next Big Thing
Emerging Trends in Data Architecture – What’s the Next Big Thing
 
Improving Data Literacy Around Data Architecture
Improving Data Literacy Around Data ArchitectureImproving Data Literacy Around Data Architecture
Improving Data Literacy Around Data Architecture
 
Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?
 
Lakehouse in Azure
Lakehouse in AzureLakehouse in Azure
Lakehouse in Azure
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
 
Building a Data Governance Strategy
Building a Data Governance StrategyBuilding a Data Governance Strategy
Building a Data Governance Strategy
 
Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data Catalog
 
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
 
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
 
Enterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data ArchitectureEnterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data Architecture
 
Glossaries, Dictionaries, and Catalogs Result in Data Governance
Glossaries, Dictionaries, and Catalogs Result in Data GovernanceGlossaries, Dictionaries, and Catalogs Result in Data Governance
Glossaries, Dictionaries, and Catalogs Result in Data Governance
 
DataOps introduction : DataOps is not only DevOps applied to data!
DataOps introduction : DataOps is not only DevOps applied to data!DataOps introduction : DataOps is not only DevOps applied to data!
DataOps introduction : DataOps is not only DevOps applied to data!
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to Mesh
 
Data Quality Best Practices
Data Quality Best PracticesData Quality Best Practices
Data Quality Best Practices
 
DI&A Slides: Data Lake vs. Data Warehouse
DI&A Slides: Data Lake vs. Data WarehouseDI&A Slides: Data Lake vs. Data Warehouse
DI&A Slides: Data Lake vs. Data Warehouse
 
Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)
 
Data Architecture for Data Governance
Data Architecture for Data GovernanceData Architecture for Data Governance
Data Architecture for Data Governance
 
Data Quality & Data Governance
Data Quality & Data GovernanceData Quality & Data Governance
Data Quality & Data Governance
 

Similar to Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines

Cloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native appsCloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native apps
VMware Tanzu
 
InfoSphere BigInsights
InfoSphere BigInsightsInfoSphere BigInsights
InfoSphere BigInsights
Wilfried Hoge
 
Self-Service Analytics with Guard Rails
Self-Service Analytics with Guard RailsSelf-Service Analytics with Guard Rails
Self-Service Analytics with Guard Rails
Denodo
 
2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics
DATAVERSITY
 
Big Data Ready Enterprise
Big Data Ready Enterprise Big Data Ready Enterprise
Big Data Ready Enterprise
DataWorks Summit/Hadoop Summit
 
Enabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSetsEnabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSets
Streamsets Inc.
 
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
DataStax Academy
 
Simplifying Real-Time Architectures for IoT with Apache Kudu
Simplifying Real-Time Architectures for IoT with Apache KuduSimplifying Real-Time Architectures for IoT with Apache Kudu
Simplifying Real-Time Architectures for IoT with Apache Kudu
Cloudera, Inc.
 
Horses for Courses: Database Roundtable
Horses for Courses: Database RoundtableHorses for Courses: Database Roundtable
Horses for Courses: Database Roundtable
Eric Kavanagh
 
About CDAP
About CDAPAbout CDAP
About CDAP
Cask Data
 
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Timothy Spann
 
Migrating from Oracle to Postgres
Migrating from Oracle to PostgresMigrating from Oracle to Postgres
Migrating from Oracle to Postgres
EDB
 
Re-Platforming Applications for the Cloud
Re-Platforming Applications for the CloudRe-Platforming Applications for the Cloud
Re-Platforming Applications for the Cloud
Carter Wickstrom
 
Hadoop and Your Enterprise Data Warehouse
Hadoop and Your Enterprise Data WarehouseHadoop and Your Enterprise Data Warehouse
Hadoop and Your Enterprise Data Warehouse
Edgar Alejandro Villegas
 
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
Denodo
 
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
MapR Technologies
 
A Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data VirtualizationA Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data Virtualization
Denodo
 
Government and Education Webinar: Optimizing Database Performance
Government and Education Webinar: Optimizing Database PerformanceGovernment and Education Webinar: Optimizing Database Performance
Government and Education Webinar: Optimizing Database Performance
SolarWinds
 
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus WebinarBuild and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
Impetus Technologies
 
CI/CD for a Data Platform
CI/CD for a Data PlatformCI/CD for a Data Platform
CI/CD for a Data Platform
Codit
 

Similar to Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines (20)

Cloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native appsCloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native apps
 
InfoSphere BigInsights
InfoSphere BigInsightsInfoSphere BigInsights
InfoSphere BigInsights
 
Self-Service Analytics with Guard Rails
Self-Service Analytics with Guard RailsSelf-Service Analytics with Guard Rails
Self-Service Analytics with Guard Rails
 
2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics
 
Big Data Ready Enterprise
Big Data Ready Enterprise Big Data Ready Enterprise
Big Data Ready Enterprise
 
Enabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSetsEnabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSets
 
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
 
Simplifying Real-Time Architectures for IoT with Apache Kudu
Simplifying Real-Time Architectures for IoT with Apache KuduSimplifying Real-Time Architectures for IoT with Apache Kudu
Simplifying Real-Time Architectures for IoT with Apache Kudu
 
Horses for Courses: Database Roundtable
Horses for Courses: Database RoundtableHorses for Courses: Database Roundtable
Horses for Courses: Database Roundtable
 
About CDAP
About CDAPAbout CDAP
About CDAP
 
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
 
Migrating from Oracle to Postgres
Migrating from Oracle to PostgresMigrating from Oracle to Postgres
Migrating from Oracle to Postgres
 
Re-Platforming Applications for the Cloud
Re-Platforming Applications for the CloudRe-Platforming Applications for the Cloud
Re-Platforming Applications for the Cloud
 
Hadoop and Your Enterprise Data Warehouse
Hadoop and Your Enterprise Data WarehouseHadoop and Your Enterprise Data Warehouse
Hadoop and Your Enterprise Data Warehouse
 
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
 
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
 
A Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data VirtualizationA Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data Virtualization
 
Government and Education Webinar: Optimizing Database Performance
Government and Education Webinar: Optimizing Database PerformanceGovernment and Education Webinar: Optimizing Database Performance
Government and Education Webinar: Optimizing Database Performance
 
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus WebinarBuild and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
 
CI/CD for a Data Platform
CI/CD for a Data PlatformCI/CD for a Data Platform
CI/CD for a Data Platform
 

More from DATAVERSITY

Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
DATAVERSITY
 
Data at the Speed of Business with Data Mastering and Governance
Data at the Speed of Business with Data Mastering and GovernanceData at the Speed of Business with Data Mastering and Governance
Data at the Speed of Business with Data Mastering and Governance
DATAVERSITY
 
Exploring Levels of Data Literacy
Exploring Levels of Data LiteracyExploring Levels of Data Literacy
Exploring Levels of Data Literacy
DATAVERSITY
 
Building a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business GoalsBuilding a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business Goals
DATAVERSITY
 
Make Data Work for You
Make Data Work for YouMake Data Work for You
Make Data Work for You
DATAVERSITY
 
Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?
DATAVERSITY
 
Data Catalogs Are the Answer – What Is the Question?
Data Catalogs Are the Answer – What Is the Question?Data Catalogs Are the Answer – What Is the Question?
Data Catalogs Are the Answer – What Is the Question?
DATAVERSITY
 
Data Modeling Fundamentals
Data Modeling FundamentalsData Modeling Fundamentals
Data Modeling Fundamentals
DATAVERSITY
 
Showing ROI for Your Analytic Project
Showing ROI for Your Analytic ProjectShowing ROI for Your Analytic Project
Showing ROI for Your Analytic Project
DATAVERSITY
 
Is Enterprise Data Literacy Possible?
Is Enterprise Data Literacy Possible?Is Enterprise Data Literacy Possible?
Is Enterprise Data Literacy Possible?
DATAVERSITY
 
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
DATAVERSITY
 
Emerging Trends in Data Architecture – What’s the Next Big Thing?
Emerging Trends in Data Architecture – What’s the Next Big Thing?Emerging Trends in Data Architecture – What’s the Next Big Thing?
Emerging Trends in Data Architecture – What’s the Next Big Thing?
DATAVERSITY
 
Data Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and ForwardsData Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and Forwards
DATAVERSITY
 
Data Governance Trends and Best Practices To Implement Today
Data Governance Trends and Best Practices To Implement TodayData Governance Trends and Best Practices To Implement Today
Data Governance Trends and Best Practices To Implement Today
DATAVERSITY
 
2023 Trends in Enterprise Analytics
2023 Trends in Enterprise Analytics2023 Trends in Enterprise Analytics
2023 Trends in Enterprise Analytics
DATAVERSITY
 
Data Strategy Best Practices
Data Strategy Best PracticesData Strategy Best Practices
Data Strategy Best Practices
DATAVERSITY
 
Who Should Own Data Governance – IT or Business?
Who Should Own Data Governance – IT or Business?Who Should Own Data Governance – IT or Business?
Who Should Own Data Governance – IT or Business?
DATAVERSITY
 
Data Management Best Practices
Data Management Best PracticesData Management Best Practices
Data Management Best Practices
DATAVERSITY
 
MLOps – Applying DevOps to Competitive Advantage
MLOps – Applying DevOps to Competitive AdvantageMLOps – Applying DevOps to Competitive Advantage
MLOps – Applying DevOps to Competitive Advantage
DATAVERSITY
 
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
DATAVERSITY
 

More from DATAVERSITY (20)

Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
 
Data at the Speed of Business with Data Mastering and Governance
Data at the Speed of Business with Data Mastering and GovernanceData at the Speed of Business with Data Mastering and Governance
Data at the Speed of Business with Data Mastering and Governance
 
Exploring Levels of Data Literacy
Exploring Levels of Data LiteracyExploring Levels of Data Literacy
Exploring Levels of Data Literacy
 
Building a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business GoalsBuilding a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business Goals
 
Make Data Work for You
Make Data Work for YouMake Data Work for You
Make Data Work for You
 
Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?
 
Data Catalogs Are the Answer – What Is the Question?
Data Catalogs Are the Answer – What Is the Question?Data Catalogs Are the Answer – What Is the Question?
Data Catalogs Are the Answer – What Is the Question?
 
Data Modeling Fundamentals
Data Modeling FundamentalsData Modeling Fundamentals
Data Modeling Fundamentals
 
Showing ROI for Your Analytic Project
Showing ROI for Your Analytic ProjectShowing ROI for Your Analytic Project
Showing ROI for Your Analytic Project
 
Is Enterprise Data Literacy Possible?
Is Enterprise Data Literacy Possible?Is Enterprise Data Literacy Possible?
Is Enterprise Data Literacy Possible?
 
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
 
Emerging Trends in Data Architecture – What’s the Next Big Thing?
Emerging Trends in Data Architecture – What’s the Next Big Thing?Emerging Trends in Data Architecture – What’s the Next Big Thing?
Emerging Trends in Data Architecture – What’s the Next Big Thing?
 
Data Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and ForwardsData Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and Forwards
 
Data Governance Trends and Best Practices To Implement Today
Data Governance Trends and Best Practices To Implement TodayData Governance Trends and Best Practices To Implement Today
Data Governance Trends and Best Practices To Implement Today
 
2023 Trends in Enterprise Analytics
2023 Trends in Enterprise Analytics2023 Trends in Enterprise Analytics
2023 Trends in Enterprise Analytics
 
Data Strategy Best Practices
Data Strategy Best PracticesData Strategy Best Practices
Data Strategy Best Practices
 
Who Should Own Data Governance – IT or Business?
Who Should Own Data Governance – IT or Business?Who Should Own Data Governance – IT or Business?
Who Should Own Data Governance – IT or Business?
 
Data Management Best Practices
Data Management Best PracticesData Management Best Practices
Data Management Best Practices
 
MLOps – Applying DevOps to Competitive Advantage
MLOps – Applying DevOps to Competitive AdvantageMLOps – Applying DevOps to Competitive Advantage
MLOps – Applying DevOps to Competitive Advantage
 
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
 

Recently uploaded

🔥Call Girl Price Pune 💯Call Us 🔝 7014168258 🔝💃Independent Pune Escorts Servic...
🔥Call Girl Price Pune 💯Call Us 🔝 7014168258 🔝💃Independent Pune Escorts Servic...🔥Call Girl Price Pune 💯Call Us 🔝 7014168258 🔝💃Independent Pune Escorts Servic...
🔥Call Girl Price Pune 💯Call Us 🔝 7014168258 🔝💃Independent Pune Escorts Servic...
Ak47
 
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance PaymentCall Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
prijesh mathew
 
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
Marlon Dumas
 
Startup Grind Princeton 18 June 2024 - AI Advancement
Startup Grind Princeton 18 June 2024 - AI AdvancementStartup Grind Princeton 18 June 2024 - AI Advancement
Startup Grind Princeton 18 June 2024 - AI Advancement
Timothy Spann
 
saps4hanaandsapanalyticswheretodowhat1565272000538.pdf
saps4hanaandsapanalyticswheretodowhat1565272000538.pdfsaps4hanaandsapanalyticswheretodowhat1565272000538.pdf
saps4hanaandsapanalyticswheretodowhat1565272000538.pdf
newdirectionconsulta
 
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
Rebecca Bilbro
 
Mumbai Call Girls service 9920874524 Call Girl service in Mumbai Mumbai Call ...
Mumbai Call Girls service 9920874524 Call Girl service in Mumbai Mumbai Call ...Mumbai Call Girls service 9920874524 Call Girl service in Mumbai Mumbai Call ...
Mumbai Call Girls service 9920874524 Call Girl service in Mumbai Mumbai Call ...
hanshkumar9870
 
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
9gr6pty
 
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
mparmparousiskostas
 
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
PsychoTech Services
 
Bangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts ServiceBangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts Service
nhero3888
 
IBM watsonx.data - Seller Enablement Deck.PPTX
IBM watsonx.data - Seller Enablement Deck.PPTXIBM watsonx.data - Seller Enablement Deck.PPTX
IBM watsonx.data - Seller Enablement Deck.PPTX
EbtsamRashed
 
Salesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - CanariasSalesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - Canarias
davidpietrzykowski1
 
Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)
GeorgiiSteshenko
 
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
mona lisa $A12
 
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call GirlCall Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
sapna sharmap11
 
MySQL Notes For Professionals sttudy.pdf
MySQL Notes For Professionals sttudy.pdfMySQL Notes For Professionals sttudy.pdf
MySQL Notes For Professionals sttudy.pdf
Ananta Patil
 
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your DoorAhmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
Russian Escorts in Delhi 9711199171 with low rate Book online
 
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call GirlCall Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
sapna sharmap11
 
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls HyderabadHyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
2004kavitajoshi
 

Recently uploaded (20)

🔥Call Girl Price Pune 💯Call Us 🔝 7014168258 🔝💃Independent Pune Escorts Servic...
🔥Call Girl Price Pune 💯Call Us 🔝 7014168258 🔝💃Independent Pune Escorts Servic...🔥Call Girl Price Pune 💯Call Us 🔝 7014168258 🔝💃Independent Pune Escorts Servic...
🔥Call Girl Price Pune 💯Call Us 🔝 7014168258 🔝💃Independent Pune Escorts Servic...
 
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance PaymentCall Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
 
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
 
Startup Grind Princeton 18 June 2024 - AI Advancement
Startup Grind Princeton 18 June 2024 - AI AdvancementStartup Grind Princeton 18 June 2024 - AI Advancement
Startup Grind Princeton 18 June 2024 - AI Advancement
 
saps4hanaandsapanalyticswheretodowhat1565272000538.pdf
saps4hanaandsapanalyticswheretodowhat1565272000538.pdfsaps4hanaandsapanalyticswheretodowhat1565272000538.pdf
saps4hanaandsapanalyticswheretodowhat1565272000538.pdf
 
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
 
Mumbai Call Girls service 9920874524 Call Girl service in Mumbai Mumbai Call ...
Mumbai Call Girls service 9920874524 Call Girl service in Mumbai Mumbai Call ...Mumbai Call Girls service 9920874524 Call Girl service in Mumbai Mumbai Call ...
Mumbai Call Girls service 9920874524 Call Girl service in Mumbai Mumbai Call ...
 
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
 
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
 
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
 
Bangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts ServiceBangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts Service
 
IBM watsonx.data - Seller Enablement Deck.PPTX
IBM watsonx.data - Seller Enablement Deck.PPTXIBM watsonx.data - Seller Enablement Deck.PPTX
IBM watsonx.data - Seller Enablement Deck.PPTX
 
Salesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - CanariasSalesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - Canarias
 
Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)
 
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
 
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call GirlCall Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
 
MySQL Notes For Professionals sttudy.pdf
MySQL Notes For Professionals sttudy.pdfMySQL Notes For Professionals sttudy.pdf
MySQL Notes For Professionals sttudy.pdf
 
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your DoorAhmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
 
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call GirlCall Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
 
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls HyderabadHyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
 

Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines

  • 1. © Stonebranch 2022. All rights reserved. ORCHESTRATE the Flow of Data Across Data Pipelines May 3, 2022 Ravi Murugesan Sr. Solution Engineer Scott Davis Global Vice President
  • 2. 2 © Stonebranch 2022. All rights reserved. DevOps Orchestration Layer 01 What is a Data Pipeline 02 How to Orchestrate a Data Pipeline 03 Data Pipeline Orchestration Demo 04 Questions and Answers 05 Agenda
  • 3. © Stonebranch 2022. All rights reserved. About Data Pipelines 3 Scott Davis Global Vice President
  • 4. 4 © Stonebranch 2022. All rights reserved. Vendor Landscape for DataOps – From Gartner Orchestrators Specialists Portfolio Cloud Service Providers Servware (Services & Software) System Integrators Integration Cataloging Governance MDM Analytics-Ready Enterprise Data Management Industrial Data Data Quality Observability Continuous Delivery Accelerators Privacy & Access Control * Based on “Gartner Data and Analytics Essentials: DataOps,” by Robert Robert Thanaraj
  • 5. 5 © Stonebranch 2022. All rights reserved. Data Pipeline: Simple View
  • 6. 6 © Stonebranch 2022. All rights reserved. Software & Tools By Stage Dashboards Looker, Tableau, Qlik, Power BI, SAP BusinessObjects Embedded Analytics Sisense, Looker, Cube.js Augmented Analytics Throughspot, Outlier, Anodot, Sisu App Frameworks Plotly Dash, Streamlit Custom Apps SMS Messages / Emails Data Science & Machine Learning Databricks, SAS, MathWork, Domino, Dataiku, DataRobot, TIBCO Software, Spark, RapidMiner, H2O.AI, AWS, GCP AI, Azure ML, IBM Watson Studio, Cloudera, Alteryx, TensorFlow, Anaconda Data Lake Databricks Delta Lake, Iceberg, Hudi, Hive Acid Data Lake within Cloud Storage AWS S3, Google Cloud Storage, HDFS, Azure Data Lake Store Data Warehouse Snowflake, BigQuery, Spark, AWS Redshift, Qubole, SAP BW, SAP DWC, Oracle ADW, Hive, Cloudera (for Hadoop) ETL (Extract, Transform, Load) Informatica, IBM, SAP Data Services, Oracle OWB, SAS, Talend, AWS Glue, Azure Data Factory, Pentaho, GCP Data Fusion Stream Data Processing ELT Kafka, Flink, Storm, GCP Pub/Sub Applications / ERP Oracle, Salesforce, SAP, ServiceNow IoT Devices / Sensors Stream Data Website & Mobile Apps Stream Data, Online Transaction Cloud Storage AWS S3, Google Cloud Storage, Azure Data Sources Data Integration & Ingestion Data Store Analyze / Computation Delivery
  • 7. How Do Enterprises Orchestrate Today? 7 © Stonebranch 2022. All rights reserved. Common Ways to Connect Data Tools Within the Pipeline Point-to-Point Integrations Custom Scripts Don’t Connect (Manual Movement)
  • 8. How Do Enterprises Orchestrate Today? 8 © Stonebranch 2022. All rights reserved. Common Ways to Connect Data Tools Within the Pipeline Point-to-Point Integrations Custom Scripts Don’t Connect (Manual Movement) Benefits of Proper Orchestration Solutions Centralized View Root-Cause Issues Proactive Support Achieve Scale
  • 9. Automation Pain Points Common Ways to Connect Data Tools Within the Pipeline Point-to-Point Integrations Custom Scripts Don’t Connect (Manual Movement) How Do Enterprises Orchestrate Today? 9 © Stonebranch 2022. All rights reserved. Benefits of Proper Orchestration Solutions Centralized View Root-Cause Issues Proactive Support Achieve Scale In-Built Schedulers Open-Source Schedulers Cloud Schedulers Legacy On-Prem Focused Schedulers Can’t schedule jobs in other tools Often batch- or time- based automation Focus on their own ecosystems Can’t automate jobs in both on-prem and cloud systems, i.e., no hybrid IT automation
  • 10. Data Pipeline Orchestration © Stonebranch 2022. All rights reserved. 10
  • 11. 11 © Stonebranch 2022. All rights reserved. Data Pipeline Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline.
  • 12. Data Pipeline Orchestration Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline. • Centrally schedule and orchestrate automated processes within each tool along the entire data pipeline • Use APIs or Agents to control the various tools used within each stage 12 © Stonebranch 2022. All rights reserved.
  • 13. Data Pipeline Orchestration Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline. • Centrally schedule and orchestrate automated processes within each tool along the entire data pipeline • Use APIs or Agents to control the various tools used within each stage What you achieve with this approach: • Observability of the logs and data for governance and security • DataOps lifecycle management (Dev-Test- Prod) - including simulations • Centralized control and visibility with visual workflows • Quickly root-cause issues with proactive alerts when something fails 13 © Stonebranch 2022. All rights reserved.
  • 14. Data Pipeline Orchestration Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline. • Centrally schedule and orchestrate automated processes within each tool along the entire data pipeline • Use APIs or Agents to control the various tools used within each stage What you achieve with this approach: • Observability of the logs and data for governance and security • DataOps lifecycle management (Dev-Test- Prod) - including simulations • Centralized control and visibility with visual workflows • Quickly root-cause issues with proactive alerts when something fails 14 © Stonebranch 2022. All rights reserved.
  • 15. Orchestration 15 © Stonebranch 2022. All rights reserved. Driven by SOAP Cloud DevOps ERP/Apps IaaS UAC SOAP META-ORCHESTRATION
  • 16. 16 © Stonebranch 2022. All rights reserved. Self-Service Automation Centralized collaboration platform for data, developers, and operations IT ops teams gain operational visibility Data teams approve and trigger automated workflows & pipelines from common business applications Data Pipeline
  • 17. Putting the Ops in DataOps 17 © Stonebranch 2022. All rights reserved. For Enterprises Ready for the Next Level of Maturity Develop/ Orchestrate Test / Simulate Production / Deploy Continuous Improvement Continuous Deployment Development Controller Production Controller
  • 18. Develop/ Orchestrate Test / Simulate Production / Deploy Continuous Improvement Continuous Deployment Development Controller Production Controller Putting the Ops in DataOps 18 © Stonebranch 2022. All rights reserved. For Enterprises Ready for the Next Level of Maturity Web GUI As Code Via in-built capabilities Promotion Options Via third-party repositories like GitHub
  • 19. Data Pipeline Orchestration Demo Ravi Murugesan Sr. Solution Engineer © Stonebranch 2022. All rights reserved. 19
  • 20. © Stonebranch 2022. All rights reserved. 20 Demonstration Update Visual Dashboard from Multiple Data Sources (both on-prem and cloud-based) Live orchestration of a data pipeline, including • Sources (cloud, on-prem, apps) • Ingestion, transformation (Informatica) • Stores (Azure blob, Snowflake) • Delivery (Tableau)
  • 21. One of the Largest Global Food & Beverage Manufacturers in the World Customer Use Case 21
  • 22. Customer Use Case: Overview One of the Largest Global Food & Beverage Manufacturers in the World Evolution & Goal • Goal: Orchestrate the full pipeline end-to-end • Objective: Identify a platform that could connect all their critical data tools Overall Strategy • On-prem to cloud digital transformation • Implemented an enterprise analytics data management environment • Hub-and-spoke model to help keep regional resource groups and services segregated • Approved services are first developed and deployed at the hub level, with further spoke deployment via containers Original Approach • Their data pipeline for the enterprise data management environment with Azure Data Factory • Azure Data Factory worked well in an Azure environment • It served as an entry point for the project • The Challenge: Data Factory did not integrate with their full stack of solutions used along the data pipeline 22 © Stonebranch 2022. All rights reserved.
  • 23. Data Pipeline Orchestration One of the Largest Global Food & Beverage Manufacturers in the World Achieving Their Goal • Secure and robust file transfer • DataOps: define pipelines as code and gain lifecycle management (test/dev/prod) capabilities • Integrate diverse data pipelines that are built using various cloud-based and on-prem services and tools • For operations: visibility into the process, improve SLAs, real-time monitoring, alerting • Unified view to design and orchestrate workflows across multiple cloud and on-prem applications Orchestration Databases 23 © Stonebranch 2022. All rights reserved.
  • 24. © Stonebranch 2022. All rights reserved. Data Pipeline Orchestration Solution Universal Automation Center 24
  • 25. Real Time Hybrid IT Automation 25 © Stonebranch 2022. All rights reserved. Universal Automation Center Platform A Platform Approach Orchestrating IT processes from on-prem, to cloud, to containerized microservices
  • 26. Find. Deploy. Extend. • Download extensions • Share extensions • Community driven • Constant additions (monthly) • Large Data Pipeline Focus • Rapid creation of new integrations Orchestration = Integration 26 © Stonebranch 2022. All rights reserved.
  • 27. What to Look for in a Data Pipeline Orchestration Solution 27 © Stonebranch 2022. All rights reserved.
  • 28. Summary Who is this for? • Want to keep using existing data tools, but are ready to graduate from opensource schedulers to enterprise grade platforms • Would like a single platform to connect Data Teams, Developers, IT Ops, and Cloud Ops teams – to help scale their data program • Need to operationalize DataOps methodologies to gain speed and improve data quality • Want to gain full visibility across the entire pipeline – to move quickly when issue arise • Have a growing or changing data tool landscape, and need the ability to rapidly build new integrations (or download pre-existing integrations) • Need to enable data scientists or business users with simple self-service capabilities via the platform or third-party tools like ServiceNow, Microsoft Teams, or Slack • Bonus: Want a central IT automation and orchestration platform (beyond data pipeline orchestration) to support cloud automation, on-prem automation, traditional job scheduling, and DevOps orchestration © Stonebranch 2022. All rights reserved. 28
  • 29. © Stonebranch 2022. All rights reserved. 29 Q & A Scott Davis Global Vice President scott.davis@stonebranch.com Stonebranch - Atlanta, USA Ravi Murugesan Sr. Solution Engineer ravi.murugesan@stonebranch.com Stonebranch – Frankfurt, Germany
  • 30. Thank You © Stonebranch 2022. All rights reserved.
  翻译：